From infrastructure setup
to inference through a CLI.
GPU inference platform. A working prototype that abstracted GPU infrastructure, model serving and routing behind a developer interface, built around vLLM.
- Context
- An independent Koeo prototype.
- Service
- Inference infrastructure, developer tooling, routing, scaling and automation.
- Industry
- AI infrastructure
- Specialty
- Improve AI infrastructure
- Status
- Working prototype. Development concluded before production.
Overview
We built a prototype that let developers work with open models through a CLI and an OpenAI-compatible API, without assembling the GPU infrastructure themselves. Testing covered several models from Hugging Face on consumer GPUs and RTX 6000 hardware.
The commercial objective was to make better use of available GPU capacity and support inference-based monetization, rather than simply rent each GPU by the hour.
Challenge
Getting started with inference involved configuring infrastructure, loading a model and connecting the application to it. Those steps needed to be handled by the platform while giving developers a straightforward interface.
The supply side also needed defined service expectations. One partner MOU covered GPU access for prototype testing, with production access conditional on conclusive tests.
Solution
The prototype used vLLM as its inference foundation, with routing, scaling and automation around it. A CLI gave developers a way to configure access and start working with models. The CLI could also be used from Claude Code or connected through MCP.
Model loading remained a separate step and varied with model size. The platform used vLLM’s PagedAttention capabilities as a foundation for improving utilization.
Results
Developer setup in minutes.
- CLI setup
- Installation and configuration typically took a couple of minutes
- Model loading
- Typically 2–3 minutes, depending on model size
- Hardware tested
- Consumer GPUs and RTX 6000 hardware
- Partner validation
- One signed MOU for prototype GPU access and conditional production access
- Delivery stage
- Working prototype; development concluded before production
These timings were observed during prototype testing and vary by model size. CLI setup and model loading are separate steps. Production performance, utilization gains and cost savings were not measured.
Relevant experience for model serving, GPU infrastructure and developer tooling.