ORBITKVGitHub

KV cache for vLLM and SGLang

Reuse KV.
Across
cache tiers.

Keep repeated documents, system prompts and conversation prefixes useful beyond GPU memory.
One Cache Manager per host. Your engine keeps control of execution.

Read the docs Get started
Current single-node path
Serving runtimevLLM or SGLang
Engine-owned HBMGPU
PAGESCUDA IPC registration
Cache APIQUERY
& SAVEUDS + iceoryx2 control
CACHE MANAGER.Pinned DRAM · SSD
Remote cache fetch via Mooncake is experimental.

01 / Capabilities

Compute once.
Keep it useful.

Explore the architecture
Reuse

Reuse prefixes.

Keep useful KV beyond one request and restore it when a compatible prefix returns.

Store

Extend capacity.

Use pinned DRAM and optional SSD, with automatic GPU storage selection and experimental Rust FP8 storage quantization.

Model-aware

Restore complete state.

Select the required attention pages, windows and checkpoints for supported model layouts, and validate them before recovery.

Integrate

Connect both engines.

Enable the vLLM connector or SGLang plugin. Both use direct GPU transfers and the same Rust cache client.

Experimental

Share across replicas.

Local inventory indexes discover compatible replicas. etcd maintains leased membership; Mooncake TENT moves cache bytes.

Observe

Measure the path.

Measure Qwen3-8B with a KV working set larger than DRAM. Compare request latency, throughput and concurrent SSD reads and writes.

Now Engine-owned HBM · pinned DRAM · optional SSD Experimental Mooncake remote fetch

02 / Deployment

Start on one host.
Build toward a pool.

Single-node caching → independent replicas → prefill/decode reuse

Use DRAM and SSD caching with official vLLM 0.31.0 or SGLang 0.5.21. Explore shared-cache deployments after validating your single-node workload; multi-node serving remains experimental.

Choose a deployment

03 / Get started

Connect.
Run. Reuse.

Build a matching CUDA wheel, start a manager, then enable your engine's adapter. The first Python release is in preparation.

vLLM quickstart →
SGLang quickstart →
Terminal / Cache Manager
# After installing OrbitKV and your inference engine
orbitkv-cache-manager \
  --addr 127.0.0.1:50055 \
  --pool-size 8gb

04 / Documentation

From first hit
to system design.

Browse all documentation →
Installation and engine setup →Architecture and ownership →Metrics and operation →Benchmarks and measured limits →Distributed cache design →Completion and validation gates →