ORBITKVGitHub

Architecture / Ownership before placement

One cache API.
Across tiers.

The engine owns GPU memory and model execution. OrbitKV owns external cache replicas, finds reusable prefixes, and holds pages until transfers finish.

From one host to a shared cache

Each engine owns HBM and uses compiled page demand and recovery evidence. Per-host Cache Managers own DRAM, SSD and catalog shards; Mooncake transfers remote bytes and etcd tracks membership. General lifetime analysis, retention and physical planning remain future work.
Single-node recovery is validated. The multi-node path is experimental; catalogs currently have one copy per shard. A shared API does not imply interchangeable KV layouts. Declared page demand is implemented; general lifetime and physical planning remain future work.
01 / Single node

Keep useful prefixes.

Run an independent Manager per node and connect engines to its shared DRAM and optional SSD cache. Engines retain HBM ownership. Standalone deployment needs neither etcd nor peer gRPC. Configure the SSD path and capacity; automatic selection tries native cuFile and falls back to io_uring without engine-side backend settings. Rust submits cuFile reads and writes through two bounded GPU slots, allowing demand reads to progress alongside writeback. Native GDS performance and multi-engine container qualification remain separate.

02 / Independent replicas

Reuse across workers.

Managers share compatible replicas through embedded catalogs. Cached candidates reduce discovery work; source authorization and Mooncake move the requested blocks. Cross-host serving is the next gate.

03 / Prefill & decode

Compose the handoff.

A P/D connector transfers the current request. The cache retains KV for later requests. Combined serving qualification follows independent-replica sharing.

How a request reuses KV

  1. Compile

    The adapter declares prefix, window and checkpoint rules. The shared contract turns them into page requirements for known token ranges from a valid HBM origin.

  2. Prepare

    The manager finds DRAM, SSD or remote candidates within query byte budgets. The same compiled requirements validate leased evidence and yield complete recovery boundaries across ranks.

  3. Restore

    Compiled ranges select SGLang's transferred and retained keys and vLLM's hybrid allocation. Leases hold pages through transfer completion.

  4. Continue

    The engine resumes computation; completed KV can be published for later reuse.

The vLLM adapter defers new admissions until an admitted restore reaches computation, avoiding deferred-queue capacity stalls. SGLang waits for leased readiness before allocating restore destinations. Each retains its own scheduler and HBM allocator.

Shared page requirements remove duplicate adapter arithmetic. They do not predict future tokens, analyze arbitrary model graphs or authorize reclamation. Automatic hybrid warming remains disabled. Exact GPU-byte gates qualify hybrid recovery; its standalone latency benefit is not yet measured. Ordinary recovery and request preparation have separate Qwen3-8B performance reports.

Read the architecture and ownership contracts →

Implementation boundaries

orbitkv-state

Versioned model/storage keys and compiled rules for demanded page ranges and leased-evidence validation, shared by SGLang and vLLM hybrid adapters.

orbitkv-channel

Rust ownership of query revisions, warming and transfer waits; versioned iceoryx2 requests plus UDS lifecycle for both adapters.

orbitkv-core

KV blocks, leases, tiers, batched GPU codecs with reusable workspace, CPU SIMD, and encoded cuFile storage.

orbitkv-catalog

Embedded directory shards, residency journals, candidate indexes and etcd membership; one copy per shard today.

orbitkv-transfer

Pinned Mooncake Transfer Engine for experimental remote cache fetch and vLLM P/D.

Cache Manager

Independent per-node service sharing external DRAM/SSD capacity across attached engines, with automatic SSD selection, health and peer transfer control.

orbitkv.sglang

Direct GPU recovery for full attention, windows and recurrent/conv checkpoints, including combined layouts; selected-range reads validate every required group.

orbitkv.vllm

External KV cache connector plus a separate experimental Mooncake P/D adapter.

What comes next

The next local optimization measures transfer costs in Rust before changing path selection. Local storage and Mooncake TE transfers share cost observations and budgets; replica reuse, P/D handoff and TP/PP consumption keep distinct completion contracts. These roles can compose within one deployment. Dynamic cost selection remains planned.

Qualify matching replicas on two real hosts, then P/D with reusable cache. Single-node preparation and stopping controls are implemented and stay opt-in after the first repeated measurements. Replicated catalog shards and controlled placement handoff precede production distributed deployment. General lifetime analysis, resharding, cross-engine byte conversion and KV-aware routing remain future work.

Implementation plan and upstream references →Transfer costs and deployment policies →Distributed catalog design →Model-aware identity →LMCache and Mooncake deployment paths →Roadmap and qualification gates →