Framework integration

Two engines.
One cache manager.

vLLM and SGLang keep control of HBM. Their adapters register GPU KV pages with a node-local Cache Manager for reuse in pinned DRAM and optional SSD.

vLLM

Enable OrbitKVConnector to save computed prefixes and restore reusable blocks into vLLM-owned GPU memory. Version 0.29.0 is the validated baseline. Full attention, sliding-window and aligned recurrent groups use compiled recovery ranges, including combined layouts. Cross-host TP is outside current support. Start with one rank and verify a restore after engine restart.

vLLM install and deployment

SGLang

Enable the OrbitKV RadixCache backend and external linker on SGLang 0.5.20. The adapter supports full attention, Full + SWA, Full + recurrent/conv and their combined layout through the same CUDA IPC path. SGLang owns GPU pages; OrbitKV prepares external state and returns a validated restore. DRAM and SSD recovery have single-rank GPU gates. Multi-rank serving requires further qualification.

SGLang install and deployment Hybrid recovery rules and validation SSD measurements and qualification Concurrent budgets and output controls

P/D and NIXL

OrbitKV's experimental PdConnector pushes live P/D KV between vLLM workers through Mooncake. vLLM also ships its own NIXL connector for P/D; OrbitKV does not ship one. These request-handoff paths are separate from OrbitKV's external-cache lookup. SGLang currently has an OrbitKV cache linker, not an OrbitKV P/D adapter.

P/D versus cache sharing

Model-aware reuse

Cache identities bind model artifacts, computation settings and storage geometry. Supported model rules compile to the page ranges needed at a legal recovery boundary, including complete windows and exact checkpoints. This uses engine-declared state requirements; arbitrary graph analysis and future-token prediction remain future work.

Checkpoint qualification records the exact engine, precision and recovery path. Larger Qwen, GLM, DeepSeek and Kimi targets have separate gates; sparse indexer state and an engine's ability to load a model do not imply a complete OrbitKV restore.

Pretrained models and qualification limits State identity plan and gates

Tune and deploy

Begin with ordinary demand and size DRAM/SSD for your working set. Optional request preparation has three matched Qwen3-8B controls per engine: vLLM improves modestly, while SGLang gains throughput with worse P95 latency. It stays disabled by default. Next comes real two-host replica sharing, followed by P/D with cache reuse and catalog resilience.

Read the implementation gates Request preparation measurements