DocumentationDevelopment

Completion and upstream integration plan

This is OrbitKV’s only execution queue. Architecture pages describe contracts; compatibility describes qualified profiles. Implementation, independent acceptance and release are different states. Deliver one coherent stage or substage, then independently review its consumed path and evidence.

The chronological records have been consolidated after verifying their complete archive. The immutable pre-refresh plan retains every frozen cohort, failed run, reproduction instruction and earlier review limit. The summaries below do not replace those evidence packages or change any qualification result.

Release baseline and reference policy

ProjectSelected release / commitRole
vLLM0.31.0 / db9527a46873454610df6dbedf79a36d6bf1a7f6Official V1, one-attention-group cache baseline; scoped dense acceptance
SGLang0.5.21 / e00930c5489053f26d86b179cee0d087f846acbbOfficial UnifiedRadixCache external linker; public lifecycle migration open
LMCache0.5.5 / 05a013b29da78cf2321b9b46ec5039dde2fb0bb0Released integration and matched-comparison reference

At each upgrade, recheck official releases, freeze the selected tag/commit and update dependencies, lockfiles, submodules and qualification together. Upstream main and submitted patches are contribution material, not release APIs. Keep historical results at their measured versions; remove superseded compatibility branches at cutover. See interface audit.

Delivery status and dependencies

StageCurrent stateRemaining delivery
S0CompleteHandoff/skill migration merged in PR #190
S1Accepted at ec3add9bPreserve external evidence and one queue
S2PartialDRAM/SSD isolation FAIL; sixteen owners UNQUALIFIED; support stays four owners. Cross-host cache HA and independent etcd failure domains remain unqualified
S3OpenPhysical termination proof, page generations, explicit registration and crash reclamation
S4PartialMeasured copy/SSD/control improvements exist; mixed contention, complete serving gains and S3-dependent faults remain open
S5PartialOfficial dense cache/restart and scoped deployment profiles accepted; public lifecycle, complete hybrid recovery, native P/D faults and broader topology remain open
S6PartialObservations and guarded limited choices exist; one consumed route/admission planner remains open
S7OpenConsumed semantic retention/checkpoint compiler beyond recovery validation
S8PartialDocumentation and wheel workflow exist; final images, artifact qualification and publication remain open

S5.1 research and S8 documentation/package preparation may proceed independently of S2/S3. New payload concurrency, reuse, reclamation and early consumption must satisfy their lifetime dependencies. Missing hardware leaves a gate unqualified; it does not stop unrelated work. No stage is closed by this documentation change.

Baseline: preserve these implementations

OwnerExisting behaviorRemaining boundary
Process channel/native clientUDS bootstrap, iceoryx2 commands, shared arenas, bounded accepted-operation ownershipPage generations and live-process stall handling
Core transfer/storageEngine-local raw DRAM Restore, Manager Publish/SSD/codecs, bounded multipart plans and final drainSafe earlier multipart/SSD/codec consumption and physical lifetime
Catalog/server clusterScoped inventory streams, local complete views, leased etcd membership and source-incarnation fencesIsolation, scale qualification and real host-failure recovery
Peer owner/TENTAuthoritative grants, DRAM/SSD READ, release reconciliation and retained source exportsPermanent requester loss and transport-fault reclamation
Engine adaptersOfficial release callbacks/linker, engine-owned allocation and native P/D candidatesPublic SGLang hooks, hybrid/rank/fault profiles
Cost/planningResource-scoped observations, shadow estimates and guarded peer choicesComplete executed route and admission ownership
State contractsDeclared recovery rules and compiled page demandsGeneral retention/read-set IR and semantic reclamation

Do not add a second Catalog, directory RPC, Python coordinator or transport facade just to close a checkbox. Recovery and component improvements do not establish a win over an engine’s resident-HBM hit; see performance.

S1 — Evidence separation and one truthful work queue

Accepted. Generated evidence is archived outside Git, benchmark output uses explicit external directories, and the experiment-output guard protects tracked source. Keep programs, reusable workloads and small fixtures in the repository. Verify archives and update every reference before retiring tracked evidence. No history rewrite or removal of a sole evidence copy is allowed.

Acceptance: archives readable; no tracked generated collections; benchmark and publication/link/search gates pass; every open obligation belongs to S2–S8 or explicit deferred research. The earlier 70-obligation migration is retained in the immutable plan.

S2 — Metadata reliability and measured capacity

Depends on: S1. Owners: orbitkv-server/src/cluster/, orbitkv-catalog/, core inventory and metadata diagnostics.

  • Extend real-etcd coverage to quota exhaustion/recovery, slow/disconnected Watches, concurrent rebuild and eviction, sustained journal overflow, member churn, index exhaustion and network partition/rejoin. Preserve incarnation and publication-cursor fences; never advertise a partial index as complete.
  • Measure publication latency, Watch lag, snapshot rebuild time, update CPU, logical index bytes/RSS and etcd growth across increasing node/key/update loads. Define a tested operating envelope instead of claiming unlimited replication.
  • Fix concrete bottlenecks and recovery gaps revealed by those runs. Keep local discovery free of directory/coordinator RPCs. Separate membership renewal from bulk synchronization and apply bounded work/backoff throughout recovery.
  • Verify leader loss with a quorum and host loss across actual failure domains. Three etcd processes on one machine prove process faults only. Include startup readiness, operational diagnosis and recovery instructions.

Acceptance: no lost final puts/deletes after convergence, no stale incarnation authorization, bounded memory/queues, local cache still usable during metadata loss, and a repeatable capacity/recovery report. A fenced runtime restarts with a new incarnation; it cannot silently resume using the expired one.

S2.1 — Quota and index-budget recovery

Independently accepted. Real etcd quota/recovery and index-budget controls retain bounded ownership and explicit completeness. A partial or retired view cannot authorize a source. Preserve negative controls and exact final-key checks.

S2.2 — Watch delay, partition and compaction recovery

Independently accepted. Delayed/disconnected membership Watches and compaction recover without reviving an expired incarnation. Preserve reconnect/bootstrap, lease-expiry and final put/delete convergence controls.

S2.3 — Measured metadata scaling smoke

Independently accepted within its smoke scope. Retain measured publication, index/queue budgets and etcd growth at the frozen scale. A smoke result is not sustained capacity, a sixteen-owner qualification or independent-domain HA.

S2.4 — Bounded live-DRAM journal and metadata-loss correctness

Independently accepted. A bounded journal supports live DRAM publication and recovery. Gaps/overflow require a complete replacement view; they cannot become silent partial success. Metadata loss must leave local cache correctness intact.

S2.5 — Full-Manager DRAM and io_uring metadata-fault correctness

Independently accepted at f3908a93. Full Manager DRAM/io_uring processes pass transient metadata loss, lease expiry, exact payload/identity checks and normal cleanup. SSD reads do not masquerade as DRAM residency. This is same-host multi-process evidence; host HA, RDMA and native GDS were not qualified.

S2.6 — Sustained metadata churn and capacity

Independently accepted within the frozen workload. Sustained puts/deletes and membership churn converge exactly with bounded queues/index ownership. Preserve the tested operating envelope and failed capacity controls rather than claiming unlimited replication.

S2.7 — Bounded coalescing and frozen comparison baseline

Independently accepted at 93abd676. Bounded coalescing trades stream bytes against visibility delay. Keep matched no-coalescing controls, all samples and independent-run uncertainty. A byte reduction alone is not a latency win.

S2.8 — Owner inventory streams and coordinated protocol cutover

Independently accepted at ad5bb8e6; production fix 344ef6c9. etcd retains protocol identity and leased membership; bounded owner inventory streams feed local indexes. An owner retired by budget revocation stays retired across membership refresh until a complete replacement snapshot commits. The mutation control rejects the former unsafe assignment. No directory RPC or second index implementation is introduced.

S2.9 — Scoped discovery and explicit coverage

Independently accepted. Scope selects inventory coverage without changing in-scope candidates or payload correctness. Coverage is explicit: absent coverage is not a proven miss. Preserve complete snapshot/reconnect and scope-change fences; scoped bytes may shrink while source authority remains identical.

S2.10 — Sustained live-store and independent-domain qualification

Partial. Same-host DRAM/io_uring all/scoped correctness is accepted at 65c51aaa, including the four-owner envelope and normal membership/stream drain. The frozen formal 25-cell campaigns are valid and independently accepted as failed isolation qualifications. Five sixteen-owner ordinary publication→install p99 measurements pass 50 ms; install→harness is reported separately. DRAM and SSD fail the predeclared per-pair and 95% CI ≤1.05 isolation contract.

Support stays four owners; sixteen owners remain UNQUALIFIED. Never delete slow samples, relax thresholds or relabel a valid failure as a pass. Preserve both formal cohorts and their actual terminal receipts.

Remaining gates: reviewed isolation diagnosis/remediation and a separately frozen replacement qualification if justified; cross-host cache/HA; real etcd leader/quorum/process faults; independent physical failure domains; final serving/rank profiles. Three processes or hosts do not prove independent power and network failure domains. Native GDS and S3-dependent failures remain separate.

S2.10a — Same-host isolation root-cause diagnosis

Implemented / qualification blocked. The bounded correlated timeline and lifecycle fixes reached 3161eaf9. Three instrumentation pilots failed the predeclared overhead guard; no formal ABBA/BAAB diagnosis or production policy repair was authorized from those pilots. Preserve the off/on controls and stop decisions; overhead failure is not a Manager-isolation root cause.

S2.10b — Lifecycle runtime acceptance and lower-overhead diagnosis

Scoped diagnosis accepted; production repair blocked. Fourteen A100 cells with 50 warm-up and 1,000 measured operations each have valid byte oracles and terminal receipts. Tails correlate with the Python observer/return boundary; order and phase effects prevent a repeatable metadata-pressure causal claim. No Manager, metadata-lock or SSD-owner repair follows from this evidence.

PR #202’s observer/helper protocol and interruption-safe cleanup are merged. Later valid 25-cell qualification still fails DRAM/SSD isolation; it does not expand owner support. Original invalid runtime/wheel-shadowing smokes remain archived rather than replaced or hidden.

Next S2 delivery: inspect the actual current handoff, finish the authorized runtime/proxy/harness remediation, freeze inputs and obtain launch review. Check direct etcd reachability without inherited proxy interference and complete native runtime closure before admitting a new fault cohort. Run only newly authorized cells, retain failures and check cleanup. Network readiness and ordinary RDMA serving do not close HA. Do not start dependent S3/S6/S7 work from an unaccepted launch or failed qualification. The immutable history preserves earlier task-specific fail-stop contracts; current external handoffs need their own review.

S3 — Transfer lifetime and generation-safe ownership

Depends on: S2’s metadata contracts. Owners: core peer/query/storage owners, transfer/sys C ABI, process channel and the consumed adapter registrations.

  • Establish transport termination evidence before reclaiming exports belonging to a permanently lost requester. Inspect the current pinned TENT and latest upstream main/issues before changing the ABI; reuse or submit a focused transport fix if needed. Keep OrbitKV metadata/control state in OrbitKV.
  • Bound and diagnose retained/quarantined source memory while proof is absent. Timeout, etcd expiry, cancellation, remote process disappearance and a control ACK are not interchangeable with native drain. An unavailable upstream primitive is an explicit blocker to safe reclamation, not permission to free memory.
  • Carry owner-authored page/allocation generations through actual registration, source grants, destination use and adapter boundaries. Keep semantic last-use separate from completion of CUDA/RDMA/SSD work. Extend existing owners instead of adding an unused generic fence framework.
  • Replace the remaining framework CUDA IPC pickle interpretation in the Manager with explicit validated registration data; remove the retired wire/API path.
  • Implementation open: expose a stable native peer-liveness contract before enabling SGLang failed-session probing. Liveness is not transfer termination.
  • Add a bounded operational response to a live Manager that never completes a Publish, preserving held pages until drain or established process death.

Acceptance: real tests for requester/source crash, delayed/lost grants and release ACKs, partial submission, repeated page reuse and restart. No use-after- free or premature recycle; resource retention is bounded and observable; safe reclamation eventually happens when its proof exists. Both adapter lifecycles consume the new generation checks. No TTL-based reclamation substitute.

S4 — Finish communication execution and demonstrate gains

Partial. Depends on S3 for new reuse/early consumption. Existing transfer, channel, SSD/codec and engine-hook owners retain responsibility. Contracts are in transport and engine-local Restore.

KDA copy work, bounded SSD host batching, separated SSD read/write submission, GPU/NUMA identity and descriptor/control measurements are implemented within their reviewed scopes. CUDA sources live in separate files. Rejected descriptor reuse, READV merging and TENT RPC-thread/lane candidates remain rejected; correct bytes do not override a performance regression.

  • Profile actual adapter hashing/conversions, destination checks, fragmentation, dispatch, scratch allocation and repeated shared-prefix H2D before selecting the next bottleneck. Keep GIL/observer timing separate from native execution.
  • Compare device-event DMA/kernel timings on measured shapes, with registration, plan/submission and complete restoration cost reported separately. Preserve exact-byte, small-payload, contention and model-output controls. Do not switch defaults from a component-only result.
  • Improve existing SSD preparation/read parallelism and completion scheduling only with a consumed workload and matched byte/drain/latency gates. Native GDS/cuFile requires physical eligibility and separate qualification.
  • Profile first peer recovery, lazy TENT setup and warm steady state separately. A connection-warming candidate must consume an explicit reviewed interface, account for setup/fabric resources and pass contention controls. Never use unowned dummy payload addresses or omit warming cost from serving latency.
  • Retain bounded multipart plans and whole-operation ownership. Earlier layer, multipart, codec or SSD readiness requires consumer dependencies and final physical drain, not only a readiness notification.
  • Run matched native/OrbitKV/LMCache serving comparisons after model recovery qualifies. Larger-model native output controls do not close this gate.

Acceptance: correct GPU bytes and outputs, unchanged resource/lifetime contracts, repeatable matched gains with uncertainty and frozen cold/small-path regression guards. Require an independent review before performance promotion. Detailed accepted/rejected cohorts remain in kernel work, SSD methodology, peer qualification and the immutable plan; historical evidence is not requalified by this summary.

S5 — Released-engine integration and upstream contributions

Partial. S5.1 may proceed independently; new lifetime behavior depends on S2/S3 and measured S4 paths. Engine adapters and native client owners implement the boundary. Use released LMCache/FlexKV integration contracts as references without duplicating their implementation layers.

S5.1 — Release and interface audit

Independently accepted at 38f8dbb2. Pins, package probes and source submodules align with official vLLM 0.31.0 / SGLang 0.5.21. Released callbacks, installed-wheel ordinary cache/restart and bounded model controls have scoped acceptance. Upstream V2/recurrent restore ordering, SGLang pre-factory graph capture, missing public pending tickets and native P/D timeout drain remain interface limits.

At every upgrade, record callback ownership, public/private dependencies, selected release and exact gates. No fork-only callback, factory, default or CI runtime may be required. A submitted or merged upstream fix is not a released contract. See interface audit and compatibility.

S5.2 — Minimal official cache backends

Scoped ordinary-cache acceptance. vLLM uses scheduler/worker KV Connector owners; SGLang uses its external linker and shared native client. Python handles callbacks, layout and GPU handoff; Rust owns state, batching and waits. Retired custom P/D connectors, proxy, handshake wrappers and forwarding layers stay removed. Single-rank admission avoids unnecessary consensus tensors while retaining actual multi-rank decisions and pending/expiry/cancel ownership.

Official dense DRAM/io_uring recovery/restart, 900-second two-engine shared-Manager pressure and explicit serial batch-1 FULL decode Graph recovery have separate H20 acceptance. Broader graph shapes, models, ranks and fault profiles remain open.

Remaining: qualify each newly claimed profile from the final installed wheel; keep cold/native-HBM/full/partial/restart/failure controls, exact outputs, actual restore bytes, installed-file integrity and truthful process cleanup. Remove obsolete code only after its replacement is consumed and passes the same contract.

S5.3 — Public lifecycle and hybrid-state contracts

Partial. Idle process-scoped vLLM external reset is accepted at 6d874384; immutable local static PEFT identity at 47fea821; the official dense query-control handshake at 1ab67a58. These do not qualify live weight changes, dynamic adapter mutation or physical multi-rank recovery.

  • Replace SGLang’s private graph-initialization and pending-query admission hooks with consumed public lifecycle interfaces. Document their behavior and released alternatives; never remove cancellation/page-reuse protection merely for cleanup.
  • Preserve model/tokenizer/computation/layout identity and generation fencing. Dynamic mutations require a consumed invalidation contract; stop/restart until such a profile qualifies. An idle reset is not that contract.
  • Establish complete prefix/window/recurrent/conv/auxiliary state at legal recovery boundaries. The current vLLM V2/multi-group/recurrent guard stays active until released scheduling/restore order protects every destination page.
  • Test cancellation, preemption, restart, partial commit and page reuse with real native state. Both semantic non-use and physical completion are required.

Qwen3.8-27B BF16/FP8 native output controls on both official engines are accepted only for frozen text-only eager TP=1/PP=1 profiles with BF16 KV. They do not prove external hybrid recovery, quality, cross-engine/precision equivalence or speed. DeepSeek-V4.1-Flash remains a separate native readiness and complete-state target. See models; retain all original input/startup/cleanup limits.

S5.4 — Official native P/D and independent cache

Release-only cutover implemented; native fault qualification open. vLLM uses NIXL/MultiConnector; SGLang uses native disaggregation. Manager shared-cache bytes continue through TENT. There is no maintained fork runtime or custom P/D proxy.

The bounded same-A100 dense model gates retain exact monolithic output, independent cache reuse after engine restart and already-running request cancellation. They do not prove cancellation during transfer, partial submission, delayed ACK or page reuse. P reads/writes cache; D is save-only in the current candidate, avoiding competing writers into native incoming pages. Test other selection orders separately.

Acceptance: independently reproduce installed official native P/D output, preemption/retraction, cancellation, restart, peer loss, partial submit, delayed ACK and page reuse for each opened profile. Preserve ownership and S3 drain proof. Fork patches remain immutable upstream contribution material, not user runtimes. Current commands and limits are in P/D.

S5.5 — Deployment matrix and upstream maintenance

Scoped acceptance, broader qualification open. Current accepted profiles are listed in compatibility; precise peer cohorts remain in shared-cache qualification.

  • Ordinary installed dense H20 DRAM/io_uring recovery/restart and shared-Manager pressure have acceptance for their frozen budgets and execution settings.
  • Registered trusted-loopback query-control authority/leases are accepted at b1ea00a7; common-prefix coordination and official TP=1 handshake at 1ab67a58. This is not physical TP/PP/EP support. Protected cross-host activation remains implementation open.
  • Physical two-H20 RDMA dense TP=1/PP=1 serving with DRAM/io_uring, consumer restart and controlled empty-source restart has scoped acceptance on both official engines. Cache HA, independent failure domains and requester-loss physical reclamation remain separate gates.
  • GPU registration uses full UUID identity across reordered/partial device views; payload placement binds the resolved GPU-local NUMA node before first-touch and fails explicitly if disallowed. Test cleanup failures/interrupts remain failures and retain ownership receipts; they cannot certify every native child drain.

Track single/multiple instances, dense/hybrid, independent replicas, native P/D, attention DP, TP, PP, MoE EP and heterogeneous combinations separately. Do not infer KV partitioning from EP size. Qualify shared budgets, source incarnations, sender/receiver cancellation, Manager restart, containers/UID/IPC/PID visibility, physical NIC counters and actual GPU destinations for each claimed cell.

Maintain upstream registration/docs/tests and upgrade gates. Record issue/PR, maintainer response, merge commit and first released version separately. Submit lifecycle and TENT fixes independently of the minimal cache backend.

Acceptance: executable scenarios with output, physical-transfer and drain receipts for every support claim. A minimal upstream cache contribution need not wait for every research cell, and its merge does not qualify those cells.

S6 — One consumed route and admission planner

Depends on: S2/S3, measured S4 paths and S5 P/D authority. Owners: existing planning/, cost/, query/, residency/peer owners and actual engine feedback.

  • Extend complete-route estimates to compatible local DRAM/SSD, peer DRAM/SSD, supported direct transfers and engine-owned recomputation. Compare only equal state coverage and completion targets; include queue/admission/authorization, transfer, decode and H2D once, with freshness and uncertainty.

  • Make the selected plan acquire the actual source/extent lease, byte credits and destination ownership. For P/D vs direct recovery, the same consumed plan must own handoff authorization; observation-only messages are insufficient.

  • Qualify bounded replanning and contending destinations, device/SSD/NIC budgets, per-instance shares, preparation deadlines and retention/write admission. Train only observed outcomes. An unknown estimate must not masquerade as zero.

  • Compare DMA/kernel and io_uring/native cuFile only when both are physically eligible and qualified. Introduce guarded switching with measured margins; retain a clear deterministic policy for unknown estimates, without a duplicate legacy planner or generic forwarding layer.

  • Keep Manager route decisions independent of request routing. After that works, integrate the pinned upstream router through a narrow consumed boundary if the routing milestone is retained; include worker evidence, replica risk and revalidation. NIXL/backend expansion is an evaluation decision, not an assumed prerequisite to finishing TENT recovery.

  • Add receiver-driven admission using a bounded in-flight byte window and pacing. One node/physical-NIC/direction budget must include Manager READ and engine-owned native P/D traffic; Manager-only admission misses the latter. Lease bounded credits in batches; each engine retains its native P/D transport and Manager peer-cache movement remains direct through TENT.

  • Keep network credits, destination buffer lifetime and source export authority distinct. Return credits on verified receive/transport progress; retain memory until its final user drains. Credit timeout is never a memory-release proof.

  • Schedule ready data using deadlines, remaining critical state/rank/layer work, per-instance fairness and aging. Pair receiver admission with source egress limits; a receiver cannot control other flows or all fabric bottlenecks. Reserve headroom for engine communication using measured physical interference.

  • Qualification open: cross-medium selection and peer SSD under contention, cancellation and RDMA. Two-host TCP byte recovery does not qualify policy gain.

  • Deferred research: router event normalization and pinned upstream router service integration; eviction externalities/replica risk; load-only versus overlap-only versus joint routing on one trace; active RDMA WRITE cache replication; topology slicing, endpoint pooling and alternate-rail retry; NIXL Mooncake-backend evaluation. These need a concrete scope decision and measured benefit. Current P/D WRITE is not cache replication. Do not introduce a new transport abstraction or deprecated KVBM to close the current plan.

Acceptance: selected routes explain their coverage, ownership and predicted complete cost; stale evidence/resource rejection replans safely; submitted payload failure does not blindly retry another medium. Fixed-policy, shadow and executed-policy comparisons demonstrate repeatable benefit under contention without violating correctness, memory budgets or cold-path regression limits.

S7 — Semantic retention and checkpoint compiler

Depends on: S3 generations/lifetimes, S5 consumed hybrid contracts and S6 physical planning. Owners: orbitkv-state, core planning/storage and adapter state declarations; retain framework scheduling ownership.

  • Define a finite, explicit may_read(query, state) IR for supported state families. Compile full attention, sliding windows/sinks and recurrent state into required ranges, legal checkpoints, retention and reclamation decisions.
  • Implement minimum persistent state for the defined model and supported families, with an exact oracle on bounded cases and explicit approximation limits elsewhere. Do not claim a general optimum for undeclared semantics.
  • Feed checkpoint placement, prefetch/placement and semantic release decisions into real storage/planning owners. Safe reclamation requires both semantic non-use and physical completion; compile-only output is not delivery.
  • Extend DSA, draft-model or auxiliary state only with a matching semantic contract and representative model gate. Unknown state stays unsupported.

Acceptance: real dense, SWA/sink and recurrent/hybrid model recovery, plus boundary/preemption/restart adversarial cases. Check required-state soundness, generation safety, bounded-case minimality, retained bytes, retention amplification and semantic reclaim latency. Retain a workload with measured state reduction; do not describe recovery validation alone as this compiler.

S8 — Product organization, final artifacts and release

Depends on: accepted scope from S1–S7; retain explicit exclusions for unqualified hardware/research work. Owners: docs/README/website, examples, containers and existing build/release workflows.

  • Organize README/docs like LMCache: product purpose, supported deployment modes, installation, short quickstart, engine/model recipes, architecture, operation and benchmark methodology. Use its separation of library, tests, examples, benchmark programs and tools as guidance, preserving OrbitKV’s Rust owners.
  • Preserve OrbitKV’s visual identity. Keep one canonical technical document per subject; remove chronological development diaries and duplicate result tables from user paths. Show full local/distributed/P/D ownership and data/control paths in diagrams, with status and limits next to capability claims.
  • Publish curated experiments from external versioned artifacts, linking model, engine/source revisions, transport, budgets, controls and reproducible scripts. Verify links/search/navigation after moving legacy result content.
  • Build independent Manager and engine images from validated artifacts. Separate Manager/client packaging only at a clear dependency/version contract; qualify shared-process resources, GPU access and SSD mounts before publishing manifests.
  • Preserve licenses and bundled upstream provenance. Keep heavy GPU/RDMA gates explicitly marked and distinguish engine support from runtime build success.
  • Build final wheels/images, install outside the checkout on both hosts and repeat the claimed sharing/P/D/restart gates with those exact hashes. Source test evidence cannot replace final artifact qualification.
  • Use the existing release workflow for publication within the user’s release authorization. Publishing is a distinct delivery action; manual candidate builds and successful CI do not mean a PyPI release has happened.

Acceptance: clean install and documented launch work for each supported profile; no undocumented source-tree/native-library dependency; CI and website checks pass; no generated experiment results are tracked; package/image hashes, external evidence, actual publication status and remaining limits are reviewable.

Review contract for every delivery

The implementing agent provides:

  1. Stage/substage, base and final commits, and the changed user-visible behavior.
  2. The concrete state/resource owner and invariants affected; removed APIs and duplicate paths; any remaining compatibility or forwarding layer justified.
  3. Exact checks and exit status. Run only the gates relevant to the change, plus required repo checks. Never build native libraries during live runtime tests.
  4. External evidence locations and artifact hashes, with failed controls and environmental limitations. No private credentials or results in the commit.
  5. Updated completion-plan/support status and the next dependency. Passing unit tests does not close a hardware, model-serving or performance cell.

The reviewer inspects implementation and call sites, repeats the smallest tests that establish the changed contract, audits lifecycle failures and benchmark fairness, and checks public claims against exact artifacts. Acceptance is accepted, changes required, or implemented / qualification blocked, with concrete reasons. The last status does not mark a gate complete.

References and scope

These sources guide organization and workflows, not OrbitKV capability claims:

Architecture contracts remain in distributed cache, transport, state planning, deployment, P/D and release documentation. Existing historical reports describe their recorded revisions only.

Consolidated documentation and retired work

The former root TODO, implementation handoff, communication sequence and roadmap have been consolidated here. Their historical revision remains available at the S1 source snapshot. All 70 open obligations were assigned to S2–S8: storage/codec execution and performance to S4; transfer generations/revocation to S3; engine identity, Radix/P-D/topology/container contracts to S5; cost/network admission and explicit router/backend research to S6; retention semantics to S7; package/image/provenance and recurring release qualification to S8. S2 retains metadata capacity/faults.

Retired proposals remain retired: a second shard Catalog, binary OrbitKV metadata inside Mooncake, unused Python state facades, a generic region RPC without a consumer, and duplicated backend/cost wrappers. Existing raw 2D DMA, bounded raw parts, single-part layer events and local global indexes need extension or qualification, not another implementation. The former direct engine-page SSD and multi-writer assembly item is implementation partial, not just untested.

Historical result tables stay at immutable revisions or external evidence storage. Current resource contracts remain in their owner documents. No compatibility stub documents or duplicate execution queues replace the deleted files.