OrbitKV Metrics Guide

This guide explains how to collect, export, and visualize metrics from OrbitKV.

Overview

OrbitKV supports two methods for exposing metrics:

Method 1: Direct Prometheus (Recommended)

Cache Manager → Prometheus → Grafana
   (/metrics)      (scrape)    (visualize)

• Simpler deployment (2 components)
• OrbitKV exposes /metrics endpoint directly
• Use examples/metric-prometheus/

Method 2: OTLP via OpenTelemetry Collector (Optional)

Cache Manager → OpenTelemetry Collector → Prometheus → Grafana
   (OTLP/gRPC)         (HTTP scrape)       (HTTP queries)

• More flexible (supports multiple backends)
• Useful if you already have OTel infrastructure
• Use examples/metric/

Available Metrics

OrbitKV exposes the following metrics for monitoring KV cache operations:

Bounded cost observations

See the P4.1 boundaries before combining measurements. All labels below come from fixed enums; device, peer, request and state identities are kept out of the exported series.

path="ssd_uring_restore" and path="ssd_cufile_restore" identify complete SSD restore routes to the same GPU completion target. Their estimates include queueing and host materialization/H2D or GPU staging/scatter/decode. Other paths continue to estimate service from submission to observed completion; stage histograms retain their queue/admission/service/total meanings. Neither failed nor cancelled, timed-out or abandoned work trains successful estimates.

Raw GPU-copy keys retain separate logarithmic buckets for input descriptors and DMA-coalesced ranges. Actual execution samples and shadow candidates use the same shape from the validated copy list; the executor’s merge iterator supplies the range count. Refining that key does not restart queue/admission timing. These fields are bounded estimator dimensions, not additional metric labels.

The bounded SSD-route key includes buckets for the complete stored source image’s bytes/fragments and the SSD-derived portion of target bytes/fragments, alongside the full task shape, representation and shared SSD/device identity. These fields are not metric labels. Stored source sizes do not turn padded raw bytes into known logical data. Composite route totals and nested operation timers overlap; physical bytes are reported only by their existing I/O owners. Do not sum either parent/child timings or logical bytes across those paths.

This family and its shadow work are disabled by default. Enable them with ORBITKV_COST_OBSERVATIONS=1 before Manager startup. Existing operational metrics remain enabled independently. See the overhead gate before enabling cost observations in serving.

Storage encoding

See codec qualification. orbitkv_storage_codec_bytes_total labels logical and stored count encoded publications; stored includes per-segment alignment and mixed raw segments. orbitkv_storage_codec_transfer_bytes_total labels d2h/h2d count actual codec-path payload transfers, including CPU fallback. orbitkv_storage_codec_reserved_bytes holds active codec operation reservations through completion. orbitkv_storage_codec_workspace_bytes measures retained GPU arenas, including idle workers and SSD decoder inputs; they are released at worker teardown. orbitkv_storage_codec_workspace_allocations_total counts arena allocations and growth, while orbitkv_storage_codec_batches_total and the orbitkv_storage_codec_batch_segments histogram report actual batch sizes. Budgets apply per worker, not across the whole Manager. Duration and decode-failure metrics track complete codec transfers, corrupt objects rejected before admission and failed GPU codec restores. SSD prefetch byte counters now count stored physical segment bytes, including encoded payloads and padding. GPU load counters count reconstructed logical bytes. Codec transfer counters cover saves through the codec worker and restores of encoded slots, including their raw sibling segments; raw-only restores use the ordinary copy backend and are excluded from the codec counter. GDS counters apply only to cuFile operations and do not prove native GDS.

Query ownership and preparation

• orbitkv_query_reserved_bytes tracks total conservative per-owner bytes. Admission and release update this unlabelled counter under the budget lock; use it to check the configured query budget. Shared physical pages may be counted for several owners. Use the pool metric for actual allocator occupancy.
• orbitkv_query_speculative_reserved_bytes tracks the speculative share under the same lock; compare it with one quarter of the query budget.
• orbitkv_query_reserved_bytes_by_phase separates warming, preloading, prepared, preparing, ready, and restoring. Phase samples are diagnostic: collection can overlap a transition, so their sum is not an atomic budget snapshot. Owned preparation moves from preloading to prepared, then into foreground ownership on claim without releasing its total reservation. Unowned warming retains no ready lease; its bytes return to zero after preparation even without polling.
• orbitkv_query_budget_waits_total and orbitkv_query_budget_bypasses_total include warmup admission attempts. A skipped warmup does not imply the later demand query will bypass restoration.
• orbitkv_query_coalesced_reads_total counts owners joining an identical backing-read plan. It is not a byte-savings counter.
• orbitkv_warmup_prepared_bytes_total, orbitkv_warmup_restored_bytes_total, orbitkv_warmup_unused_bytes_total, and orbitkv_warmup_pending_bytes follow each speculative-origin page through successful local H2D or last-owner release. A query hit is not use; restored bytes count unique page footprints, not individual layer copies or exact H2D traffic. Both unowned warming and owned preparation use these physical-page counters; compare them in separate runs.
• orbitkv_warmup_wait_byte_seconds_total records byte-weighted time from preparation to first successful H2D or unused release, labelled by outcome. Unresolved live intervals are excluded. See the accounting contract.
• orbitkv_warmup_foreground_skips_total counts unowned warming hints skipped because foreground preparation, leases or GPU transfers already own query bytes. Owned preparation can overlap foreground work within its speculative share and total byte limits. See request preparation.

Optional request timelines correlate enqueue, preparation, restore and engine consumption. Cache-tier probe counts include warmups and demand; they are not end-user request hit rates.

Pool Metrics (Pinned Memory)

• orbitkv_pool_used_bytes (Gauge)

  • Current pinned memory pool usage in bytes
  • Use case: Monitor memory pressure

• orbitkv_pool_capacity_bytes (Gauge)

  • Total pinned memory pool capacity in bytes
  • Use case: Derive pool utilization

• orbitkv_pool_largest_free_bytes (Gauge)

  • Largest contiguous free region in pinned pool (fragmentation signal)
  • Use case: Distinguish true exhaustion vs fragmentation (largest_free << free_bytes)

• orbitkv_pool_alloc_failures_total (Counter)

  • Total allocation failures after eviction retries
  • Use case: Detect memory exhaustion issues

Cache Metrics (Block-level)

• orbitkv_cache_candidate_hits_total, orbitkv_cache_candidate_misses_total (Counters)

  • Metadata-only recovery discovery across attention and auxiliary groups
  • Attention counts the available contiguous prefix; auxiliary groups count independently available positions. The remaining positions count as misses.
  • Candidates hold no leases and can become stale. These counters describe planning availability, not completed reads, transferred bytes, or GPU reuse.

• orbitkv_cache_block_hits_total (Counter)

  • Blocks returned by terminal prefix reads, including warmup reads
  • Excludes metadata-only candidate discovery; does not prove GPU transfer

• orbitkv_cache_block_misses_total (Counter)

  • Missing suffix blocks reported by terminal prefix reads
  • Use candidate-miss counters for recovery plans rejected before payload reads

• orbitkv_cache_tier_block_requests_total (Counter)

  • Per-decision query_prefetch block attribution by tier (ram, remote, ssd, or miss)
  • Use case: Calculate overall hit ratio and each cache tier’s contribution from one consistent denominator
  • Invariant: for each attributed decision, ram + remote + ssd + miss == block_hashes.len()
  • Semantics: this is decision attribution. tier="remote" and tier="ssd" mean the block was selected to be satisfied by that backing tier; they do not guarantee the later backing operation succeeded.

• orbitkv_cache_block_insertions_total (Counter)

  • New blocks inserted into cache
  • Use case: Track cache growth

• orbitkv_cache_block_evictions_total (Counter)

  • Blocks evicted from cache due to memory pressure
  • Use case: Monitor eviction frequency, tune pool size

• orbitkv_cache_block_evictions_by_class_total (Counter)

  • Blocks evicted from cache due to memory pressure, labelled by replacement class (reclaimable, probationary or retained)
  • Use case: Verify remote-fetched replicas are reclaimed before locally produced blocks

• orbitkv_cache_block_evictions_still_referenced_total (Counter)

  • Evicted blocks that still had external references (eviction did not immediately reclaim pinned memory)
  • Use case: Explain “evictions spike but pool_used_bytes doesn’t drop”

• orbitkv_cache_eviction_reclaimed_bytes_total (Counter)

  • Estimated bytes actually reclaimed in pinned allocator after cache eviction
  • Use case: Measure effectiveness of eviction under real reference patterns

• orbitkv_cache_resident_blocks (Gauge)

  • Current number of sealed blocks resident in cache, labelled by replacement class (reclaimable, probationary or retained)
  • Use case: Track cache size and source-based replacement pressure in blocks

• orbitkv_cache_resident_bytes (Gauge)

  • Current sealed block bytes resident in cache (sum of footprints)
  • Use case: Attribute pinned pool usage to cache residency

• orbitkv_cache_protected_bytes (Gauge)

  • Bytes in the protected segment when --cache-protected-percent is enabled; bounded by that percentage of pool capacity. Zero with protection disabled. These bytes are part of residency, not an additional pool reservation.

• orbitkv_cache_policy_promotions_total, orbitkv_cache_policy_demotions_total (Counters)

  • Foreground promotions into the protected segment and capacity demotions back to probation. Speculative peeks do not promote. Catalog demotion and cleanup affect protected bytes but are not policy-capacity demotions.

• orbitkv_ssd_write_admission_skips_total (Counter)

  • Pages skipped before the SSD write queue, by reason: cold, resident, pending or duplicate. Intentional policy decisions, separate from orbitkv_ssd_write_queue_full_total pressure drops. See policies.

• orbitkv_cache_residence_duration_seconds (Histogram)

  • RAM resident block lifetime from its first successful cache insertion to removal, measured in seconds
  • Labels: reason (pressure or cleanup)
  • pressure: allocator pressure removed the block through LRU reclaim
  • cleanup: the memory-cache cleanup endpoint removed the block
  • Buckets: 1s, 5s, 10s, 30s, 1m, 2m, 5m, 10m, 30m, 1h, 2h, 6h, 12h, 24h, and +Inf
  • Use case: Track typical and tail cache residence time and visualize the eviction-age distribution
  • Scope: each RAM residence is a separate lifetime. A block reinserted after eviction starts a new lifetime; cache hits, duplicate inserts, and replacement-class changes do not reset the original insertion time.
  • The lifetime ends when the block leaves the resident cache, even if an outstanding Arc keeps its pinned memory allocated. Correlate orbitkv_cache_block_evictions_still_referenced_total and orbitkv_cache_eviction_reclaimed_bytes_total to diagnose delayed memory reclamation.
  • SSD ring-cache overwrite and blocks still resident at server shutdown are not observed.

• orbitkv_pinned_for_load_entries (Gauge)

  • Current number of pinned_for_load entries (instance_id, block_key)
  • Use case: Diagnose load-path pins keeping evicted blocks alive

• orbitkv_pinned_for_load_refs (Gauge)

  • Current outstanding pinned_for_load consumer refcount (sum of per-entry counts)
  • Use case: Detect stuck consumers / missing release on load path

• orbitkv_pinned_for_load_unique_blocks (Gauge)

  • Current number of unique blocks referenced by pinned_for_load
  • Use case: Understand how many distinct blocks are being kept alive by pins

• orbitkv_pinned_for_load_unique_bytes (Gauge)

  • Current bytes referenced by pinned_for_load (unique blocks; sum of footprints)
  • Use case: Attribute pinned pool usage to load-path pins

HLL Reuse Metrics

• orbitkv_hll_cardinality (Gauge)

  • Estimated distinct (namespace, block hash) objects classified as misses in a configured sliding window
  • Labels: window (15m, 1h, 1d by default)
  • Use case: Derive approximate prefix reuse over longer windows without storing every block hash

• orbitkv_hll_total_requests (Gauge)

  • Total queried blocks in the same configured sliding window, including ready blocks and duplicates
  • Hybrid recovery records attention-prefix discovery once. Materializing the selected ranges does not count the same lookup again; auxiliary groups and speculative preparation do not enter this denominator. A dense prepared claim counts the ordinary lookup once. Queries stopped before finishing their evidence, including finite-batch prepared claims, do not enter this estimate: an unread page is unknown, not a demonstrated miss.
  • Labels: window (15m, 1h, 1d by default)
  • Use case: Denominator for HLL-based reference reuse rate

• orbitkv_hll_estimated_hit_rate (Gauge)

  • Server-computed miss-based infinite-cache reuse reference from the same HLL snapshot as the two gauges above
  • Labels: window
  • Value is clamped to [0, 1]

The existing cardinality and total metrics are retained. Existing PromQL continues to work, but new dashboards should prefer the direct gauge because it applies the same cardinality clamp as the tracker:

1 - (
  orbitkv_hll_cardinality{window="1h"}
  /
  clamp_min(orbitkv_hll_total_requests{window="1h"}, 1)
)

orbitkv_hll_estimated_hit_rate{window="1h"}

This is a metrics semantic update, not an /metrics protocol breaking change: metric names, types, existing labels, and the HTTP endpoint are unchanged; the new gauge is additive. The default HLL size changes from 16,384 registers (bucket_bits=14, about 0.8% standard error) to 65,536 registers (bucket_bits=16, about 0.4%). Three default windows use about 192 KiB of register storage; sliding slots make the live tracker a few MiB per server. The setting remains configurable with --metric-hll-bucket-bits.

Save Metrics (GPU → CPU)

• orbitkv_save_bytes_total (Counter)

  • Logical bytes published from GPU pages, before storage encoding
  • Use case: Monitor save throughput

• orbitkv_save_duration_seconds (Histogram)

  • Save operation latency distribution
  • Use case: Track save performance (p50, p99)

Load Metrics (cache → GPU)

• orbitkv_load_bytes_total (Counter)

  • Logical bytes restored from DRAM or SSD into engine GPU pages
  • Use case: Monitor load throughput

• orbitkv_load_duration_seconds (Histogram)

  • GPU restore duration, including SSD reads on the selected cuFile or explicit io_uring restore lane
  • Use case: Track load performance (p50, p99)

• orbitkv_load_failures_total (Counter)

  • Load operation failures (e.g., transfer errors)
  • Use case: Detect data transfer issues

SSD Cache Metrics

• orbitkv_ssd_backend_fallbacks_total (Counter) - Automatic cuFile admission disabled after initialization or operation failure. The default policy uses io_uring thereafter; an explicit cuFile read route does not silently change. Logs record the reason.
• orbitkv_ssd_cufile_write_bytes_total (Counter) - Physical cuFile write bytes, including padding.
• orbitkv_ssd_cufile_write_seconds (Histogram) - Async submission-to-completion latency, including GPU gather and polling; excludes waiting for a slot.
• orbitkv_ssd_cufile_write_failures_total (Counter) - Failed or short GPU-backed writes.
• orbitkv_ssd_cufile_read_bytes_total (Counter) - Physical cuFile read bytes, including aligned edges. Does not distinguish native GDS from CPU compatibility.
• orbitkv_ssd_cufile_read_seconds (Histogram) - Async submission-to-completion latency, including GPU scatter and polling; excludes waiting for a slot.
• orbitkv_ssd_cufile_read_failures_total (Counter) - Failed or short cuFile reads.
• orbitkv_ssd_read_pinned_bytes (Gauge) - SSD extents owned by restore leases.
• orbitkv_ssd_gpu_staging_bytes (Gauge) - Registered GPU storage staging memory.
• orbitkv_ssd_cufile_inflight_batches (Gauge) - Occupied staging slots until I/O and scatter completion, at most two per instance/device.
• orbitkv_ssd_gpu_write_fallbacks_total (Counter) - Write jobs using host publication after the eight-job GPU write admission limit is reached.
• orbitkv_ssd_pinned_write_skips_total (Counter) - Reservations rejected to protect an active SSD read or write. See GPU storage recovery.
• orbitkv_ssd_write_bytes_total (Counter) - Bytes written to SSD cache
• orbitkv_ssd_write_duration_seconds (Histogram) - Per-block io_uring write submission and completion latency; cuFile writes use their own duration histogram. Concurrent block writes overlap; summing these durations does not measure wall-clock flush time.
• orbitkv_ssd_prefetch_bytes_total (Counter) - Successfully read and validated SSD bytes, including reads an engine may not subsequently consume.
• orbitkv_ssd_prefetch_success_total (Counter) - Successful SSD prefetches
• orbitkv_ssd_prefetch_failures_total (Counter) - Failed SSD prefetches
• orbitkv_ssd_prefetch_duration_seconds (Histogram) - Prefix prefetch or explicitly leased host-read latency, including queueing, pinned-memory allocation, reads, and block reconstruction. Excludes H2D and does not prove a successful GPU restore; correlate with read bytes, failures, and load bytes.

Tier Attribution Semantics

orbitkv_cache_tier_block_requests_total{tier} is the canonical metric for explaining how each cache tier contributes to prefix-query hit ratio. It is emitted once for each query_prefetch decision and uses exactly one label: tier.

Tier values:

• ram: blocks already present in the resident RAM cache at the decision point
• remote: blocks selected to be satisfied by Mooncake remote fetch
• ssd: blocks selected to be satisfied by SSD prefetch
• miss: blocks no tier selected for that decision, including SSD prefetch backpressure and residual blocks after Mooncake partial availability

This metric intentionally records decisions, not completed service outcomes. For backing failure correlation, use:

• orbitkv_candidate_cache_lookups{result="hit|miss"} counts key checks before lookup coalescing; orbitkv_candidate_lookup_rpcs{result="ok|error|timeout"} counts actual batched directory RPCs (an OK RPC can still contain a miss).
• orbitkv_remote_fetch_total{status="rejected"} counts source authorization rejection before payload submission; status="error" counts other fetch failures.
• orbitkv_remote_fetch_plan_segments includes attempted alternative-source segments; orbitkv_remote_fetch_plan_completed_segments counts completed ones.
• orbitkv_ssd_prefetch_failures_total for SSD prefetch failures
• orbitkv_remote_stage_duration_seconds{stage,status} separates discovery_rpc, authorization, allocation, read, rebuild and release. Allocation/read/rebuild describe completed successful transfers; discovery includes timed-out attempts. Authorization includes first-use window setup and its bounded single-flight wait. Release measures time from native completion/abandonment to acknowledgement, including retry delays.
• orbitkv_transfer_reserved_bytes counts whole source allocations once per session, including overdue holds. orbitkv_transfer_expired_sessions counts overdue sessions still retaining memory; orbitkv_transfer_lock_timeouts_total counts the transition once. Timeout is not a release condition.
• orbitkv_transfer_completion_outstanding counts requester slots from window setup through source release acknowledgement (1024 total, at most 64 per source incarnation). orbitkv_transfer_completion_retries_total counts failed release attempts; orbitkv_transfer_completion_rejections_total counts new authorizations skipped at capacity. Outstanding records must drain as well as source pins.
• orbitkv_transfer_lock_rejections_total{reason="bytes|sessions"} counts bounded source admission failures. Source pins and query reservations must both drain after successful remote restoration.

orbitkv_cache_block_hits_total and orbitkv_cache_block_misses_total count terminal prefix reads. Metadata discovery has separate candidate counters, so a cold hybrid lookup can record a candidate miss without issuing a payload read. Use orbitkv_cache_tier_block_requests_total{tier} for read-tier decisions; do not add candidate counters to that denominator. Verify actual GPU reuse with orbitkv_load_bytes_total, not candidate or query hits alone.

Configuration

Cache Manager Parameters

Metrics Parameters:

• --http-addr: HTTP server address for health check and Prometheus metrics (default: 0.0.0.0:9091)

  • Always enabled for health check at /health
  • /metrics is enabled by default

• --enable-prometheus: Enable Prometheus /metrics endpoint (default: true)

  • When enabled, metrics are available at http://<http-addr>/metrics
  • Health check is always available at http://<http-addr>/health

• --metrics-otel-endpoint: OTLP gRPC endpoint for metrics export (optional)

  • Example: http://127.0.0.1:4321
  • Leave unset to disable OTLP export

• --metrics-period-secs: Metric export interval in seconds (default: 10)

  • Only used when --metrics-otel-endpoint is set

• --metric-hll-windows: Comma-separated HLL sliding windows for estimated prefix reuse (default: 15m,1h,1d)

  • Supported units: s, m, h, d
  • Each configured duration becomes a canonical window label. For example, the default config exports window="15m", window="1h", and window="1d".
  • Empty entries such as 15m,,1h and duplicate durations such as 1h,60m are rejected at startup.

• --metric-hll-bucket-bits: HLL bucket index bits (default: 16)

  • 2^16 = 65,536 registers per window and about 0.4% standard error.
  • Higher values use more memory and lower estimation error; 18 remains the supported maximum.

Example: Prometheus Metrics

cargo run -r --bin orbitkv-cache-manager -- \
  --addr 127.0.0.1:50055 \
  --devices 0 \
  --pool-size 30gb \
  --http-addr 0.0.0.0:9091 \
  --enable-prometheus

For an existing OpenTelemetry deployment, add --metrics-otel-endpoint http://127.0.0.1:4321 with the collector’s configured endpoint. Direct Prometheus remains available.

Distributed inventory recovery

Manager synchronization counters are:

These are background synchronization metrics, separate from request discovery and Mooncake data transfer. Lost replies can undercount applied records; a snapshot may retransmit already known entries. Repeated snapshot starts without commits indicate failure to converge. See directory recovery.

Environment Variables

• RUST_LOG: Control logging verbosity (e.g., info,orbitkv_core=debug)

Quick Start: Direct Prometheus (Recommended)

The examples/metric-prometheus/ directory provides a simple monitoring stack.

1. Start Cache Manager

# From repository root
cargo run -r --bin orbitkv-cache-manager -- \
  --addr 127.0.0.1:50055 \
  --devices 0 \
  --pool-size 30gb \
  --http-addr 0.0.0.0:9091 \
  --enable-prometheus

2. Start the Monitoring Stack

cd examples/metric-prometheus

docker compose up -d
# To stop: docker compose down

This starts two services:

• Prometheus (port: 9090) - Scrapes metrics from OrbitKV
• Grafana (port: 3000) - Visualizes metrics

3. Access Grafana Dashboard

1. Open browser: http://localhost:3000
2. Login: admin / admin
3. Navigate to Dashboards → OrbitKV Metrics

4. Test Metrics Endpoint

curl http://localhost:9091/metrics

Architecture details

Direct Prometheus

┌─────────────────┐
│ Cache Manager   │
│ :9091 /metrics  │
└────────┬────────┘
         │ Prometheus scrape
         ▼
┌─────────────────┐
│   Prometheus    │
│     :9090       │
└────────┬────────┘
         │ PromQL queries
         ▼
┌─────────────────┐
│    Grafana      │
│     :3000       │
└─────────────────┘

Port Reference

PromQL Query Examples

# Overall cache hit ratio from decision attribution (last 5 minutes)
sum(rate(orbitkv_cache_tier_block_requests_total{tier!="miss"}[5m])) /
sum(rate(orbitkv_cache_tier_block_requests_total[5m]))

# RAM contribution to total requested blocks
sum(rate(orbitkv_cache_tier_block_requests_total{tier="ram"}[5m])) /
sum(rate(orbitkv_cache_tier_block_requests_total[5m]))

# Remote contribution to total requested blocks
sum(rate(orbitkv_cache_tier_block_requests_total{tier="remote"}[5m])) /
sum(rate(orbitkv_cache_tier_block_requests_total[5m]))

# SSD contribution to total requested blocks
sum(rate(orbitkv_cache_tier_block_requests_total{tier="ssd"}[5m])) /
sum(rate(orbitkv_cache_tier_block_requests_total[5m]))

# Miss ratio from the same denominator
sum(rate(orbitkv_cache_tier_block_requests_total{tier="miss"}[5m])) /
sum(rate(orbitkv_cache_tier_block_requests_total[5m]))

# Average save latency (p50)
histogram_quantile(0.5, rate(orbitkv_save_duration_seconds_bucket[5m]))

# Average load latency (p99)
histogram_quantile(0.99, rate(orbitkv_load_duration_seconds_bucket[5m]))

# Save throughput (MB/s)
rate(orbitkv_save_bytes_total[1m]) / 1e6

# Pool memory utilization
orbitkv_pool_used_bytes / orbitkv_pool_capacity_bytes

# RAM cache residence-time quantiles for pressure evictions
histogram_quantile(
  0.50,
  sum by (le) (
    rate(orbitkv_cache_residence_duration_seconds_bucket{reason="pressure"}[5m])
  )
)

histogram_quantile(
  0.95,
  sum by (le) (
    rate(orbitkv_cache_residence_duration_seconds_bucket{reason="pressure"}[5m])
  )
)

histogram_quantile(
  0.99,
  sum by (le) (
    rate(orbitkv_cache_residence_duration_seconds_bucket{reason="pressure"}[5m])
  )
)

# RAM cache residence-time buckets for a Grafana heatmap
sum by (le) (
  rate(orbitkv_cache_residence_duration_seconds_bucket{reason="pressure"}[5m])
)

# HLL estimated hit rate for the 1h window (preferred)
orbitkv_hll_estimated_hit_rate{window="1h"}

# Backward-compatible derivation from the retained gauges
1 - (
  orbitkv_hll_cardinality{window="1h"}
  /
  clamp_min(orbitkv_hll_total_requests{window="1h"}, 1)
)

# HLL estimated hit rate for every configured window
1 - (
  orbitkv_hll_cardinality
  /
  clamp_min(orbitkv_hll_total_requests, 1)
)

Troubleshooting

Metrics not appearing (Direct Prometheus)

1. Check OrbitKV is exposing metrics:

   curl http://localhost:9091/metrics

2. Check Prometheus targets:

   • Open http://localhost:9090/targets
   • Verify orbitkv target is UP

3. If Docker cannot reach host, ensure extra_hosts is configured:

   extra_hosts:
     - "host.docker.internal:host-gateway"

Best Practices

1. Monitor tier-attributed hit ratio using orbitkv_cache_tier_block_requests_total

   • Low hit rate → consider increasing --pool-size

2. Watch eviction rate: High evictions indicate memory pressure

   • Use rate(orbitkv_cache_block_evictions_total[5m])

3. Track allocation failures: Any failures indicate critical issues

   • Alert on orbitkv_pool_alloc_failures_total > 0

4. Analyze latency distributions: Use histogram quantiles

   • p50: Typical case performance
   • p99: Worst-case user experience

References

• Prometheus Query Language
• Grafana Dashboard Guide
• OpenTelemetry Documentation