Testing & Validation Matrix

This document inventories NoKV’s automated coverage and provides guidance for extending tests. It aligns module-level unit tests, integration suites, and benchmarking harnesses with the architectural features described elsewhere.

1. Quick Commands

# All unit + integration tests (uses local module caches)
GOCACHE=$PWD/.gocache GOMODCACHE=$PWD/.gomodcache go test ./...

# Focused distributed transaction suite
go test ./percolator/... ./raftstore/client/... -run 'Test.*(Commit|Prewrite|TwoPhaseCommit)'

# Focused distributed migration / membership / restart suite
go test ./raftstore/integration -count=1

# Crash recovery scenarios
RECOVERY_TRACE_METRICS=1 \
go test ./... -run 'TestRecovery(RemovesStaleValueLogSegment|FailsOnMissingSST|FailsOnCorruptSST|ManifestRewriteCrash|SlowFollowerSnapshotBacklog|SnapshotExportRoundTrip|WALReplayRestoresData)' -count=1 -v

# Protobuf schema hygiene
make proto-check

# gRPC transport chaos tests + watchdog metrics
CHAOS_TRACE_METRICS=1 \
go test -run 'TestGRPCTransport(HandlesPartition|MetricsWatchdog|MetricsBlockedPeers)' -count=1 -v ./raftstore/transport

# Sample Coordinator service for shared TSO / routing in distributed tests
go run ./cmd/nokv coordinator --addr 127.0.0.1:2379 --id-start 1 --ts-start 100 --workdir ./artifacts/coordinator

# Local three-node cluster (includes catalog bootstrap + Coordinator)
./scripts/dev/cluster.sh --config ./raft_config.example.json
# Tear down with Ctrl+C

# Docker-compose sandbox (3 nodes + Coordinator)
docker compose up -d
docker compose down -v

# Build RocksDB locally (installs into ./third_party/rocksdb/dist by default)
./scripts/build_rocksdb.sh
# YCSB baseline (records=1e6, ops=1e6, warmup=1e5, conc=16)
./scripts/run_benchmarks.sh
# YCSB with RocksDB (requires CGO, `benchmark_rocksdb`, and the RocksDB build above)
LD_LIBRARY_PATH="$(pwd)/third_party/rocksdb/dist/lib:${LD_LIBRARY_PATH}" \
CGO_CFLAGS="-I$(pwd)/third_party/rocksdb/dist/include" \
CGO_LDFLAGS="-L$(pwd)/third_party/rocksdb/dist/lib -lrocksdb -lz -lbz2 -lsnappy -lzstd -llz4" \
YCSB_ENGINES="nokv,badger,rocksdb" ./scripts/run_benchmarks.sh
# One-click script (auto-detect RocksDB, supports `YCSB_*` env vars to override defaults)
./scripts/run_benchmarks.sh
# Quick smoke run (smaller dataset)
NOKV_RUN_BENCHMARKS=1 YCSB_RECORDS=10000 YCSB_OPS=50000 YCSB_WARM_OPS=0 \
./scripts/run_benchmarks.sh -ycsb_workloads=A -ycsb_engines=nokv

Tip: Pin GOCACHE/GOMODCACHE in CI to keep build artefacts local and avoid permission issues.

2. Module Coverage Overview

3. System Scenarios

4. Observability in Tests

• RECOVERY_METRIC logs – produced when RECOVERY_TRACE_METRICS=1; helpful when triaging targeted recovery suites and CI failures.
• TRANSPORT_METRIC logs – emitted by transport chaos tests when CHAOS_TRACE_METRICS=1, capturing gRPC watchdog counters during network partitions and retries.
• Stats snapshots – stats_test.go verifies JSON structure so CLI output remains backwards compatible.
• Benchmark artefacts – stored under benchmark/results/ for shared suites and under suite-local result directories where applicable, for example benchmark/results/ycsb/.

5. Extending Coverage

1. Property-based testing – integrate testing/quick or third-party generators to randomise distributed 2PC sequences (prewrite/commit/rollback ordering).
2. Stress harness – add a Go-based stress driver to run mixed read/write workloads for hours, capturing metrics akin to RocksDB’s db_stress tool.
3. Distributed readiness – strengthen raftstore fault-injection and long-run tests (leader transfer, transport chaos, snapshot catch-up) with reproducible CI artifacts.
4. CLI smoke tests – simulate corrupted directories to ensure CLI emits actionable errors.

6. Distributed Test Layers

• Protocol unit tests: package-local tests under raftstore/peer, raftstore/store, raftstore/admin, raftstore/snapshot, and raftstore/migrate validate one protocol surface at a time.
• Node-local integration tests: store/admin tests verify snapshot install, membership application, and region runtime publication without booting a full cluster.
• Multi-node deterministic data-plane integration tests: raftstore/integration uses raftstore/testcluster to boot real stores, wire transports, and drive migration/member flows against live runtimes.
• Multi-node deterministic control-plane integration tests: coordinator/integration/*_test.go uses coordinator/testcluster to boot 3 coordinator + replicated meta, exercise rooted watch/reload propagation, follower write rejection, allocator-fence/remove-region propagation, and control-plane read staleness without mixing those cases into store/data-plane tests.
• Restart and recovery suites: raftstore/integration/restart_recovery_test.go covers restarted followers, removed-peer dehost persistence, and leader restart with subsequent membership changes.
• Control-plane degradation and publish-boundary tests: raftstore/integration/coordinator_degraded_test.go and raftstore/integration/snapshot_interruption_test.go cover live Coordinator outage after startup and failpoint-driven snapshot interruption before peer publication.

When adding new distributed tests:

• use raftstore/testcluster for store/data-plane behavior
• use coordinator/testcluster for control-plane / replicated-root behavior
• avoid embedding ad-hoc cluster bootstrap helpers into feature-specific test files

7. Distributed Fault Matrix

Next fault-matrix additions should focus on:

• more publish-boundary failpoints around snapshot install and migration init
• deeper transport/interleave chaos beyond partition + recovery, especially more concurrent membership combinations and repeated multi-link flaps

Keep this matrix updated when adding new modules or scenarios so documentation and automation remain aligned.