RMSNorm
Standalone normalization
Rust-first · CUDA-backed · Engine-ready
Own the
operator boundary.
Loom Kernels turns inference-critical fusion points into explicit Rust contracts, handwritten CUDA, and reproducible engine evidence. No framework reinvention. No benchmark-free claims.
pub struct SiluAndMulDynamicFp8Spec {
rows: usize,
width: usize,
group_size: 64 | 128,
input_dtype: F16 | Bf16,
output_dtype: Fp8E4M3Fn,
}
// Contract → oracle → CUDA → engine gate01Contract
02Bridge
03Safe Rust
04CUDA
11qualified operator paths
Nativeclean-install wheel gate
SM90NVIDIA H20 evidence
0.24–0.25qualified vLLM range
Supported now
Small surface.
Real fusion value.
Loom admits operators where layout knowledge, quantization, or launch reduction matters to an inference engine. Dense, quantized, sparse, and grouped GEMM always stay with vendor libraries; Loom owns only the measured memory-bound work around them.
Add + RMSNorm
Residual update + normalization
RMSNorm + dynamic FP8
Normalization + GEMM input quantization
SiLU-and-Mul
Split-half SwiGLU activation
SiLU-and-Mul + block FP8
Activation + group-64/128 quantization
RoPE + paged-KV write
Packed Q/K rotation + native cache write
Greedy + sampled logprob
Argmax + logsumexp + gather + tie rank
Selected-token logprob + rank
Engine-owned sampling + one-pass normalization
Min-P filtering
In-place row-max threshold; shape-gated in vLLM
Paged MQA/GQA decode
GQA packing + local split-K/LSE; short shapes route into vLLM
Greedy speculative verify
Ragged acceptance + mismatch/bonus-token compaction
Architecture
One contract.
One execution path.
Every framework operator crosses the same versioned bridge into safe borrowed Rust dispatch. Adapters admit exact contracts before entering Loom; they never select an alternate implementation inside the library.
Backend-independent truth
Dtypes, shapes, layouts, aliasing rules, capabilities, and CPU oracles.
Internal launch boundary
Packaged handwritten CUDA and a private ABI used only by the safe Rust backend.
Safe execution
Owned or borrowed streams and memory, explicit layouts, checked dispatch, and benches.
Only framework boundary
Versioned, panic-contained entrypoints from engine tensors into safe borrowed Rust.
Translation only
PyTorch current-stream operators and narrow vLLM admission points without CUDA launches.
Measured on H20
Evidence before adjectives.
Correctness is checked before timing. Named baselines share the same dispatcher boundary, GPU warm-up, sample count, and provider-order reversal.
Add + RMSNorm
2.914 µsRaw H20 kernel median
BF16 · 8 × 4096RMSNorm + FP8
1.057–1.076×CUDA Graph ratio vs vLLM
BF16 · 8 × 4096SiLU + Mul + FP8
1.037–1.082×CUDA Graph ratio vs vLLM fused
BF16 · 8 × 11008 · G128Qwen2.5 FP8 engine
0.999–1.004×Exact-token path hit; end-to-end parity
0.5B · batches 1 / 8 / 32RoPE + paged-KV write
2.30–2.40×Dispatcher ratio vs separate vLLM ops
BF16 · Qwen2.5-style · 1–512 tokensGreedy + sampled logprob
1.129–1.250×Order-stable real-engine batch-latency ratio
Qwen2.5-0.5B · batches 1 / 8 / 32Selected-token logprob + rank
1.044–1.125×vLLM-owned sampling; order-stable engine ratio
Qwen2.5 top-k/top-p · batches 1 / 8 / 32Min-P filtering
1.885×0 tensor-sized temp; smaller batches route back to vLLM
F32 · 151,936 vocab · 128 rowsPaged MQA/GQA decode
1.154–2.374×24/24 routed vLLM backend cases win; other shapes fall back to FA3
FP16/BF16 · Hq/Hkv 32/8 · context ≤ 32Paged decode split-K/LSE
1.14–6.22×CUDA Graph ratio vs legacy Loom; FA3 remains the engine fallback
BF16 · batch 1–8 · context 128–1,024Greedy speculative verify
1.101–1.128×Bit-exact verifier-level ratio vs vLLM 0.24
H20 · batch 1–256 · draft length 1 / 4 / 8Real-model speculative decode
0.048–0.200%Verifier share of batch latency; exact native/Loom path, no end-to-end win
Qwen2.5 1.5B target + 0.5B draft · batch 1 / 8 / 32The pinned Qwen2.5 greedy and top-k/top-p selected-logprob gates show order-stable real-engine latency and TPOT wins. FP8 and RoPE+KV prove exact invocation but remain near parity. Min-P is shape-gated from its measured crossover. Paged decode now has a native-cache, context-32 vLLM route, synthetic-engine path proof, and a local 128–1,024-token split-K path that is 1.14–6.22× faster than legacy Loom. FA3 still wins the long engine envelope, and a broader Qwen2.5 geometry failed token and latency gates, so neither route was widened; serving-scale concurrency, memory, and goodput stay separate. Greedy speculative verification is bit-exact and faster at the operator boundary. Its real Qwen2.5 draft/target gate now proves exact native/Loom invocation, but the verifier is below 0.2% of batch latency, end-to-end ratios cross parity, and speculative decode loses to target-only for that workload.
Next boundaries
Build where engines pay.
With the ABI2 native-wheel and FP8 KV implementation/integration gates complete, the next K3 proof is a native-versus-FP8 capacity, quality, memory, TTFT, and TPOT study. The complete sampling tail follows. Tree, stochastic, and speculative-KV work waits for a named profile that shows material cost.
FP8 KV-cache compression
Reduce cache bytes and improve admitted context or batch size with quality and TPOT reported together.
Complete sampling tail
Own penalties, top-k/top-p, deterministic RNG, and top-k logprobs without host round trips.
KV-cache movement
Measure scheduler-facing block copy, gather, scatter, and compact work for prefix reuse and preemption.
Speculative extensions
Add tree, stochastic, or KV metadata only when a named profile exposes material non-GEMM cost.
Quantization plumbing
Remove scale, pack/unpack, dequant/requant, and layout passes around an unchanged vendor GEMM.
MoE routing + movement
Own routing, histogram/prefix sum, permutation, and combine while grouped GEMM stays vendor-owned.
Rust decode step
Prove zero-copy engine-neutral orchestration over borrowed tensors and streams without building an inference engine.
Start at the contract.
Inspect the operator surface, run the CPU gates, then qualify CUDA.