Fit Calculator

How these numbers were taken

All measurements are from one RTX 3090 (24 GiB, sm_86, CUDA 12.8) on this fork, with ninfer_bench and ninfer-perplexity.

• Memory is read from the engine's own capacity report at load, not inferred. Per-token cache cost is kv_payload_bytes / max_context, linear with no intercept — but the cache itself is reserved in whole 64-token physical pages (src/core/paged_kv_cache.h), so this page rounds a requested context up to its page boundary before pricing the cache, and rounds a reported maximum context down to one so it is never a page-partial figure the engine would round past. Two more things about the capacity report are easy to get wrong, so both were checked at four context lengths rather than assumed: the engine's sequence capacity contains the KV payload rather than sitting beside it, so adding the two columns double-counts the cache; and the remainder above the cache is , a fixed block rather than a per-token cost. Measuring it at a single context length would have made it look like roughly 4 KB per token on the 27B, which understates memory at short contexts and overstates it badly at long ones. It is identical across all seven formats.
• Perplexity is ninfer-perplexity on the fixed ninfer-ppl-1m-v1 corpus with --quick, context/stride 4096/2048, 261,167 scored tokens — the same corpus and window for all seven formats, so the differences are comparable to each other. Absolute values are not comparable to perplexity numbers from other tools. All seven are on Qwen3.8-27B (the rk4v4 figure is ratio-derived from a later session, not read directly): the ranking is a property of how each format quantizes, so it carries across models, but the absolute figures are the 27B's and are shown as such when another model is selected.
• Decode speed is a -pg P,128 run: prefill P tokens, then time 128 decode steps on top of that cache. Prefill and decode are timed separately, so these are decode-only rates at a genuine cache depth.

The model is checked against the engine, not just against itself. Asking for a 262,144-token INT8 context on the 27B is refused with minimum Engine runtime reservation requires 9197389568 bytes in addition to 1073741824 bytes of automatic headroom. Adding this page's own terms — 262,144 × 33,792 for the cache, 166,438,656 fixed, 16,384 for the per-token remainder, 159,981,568 of workspace and 12,582,912 of graph allowance — gives 9,197,389,568. An exact match, so the arithmetic here is the arithmetic the engine actually applies.

One difference worth knowing: --kv-capacity auto holds back a further 1 GiB of sizing headroom on top of everything above. The largest context this page reports is what physically fits; the largest the engine will auto-size to is whatever fits with that extra gibibyte also spoken for. Ask for a context explicitly and you get the former.

What this does not model

Single-lane serving only. Concurrency changes the picture: --kv-capacity is a shared pool while --max-context is a per-request cap, so a second lane does not cost twice the memory unless you also want twice the per-request context. Vision in overlay residency costs no resident capacity. Prefill throughput, time-to-first-token and multi-GPU expert offload are all out of scope here — see docs/performance.md.

Speculative modes are now modelled per mode, and cross-checked against the engine. Every term — resident weights, CUDA graph allowance, the fixed and per-token parts of the non-KV sequence block, and the extra cache each backend reserves — is measured separately for each speculation option by scripts/sweeps/speculative-memory-terms.ps1. Two configurations are pinned against the engine's own refusal arithmetic: MTP3 with the draft head on the 27B at a 262,144-token context agrees to 0.044%, and DFlash-3 on the 35B agrees exactly. Before this, the page reused the 12 MiB no-speculation graph allowance and added no extra cache for any mode, which understated that DFlash-3 case by 1,289 MiB.

The two backends behave differently and the page reflects that. MTP buffers extra draft tokens in the KV format, so its cost is a multiplier — +6.30% per token on the 27B, +10.08% on the 35B — identical across storage formats. DFlash v1 instead reserves a fixed 4,096 bytes per token regardless of format, which as a ratio would read as +38.8% on int8 and +71.1% on nvfp4. DFlash2 costs nothing per token and pays entirely in a graph allowance that is itself context-dependent: 288 MiB up to 8,192 tokens and 384 MiB beyond, against 12 MiB with speculation off.

Weight profile. Every model here is measured against the one groupwise-int-quantized weight artifact its download script fetches. The runtime also accepts other weight profiles for the same model family with different resident sizes (see src/runtime/engine/context_cost_defaults.cpp); these figures do not apply to a differently-quantized weight artifact.

Estimates between measured depths are interpolated, and anything past the deepest measured point is extrapolated and marked as such. Both are done in seconds-per-token rather than tokens-per-second, because step time is what grows linearly with cache depth; extrapolating the rate directly would have it fall through zero a few hundred thousand tokens out. Treat an extrapolated figure as the shape of the curve, not a promise.