Skip to content

[Refactor] Compute NAX prefill tile counts on demand - #1372

Open
suntp wants to merge 2 commits into
vllm-project:mainfrom
suntp:refactor/nax-tile-stats-on-demand
Open

suntp wants to merge 2 commits into
vllm-project:mainfrom
suntp:refactor/nax-tile-stats-on-demand

Conversation

@suntp

@suntp suntp commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Problem

Ordinary SDPA counts NAX tiles for some prefill calls that run on a different kernel. In a mixed batch it walks the prefill rows twice to supply the whole-batch and prefill-suffix hints. Explicit TurboQuant also counts tiles for rows that stay compressed and for dequantized prefill when NAX is disabled.

Solution

For ordinary SDPA, check whether a prefill can consume a NAX hint before counting tiles. Capture both counts in one pass and reuse them across layers with the same sliding window. For TurboQuant, compute the hint on first use from the query and KV lengths of the selected dequantized rows. A pure decode step proceeds directly to its existing decode routing. The native dispatcher selects the kernel using the existing hint interface.

Validation results

On a real mixed batch, Gemma's two attention windows required 4 → 2 tile-count calls with NAX enabled and 4 → 0 with NAX disabled. For explicit TurboQuant the mixed batch required 2 → 0 calls with NAX disabled and 2 → 1 with NAX enabled. Dispatch families, decode partitions and recorded NAX grids matched on identical row views. Pure decode made zero NAX planning queries.

In an isolated CPU measurement with NAX disabled, Gemma's 24 readiness checks took 0.109 µs per mixed forward, compared with 5.23 µs for the four tile counts on main. Qwen's 28 checks took 0.126 µs, compared with 1.46 µs for the two tile counts on main. These are medians after subtracting loop overhead.

M5 Pro (20 GPU cores), 4-bit weights. Gemma 4 E4B used BF16 KV and NAX. Qwen3-0.6B used explicit TurboQuant with Q8_0 keys, Q3_0 values and tiled prefill. Each foreground request hit 512 cached tokens, appended 256 tokens and generated 32. Mixed batches had two foreground requests alongside two ongoing decodes.

Serving latency, throughput and same-policy drift
Model and load Metric Main This PR Change Adjacent A/B changes Same-policy drift, main / PR
Gemma, single TTFT (ms) 83.231 83.337 +0.13% -0.19% / +0.44% -0.02% / +0.61%
Gemma, single TPOT (ms/token) 12.332 12.361 +0.24% +0.35% / +0.12% +0.39% / +0.16%
Gemma, single Foreground tokens/s 68.403 68.191 -0.31% -0.31% / -0.31% -0.31% / -0.31%
Gemma, mixed TTFT (ms) 156.727 156.663 -0.04% +0.01% / -0.09% +0.42% / +0.32%
Gemma, mixed TPOT (ms/token) 15.523 15.534 +0.07% +0.10% / +0.03% +0.11% / +0.04%
Gemma, mixed Foreground tokens/s 100.176 100.079 -0.10% -0.02% / -0.17% -0.11% / -0.26%
Qwen TQ, single TTFT (ms) 49.523 49.010 -1.03% -0.35% / -1.71% +1.27% / -0.10%
Qwen TQ, single TPOT (ms/token) 3.283 3.193 -2.74% -0.42% / -4.96% +4.90% / +0.12%
Qwen TQ, single Foreground tokens/s 208.869 213.208 +2.08% +0.41% / +3.80% -3.53% / -0.28%
Qwen TQ, mixed TTFT (ms) 65.225 65.215 -0.02% +0.26% / -0.29% +0.57% / +0.01%
Qwen TQ, mixed TPOT (ms/token) 4.299 4.309 +0.24% -0.06% / +0.55% +0.32% / +0.93%
Qwen TQ, mixed Foreground tokens/s 321.462 321.221 -0.08% -0.03% / -0.12% -0.53% / -0.62%

Latency values average the per-run request medians. Both adjacent changes use the PR relative to main. Drift compares the later and earlier runs of the same build.

Limitations & follow-ups

Gemma single-request TPOT rose 0.24%. Qwen single-request TPOT fell 2.74%, while main itself drifted 4.90%. The measured reduction is in tile counting. Serving latency varied with workload.

All foreground token IDs matched. One Gemma background stream differed at output token 36 between two main runs with different active decode row counts. Replaying both observed schedules reproduced each HTTP output on both builds. At two decode rows the two leading BF16 logits tied at 26.5. At four rows they were 26.625 and 26.5.

Timing covers this M5 Pro and cache length. Grouped NAX width selection uses the existing host length metadata in the native dispatcher and can land independently. Related PR #1300

Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant