Repository navigation
Conversation
Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Ordinary SDPA counts NAX tiles for some prefill calls that run on a different kernel. In a mixed batch it walks the prefill rows twice to supply the whole-batch and prefill-suffix hints. Explicit TurboQuant also counts tiles for rows that stay compressed and for dequantized prefill when NAX is disabled.
Solution
For ordinary SDPA, check whether a prefill can consume a NAX hint before counting tiles. Capture both counts in one pass and reuse them across layers with the same sliding window. For TurboQuant, compute the hint on first use from the query and KV lengths of the selected dequantized rows. A pure decode step proceeds directly to its existing decode routing. The native dispatcher selects the kernel using the existing hint interface.
Validation results
On a real mixed batch, Gemma's two attention windows required 4 → 2 tile-count calls with NAX enabled and 4 → 0 with NAX disabled. For explicit TurboQuant the mixed batch required 2 → 0 calls with NAX disabled and 2 → 1 with NAX enabled. Dispatch families, decode partitions and recorded NAX grids matched on identical row views. Pure decode made zero NAX planning queries.
In an isolated CPU measurement with NAX disabled, Gemma's 24 readiness checks took 0.109 µs per mixed forward, compared with 5.23 µs for the four tile counts on main. Qwen's 28 checks took 0.126 µs, compared with 1.46 µs for the two tile counts on main. These are medians after subtracting loop overhead.
M5 Pro (20 GPU cores), 4-bit weights. Gemma 4 E4B used BF16 KV and NAX. Qwen3-0.6B used explicit TurboQuant with Q8_0 keys, Q3_0 values and tiled prefill. Each foreground request hit 512 cached tokens, appended 256 tokens and generated 32. Mixed batches had two foreground requests alongside two ongoing decodes.
Serving latency, throughput and same-policy drift
Latency values average the per-run request medians. Both adjacent changes use the PR relative to main. Drift compares the later and earlier runs of the same build.
Limitations & follow-ups
Gemma single-request TPOT rose 0.24%. Qwen single-request TPOT fell 2.74%, while main itself drifted 4.90%. The measured reduction is in tile counting. Serving latency varied with workload.
All foreground token IDs matched. One Gemma background stream differed at output token 36 between two main runs with different active decode row counts. Replaying both observed schedules reproduced each HTTP output on both builds. At two decode rows the two leading BF16 logits tied at 26.5. At four rows they were 26.625 and 26.5.
Timing covers this M5 Pro and cache length. Grouped NAX width selection uses the existing host length metadata in the native dispatcher and can land independently. Related PR #1300