Repository navigation
Conversation
| * heads_per_group * grouped.tail_stream; | ||
| } | ||
|
|
||
| inline NaxGqaScore nax_gqa8_work_score( |
There was a problem hiding this comment.
I would consolidate GQA8 admission without losing the measured gains. The plan selector uses this score as its starting choice. Rerun Q16/Q64/Q128 serving on the final head before carrying over the performance claims.
| "nax_work_plan", False | ||
| ): | ||
| if ctx.nax_work_plan is None: | ||
| ctx.nax_work_plan = ops.nax_prefill_plan( |
There was a problem hiding this comment.
Before changing the no-plan fallback, pass this TurboQuant lane’s own host lengths into the plan path. Preserve its existing GQA6 route for compact KV storage and add a regression case.
|
|
||
| // Each sequence keeps its own causal origin, full blocks and final Q/KV | ||
| // tails. Concatenating Q lengths would invent complete blocks across requests. | ||
| template <typename Sequences> |
There was a problem hiding this comment.
I would make this take one NaxSequence and keep only fields the selector reads. Production rejects batches and nonzero offsets before this call, so remove the unused aggregation and offset handling with their test.
| if (layout.heads_per_group > 1) { | ||
| if (geometry.kernel_page == 16) { | ||
| switch (layout.heads_per_group) { | ||
| case 2: rate = 1.0169304348373918; break; |
There was a problem hiding this comment.
I find these raw coefficients hard to maintain. Name the fitted values, normalizers, and margins. Explain the cost terms briefly beside the code and link this PR for the calibration record.
| @@ -0,0 +1,260 @@ | |||
| # Earlier grouped NAX measurements | |||
There was a problem hiding this comment.
Remove this file. These tables describe older policies. Keep those measurements in the PRs that produced them.
| every Q16/Q64 and prefix32 residue, and longer boundaries. Multi-sequence and | ||
| mixed automatic admission remains disabled pending its own performance study. | ||
|
|
||
| ## Calibration and independent validation |
There was a problem hiding this comment.
I would keep this page to the flag, supported widths, minimum 15% predicted margin, capacity reserve, and limits. Keep measurement details in the PR body and clearly label results from before the rebase.
| | `VLLM_MLX_DEVICE` | `gpu` | MLX device (`gpu` or `cpu`) | | ||
| | `VLLM_METAL_DISABLE_NAX` | `0` | Emergency override for automatic M5 NAX prefill attention. Set to `1` to force the non-NAX fallback. | | ||
| | `VLLM_METAL_NAX_GQA_PREFILL` | `1` | Automatically select three- or six-head grouped NAX prefill on the MLX backend for single-sequence causal attention without sliding windows, softcap or sinks, using BF16, head dimension 256, GQA ratio 6 and 16-token kernel pages. The work-based selector compares query and KV lengths, query/KV heads, detected GPU cores and the actual upstream pipeline. Unsupported features, mixed batches/suffix offsets, unknown capacity or insufficient pipeline limits retain upstream. Extra six-head admission uses complete groups per GPU core and full-tile work coverage for power-of-two KV-head layouts; its guarded-query allowance is calibrated only at 10 cores. Other layouts retain the existing work score; total KV length has no fixed 32K admission cap. Complete tiles omit bounds checks while partial Q/KV tiles remain guarded. Set `0` to retain upstream NAX. Read once at worker startup; restart after changing it. `VLLM_METAL_DISABLE_NAX=1` disables all NAX. | | ||
| | `VLLM_METAL_NAX_GQA_PREFILL` | `1` | Automatically select grouped NAX prefill for MLX BF16 D256, single-sequence ordinary causal requests. GQA6/page16 enumerates h2/h3/h6; GQA8/page32 enumerates h2/h4/h8. With exact CPU lengths and calibrated interleaved KV strides, a shared work/resource cost compares all candidates with actual upstream and the retained h3/h6/h4 route, including GPU capacity and a replacement margin. Other valid layouts retain the established policy; unsupported features, mixed batches, unknown capacity and insufficient pipeline limits retain upstream. Larger physical pages are translated; partial tails are guarded and there is no fixed 32K KV limit. Set `0` to retain upstream NAX. Read once at worker startup. `VLLM_METAL_DISABLE_NAX=1` disables all NAX. See [scope and validation](gqa-prefill.md). | |
There was a problem hiding this comment.
Keep this cell about the user control: what 1 and 0 do, supported geometries, and the required worker restart. Put scoring details beside the implementation.
| " }\n" | ||
| " return 0;\n" | ||
| " }\n" | ||
| ' if (argc == 2 && std::string(argv[1]) == "resources") {\n' |
There was a problem hiding this comment.
I would keep one small helper that calls the plan-based selector, with inputs and expected configs in a parameter table. Remove the resource-reporting modes and copied resource formulas. Keep the independent work-count oracle and pipeline-capacity coverage.
| whole_hint = int(kind == "prefill") | ||
| assert spy.calls[-1].nax_split_hint == (whole_hint if forwarded else -1) | ||
| assert spy.calls[-1].nax_prefill_split_hint == (1 if subset else -1) | ||
| if forwarded and supported == "subset": |
There was a problem hiding this comment.
I think explicit plan-call and forwarding expectations would make this matrix clearer. Put them in the parameter table and assert them directly. Keep the shared setup.
| num_decode_tokens=0, | ||
| max_decode_context_len=0, | ||
| ) | ||
| if width in (2, 8): |
There was a problem hiding this comment.
Pass the plan for every grouped case, including h3/h4/h6. Put the expected dispatch, width, compact-cache fallback, and kernel suffix in the parameter table. Each case should state the route it checks.
5190e1e to
47e3435
Compare
|
@LxYuan0420, updated to
Could you please take another look at the simplified selector and updated evidence? |
WindChimeRan
left a comment
There was a problem hiding this comment.
The performance gain from the PR description looks good, but I haven't check all the workload. Currently I'm not sure if this is the right direction.
Design issue:
- Two policies determine one decision
- Historical behavior becomes part of the architecture
- The calibration burden grows
- The interface gains another specialized input
the description mentions the cost model, but it does not clearly explain the architectural change.
Overall, the PR is hard to review. The author should cleanup the review noise, and make the design better motivated.
Score ordinary NAX and the compiled grouped widths from the same causal tile counts and GPU capacity. Use the existing length bound for cost only, including compact TurboQuant caches, and remove the previous GQA6 selector and per-layout rates. Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
47e3435 to
9ff66a2
Compare
Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Add pinned selector cases for grouped costs 0.34% below, 0.49% above and exactly at the 90% admission ratio, extend the policy driver with value-dim, token-stride and KV-length overrides, and assert ordinary NAX for each violated geometry guard. Mutation checks: changing the margin to 9%/11%, admitting equality or dropping any of the three guards now fails the suite. Signed-off-by: suntp <20786998+suntp@users.noreply.github.com>
Problem
Main groups three or six heads for GQA6 through heuristic admission and uses ordinary NAX for GQA8; the two cases share no common decision rule. Wider groups share KV reads but expose fewer independent threadgroups.
Solution
A single selection pipeline replaces the GQA6 heuristics and covers GQA8:
The legacy GQA6 selector and its per-layout rates are deleted rather than wrapped, and grouped launches reuse the existing host length bound, so the interface gains no new input. Supported shapes and switch.
Validation results
Mean TTFT change versus main (negative is faster). BF16, APC, one sequence, 16 output tokens; cached prefixes 31,680 (35B) and 31,360 (27B) tokens. 35B and M5 27B rows compare
2a311af/f6e94a2; Pro 27B compares9d3d0bb/e595597; Max 27B cached compares9d3d0bb/0e01a53(test-only head, same production code) over four AB/BA service pairs — 192 formal requests, cache hits, widths and outputs verified per request.Q64 -10.63% / -8.31%
Q128 -18.12% / -18.32%
Q64 -5.81% / -5.69%
Q128 -9.60% / -9.32%
Q64 -8.22% / -8.45%
Q128 -8.24% / -8.27%
Q144 +0.14% / +3.12%
Q1328 -2.04% / +0.11% †
Q144 +0.19% / +0.19% †
Q1328 +0.67% / +2.60%
Q144 -0.13% ±1.84 (h3→h2)
Q1328 +0.37% ±4.50 †
† Both arms used the same widths; ± is the 95% CI over the four pairs; ‡ see Limitations. CI passed 6,030 tests at
e595597; the re-run at0e01a53was cancelled by macOS runner capacity, not a test failure — that head only adds tests (native artifacts byte-identical, 52/52 local). Native parity covers 122 cases per switch state, and the new tests pin the 90% admission margin (89.71% / 90.49% / exactly 90%) and the geometry guards; mutating any of them fails the suite.Limitations & follow-ups
AI assistance: OpenAI Codex assisted with implementation, tests and this description.