Repository navigation
tune(sm90): refine small-M indexed MoE residency with work guards - #143
Andrew88Lin wants to merge 3 commits into
Conversation
Port the H200 A8 indexed MoE policy from andrewchen_microsoft/humming#2 to the current upstream candidate framework. Apply the measured CTA bounds before routed-wave selection, preserve sparse grid and specialized prefill behavior, and add regression coverage. Signed-off-by: Andrew Chen <andrewchen@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
We don't need to add those tests.
There was a problem hiding this comment.
Removed the added tests in 73e928d; the PR only changes the policy file.
Remove the test additions as requested in review. Preserve all existing tests and leave the optimization policy unchanged. Signed-off-by: Andrew Chen <andrewchen@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
The speedup in your introduction are based on a comparison with version 0.1.15. However, Humming has made several optimizations for H200 over the past few months. Could you provide the full benchmark results comparing against the latest main branch? |
|
Superseded by the guarded update |
Require enough output waves before adding a third CTA. Preserve two-CTA schedules for sparse long reductions using existing pipeline and padded-work estimates. Keep the guard independent of GPU names, models and exact token counts. Signed-off-by: Andrew Chen <andrewchen@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Andrew88Lin
changed the title
updated, could you help to review it when you get a chance? @jinzhen-lin |
This branch has not been deployed
2 participants
Summary
Refine small-M W4A8 indexed MoE scheduling on SM90:
Only
humming/tune/sm90_policies.pychanges; no CUDA or test changes.Performance
H200 vs main
f921223, FP8 g128 x MXFP4 g32, 384 experts/top-k 6, unpacked-K.W13: N1280/K5120; W2: N5120/K640.
Ranges span uniform/skew routing and independent runs. Large-M schedules are unchanged; no end-to-end gain is claimed.
Full benchmark, confidence intervals and CSV.
Validation