fix(ci): resolve kernel failures, expand test coverage, and speed up compilation by jinzhen-lin · Pull Request #113 · vllm-project/humming

jinzhen-lin · 2026-10-04T11:18:11Z

Fix various CI failures.
Add CI tests on H200 and L4.
Fix tests accidentally disabling parallel compilation, significantly speeding up test runs.
Add batch-invariant testing to Base Tests.
Add optional sampled tests that sample 32 configurations from the available tuning parameter space for each test case.
Integrate UMMA tests into the existing test suite and remove redundant cases already covered by other tests.

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: Codex (GPT-6) <codex@openai.com>

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

…s MMA Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

jinzhen-lin and others added 2 commits October 4, 2026 11:17

fix(tune): constrain Blackwell MMA schedules and warp groups

e4b4162

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(umma): support dequantized floating weights with FP8 activations

a86bef3

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

jinzhen-lin changed the title ~~fix(tune): constrain Blackwell MMA schedules and warp groups~~ fix(blackwell): correct kernel schedules and FP8 UMMA weight handling Oct 4, 2026

jinzhen-lin and others added 27 commits October 4, 2026 12:08

test(kernels): remove redundant PPU and SM90 regressions

8829f93

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(umma): remove redundant Stream-K bias regression

6f9b482

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): consolidate UMMA coverage into shared suites

17f0776

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(tests): preserve parallel JIT compilation during collection

4197f29

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(umma): support zero-point dequantization for low-bit activations

2be5168

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(moe): cover native FP4 input modes on SM10x and SM11x

fd84b21

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): tolerate sparse BF16 Stream-K outliers

440f6bb

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(umma): align channel zero-point register loads and reads

11acaf4

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): remove standalone Stream-K tolerance test

de36ced

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): add batch-invariant kernel test jobs

da269d8

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): gate sampled tests and cap sampling at 32

fbeb9bb

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

feat(testing): combine tuning sources and configure sample counts

6dbf307

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): add L4 and H200 jobs and adjust sampling defaults

a31b5be

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): isolate sampled approvals and configure H200 image

438dabf

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

feat(jit): limit parallel compilation to container CPU capacity

de7b759

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(tests): require native block-scaled hardware for MX shape cases

5649859

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(sm90): preserve batch-invariant K partitioning

a1cf9be

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): use unaligned batch size 4090 in default coverage

20923c5

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(jit): require async transfers for warp specialization

206e6a4

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(tuning): require at least 64 registers per thread

95e1cf8

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(tuning): lower minimum register budget to 56

b83f5cc

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): separate sampled test concurrency groups

b14af48

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(wgmma): validate instruction register budget against launch bounds

c87af57

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(sm90): reuse MMA heuristics for int4 activations

eae4d09

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(kernels): flush group scales only at the final MMA instruction

e047187

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): use 35GB H200 queue

23aa518

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): filter upstream TorchScript deprecation warning

4043dd8

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

jinzhen-lin and others added 9 commits October 5, 2026 03:12

ci(buildkite): rename test groups and add sampled1 to base tests

a05fced

Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: Codex (GPT-6) <codex@openai.com>

fix(wgmma): account for input scale register buffers

5cd14ca

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(tuning): require cp.async for warp specialization and asynchronou…

4c8bf35

…s MMA Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(wgmma): align preferred register budget to eight

b8362df

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): remove sampled1 from base tests

57b3c62

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

test(kernels): replace batch sizes 17 and 64 with 30

8df59b3

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(umma): prepare channel scales before reordered output

9d6bb15

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

fix(umma): fence channel reads before async buffer reuse

2224c81

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

ci(buildkite): use 18GB H200 for base tests and refresh sampling seed

c0342a8

Co-authored-by: Codex (GPT-6) <codex@openai.com> Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>

jinzhen-lin changed the title ~~fix(blackwell): correct kernel schedules and FP8 UMMA weight handling~~ fix(ci): resolve kernel failures, expand test coverage, and speed up compilation Oct 5, 2026

jinzhen-lin merged commit f197d9f into main Oct 5, 2026
4 checks passed

jinzhen-lin deleted the fix/blackwell-ci-heuristics branch October 5, 2026 08:42

Navigation Menu

Repository navigation

fix(ci): resolve kernel failures, expand test coverage, and speed up compilation - #113

fix(ci): resolve kernel failures, expand test coverage, and speed up compilation#113
jinzhen-lin merged 38 commits into
mainfrom
fix/blackwell-ci-heuristics

jinzhen-lin commented Oct 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jinzhen-lin commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jinzhen-lin commented Oct 4, 2026 •

edited

Loading