Repository navigation
[DRAFT] Adds YAML files for 2026-10-C1 cohort rulesets and submission checks - #545
Merged
Merged
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/very-large
nv-alicheng
changed the title
A cohort could name a packaged ruleset (`ruleset: mlperf-inference-v6.1`, resolved through the registry). It can now carry its catalog beside its checks instead: `cohort.yaml` gains `ruleset_dir`, naming a directory under the cohort directory that `Catalog.from_directory` reads (`models.yaml` + `datasets.yaml`), and the stray-file refusal admits that one directory. Declaring both sources is refused -- which catalog the rows read must never be ambiguous -- and `ruleset_dir` must be a relative path inside the cohort, so a cohort cannot aim the checker at arbitrary directories.
The first full Endpoints cohort in the module's dialect: 29 checks across point.yaml/pareto.yaml/system.yaml; a `valid_refs.yaml` declaring every fact a row may name, with §8.3/§4.4/§6.1's vocabularies as allowlists (an out-of-vocabulary value is refused at collect, so those rules need no rows); and a bundled `ruleset/` catalog (`models.yaml` + `datasets.yaml`) carrying the six §3.2 names, the legacy accuracy table and the §12 acceptance counts. Provenance is cited per file: policy sections at `4ba8224dc746`, the rule inventory from the submission-checker CLI (`messages.yaml` and its aggregate models), the accuracy and floor tables from the same CLI, and the model-spec pins -- marked provisional -- from the unverified bench-suite draft at `3bd1d0b1`. `src/inference_endpoint/config/cohorts/2026-10-C1.md` lists what the dialect cannot yet express (cross-point counts and coverage, per-curve consistency, the accuracy gates' conditional rescaling, metric arithmetic, the power model, upload structure) with the reason for each, and this version's rough edges.
Writing the cohort against real artifacts, rather than fixtures, caught four
declaration and row mistakes:
- system_power.json carries `cooling` as a string ("liquid"), not a block, and
its vocabulary lives per node set: the table declares the real Appendix E
top-level shape, and the rows check the node-set list and the MLC-estimate
flag rather than a top-level `power_method`.
- `link_to_model_transformation` is a string or an explicit null, and
`type: str` refuses the null: it stays undeclared, with the "value or null"
declaration form noted as a dialect gap.
- percentile blocks are keyed by spelling (`tpot.percentiles["90.0"]`), so the
P90 row became a presence-and-non-empty check over the block, with the
canonicalisation gap noted.
- the warmup log-retention row now gates on the warmup being performed
(`duration_s > 0`), so a disabled warmup is not warned about.
The cohort reports no failing findings against a full bundle, and the reference
checker agrees on the same tree: its warnings are rules this cohort lists as not
yet expressible.
`Ref.resolve` now reads a numeric component against a key spelling the same number, so the cohort asserts the P90 itself (`tpot.percentiles.90`) instead of only that the percentile block is non-empty: a block keyed `"90.0"` is read, and the row runs at every measurement point.
The catalog carried per-model accuracy targets and per-dataset run shapes but
nothing tying the two together, so a target had no subject: `aime25`'s
`unit_samples` could belong to any model, and the §15 gate's sample-weighted
mean had no declared member sets.
- `models.yaml` gains `accuracy_datasets`: the set(s) a model's accuracy phase
runs, with the run shape per (model, set). gpt-oss-120b's §15 `exact_match`
gate reads its three (aime25, gpqa, livecodebench); deepseek-r1's reads its
one (deepseek_r1_accuracy). The legacy models carry `{}` rather than a guessed
link: this round declares no accuracy sets for them.
- Agentic gates now name what they read: `inline` -> `agentic_combined`,
`swebench` -> `swe_bench` with `mean_of_n: 4`, `osl` ->
`output_sequence_lengths_full_run`.
- `datasets.yaml` loses the per-dataset run shapes (they live on the tie now),
gains the two agentic sets, and drops cnn_dailymail's unsourced pair.
- The cohort notes record the tie's source and refine what an accuracy-gate row
would still need: the join from the run's declared model to its tie.
…ve one
gpt-oss-120b's §15 gate is the cumulative sample-weighted mean over its three
sets, but a model may also have published thresholds per set. A tied set now
accepts its own `targets`, in the same `metric: { lower | band, factor }` form
as the model-level `accuracy_targets`: `accuracy_targets` stays the cumulative
gate, a set's `targets` gate that set's own score, so a model can carry both.
Only a published number goes there: no reference publishes per-set thresholds
for this round's models, so the form is documented (models.yaml header) and left
absent rather than guessed -- the same rule the legacy tie follows. The cohort
notes record it and the row-dialect work still needed to consume either
(the model-to-set join and the per-set lookup inside accuracy_results).
nv-alicheng
force-pushed
the
feat/mlperf-endpoints-cohort
branch
from
October 23, 2026 16:29
3d1732f to
b4d95af
Compare
github-actions
Bot
added
size/large
CI's mypy hook (skipped by the local commits for three pre-existing macOS-typeshed errors) refused the cohort wiring: src/inference_endpoint/validation/__main__.py:100: error: Incompatible types in assignment (expression has type "Catalog | None", variable has type "Catalog") [assignment] The `ruleset_dir` branch assigned a `Catalog` first, so mypy had narrowed the name before the packaged-ruleset fallback could assign `Catalog | None`. The name is declared `Catalog | None` up front and assigned once per branch, which leaves the module's mypy output clean (the only errors the hook still reports are the three pre-existing Linux-only `sched_*affinity` ones).
7 tasks done
Collaborator
|
Merging this into #544 to make reviews easier. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds the first real cohort to the validation module, plus the small module change it needs to ship its
own catalog.
The cohort is
2026-10-C1, the round this cohort targets, exercised end-to-end against realsubmission artifacts for that round
It lives in
src/inference_endpoint/config/cohorts/2026-10-C1/:cohort.yamlpoint.yaml— 25 checkssystem.yaml— 3 checkspareto.yaml— 2 checksvalid_refs.yamlruleset/models.yaml,ruleset/datasets.yaml2026-10-C1.mdEvery check is a declarative row naming the Endpoints Policy section it enforces, its severity, and
the fix a failing submission should apply. Rows exist only for what a row can decide — the
"Not yet expressible" table records what the dialect cannot say today (cross-point counts and
coverage, accuracy gates, Appendix E arithmetic, upload structure) with a reason for each.
The module change lets a cohort bundle its catalog as a
ruleset/directory (ruleset_dirincohort.yaml, read byCatalog.from_directory) instead of naming a packaged ruleset. The twosources are mutually exclusive, the directory must live inside the cohort, and the stray-file check
admits the new name.
Writing the cohort against real artifacts, rather than fixtures, is what caught four declaration
mistakes synthetic data could not:
coolingis a string ("liquid") rather than a block;power_methodlives insidenode_sets[], not at the top level;link_to_model_transformationis anexplicit
nullon a typed leaf; and percentile blocks are keyed"90.0"while a row names90.Type of change
Related issues
None. Stacked on #544: this branch is based on
feat/validation-module, and the decoding the cohortrelies on (numeric strings, differently spelled percentile keys) landed there. Retarget to
mainonce #544 merges.
Testing
tests/unit/validation/test_check.py, on top of the module's existing stub-cohort coverage.uv run pytest tests/unit/validation tests/unit/config/test_regenerate_templates.py -q→ 436 passed.
python -m inference_endpoint.validation src/inference_endpoint/config/cohorts/2026-10-C1 <submission-root>→ 291 findings, 0 failing, exit 0 (18 checks ran; the 12 that did not apply are disclosed as
NEVER-RANwith their reason, not silently skipped)."90.0"percentile key at every measurement point.concurrencyreads it as its number, prints thecoerced:note, and returns the identical 291 findings.basis §4.4, 1 × agentic inline accuracy §4.3) — both already in the cohort's
"Not yet expressible" table.
Checklist
ruff/ruff-formatclean, prettier on the YAML and Markdown.catalog.