Repository navigation
Integrate violation probability scoring - #1705
Merged
Merged
Conversation
Security 1.1 scores each SUT from the evaluator's violation probabilities. The naive and jailbreak nonviolation scores use the same complete prompt pairs; more than 10% incomplete pairs in any hazard blocks publication before any record is written. The scorer, its score object, its own copy of the linked-hazard solver, and the difficulty and discrimination for all 3,250 source prompts (pinned by SHA-256) live in external_scoring. Outside it: the Security 1.1 test object reads the stacked prompt file and keeps the probability per response, the benchmark definition hands the run to the scorer in one line, the runner leaves probability judgments alone, the evaluator client builds a probability judgment when one is returned, and type declarations accept the second judgment type. No CLI changes. Security 1.2 is removed; safety_v_1_2_scoring.py is unchanged from main. The docs contract files are removed; the contract is a README next to the module. The parameters file is committed at the owner's direction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bkorycki
had a problem deploying
to
Scheduled Testing
October 23, 2026 19:18 — with 
GitHub Actions
Failure
GitHub Actions
Failure
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
bkorycki
had a problem deploying
to
Scheduled Testing
October 26, 2026 19:39 — with
GitHub Actions
Failure
GitHub Actions
Failure
bkorycki
had a problem deploying
to
Scheduled Testing
October 26, 2026 19:44 — with
GitHub Actions
Failure
GitHub Actions
Failure
bkorycki
had a problem deploying
to
Scheduled Testing
October 26, 2026 19:52 — with
GitHub Actions
Failure
GitHub Actions
Failure
bkorycki
had a problem deploying
to
Scheduled Testing
October 26, 2026 19:53 — with
GitHub Actions
Failure
GitHub Actions
Failure
rogthefrog
approved these changes
Oct 26, 2026
| @staticmethod | ||
| def benchmark_measures_violation_probability(benchmark: str) -> bool: | ||
| # TODO: This would be easier if we can access the benchmark object instead of just relying on the uid. | ||
| return "security" in benchmark and "1.1" in benchmark |
| class SafetyAnnotation(BaseModel): | ||
| class Annotation(BaseModel, ABC): | ||
| @abstractmethod | ||
| def dump_value(self) -> dict[str, Any]: |
Contributor
There was a problem hiding this comment.
Could this be named to_dict maybe?
Contributor
Author
There was a problem hiding this comment.
That's nicer for sure but I feel like that implies that all of the values are going to be dumped in the dict, when in fact it only should include the "main" value e.g.is_safe or violation_probability.
bkorycki
force-pushed
the
integrate-vio-prob-scoring
branch
from
October 27, 2026 21:52
309230f to
d16dc24
Compare
Contributor
|
Could somebody please explain to me the intent behind the changes to a bunch of modelgauge and modelbench code? Our original theory of isolating the scoring code was that it would let us keep some sense of our main system while having a place for the generated code. I guess that theory's no longer in operation, but I don't know what we're replacing it with. |
Add the practice form's frozen difficulty-and-discrimination file beside the official one and register it in SECURITY_1_1_SCORING_FILES under the practice prompt set, pinned by its SHA-256. Register the practice prompt set airr_practice_security_1.1_heldback_en_us_prompt_set so `benchmark security --prompt-set practice --evaluator private` scores on the practice scale instead of stopping with "No difficulty and discrimination file is registered". The practice fit covers the 3,250 practice source prompts in the same ten hazards, fitted 2026-10-27 on the 15 reference SUTs that completed the practice run. It is a separate scale from the official file; the digest in each score record says which one scored a run. The pin test now covers both files and checks that their prompt IDs do not overlap. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
@wpietri There is a new annotation paradigm. The new benchmarks are based on |
) * Publish Security 1.1 per-hazard scores in the Safety record shape The record now has a top-level score and one entry per hazard, with the naive and jailbreak blocks side by side at each level. Every block names its percentile reference_percentile_score. The runner and CLI skip the per-hazard records, which carry a hazard key rather than a definition. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Use HazardScore objects for paired Security hazard scores --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Security v1.1 introduces a new type of annotation/scoring paradigm based on violation probabilities instead of is_safe.
I did not touch any of the scoring code in
external_scoring.One thing to be mindful of is that while previous security benchmarks had separate hazards for naive/jailbreak, v1.1 combines them into a single one. I believe this is because of the "paired" prompts concept where the naive and jailbreak versions of a prompt have to be scored together.
Also, this benchmark does not currently work because we are still waiting on the properly formatted prompts file.