Repository navigation
Grader audit helpers: a shortcut probe and a training-signal report #3071
juanpablozarza
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I am building RL environments, porting an openFDA environment to v1. Training a 7B model with GRPO on it, I lost runs to two things I only found afterwards: the model gamed my partial-credit reward, and many groups had no reward variance, so they gave no signal. I'd like to offer two small library helpers for v1 that catch both before training, and want to check fit before opening a PR.
Shortcut probe.
probe_task(task)puts degenerate replies (empty, the prompt echoed, yes, a refusal, a list of numbers) into a trace and callsTask.score. No model or sandbox. Run on the bundled examples (8 of 9; wiki_search needs a judge API key), it flagged the two shortcuts in deepwiki and scratchpad examples: substring rewards pass without calling the tool they're meant to test #2948 (deepwiki and scratchpad) and also showed deepwiki scoring 1.0 on a plain prompt echo (not in deepwiki and scratchpad examples: substring rewards pass without calling the tool they're meant to test #2948).Training-signal report.
vf.load_signal_report(dir)reads a run'straces.jsonl, groups rollouts bytask.key, and names why each flat group gives no signal: all errored, all truncated or dropped, all format violations, all 0, all solved, or constant partial credit. It also shows which reward components never vary, agreement between components, and judge call failures and cost. prime-rl prunes zero-advantage samples; this explains them before training.Evidence, from vf-run on a deliberately small local setup (a 0.5B model, 256-token cap, 4 rollouts per task) to show what the report catches: on gsm8k, it showed right away that 6 of 12 tasks gave no signal because every rollout hit the token cap, which a mean reward alone doesn't tell you. With a reference judge added, the gold check varied in 1 of 10 tasks and the judge in 7, and they agreed on 41% of rollouts. One rollout errored when the judge replied with code instead of yes or no. Tested on main at 5dad16d. Full output: https://gist.github.com/juanpablozarza/5cab3fd8b96e8c3bc03adc5eda582b6e
Both would be plain functions plus a section in docs/debugging.md, each with its own tests. No CLI.
I have one question. Is
verifiers/v1/utilsthe right home, or would you rather these live in prime-rl's eval?All reactions