- Code is the source of truth: before writing anything, read the v1 code for existing helpers, harnesses, and interfaces instead of reinventing them. Prefer verifiers-native task, trace, server, harness, and runtime interfaces over repeated path/import/discovery plumbing in user packages.
- Minimal config surface: expose as few knobs as possible, but as many as needed.
- Keep tasksets small: a basic taskset fits in a few dozen idiomatic lines — typed data/task/config classes,
load(), and decorated scoring on the task. Don't overrideTaskset.__init__(implementload()); don't overrideHarness.__init__(usesetup()).
- Module loggers: create them with
get_logger(__name__)fromverifiers.v1.utils.logging. It prefixes every line with its scope:[<episode>]inside an episode,[<episode>/<trace> <agent>]inside an agent run,[<trace>]for an agent run outside an episode. Never put a trace or episode id in the message itself. - Levels by frequency: INFO is for process lifecycle (servers, pools) and, per episode,
Starting episode,Finished episodeand oneFinished rolloutline per agent run. Steps inside a rollout (runtime start/stop, setup, scoring) and per-turn or per-request events are DEBUG. Every failure and retry is a WARNING. ERROR is for a process component that can't continue, never for a rollout's outcome. - Message style: an active sentence with no trailing period (
Started prime runtime abc in 9.8s), thenkey=valuefields. Durations useformat_time()and readin 1.2sorin 3m 46s. Error details come last, after-:Failed to delete prime runtime abc - APIError: .... Clip error messages withclip(). - Retries:
Retrying <what> in <delay> (attempt <next>/<total>) - <Type>: <message>. - Tracebacks: attach one for unexpected errors only. For an expected failure (teardown, cleanup, a refused call), pass
exc_info=logger.isEnabledFor(logging.DEBUG). - Config warnings: warn once per process, not once per rollout.
- Always use uv: run code and commands with
uv run, never rawpython. Make sureuvis installed (docs). - Check tasksets: check a taskset without a model with
uv run vf-validate <taskset>. Run a few episodes withuv run vf-run <taskset> -n 1 -r 1. Run full model evals with prime-rl'suv run eval. - Validate TOML first: validate config
.tomlfiles before running them. - Don't add dependencies: never add dependencies or optional extras to the top-level
pyproject.toml.
- Kept intentionally minimal:
docs/andskills/are deliberately sparse. Don't touch them unless your change breaks their assumptions. - Docs reflect
main, not history: describe the current state of the codebase only — no removed/legacy fields, migration paths, or "this used to be X" anecdotes.
- Use bundled skills first: the
releaseskill inskills/documents the stable-version release workflow, and thebumpskill documents how to bump a pinned harness version.
- Two suites:
tests/unitruns one component in isolation;tests/integrationruns real evals (a live model, real runtimes) per behavior across the harness x runtime grid intests/integration/grid.py. Prefer an integration cell over a unit test; a unit test earns its place by stating one component's intended behavior, not by asserting an implementation detail. To check your own work, write a temporary script instead of committing new tests. - Run the contributor checks: run
uv run pre-commit installonce, then for touched areasuv run ruff check --fix .,uv run pytest tests/unit, anduv run pre-commit run --all-files.