[Wings-Of-Disaster]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Add six check kinds to twinkle_agentic.verifier.result_check, expanding _KINDS from 10 to 16.

The new checks support held-out validation, exact and numeric answers, verifier-executed SQL, retrieval receipts, and compiler-checked Lean proofs.

Scope: 2 files, +639/−2 lines, 51 new tests. The implementation introduces no new imports and can be reviewed and merged independently of world or profile changes.

Why this is needed

The existing ten kinds grade a workspace and an answer string. Agentic tasks also need checks that can establish whether:

• Validation was held out from the task proposer.
• A claimed value agrees with a query executed by the verifier.
• A cited source was actually retrieved during the episode.

They also need precise answer matching and a compiler verdict for proof tasks. The six additions cover these cases.

New check kinds

Kind	What it checks	Why it is useful
hidden_python	Runs a Python check withheld from the proposer.	Supports validation the proposer cannot tailor its task around.
answer_one_of	Requires the answer to equal a member of an exact set.	Avoids substring matches that accept both “the answer is 42” and “the answer is not 42.”
answer_number	Requires one unambiguous finite number within a tolerance, with an optional required unit.	Accepts equivalent numeric forms such as 3 and 3.0, while detecting incorrect units.
sql_query	Executes the query in the verifier and compares the resulting rows.	Checks the underlying data rather than accepting the model's stated result.
web_receipt	Requires every cited URL to appear in retrieval receipts for the episode.	Distinguishes a retrieved source from a plausible but unserved citation.
lean	Compiles the submitted proof with lake env lean and rejects disallowed proof shortcuts.	Uses the compiler to validate the proof rather than matching proof-like text.

Implementation and compatibility

Reuse the existing Python execution path

hidden_python shares the existing shell/Python execution path. The dispatch change selects the interpreter explicitly:

if kind in ('shell', 'python', 'hidden_python'):
    interpreter = 'shell' if kind == 'shell' else 'python'

For the existing shell and python kinds, interpreter == kind, preserving their behavior. These two lines are the only modification to existing code; the remaining implementation is additive.

Consume retrieval receipts without a world dependency

web_receipt receives receipts through CheckContext. It does not import or depend on a particular world implementation, and its tests supply fabricated receipts.

In production, this check relies on receipts recorded by the retrieval provider when it serves a result. Policy-authored claims are not evidence of retrieval. A failed fetch records no URL and therefore cannot satisfy a citation check.

This establishes that a cited URL was served during the episode; it does not, by itself, establish that the source supports the answer.

Compile Lean proofs and reject disallowed shortcuts

The lean kind runs lake env lean and also refuses sorry and native_decide. Compilation alone is insufficient under this check's rules: a submitted proof must pass both the compiler and the shortcut restrictions.

Actionable failure messages

Failures explain what needs to change rather than returning only “did not match.”

Kind	Diagnostic detail
sql_query	Row counts and the first three rows of both expected and actual results.
answer_number	Distinguishes a non-number, a non-finite value, an incorrect unit, and a value outside tolerance.
web_receipt	Identifies cited URLs that were not served.

These messages help a proposer respond to rejected checks and make evaluation logs easier to diagnose.

Verification

The following results are reported for the PR and the clean upstream baseline. No training results are included; this change is evaluated through verifier tests.

pre-commit run --all-files

pytest tests/twinkle_agentic tests/processor tests/utils tests/docs \
       tests/server/static tests/cli tests/advantage tests/metric

Check	Result
Pre-commit	All hooks passed
Clean main, same test selection	391 passed, 9 skipped
PR, same test selection	442 passed, 9 skipped, 0 failed
Test delta	51 additional passing tests; no additional skips

Five files matching test_async_rl_* or named test_vllm_sampler_tq_generation.py were deselected. They fail collection identically on clean main with ModuleNotFoundError: No module named 'transfer_queue'. The reported results cover the stated selection, not the entire repository test suite.

Scope and dependencies

result_check.py introduces no new imports. Tests use upstream's LocalEnv and fabricated receipts, so neither the implementation nor its tests require the world or profile additions.

The PR can be reviewed and merged on its own. Deployment still needs to supply the resources used by a selected check, including retrieval receipts for web_receipt and the Lean toolchain for lean.