Repository navigation
feat(cookbook): enable optional RSI worlds and knowledge cards - #330
Open
Wings-Of-Disaster wants to merge 13 commits into
Open
Wings-Of-Disaster wants to merge 13 commits into
Wings-Of-Disaster wants to merge 13 commits into
Conversation
Two problems, one commit, because the second exists to make the first safe to
hand out.
An agentic run needs tasks whose answers the policy cannot invent. A task that
says "report the failure rate in the ops database" is only a real task if the
database exists and the policy has to read it; otherwise the policy writes
plausible numbers and a check that compares strings passes either way. Measured on
one run's 256 generated tasks: every one of them verified with os.path.isfile(),
and 27% parsed a SQLite file the model had itself just written -- so a high score
meant the answer agreed with its own premise.
And every environment the run handed out was the same environment: the same tools,
an empty workspace, and a topic that had no bearing on either. A task about
querying an operations database and a task about office-chair ergonomics then
differ only in their prompt, and the policy learns that the fixture is something
it can invent -- because for most topics it was.
world.py -- HostWorld is a read-only SQLite database, a document set and a frozen
search corpus, served from the training host and exposed as seven tools
(world_sql, world_schema, world_doc, web_search, fetch_page, db_query, db_schema).
Host-side is the point, not an implementation detail: a tool call already
round-trips through Env.step_batch, so the host sees every call before and after
the sandbox does and needs no network path into it. That keeps a networkless
environment honestly networkless, adds no listening port to defend, and keeps the
data out of the sandbox where it could be edited or left behind in a snapshot.
Every retrieval answer is recorded as a receipt, with the urls taken from the
tool's own output rather than its arguments -- web_search is asked a query and
decides which urls to return, so what was actually served is only knowable from
the result. A fetch that failed records the call with no url, so "I fetched this"
has to be earned. This is what makes a citation gradeable rather than assertable.
WorldToolsMixin exposes it on any Env. A mixin rather than a change to AgentEnv,
so it applies to LocalEnv and OpenEnv too and the host half is testable without a
sandbox host -- which is how all 19 of its tests run. It strips the
{server}---{tool} suffix a harness adds, and tracks step_batch positions
explicitly so an observation cannot be paired with the wrong call. Tools are
advertised only when their backing store is present: offering one that always
answers "not configured" costs the policy a turn, and a difficulty measurement
built on that episode counts the wasted turn as task work.
profiles.py -- SandboxProfile is an immutable manifest: capabilities, task forms,
allowed tools, which verifier kinds can grade it, and what the workspace must
contain before the episode starts. Validated at construction rather than at use;
a profile granting raw_network is refused outright, because serving the world from
the host is what keeps the sandbox networkless and a profile could otherwise undo
that quietly. Each profile and the registry carry a content digest, so a run can
record which profile set it used and a reader can tell whether two runs are
comparable. Thirteen ship in default_profile_registry().
ProfiledEnvMixin narrows tools() to the profile and refuses anything else in
step()/step_batch(), because advertising is not enforcement -- a policy can name a
tool it was never shown. switch_profile runs the profile's setup manifest, and a
step that does not resolve, or a canary that fails, raises rather than being
recorded: a slot missing its fixture produces tasks that read as hard, not as
broken.
Order matters and is documented at both classes: ProfiledEnvMixin has to precede
WorldToolsMixin, so the world contributes its tools first and the profile then
decides which survive. Reversed, a profile could not restrict the world at all.
seeds.py -- the programs those manifests name. A seed is what makes a task
checkable against data the policy did not author: seed-sql-analytics builds a
shop.db with customers, products and orders; seed-csv-dirty writes a CSV with a
BOM, duplicate rows and empty cells; seed-broken-repo lays down a repository whose
tests fail; seed-log-corpus writes two logs that disagree; seed-docs-corpus plants
a document set with decoys in it. Programs rather than data files, so the fixture
is reproducible from the profile alone and a run ships no bytes.
Each seed is a template holding one {workspace!r} placeholder, substituted by
exact str.replace and never by str.format: the programs are real python and
contain set and dict comprehensions whose braces format() would read as
replacement fields.
The live search backend and the per-episode database role provisioning are
deployment-specific, so both name their module directory explicitly
(TWINKLE_WORLD_SEARCH_MODULE, TWINKLE_WORLD_DB_MODULE) and raise WorldUnavailable
when it is absent. Neither provider is named anywhere here. psycopg2 and the
search plug-in are imported lazily, so a corpus-only run needs no database driver
and no search dependency.
Only envs/__init__.py is touched among existing files, to export the new names. No
environment's behaviour changes unless a caller mixes one of these in.
…utable novelty score Three modules, no dependencies between them and none on anything else new. Each answers a measured failure rather than an imagined one. knowledge.py -- a challenger left to invent topics returns to the ones it can already verify. Left to draw uniformly from a bank of cards it does not do much better: measured on a 19-iteration run against a 256-card bank, uniform drawing reached 13 distinct cards, 5% coverage, and the tasks got easier each iteration only because the policy kept meeting the same ones. KnowledgeStore walks one seeded permutation instead of drawing from it. A cursor rather than a sample means every card is reached in len(cards) / blocks_per_epoch epochs and none repeats within a pass, so a 30-iteration run covers 240 of 256 where the same run by chance covered 13. route() returns an arm as well as a card. Two consecutive groups share an anchor and differ in arm, so a block is a matched pair -- one group handed the card, one told to propose from scratch -- rather than two unrelated draws. Arm order and anchor cursor are both hashed on the epoch, which stops a loop that rebuilds its challenger per iteration from replaying the identical sequence. The bank is authenticated rather than trusted. Every card carries the checkpoint and tokenizer sha256 it was mined from, the store hashes the file it read, and TWINKLE_KNOWLEDGE_CARDS_SHA256 pins that hash for a run that wants to fail loudly on a substituted bank. A run that cannot say which bank it used cannot be reproduced from its own record. KnowledgeCardSeeder adapts the store to a no-argument seeder protocol. The control arm returns None, which such a protocol already reads as "propose from scratch", so the comparison needs no special handling downstream. One fix came out of writing the tests: block was group_id // len(arms) with no remainder, which silently assumes group_id restarts each epoch -- true for a loop that rebuilds its challenger per iteration, false for one global counter, where the cursor aliases back onto the first blocks_per_epoch anchors forever. task_bank.py -- novelty is meaningless without something to be novel against. Within one run the proposals of a group can be compared to each other; the slower failure is iteration k+1 re-proposing what iteration k already trained on, and that needs a file that outlives a run. It is also the producer for verifier/rubric_score.build_rubric_prompt(references=...), which skips its novelty criteria when the reference set is empty and which nothing in the package currently supplied. ingest_recorder fills the bank from the recorder AgenticChallenger already writes, rather than from a hook into it: the challenger stays unaware a bank exists, a resumed run gets its history at once, and add() dedupes so an ingest-every-iteration call appends each task once. The similarity used to CHOOSE which stored tasks the judge reads is 3-gram Jaccard. It is weak and known to be: over one run's 207 kept statements the closest pair scored 0.077, so ranking by it is nearly ranking at random. It never scores novelty -- the judging is rubric_score's, whose criteria compare task shapes. novelty_metric.py -- if the novelty term comes from an LLM judge, measure it before trusting it. On two recorded runs the judge scored 93.3% and 98.5% of proposals exactly zero (1439 and 4589 judged proposals), and 66.2% and 91.6% of groups (225 and 783) scored identically on every sibling. GRPO subtracts the group mean and divides by the group standard deviation, so a term constant across a group contributes exactly zero gradient -- not a small effect, a mathematically zero one. Both runs had a novelty term in the reward and in those groups it did nothing. constant_group_rate is that measurement, exported so any replacement has to be judged the same way. NoveltyScorer replaces it with two terms computed inside the group, because that is the contrast GRPO uses: distance (1 - max cosine to the closest sibling) and spread (1 / cluster size over siblings within a threshold). The vectoriser is fitted on the whole corpus rather than per group, and that is load-bearing: these statements share their scaffolding verbatim, so per-group IDF weights the boilerplate above the content and every pair of siblings looks like a duplicate. cluster_threshold is calibrated from the observed sibling cosines by default -- on one recorded run a hand-picked 0.55 left 98% of proposals in singleton clusters and the spread term measured nothing. scikit-learn is imported inside the methods that use it, so a run that never scores novelty pays nothing, and it is added to the `test` extra so CI runs these tests rather than skipping them. The tests importorskip, so they degrade rather than fail where it is absent. Not wired into a reward here: which term a challenger_reward mixes, and at what weight, is a decision for the caller rather than a default these modules should set.
…can host it task_contract.py -- TaskIntent, FixtureSpec, VerifierSpec, FreshnessSpec and ProfileProvenance, validated as a unit by validate_task_contract: a verifier the profile cannot run, a fixture it cannot mount, an output beyond what the intent declared or the profile can produce, provenance whose digest does not match the resolution, a no-tool task carrying a fixture, a web-research task with no freshness policy or no receipt check. The provenance check is what makes a run auditable: the contract has to name the environment that served the task, digest and all, not the one its author meant. This module has no in-tree caller. It is the declaration format a task builder fills in -- the same shape as verifier/rubric_score.py, which upstream also exports without a caller. 30 tests pin the format so the first builder to use it does not define the semantics by whatever it happens to pass. topic_profile.py -- routes a topic to a profile. PREFERENCE is the tiebreak order: a card mined for database work also matches the generic data patterns, so if compute ranks first it lands in a plain python slot. On one 429-card bank, 44 of the 332 switchable cards are eligible for both python-compute and a specific profile, so list order alone decides 13% of the routing. starting_state / host_tools_of / seed_steps_of report what a task author has to be told about the environment it is about to get -- host, seeded or empty -- and are derived from the registry rather than declared per profile. _service_reachable decides whether db-analyst@1 can run here, and checks two things: that the server accepts a connection, and that the provisioning module named by TWINKLE_WORLD_DB_MODULE is set. Checking only the first was measured to be wrong the expensive way -- with PostgreSQL listening and no module configured the profile stayed in the rotation, every db_query returned "cannot reach the shared database", each rollout scored 0 and the group was dropped. 91 of 332 switchable cards route there, so a quarter of the groups were burned and what the proposer learned was that database topics do not pay. Three defects the tests found: - the "unknown profile name falls back" path caught KeyError where ProfileRegistry.get raises ValueError, so it was unreachable and an unknown name would have killed a run that had already loaded its weights; - legacy-agentic@1 named a setup step nothing resolves, so switching into it raised "unknown setup step" instead of the intended "retired"; - _load dropped cards with no switchable profile, so describe() under-reported exactly the case worth noticing.
…RSI run
Until now every layer this series adds was library-only: a run could import them,
but rsi_grpo.py built plain AgentEnv / LocalEnv slots, never opened a knowledge
bank and never kept a task bank.
Slot construction moves to agentic/slot.py rather than growing a branch per
combination inside rsi_grpo.py. _slot_type returns the base class itself when
there is nothing to mix in, not an empty subclass of it, so an unconfigured run
gets the objects it got before -- which is the whole of the backwards-compatibility
guarantee, and a test asserts type(env) is LocalEnv rather than issubclass.
Mixin order is why this is a module: ProfiledEnvMixin has to precede
WorldToolsMixin so the world contributes its tools first and the profile then
decides which survive. Reversed, a profile could not restrict the world at all.
Five environment variables, all off by default, rather than CLI flags -- because
twinkle.cli's schema is shared with every cookbook and these name artefacts rather
than change how a run trains:
TWINKLE_WORLD_ROOT the host-served world
TWINKLE_WORLD_SEARCH_ROOT the frozen search corpus
TWINKLE_RSI_PROFILES which profiles the slots rotate through
TWINKLE_TOPIC_AFFINITIES read only when the list above is empty; the rotation
becomes every profile this machine can actually run
TWINKLE_KNOWLEDGE_CARDS a mined bank to seed topics from
TWINKLE_TASK_BANK where kept statements are recorded
An explicit profile list wins over the affinity table, because naming profiles is
how a run pins what it measured. Slots rotate rather than all taking one profile:
a pool whose slots all hold the same fixture can only produce one kind of task,
and the policy learns the fixture is part of the world rather than the question.
Both banks are opened at startup, before the model is on the GPUs. A bank that
fails its manifest check, and a profile list with a typo in it, have to stop the
run there rather than several iterations in.
The knowledge seeder goes first in the chain, so a card reads as the subject and
the keyword pool as the angle. Its control arm returns None, which ChainSeeder
skips -- but with other seeders configured the control arm still receives their
text, so a clean comparison wants --num-keywords 0 and no seed dataset.
cookbook/rsi/README.md is new: the layers, every variable and its default, the
thirteen profiles with the starting state each derives to, and the four things
that will bite.
Give each slot a shallow fork of loaded world data with independent retrieval receipts and database credentials. Owned worlds discard those mutable fields when an environment is reset, preventing one episode from inheriting another's retrieval evidence or role. The cookbook opts into this ownership explicitly. Remove the unused split_calls helper; WorldToolsMixin already dispatches batches while preserving indices. Keep manifest validation, all seven tool schemas, 13 profiles, all workspace seed programs and the tool-alias limitation intact. Condense repeated design discussion into API and deployment guidance. Validation: world/slot targeted run passed 39 cases, including world-fork and reset regressions; no training or broad regression suite was rerun.
Keep both supported bank formats, release/provenance authentication, deterministic knowledge/control routing, task-ledger persistence and the full novelty scorer. Remove repeated experiment narratives from code comments while retaining caller contracts and calibration/vocabulary limitations. These modules provide reusable APIs. The cookbook records the ledger but does not consume its references for reward; NoveltyScorer does not change training. Unverifiable historical coverage figures are not used to justify this revision. Validation: executable ASTs match the original modules after stripping docstrings. Previously passing functional tests were not rerun for prose edits.
Keep TaskContract and its validation tests. Document the distinction between per-card advisory choices and the host-supported rotation used by the cookbook. The offline affinity classifier remains a deployment input, as in this port. Preserve all exports while keeping the existing compact __all__ layout. Remove repeated run narratives; keep profile ordering and service probes intact. This revision neither adds a task-contract consumer nor changes reward policy. Validation: topic-profile executable AST is unchanged and the exported name set is preserved. Previously passing contract/routing tests were not rerun.
Validate world/knowledge artifacts and profile IDs before creating distributed workers or loading model weights. Pass the validated KnowledgeStore into the seeder so it is not reopened after initialization. Fork loaded world data per slot, enable database tools only for profiles that request them, and release each slot's own role at reset/close. Preserve profile filtering before world dispatch and the exact base classes for unconfigured runs. Import KnowledgeCardSeeder directly from the challenger package, eliminating a cookbook-only re-export module and its seed/__init__ edit. Remove the single-use slot-class helper and condense the README to setup, configuration and API limits. TaskContract and novelty remain available; this does not wire them into training. Validation: 20 slot cases passed, including cross-slot/episode receipt isolation, database tool discovery and failures before distributed initialization. The previous GPU smoke run was not repeated.
Use single quotes where no inner quotes require double quotes. Python 3.11 exposes complete f-string tokens to the string-fixer hook; the local Python 3.12 tokenizer handles these differently. The executable AST is unchanged. Full pre-commit checks pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR information
Wire worlds, profile rotation, knowledge seeding and kept-task recording into
rsi_grpo.py.Validate artifacts before distributed initialization and keep each slot's world state independent.
Depends on #323, #328 and
pr4-task-contract. The current diff includes these dependencies.TWINKLE_WORLD_ROOT=/data/world \ TWINKLE_RSI_PROFILES=world-forensics@1,sql-analytics@1 \ TWINKLE_KNOWLEDGE_CARDS=/data/bank/knowledge_cards.jsonl \ TWINKLE_TASK_BANK=/data/run/tasks.jsonl \ python cookbook/rsi/rsi_grpo.py --model-id /data/models/Qwen3.5-4B \ --sandbox-template twinkle-rsi-msagent --sandbox-api-url http://sandbox:8000 \ --save-dir /data/run --num-keywords 0 --micro-batch-size 1 --max-length 16384Requires the environment and knowledge APIs and deployment-supplied data. Existing model/GPU
arguments still apply. With these variables unset, the existing environment classes are used.
Topic profiles supply a rotation; per-card switching, contract construction and novelty rewards are not wired.
Experiment results