Repository navigation
feat(envs): add task contracts and topic profile lookup - #329
Open
Wings-Of-Disaster wants to merge 5 commits into
Open
Wings-Of-Disaster wants to merge 5 commits into
Wings-Of-Disaster wants to merge 5 commits into
Conversation
Two problems, one commit, because the second exists to make the first safe to
hand out.
An agentic run needs tasks whose answers the policy cannot invent. A task that
says "report the failure rate in the ops database" is only a real task if the
database exists and the policy has to read it; otherwise the policy writes
plausible numbers and a check that compares strings passes either way. Measured on
one run's 256 generated tasks: every one of them verified with os.path.isfile(),
and 27% parsed a SQLite file the model had itself just written -- so a high score
meant the answer agreed with its own premise.
And every environment the run handed out was the same environment: the same tools,
an empty workspace, and a topic that had no bearing on either. A task about
querying an operations database and a task about office-chair ergonomics then
differ only in their prompt, and the policy learns that the fixture is something
it can invent -- because for most topics it was.
world.py -- HostWorld is a read-only SQLite database, a document set and a frozen
search corpus, served from the training host and exposed as seven tools
(world_sql, world_schema, world_doc, web_search, fetch_page, db_query, db_schema).
Host-side is the point, not an implementation detail: a tool call already
round-trips through Env.step_batch, so the host sees every call before and after
the sandbox does and needs no network path into it. That keeps a networkless
environment honestly networkless, adds no listening port to defend, and keeps the
data out of the sandbox where it could be edited or left behind in a snapshot.
Every retrieval answer is recorded as a receipt, with the urls taken from the
tool's own output rather than its arguments -- web_search is asked a query and
decides which urls to return, so what was actually served is only knowable from
the result. A fetch that failed records the call with no url, so "I fetched this"
has to be earned. This is what makes a citation gradeable rather than assertable.
WorldToolsMixin exposes it on any Env. A mixin rather than a change to AgentEnv,
so it applies to LocalEnv and OpenEnv too and the host half is testable without a
sandbox host -- which is how all 19 of its tests run. It strips the
{server}---{tool} suffix a harness adds, and tracks step_batch positions
explicitly so an observation cannot be paired with the wrong call. Tools are
advertised only when their backing store is present: offering one that always
answers "not configured" costs the policy a turn, and a difficulty measurement
built on that episode counts the wasted turn as task work.
profiles.py -- SandboxProfile is an immutable manifest: capabilities, task forms,
allowed tools, which verifier kinds can grade it, and what the workspace must
contain before the episode starts. Validated at construction rather than at use;
a profile granting raw_network is refused outright, because serving the world from
the host is what keeps the sandbox networkless and a profile could otherwise undo
that quietly. Each profile and the registry carry a content digest, so a run can
record which profile set it used and a reader can tell whether two runs are
comparable. Thirteen ship in default_profile_registry().
ProfiledEnvMixin narrows tools() to the profile and refuses anything else in
step()/step_batch(), because advertising is not enforcement -- a policy can name a
tool it was never shown. switch_profile runs the profile's setup manifest, and a
step that does not resolve, or a canary that fails, raises rather than being
recorded: a slot missing its fixture produces tasks that read as hard, not as
broken.
Order matters and is documented at both classes: ProfiledEnvMixin has to precede
WorldToolsMixin, so the world contributes its tools first and the profile then
decides which survive. Reversed, a profile could not restrict the world at all.
seeds.py -- the programs those manifests name. A seed is what makes a task
checkable against data the policy did not author: seed-sql-analytics builds a
shop.db with customers, products and orders; seed-csv-dirty writes a CSV with a
BOM, duplicate rows and empty cells; seed-broken-repo lays down a repository whose
tests fail; seed-log-corpus writes two logs that disagree; seed-docs-corpus plants
a document set with decoys in it. Programs rather than data files, so the fixture
is reproducible from the profile alone and a run ships no bytes.
Each seed is a template holding one {workspace!r} placeholder, substituted by
exact str.replace and never by str.format: the programs are real python and
contain set and dict comprehensions whose braces format() would read as
replacement fields.
The live search backend and the per-episode database role provisioning are
deployment-specific, so both name their module directory explicitly
(TWINKLE_WORLD_SEARCH_MODULE, TWINKLE_WORLD_DB_MODULE) and raise WorldUnavailable
when it is absent. Neither provider is named anywhere here. psycopg2 and the
search plug-in are imported lazily, so a corpus-only run needs no database driver
and no search dependency.
Only envs/__init__.py is touched among existing files, to export the new names. No
environment's behaviour changes unless a caller mixes one of these in.
…can host it task_contract.py -- TaskIntent, FixtureSpec, VerifierSpec, FreshnessSpec and ProfileProvenance, validated as a unit by validate_task_contract: a verifier the profile cannot run, a fixture it cannot mount, an output beyond what the intent declared or the profile can produce, provenance whose digest does not match the resolution, a no-tool task carrying a fixture, a web-research task with no freshness policy or no receipt check. The provenance check is what makes a run auditable: the contract has to name the environment that served the task, digest and all, not the one its author meant. This module has no in-tree caller. It is the declaration format a task builder fills in -- the same shape as verifier/rubric_score.py, which upstream also exports without a caller. 30 tests pin the format so the first builder to use it does not define the semantics by whatever it happens to pass. topic_profile.py -- routes a topic to a profile. PREFERENCE is the tiebreak order: a card mined for database work also matches the generic data patterns, so if compute ranks first it lands in a plain python slot. On one 429-card bank, 44 of the 332 switchable cards are eligible for both python-compute and a specific profile, so list order alone decides 13% of the routing. starting_state / host_tools_of / seed_steps_of report what a task author has to be told about the environment it is about to get -- host, seeded or empty -- and are derived from the registry rather than declared per profile. _service_reachable decides whether db-analyst@1 can run here, and checks two things: that the server accepts a connection, and that the provisioning module named by TWINKLE_WORLD_DB_MODULE is set. Checking only the first was measured to be wrong the expensive way -- with PostgreSQL listening and no module configured the profile stayed in the rotation, every db_query returned "cannot reach the shared database", each rollout scored 0 and the group was dropped. 91 of 332 switchable cards route there, so a quarter of the groups were burned and what the proposer learned was that database topics do not pay. Three defects the tests found: - the "unknown profile name falls back" path caught KeyError where ProfileRegistry.get raises ValueError, so it was unreachable and an unknown name would have killed a run that had already loaded its weights; - legacy-agentic@1 named a setup step nothing resolves, so switching into it raised "unknown setup step" instead of the intended "retired"; - _load dropped cards with no switchable profile, so describe() under-reported exactly the case worth noticing.
Give each slot a shallow fork of loaded world data with independent retrieval receipts and database credentials. Owned worlds discard those mutable fields when an environment is reset, preventing one episode from inheriting another's retrieval evidence or role. The cookbook opts into this ownership explicitly. Remove the unused split_calls helper; WorldToolsMixin already dispatches batches while preserving indices. Keep manifest validation, all seven tool schemas, 13 profiles, all workspace seed programs and the tool-alias limitation intact. Condense repeated design discussion into API and deployment guidance. Validation: world/slot targeted run passed 39 cases, including world-fork and reset regressions; no training or broad regression suite was rerun.
Keep TaskContract and its validation tests. Document the distinction between per-card advisory choices and the host-supported rotation used by the cookbook. The offline affinity classifier remains a deployment input, as in this port. Preserve all exports while keeping the existing compact __all__ layout. Remove repeated run narratives; keep profile ordering and service probes intact. This revision neither adds a task-contract consumer nor changes reward policy. Validation: topic-profile executable AST is unchanged and the exported name set is preserved. Previously passing contract/routing tests were not rerun.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR information
Add validated task declarations, profile provenance and advisory topic/profile lookup.
Depends on #323. The current diff includes that PR; merge the dependency first.
resolve_intentandvalidate_task_contractcheck compatibility with the profile registry.Depends on the world/profile additions. Contracts are library APIs with no training caller.
Affinity JSONL is supplied externally; per-card lookup and host-supported rotation are separate APIs.
Experiment results