Skip to content

fix(cron): find unified-runtime cron runs by their scheduler admission - #138968

Closed
dokterdok wants to merge 4 commits into
NousResearch:mainfrom
dokterdok:fix/cron-unified-run-history
Closed

dokterdok wants to merge 4 commits into
NousResearch:mainfrom
dokterdok:fix/cron-unified-run-history

Conversation

@dokterdok

@dokterdok dokterdok commented Oct 15, 2026 •

Copy link
Copy Markdown

What does this PR do?

The problem. Since #106742 merged, new cron runs are missing from their job's run history (#138812).

  • Desktop Cron page and sidebar: every run fired after the switch shows as a dead row, with no click and no transcript. Older runs still open.
  • Runs API: each new fire appears only as an output-file row.
  • Session detail: no longer marks these sessions as cron-owned.

The runs themselves are fine; nothing can find them. An older bug is fixed along the way (#92133): job backup's history also lists the runs of job backup_weekly.

Why. On the unified runtime a cron run gets a local-<hash> session id, but every history reader still looks for the old cron_<job>_<timestamp> id. The new id must stay: it is the hash of the fire's admission id, which exact-once retry depends on.

The fix. Runs are found through the record the runtime already keeps for every fire: the scheduler's admission, filed as cron:<job>:<execution>.

Proof. On main the new tests fail: the local-* run is missing, detail ownership is None, and job daily lists the runs of daily_digest. The Desktop test fails too. All pass here. The first commit is #138821's (@KoNit-K), kept with its authorship.

Before / after:

Surface Before After
SessionDB.list_cron_job_runs / GET /api/cron/jobs/{id}/runs post-switch runs missing; each shows up only as a synthetic cron_output doc row listed as session rows, alongside legacy runs; output-doc and ledger rows are matched to their session by execution id, so a run appears once
Desktop Cron page and sidebar new runs are inert rows (no click, no transcript) open their transcript; a run is recognised by its row (source: 'cron'), with the legacy id as fallback
Session detail scheduler_owned None for local-* runs (not recognised as cron) works; ownership of a unified run is an exact execution match
Job backup vs job backup_weekly (legacy ids, #92133) backup's history also lists backup_weekly's runs a legacy id must be the prefix plus exactly YYYYmmdd_HHMMSS

Related Issue

Fixes #138812
Fixes #92133

Refs #134889 (CRON-1): work package #139162, lane 3. CRON-1 is #138812.

Thanks to @heerokod668 for the root-cause trace in #138812.

Overlap:

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • hermes_state_keys.py: one shared definition of the scheduler's request id (cron:<job>:<execution>) and principal, used by producer and reader.
  • hermes_state_common.py: partial covering index idx_session_admissions_cron_request on the scheduler's admissions.
    • It is declared in SCHEMA_SQL without a version bump, the documented convention for an index; an older store gains it on its next writable open and answers correctly, only unindexed, until then.
  • hermes_state_portability.py:
    • list_cron_job_runs merges unified runs (an admission range scan) with legacy runs (the id-range scan, now constrained to the exact timestamp suffix).
    • Rows carry cron_job_id and cron_execution_id.
    • cron_run_identity() answers a single session.
  • gateway/session_cron.py: refuses an execution id containing :. The request id splits at its last :, so fire x:y of job a would be filed under job a:x.
  • hermes_cli/web_routers/cron.py, hermes_cli/web_routers/sessions.py: pair runs with their execution, and stamp detail's scheduler_owned for unified runs.
  • Desktop (cron/open-cron-run.ts, cron/index.tsx, chat/sidebar/cron-jobs-section.tsx, use-prompt-actions/submit.ts): one predicate decides which rows open a transcript, keyed on the row rather than the id shape. The submit.ts change is a rename at a call site.

For reviewers:

  • Retention: admission rows are deleted only with their session (retire_terminal_receipts), so a unified run stays in history exactly as long as a legacy run does.
  • Spoofing: the lookup filters on the scheduler principal. Only session_cron writes cron: request ids under it.
  • Known limitation, unchanged from main: only a run's first session is listed. If a cron run compresses into a continuation, liveness and detail identity follow the first session, not the compression tip. This is documented on the method and will be a follow-up.
  • An independent adversarial review of this branch checked the schema convention, retention, spoofing, wildcard job ids (_, %, :) and dedupe. Its findings (the : guard, the legacy collision, a lighter Desktop import) are folded in.

How to Test

scripts/run_tests.sh tests/hermes_state/test_cron_run_sessions.py tests/hermes_cli/test_cron_run_history_sessions.py
cd apps/desktop && npx vitest run --project ui src/app/cron/open-cron-run.test.ts
  1. Fail first: run against main, the new Python tests fail. The local-* run is not listed, the web runs list shows ['cron_output', 'cron'] for one fire, detail scheduler_owned is None, an execution id with : is accepted and misfiled, and job daily lists cron_daily_digest_…. The legacy-id control passes. On fix(cron): list unified runtime run sessions #138821's commit alone, detail ownership, the failed-neighbour row and the index check still fail. The Desktop local-* gate fails on main ("expected false to be true").
  2. On this branch:
    • New tests plus the cron ownership, cron-profile, session_cron, owner-loop, recovery and worker-index suites: 80 passed.
    • tests/hermes_state/test_hermes_state.py: 294 passed, 3 skipped.
    • vitest on 12 cron and sidebar files: 254 passed.
    • eslint and tsc --noEmit clean.
  3. scripts/check --base upstream/main: 12 checks ok; code health 0 blocking, 0 advisory.

Checklist

Code

Documentation & Housekeeping

  • Documentation — the limitation is documented on list_cron_job_runs
  • cli-config.yaml.example — N/A
  • CONTRIBUTING.md / AGENTS.md — N/A
  • Cross-platform impact considered — SQL and row-shape changes only
  • Tool descriptions/schemas — N/A

KoNit-K and others added 4 commits October 15, 2026 09:58
Builds on the previous commit, which listed `local-*` cron sessions by
decoding the frozen policy receipt of every `gateway.local_policy.v1:*`
row on each request. That scan covers every local session (Desktop, CLI,
TUI and cron alike), parses each multi-KB receipt twice, and its OR
defeats the legacy id-range index, so the cost grows with all history.
On a synthetic store it took ~56 ms at 2k local sessions and ~1.5 s at
40k, for a 20-row page.

The durable association already exists and is the run's identity: the
scheduler principal admits each fire under `cron:<job>:<execution>`, and
that request id is the pre-image of the run's `local-*` session id. The
lookup is now a range scan over a partial covering index of those
admissions (`idx_session_admissions_cron_request`, added declaratively in
SCHEMA_SQL; an older store gains it on its next writable open and
answers correctly, only unindexed, until then). Legacy
`cron_{job}_{ts}` sessions keep their id-range scan. Both scale with the
job's own runs (<1 ms in the same benchmark).

Each run stays filed under its own job. The producer now refuses an
execution id containing `:` (the request id splits at its last `:`, so
fire `x:y` of job `a` would read as job `a:x`), the admission scan keeps
job `a` from claiming job `a:b`'s fires, and a legacy id must end in
exactly `YYYYmmdd_HHMMSS` after the prefix, so job `daily` no longer
claims `cron_daily_digest_<ts>`. The legacy list test now seeds ids in
the scheduler's real shape. As before, only a run's first session is
listed; a compression continuation is not (documented on the method).

Rows now carry `cron_job_id` and `cron_execution_id`, and
`SessionDB.cron_run_identity()` answers the same for one session. The
request-id format and principal move to `hermes_state_keys`, and the
producer in `gateway/session_cron.py` uses them, so reader and producer
cannot drift. The session id scheme is unchanged.

The receipt-fixture test from the previous commit is superseded by tests
that create runs through the real producer.

Refs NousResearch#138812
Run history and session detail still recognised a run session only by
its legacy `cron_{job}_{ts}` id:

- session detail's `scheduler_owned` (`cron_run_scheduler_owned`) skipped
  every `local-*` run, so a restored tab could not tell a live run from a
  zombie. The endpoint now stamps the row's cron identity first.
- ownership of a unified run is exact: the in-flight ledger attempt must
  be the run's own execution, not merely claimed before it started.
- output docs and ledger-only rows were paired with sessions by a 300 s
  window. A doc or attempt row now carries its execution id and pairs
  with the session of that execution, so a fire that failed a minute
  after a unified run (before it created a session) keeps its own row.
  Legacy sessions, which name no execution, keep the window.

Refs NousResearch#138812
A unified-runtime cron run keeps a runtime-minted `local-*` id, so the
`cron_{job}_{ts}` regex no longer recognised it. A tab restored after a
restart then skipped the pre-send write gate (NousResearch#88443) and could send
into a run the scheduler no longer owns.

`isCronRunSession` now asks the cached row (`source === 'cron'`, written
only by the scheduler) and keeps the legacy id as the fallback for a tab
restored before any list loaded its row. The Cron page and the sidebar
peek share `isCronRunRow` to decide which run rows open a transcript.

Refs NousResearch#138812
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery comp/desktop Electron desktop app (apps/desktop/*) area/sessions Session lifecycle, resume, persistence, history sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Oct 15, 2026
unsupportedpastels added a commit that referenced this pull request Oct 15, 2026
On the unified gateway runtime an agent cron fire keeps its
runtime-minted `local-*` session id (the hash of its admission id, which
exact-once retry re-derives), but run history still found a job's runs
only by the legacy `cron_{job_id}_{ts}` id prefix. Post-switch runs fell
back to inert `cron_output` doc rows in Desktop and session detail could
not tell whether the scheduler still owned them.

Runs are now keyed by the record the gateway already writes for every
fire: the scheduler principal's admission `cron:<job>:<execution>`.

- `SessionDB.list_cron_job_runs` merges an admission range scan (new
  partial covering index `idx_session_admissions_cron_request`) with the
  legacy id-range scan; rows carry `cron_job_id`/`cron_execution_id`.
  Legacy ids must end in exactly `YYYYmmdd_HHMMSS`, matching
  `hermes_state_ids` and keeping job `daily` from listing `daily_digest`.
- `SessionDB.cron_run_identity()` answers one session; session detail
  stamps it so `scheduler_owned` works for `local-*` runs.
- The web runs list pairs output docs and ledger rows with sessions by
  execution id (time window only for legacy sessions).
- `gateway/session_cron.py` shares the request-id format with the reader
  (`hermes_state_keys`) and refuses an execution id containing `:`.
- Desktop decides a row is a cron run from `source: 'cron'`, with the
  legacy id as fallback for a restored tab.

Read-only on the client side; the gateway stays the only writer and the
session id scheme is unchanged. Salvaged from #138968, which builds on
#138821.

Refs #134889 (CRON-1)
Refs #139162
Fixes #138812

Co-authored-by: David Dudok de Wit <5354424+dokterdok@users.noreply.github.com>
Co-authored-by: KoNit-K <konit.block@protonmail.com>
(cherry picked from commit 1225bc3e05c2b91069394f6c2c2d004cf10425c7)
@dokterdok

Copy link
Copy Markdown
Author

Landed via #139273, which salvages this PR whole with authorship kept, plus two review fixes. Thanks @unsupportedpastels. Closing. The gap #139273 notes (a cron run that compresses mid-run is listed by its first session only) is fixed in draft #139268, which I'll rebase onto main.

@dokterdok dokterdok closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/cron Cron scheduler and job management comp/desktop Electron desktop app (apps/desktop/*) comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

3 participants