Skip to content

fix(kanban): report a fully held ready queue instead of an idle board - #123964

Open
yoyodine-industries wants to merge 34 commits into
NousResearch:mainfrom
yoyodine-industries:fix/kanban-active-pr-guard
Open

yoyodine-industries wants to merge 34 commits into
NousResearch:mainfrom
yoyodine-industries:fix/kanban-active-pr-guard

Conversation

@yoyodine-industries

@yoyodine-industries yoyodine-industries commented Sep 26, 2026 •

Copy link
Copy Markdown

A ready lane whose every spawnable row is held by a respawn guard spawns nothing,
tick after tick, and no surface outside gateway.log says so. describe_suppression()
summarises the tick from skip buckets it only partially reads, and there is no
board-level read that separates "all of this work is held" from "there is no work".
This PR makes the hold visible, bounds the event it writes, and lets a deliberate
re-queue release it.

One concern: 0 spawned must not be the only number available, and a board that is
fully held must not read as idle.

Related Issue

Fixes #123963. The PRs that share this symptom are cross-linked under "Related work"
below; none of them carries the board-level read, and no covering issue existed, which
is the half that issue is filed against.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

What changes

hermes_cli/kanban_db_dispatch.py

  • active_pr lifts on status, promoted or unblocked as well as on the handoff
    kinds (assigned, changes_requested, review_reopened). reclaimed is
    deliberately excluded, unlike guard 3: a crash or reclaim is not a decision, and
    the PR is still open and unattended.
  • _RESPAWN_GUARD_PR_WINDOW is untouched at 86400, and a test asserts it.
  • respawn_guarded is written once per hold episode: re-emitted when the reason
    changes, when a new episode starts, or after an hour.
  • describe_suppression() counts skipped_per_profile_capped, skipped_nonspawnable
    and skipped_unassigned in addition to the buckets it already counted.
  • board_health() returns (ready_total, spawnable, suppressed_by_reason) with a
    state verdict, and observe_dispatch_tick() files one routed card after six
    consecutive stalled ticks with repeat spacing.

hermes_cli/kanban.py, hermes_cli/kanban_parser.py

  • hermes kanban health renders the verdict and the per-reason counts.

plugins/kanban/dashboard/plugin_api.py, dist/index.js, dist/style.css

  • the board payload carries dispatch_health, and a strip renders it, so a starved
    board does not look like a quiet one.

No config key, no env override, no GitHub API call from the tick.

Root cause

check_respawn_guard matches any GitHub PR URL in a card's thread
(hermes_cli/kanban_db_dispatch.py:1784-1802, regex at :85) inside a 24 hour window
(:83). The URL does not have to belong to this card and its state is never consulted:
a card naming a parent's PR, or a PR that has since merged, is held exactly like one
with its own open PR. The lift set at :1797 is stricter than the sibling guard's at
:1766-1776 over a window 24 times longer.

Verification

Base is d0288be5b3, the same commit as upstream/main at branch time.

  • Premise A/B, one test file, base-blob swap: base d0288be5b3 = 21 failed / 4 passed
    (the 4 passing are the controls: handoff vocabulary, the reclaimed negative, the
    empty tick, the window assertion), tip = 25 passed.
  • scripts/run_tests.sh over the nine kanban test files: 129 passed, 1 failed,
    1 skipped, with test_kanban_dispatch_health.py at 25 passed. The single failure
    (test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_card) is
    pre-existing: the same file, run the same way at d0288be5b3, fails identically.
  • macOS, Python 3.11.16.

How to Test

  1. scripts/run_tests.sh tests/hermes_cli/test_kanban_dispatch_health.py — 25 passed.
  2. On a board with a ready row held by the respawn guard, hermes kanban health
    reports state=starved and names the guard in suppressed_by_reason; an empty
    board reports state=idle. The state line does not depend on how many rows are held.
  3. To watch the lift: hermes kanban unblock <id> (or promote) on a held row, then
    run a dispatch tick — the row starts, and the guard event is not written again.

Related work

Searched for existing PRs before writing this one. Each of these is open and covers a
part of the same symptom:

The root cause here is complementary rather than the same: those change when a hold is
released or how the identical events are collapsed, while this one adds the board-level
read and the stalled-board escalation, so a fully held queue is reported as its own
condition instead of as an idle board. If a maintainer would rather take the visibility
half alone, it splits cleanly at board_health() plus the dashboard and CLI renderers.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(kanban):, one commit, seven paths)
  • I searched for existing PRs to make sure this isn't a duplicate — the ones that share the symptom are in "Related work" above; none of them carries the board-level read
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've added tests for my changes — tests/hermes_cli/test_kanban_dispatch_health.py, 25 cases, 21 of which fail at the base commit
  • I've tested on my platform: macOS 26.5.2, Python 3.11.16
  • I ran the files this change touches, not the full suite — scripts/run_tests.sh over the nine kanban test files → 129 passed, 1 failed, 1 skipped, with tests/hermes_cli/test_kanban_dispatch_health.py alone → 25 passed. The one failure, tests/hermes_cli/test_kanban_db.py::test_infrastructure_spawn_refusal_never_charges_the_card, reproduces identically at the base commit d0288be5b3, so it is pre-existing rather than introduced here. The full pytest tests/ -q was not run on this host.

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A: this tree has no docs/ directory and no README content covers these reads. The new subcommand's help and description ship in hermes_cli/kanban_parser.py, and _cmd_health carries its docstring, both in the diff.
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A: no config key is added, on purpose, so the behaviour cannot be configured away.
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A: no architecture or workflow changed. The new read is additive, and the escalation only fires after six stalled ticks.
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A: pure Python plus the dashboard's existing JS bundle, no path, shell or OS-specific code added.
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A: no tool schema is touched.

For New Skills

Not applicable — this PR adds no skill.

Screenshots / Logs

scripts/run_tests.sh tests/hermes_cli/test_kanban_dispatch_health.py on the branch tip:

Discovered 1 test files (~25 tests) under ['tests/hermes_cli/test_kanban_dispatch_health.py']; running with -j 14
[100.0% |    25/~25 | ✓25 | ✗ 0] ✓ tests/hermes_cli/test_kanban_dispatch_health.py (25✓, 4.0s)
=== Summary: 1 files, 25 tests passed, 0 failed (100% complete) in 4.1s (14 workers) ===

Authored with AI assistance (Hermes Agent); diff verified manually, conventional commits
per the contributing guide.

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins labels Sep 26, 2026
@Enough1122

Copy link
Copy Markdown

AI code review — automated follow-up for reference; not a maintainer.

The diagnosis is right and the guard-vocabulary unification (guard 4 honouring _RESPAWN_GUARD_REQUEUE_KINDS, one event per hold episode) is well argued. board_health has a scoping bug that makes the new signal lie in exactly the multi-tenant case the dashboard already supports.

1. Blocker: board_health ignores the tenant filter, so dispatch_health is wrong for every filtered board — in both directions.

get_board accepts tenant and scopes the rendered columns with it (plugin_api.py:283-294, via list_tasks's AND tenant = ?), then embeds kbd.board_health(conn, board=board).as_dict() in that same payload (plugin_api.py:341-344). But board_health runs an unfiltered SELECT id, assignee, claim_lock FROM tasks WHERE status = 'ready' (kanban_db_dispatch.py:2042-2044); its board= parameter is documented as "accepted for the caller's convenience" and never reaches the query. DispatchHealthStrip renders off that number regardless of the selected tenant.

Drove the real board_health (AST-extracted from the head blob, real sqlite, stubbed guard) on a two-tenant board:

operator view rows seen true verdict reported
?tenant=acme (1 ready, held) 1 starved dispatchable — strip suppressed
?tenant=globex (0 ready, 3 running) 0 idle starved — strip on a clean board

Both failure modes from one cause: the strip shows on healthy tenants while the genuinely starved one stays silent — the inverse of the PR's goal. The CLI and /health share the gap (get_board_health takes only board, plugin_api.py:369); threading tenant into board_health's query fixes all three call sites. The 26 new tests never set tenant (grep -c tenant = 0).

Minor: starved is startable > 0 and suppressed == startable (:2082); the loop continues after the cap, so a row cannot be both capped and guard-held today. Noted only because the two sit in different buckets and a future reorder could change the verdict.

Unverified / please confirm: the table is executed, not inferred — real board_health from a3992bb3, real sqlite, guard stubbed to return active_pr for the held id only; probe ran in 0.5s. Sources self-proved with api .size == wc -c. Please confirm tenant-scoped dashboards are supported; if a home is single-tenant in practice, this is narrower than I read it.

@yoyodine-industries

Copy link
Copy Markdown
Author

Thanks — confirmed, this is a real defect and I reproduced it independently at head 1b64a6407d.

Cause (verified read-only at head): get_board scopes the rendered columns with list_tasks(..., tenant=tenant) (plugin_api.py:293) but then embeds kbd.board_health(conn, board=board).as_dict() in the same payload (plugin_api.py:344). board_health runs an unfiltered SELECT id, assignee, claim_lock FROM tasks WHERE status = 'ready' — its board= parameter is documented convenience-only and never reaches the query, and it takes no tenant parameter at all. /health shares the gap (plugin_api.py:369-377).

Independent probe — the real board_health AST-extracted from the head blob (hermes_cli/kanban_db_dispatch.py @ 1b64a6407d), driven against a real in-memory two-tenant board with the guard stubbed to hold the one id. Both failure directions, one cause:

operator view columns rendered true verdict reported
?tenant=acme — 1 ready, held (sibling tenant has a startable row) 1 ready starved dispatchable — strip suppressed
?tenant=globex — 0 ready (acme's held row is the board's only ready) 0 ready idle starved — strip on a clean board

On your unverified note: tenant-scoped dashboards are supported — get_board takes tenant, scopes rows with it, and the payload already enumerates tenants (plugin_api.py:336). So this is the multi-tenant case, not a single-tenant hypothetical.

The fix (thread tenant into board_health and both call sites — plugin_api.py:344 and get_board_health at :369 — plus a test that sets tenant and is RED without the change) is routed to the authoring lane. New head sha will follow here when it lands. The minor starved/cap-bucket invariant you flagged is carried with it.

yoyodine-industries added a commit to yoyodine-industries/hermes-agent that referenced this pull request Oct 7, 2026
board_health scanned every ready row on the board, so its starved-vs-idle
verdict lied for a ?tenant= view in BOTH directions: a tenant with one held
ready row read "idle", and a tenant with no ready rows read "starved" because
another tenant was stalled.

Thread tenant through board_health (exact match, the same semantics as
kanban_db.list_tasks), the embedded /board dispatch_health tuple and /health,
and the CLI health read; add --tenant to the health verb. The scoped receipt
names the tenant it measured, so it cannot be mistaken for the unscoped one.
The per-profile-cap running count stays board-wide on purpose (the cap is a
physical per-profile limit, not a tenant one). Pin the cap-branch-before-guard
invariant the reviewer flagged.

Reviewer: Enough1122 (PR NousResearch#123964).
@yoyodine-industries

Copy link
Copy Markdown
Author

Fixed — tenant is now threaded end to end.

New head: 591e7b80a7866b2c734c31cd901bca58b82464a5 (branch fix/kanban-active-pr-guard, fast-forward from 08d06f94c5; still conflict-free).

What changed — 4 source files, 2 test files:

  • hermes_cli/kanban_db_dispatch.py — board_health(conn, *, board=None, tenant=None). The ready-row scan adds AND tenant = ? only when tenant is given (exact match, the same semantics as list_tasks's tenant= — a NULL-tenant row matches no tenant name). The per-profile-cap running count stays board-wide on purpose: the cap is a physical per-profile limit, not a tenant one, so a running row in another tenant still consumes the slot this tenant's ready rows would claim. The docstring now says so.
  • plugins/kanban/dashboard/plugin_api.py — the embedded dispatch_health (:344) passes the request's tenant; get_board_health gains a tenant query param. Both echo tenant in the payload only when scoped, so a scoped receipt and the unscoped tuple cannot be mistaken for each other.
  • hermes_cli/kanban.py / hermes_cli/kanban_parser.py — hermes kanban health --tenant <t> carries the same scope; the CLI previously had no way to express it.
  • The minor cap/guard bucket invariant is pinned with a comment beside the starved = ... line plus a regression assertion (suppressed <= startable), so a future reorder of those two branches fails a test instead of silently changing the verdict.

Both directions, RED without the fix:

  • a tenant whose single ready row is held reads starved;
  • a clean tenant reads idle while a sibling tenant is starved;
  • the unscoped read still reports the whole board, and its as_dict() is byte-identical — the eight keys, unchanged; tenant is echoed only by scoped callers.

Tests. pytest tests/hermes_cli/test_kanban_dispatch_health.py tests/plugins/test_kanban_dashboard_plugin.py -q → 69 passed. Reverting just the four source files to the base commit while keeping the new tests fails exactly the five new assertions — the site test on the value (assert 'starved' == 'idle') and the three unit/CLI tests on the missing tenant kwarg/flag — with all pre-existing tests still green.

yaan-platform-coder Bot and others added 2 commits October 11, 2026 15:00
…s event spam

The ready-lane respawn guards could hold an entire board's queue while every
surface still read as a correctly idle board, and one of them wrote an event
per tick, so the only evidence was both unfindable and unreadable.

- active_pr now lifts on a deliberate re-queue (status / promoted / unblocked)
  in addition to a handoff (assigned / changes_requested / review_reopened).
  reclaimed stays out of it: crash recovery is not a decision.
- respawn_guarded is written once per hold episode instead of every tick.
- describe_suppression names every bucket that can hold a ready row.
- board_health reports (ready_total, spawnable, suppressed_by_reason) plus a
  state verdict, surfaced as `hermes kanban health`, on the board payload and
  as a dashboard strip, so starved != idle at any row count.
- six consecutive stalled ticks file one routed escalation card with repeat
  spacing, instead of one more line in a log nobody is tailing.
board_health scanned every ready row on the board, so its starved-vs-idle
verdict lied for a ?tenant= view in BOTH directions: a tenant with one held
ready row read "idle", and a tenant with no ready rows read "starved" because
another tenant was stalled.

Thread tenant through board_health (exact match, the same semantics as
kanban_db.list_tasks), the embedded /board dispatch_health tuple and /health,
and the CLI health read; add --tenant to the health verb. The scoped receipt
names the tenant it measured, so it cannot be mistaken for the unscoped one.
The per-profile-cap running count stays board-wide on purpose (the cap is a
physical per-profile limit, not a tenant one). Pin the cap-branch-before-guard
invariant the reviewer flagged.

Reviewer: Enough1122 (PR NousResearch#123964).
@yoyodine-industries
yoyodine-industries force-pushed the fix/kanban-active-pr-guard branch from 591e7b8 to 0d43a06 Compare October 11, 2026 19:01
pr-evergreen and others added 16 commits October 11, 2026 18:48
…resolution)

Conflict: hermes_cli/kanban_db_dispatch.py `check_respawn_guard` (guards 3 and
4). This one is a REAL semantic overlap — two independent reworks of the same
region — so it is resolved as a union, not by taking a side:

* Upstream (after this branch's base) routed both guards through a new
  `_rows_since()` helper and hardened every guard timestamp with
  `_kb._to_epoch()` (NousResearch#97455: a TEXT stamp sorts above every INTEGER, so the raw
  `int()` reads this branch still carried would raise on one).

* This branch introduced the named kind vocabulary
  (`_RESPAWN_GUARD_REQUEUE_KINDS` / `_RECOVERY_KINDS` / `_HANDOFF_KINDS`) with
  the `_kind_clause()` placeholder helper, made guard 4 honour a deliberate
  re-queue, and added the `respawn_guarded` episode throttle.

Resolution: keep upstream's `_rows_since()` structure (so the `_to_epoch`
hardening applies to every guard read), and re-express this branch's named kind
tuples through it — guard 3 keeps its requeue+recovery vocabulary and guard 4
keeps its requeue-lifts / `reclaimed`-does-not semantics verbatim, including the
payload-aware `_is_handoff_event` loop. `_rows_since()` gains one keyword,
`params`, so a caller-supplied `where` may carry its own placeholders, bound
ahead of `since`.

Validation: the six suites that exercise the guard pass (150 passed, 1 skipped);
the one failure, test_infrastructure_spawn_refusal_never_charges_the_card, also
fails at this branch's pre-merge tip and at pristine upstream/main, so it is a
pre-existing environment failure and not a resolution artefact.

Fork policy: the resolved branch is pushed to the FORK branch only; nothing is
merged to fork main, which is an upstream mirror.

Agent: platform-coder

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kanban: an all-held ready queue reads as idle, and nothing outside gateway.log says so

3 participants