fix(kanban): prevent archived board watcher resurrection

[bezeusgptr]

Summary

• make named-board gateway watcher/notifier reads fail closed when a board disappears after discovery
• extend the same existing-only contract through gateway auto-decomposition: triage listing plus every decomposition read/write open
• prevent stale watcher snapshots from recreating an archived board directory/SQLite DB or silently falling back to the default board
• preserve create-on-demand defaults for explicit CLI/dashboard operations
• add behavioral archive-after-discovery coverage for triage listing and decomposition write paths

Root cause (5 whys)

1. Why did an empty active board reappear after archive? A long-lived gateway tick still held a board slug captured before the archive.
2. Why could that stale tick recreate state? Multiple sibling watcher paths opened the named board after discovery: notifier/dispatcher direct opens and auto-decomposer calls through list_triage_ids() / decompose_task().
3. Why were auto-decomposer opens still create-capable? connect_closing() did not expose create_if_missing, so its callers always inherited connect()'s create-on-demand default.
4. Why was passing only HERMES_KANBAN_BOARD insufficient? After archive, current-board resolution intentionally ignores a missing named-board environment slug and may fall back to the default board; a stale watcher therefore needed an explicit board pin as well as existing-only opens.
5. Why did the first remediation miss this? It fixed direct notifier/dispatcher connections but did not trace the full sibling call graph through the decomposer helper and each of its later read/write reconnections.

Root cause: the archive race affected a class of stale named-board watcher opens, but the initial fix covered only direct DB consumers. Auto-decomposition retained an indirect create-capable path through connect_closing() and could also resolve away from the archived slug.

Corrective action: connect_closing(), list_triage_ids(), and decompose_task() now accept and forward an existing-only mode. The gateway auto-decomposer explicitly pins the discovered slug and passes create_if_missing=False through triage listing and every decomposition DB open. Defaults remain create-capable for explicit user/dashboard operations. Regression tests archive after discovery and during decomposition, covering list/read and both specify/fanout write paths. No new lock, daemon, configuration, restart procedure, or cleanup process was added.

Validation

Exact head: 6bb0994137f1afd86e2d6962012dca471dbd2451

• scripts/run_tests.sh tests/hermes_cli/test_kanban_boards.py tests/hermes_cli/test_kanban_decompose.py tests/gateway/test_kanban_notifier.py tests/gateway/test_kanban_watchers_mixin.py -q — 84 passed, 0 failed
• broader canonical Kanban suite from the remediation run — 1020 passed, 0 failed
• Ruff on all 5 changed files — passed
• git diff --check 742918a47501dbc47091ca3643c3a79f7ca9fb82..6bb0994137f1afd86e2d6962012dca471dbd2451 — passed
• added-line security pattern scan — no matches

SonarQube

• project: hermes-agent-t-936d9597-archive-ghost
• Compute Engine task: 8aa9d0ed-6a65-4c88-a989-40696612fa86 (SUCCESS)
• analysis: 4fcc3fbc-6f06-4082-b6e1-1a2f4af31774
• SCM revision: 6bb0994137f1afd86e2d6962012dca471dbd2451
• Quality Gate: OK
• coverage: 84.4%
• new bugs: 0; new vulnerabilities: 0; new security hotspots: 0
• one pre-existing fence-regex hotspot was reviewed as safe because it receives only an internal auxiliary-LLM response capped at 4000 tokens; the remediation does not alter that regex path

Scope and safety

This PR does not modify archived databases, delete historical ghost stubs, deploy code, or restart the gateway. The unrelated local package-lock.json modification remains untouched. A later independently approved exact head still requires the externally managed gateway restart and isolated post-restart smoke described in the Kanban task.

[teknium1]

Thanks for tracing the stale named-board connection race and hardening the direct notifier/dispatcher opens. The approach is sound for those call sites, but one gateway dispatcher path remains create-capable.

Problems

• gateway/kanban_watchers.py:1147-1164 independently snapshots boards for auto-decomposition, pins the selected slug, and calls list_triage_ids(). That helper opens connect_closing() at hermes_cli/kanban_decompose.py:461, which delegates to connect() at hermes_cli/kanban_db.py:1813; connect() creates the parent directory at :1709. Therefore an archive after the auto-decomposer snapshot can still recreate the named board. The later decompose_task() opens at hermes_cli/kanban_decompose.py:284, :364, and :433 have the same issue.

Suggested changes

• Thread the existing-only mode through connect_closing(), list_triage_ids(), and auto-decomposer calls made by the gateway watcher, while retaining the default create behavior for explicit user operations.
• Cover the auto-decompose-enabled archive-before-connect race.

Automated hermes-sweeper review.

[bezeusgptr]

Remediation pushed for the auto-decomposer sibling path identified in review. Exact head: 6bb0994137f1afd86e2d6962012dca471dbd2451. Existing-only mode now propagates through connect_closing(), list_triage_ids(), and every watcher-invoked decompose_task() DB open. Validation: 84/84 targeted tests, 1020/1020 broader Kanban tests, Ruff and diff-check pass; Sonar CE 77705907-6d4a-491b-893f-783adda21a85, analysis f06265a2-2952-4f66-9034-45760f75d2f2, Quality Gate OK, 84.4% coverage / 100% new-code coverage. The PR body now contains the full sibling-path 5-whys RCA. Requesting exact-head re-review; do not merge until Kanban review approval.

[bezeusgptr]

Independent review — APPROVE

Reviewed exact pushed head 6bb0994137f1afd86e2d6962012dca471dbd2451.

Findings: none.

Verified:

• exact remediation commit changes only the requested five files;
• connect_closing, list_triage_ids, and all three decompose_task DB opens forward board and create_if_missing;
• gateway auto-decomposition pins the discovered slug and uses create_if_missing=False for list and decomposition paths;
• explicit CLI/dashboard call sites retain the create-enabled defaults;
• archive-after-discovery and archive-during-write regressions cover named-board non-recreation, with the gateway regression also asserting no default-board fallback;
• canonical targeted suite: 84 passed, 0 failed;
• Ruff: passed;
• git diff --check: passed;
• added-line security scan: 0 matches;
• remote branch and PR head equal the reviewed SHA;
• SonarQube CE SUCCESS, Quality Gate OK, revision matches exact SHA, coverage 84.4%, new bugs/vulnerabilities/security hotspots all 0;
• PR body includes a concise five-whys RCA with root cause, corrective action, and fix location, without extra process machinery.

Formal GitHub approval is skipped because the authenticated reviewer and PR author are both bezeusgptr; this comment plus the Kanban reviewer handoff records the independent approval.

[GottZ]

This was generated by AI during triage.

Summary

Thirteen PRs address or reference four Kanban gateway issues: singleton-dispatcher locking, false stuck telemetry, hidden or unscoped auto-decomposition failures, concurrency parsing, activity and notifier reporting, archived-board resurrection, and worker-log cleanup. The visible diffs show several distinct causes, with substantive overlap only in the locking, false-stuck, max_spawn-coercion, and auto-decompose secret-scope groups.

Related pull requests

• fix(gateway): add flock-based singleton guard to kanban dispatcher #41679 [closed] partial — (+273/-62) — n/a: Adds a same-profile dispatcher lock and release tests for [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448; this closed predecessor remains relevant because the stronger machine-global implementation was later shipped through fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068 at dfa5610 and 226e932.
• fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #44919 [closed] fixes — (+106/-0) — n/a: Implements the machine-global cross-platform lock at the shared Kanban root, matching [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448 across profiles; it is closed because its reviewed commits were salvaged with attribution into fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068.
• fix(gateway): avoid false kanban stuck alerts at concurrency cap #46892 best fix — (+256/-1) — n/a: The recorded best fix for fix: _kanban_dispatcher_watcher fires false "stuck" alert when hook fills concurrency cap #46800 detects concurrency-cap occupancy, but the contributor keep_open review identifies cross-board aggregation that can suppress a real warning and requests per-physical-board accounting plus daemon-path handling.
• fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068 [merged] best fix — (+107/-0) — n/a: This merged reference implementation carries fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #44919's machine-global lock, cross-platform acquisition, and explicit release into main, resolving the duplicate-dispatcher cause in [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448.
• fix(kanban): promote auto-decompose failures from DEBUG to rate-limited WARNING #49861 best fix — (+90/-3) — n/a: The recorded best fix for [Bug]: Auto-decompose failures log at DEBUG — invisible when gateway runs at INFO #49859 changes non-successful auto-decompose outcomes from DEBUG-only to rate-limited WARNING; the visible diff now includes controlled-time tests for the initial warning, suppression, and 300-second boundary requested by the keep_open verdict.
• fix(kanban): run auto-decompose tick under the profile secret scope #57837 related — (+170/-50) — n/a: Installs and resets a profile secret scope around the threaded auto-decompose tick, addressing multiplexed UnscopedSecretError failures; the focused test and field report support the documented keep-open salvage path.
• fix(gateway): cast max_spawn config to int to prevent TypeError (#59499) #59588 related — (+98/-3) — n/a: Casts raw max_spawn values, but accepts non-positive values and tests synthetic parsing rather than the production watcher path; despite its keep_open review, fix(kanban): close parent log handle after spawn and validate max_spawn #65644 contains the stronger positive-value validation and real-path test.
• fix(kanban): log dispatcher tick summary on crash/timeout/reclaim/auto-block, not only on spawn #62893 related — (+80/-9) — n/a: Expands dispatcher tick summaries beyond spawn-only activity to crashes, timeouts, reclaims, auto-blocks, and stale tasks, with the visible diff adding the stale-only regression requested by the keep_open review.
• fix(kanban): report truthful gave-up causes #64605 related — (+48/-7) — n/a: Replaces the unsupported repeated-spawn-failure explanation for every gave_up event with the supplied sanitized error or a cause-neutral fallback, preserving the notifier-focused keep-open salvage identified by review.
• fix(kanban): prevent archived board watcher resurrection #65321 related — (+392/-32) — n/a: Makes stale named-board watcher and auto-decomposer opens existing-only so archive races cannot recreate a removed board or fall back to default; the visible diff addresses the keep_open review by threading board and create_if_missing through all decomposition opens and adding race coverage.
• fix(kanban): avoid false dispatcher stuck warnings #65581 best fix — (+485/-35) — n/a: The recorded best fix for fix: _kanban_dispatcher_watcher fires false "stuck" alert when hook fills concurrency cap #46800 adds board-local gateway health accounting and daemon deferral handling, addressing the contributor keep_open review's cross-board objection, but it also carries unrelated gateway-status changes and its explicit global-deferral flag is tied to max_in_progress rather than the issue's externally occupied max_spawn case.
• fix(kanban): close parent log handle after spawn and validate max_spawn #65644 related — (+80/-8) — n/a: Positively validates max_spawn through the real watcher path and closes the parent log handle after successful Popen, but the contributor keep_open review still requires finally-based cleanup and a regression for a non-FileNotFoundError Popen failure.
• fix(gateway): respect explicit platform disable for HA and wrap kanban auto-decompose in profile scope #76864 [closed] related — (+62/-57) — n/a: Its Kanban half duplicates fix(kanban): run auto-decompose tick under the profile secret scope #57837's auto-decompose secret-scope repair and its Home Assistant half is unrelated; this PR remains relevant as a closed split source because the author moved those fixes to fix(kanban): run auto-decompose tick under the profile secret scope #57837 and fix(gateway): respect explicit platform disable for Home Assistant in env overrides #76871.

Duplicates

#41679 and #44919 overlap on #41448, with #49068 as the merged machine-global consolidation; #46892 and #65581 overlap on #46800 but require consolidation of complementary scope; #59588 is substantially duplicated by #65644's stronger max_spawn validation; the Kanban half of closed #76864 duplicates #57837.

Suggested consolidation

Keep #65321 open with a salvage path: obtain exact-head maintainer re-review of the post-review existing-only auto-decompose propagation and archive-race tests. Keep #49861, #57837, #62893, and #64605 open on their documented focused salvage paths; author action on #46892/#65581 should split or rebase a consolidated board-local gateway-and-daemon false-stuck fix without the unrelated gateway-status changes, while #65644 should add finally-based exceptional-Popen cleanup before #59588 can close as its duplicate; #41679 and #44919 remain closed in favor of merged #49068, and #76864 remains closed in favor of #57837 and #76871.

Cross-PR triage: Reviewed 13 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 142 kB of PR diffs, 45 kB of issue/PR text, 29 kB of discussion (31 comments), 11 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.