fix(kanban): avoid false dispatcher stuck warnings

[jtstothard]

Problem

The kanban dispatcher's health telemetry raises "stuck" warnings whenever the ready queue is non-empty and no workers spawn on a tick — even when the non-spawn is intentional and correct (concurrency caps reached, profile already at max_in_progress_per_profile, a task is locked, or respawn is guarded). This produces noisy false-positive stuck warnings that make real stalls harder to spot and erode trust in health monitoring.

What changed

• DispatchResult gains skipped_global_capped — set when the board's global max_in_progress is already filled, mirroring the existing skipped_per_profile_capped.
• Dispatcher health (both gateway and daemon paths) now treats intentional deferrals (global cap, per-profile cap, locked, respawn-guarded) as healthy ticks. bad_ticks only increments when spawnable work is genuinely stuck, not deliberately deferred.
• hermes kanban dispatch CLI surfaces skipped_global_capped so operators see "Deferred (global in-progress cap reached)" instead of silent nothing.
• hermes gateway status now scopes its running/not-running messaging to the active profile, and when the active profile is down but another profile's gateway is live, it says so explicitly ("not a system-wide outage") with a "Live gateways in other profiles" section.

Why it matters

On multi-profile hosts the gateway-status wording previously implied a system-wide outage when only one profile was idle, and the health telemetry cried "stuck" on every tick where caps correctly prevented over-spawning. Both undermine operator confidence in the very signals meant to catch real problems.

Verification

• tests/hermes_cli/test_kanban_per_profile_cap.py — new test_global_cap_marks_dispatch_as_intentionally_deferred; all 10 tests pass
• tests/hermes_cli/test_gateway_service.py — new test_gateway_status_promotes_live_other_profile; all 189 tests pass
• Combined: 199 passed, 0 failed

Risks / follow-ups

• A genuine stall that coincides with a cap/lock/respawn-guard state on the same tick will not increment bad_ticks. This is a deliberate trade (false-positive reduction) but means a real stall masked by concurrent deferral would rely on subsequent ticks where the deferral clears.

[alt-glitch]

This was generated by AI during triage.

Related: #46892 covers the global-cap false-stuck case and #44899 covers respawn-guard readiness. This PR combines those areas through DispatchResult flags and also changes gateway-status wording; maintainers should choose the preferred scope.

[teknium1]

Thanks for tracing the false-stuck telemetry path. The current-main premise is real: a full global cap returns without spawning at hermes_cli/kanban_db.py:7600-7605, while the gateway increments its aggregate streak when any board still has spawnable ready work at gateway/kanban_watchers.py:1253-1258.

Problems

• gateway/kanban_watchers.py:1255 computes intentionally_deferred with any(...) across every board, while ready_pending is also an any-board probe. A cap, lock, or respawn guard on board A can therefore clear the aggregate streak while board B has genuinely stuck spawnable work.
• The added flag test in tests/hermes_cli/test_kanban_per_profile_cap.py:121 does not exercise either changed health predicate.

Suggested changes

• Make gateway health accounting board-local, so only the deferred board is exempted from a stuck warning.
• Add a multi-board regression plus direct health-predicate coverage for gateway and the legacy daemon path.

Automated hermes-sweeper review.

[teknium1]

results spans every board, while ready_pending is also an any-board probe. A cap/lock/guard on board A therefore resets bad_ticks even if board B has spawnable work that fails every tick. Please account for health per board, or suppress only when the board containing ready work is itself intentionally deferred.

[jtstothard]

Addressed in b84456b. The gateway health predicate is now board-local: ready work on a board only exempts bad_ticks when that same board is intentionally deferred, so a cap/lock/guard on board A no longer masks stuck work on board B. Added a multi-board regression test (board A deferred, board B stuck → warning still fires) plus direct health-predicate coverage for both gateway and daemon paths.

[GottZ]

This was generated by AI during triage.

Summary

Thirteen PRs address or reference four Kanban gateway issues through distinct diffs covering dispatcher exclusivity, false stuck telemetry, auto-decompose diagnostics and secret scope, concurrency parsing, activity and notifier reporting, archived-board races, and worker-log cleanup. Material overlap is concentrated in the singleton-lock, false-stuck, max_spawn-coercion, and auto-decompose secret-scope groups.

Related pull requests

• fix(gateway): add flock-based singleton guard to kanban dispatcher #41679 [closed] partial — (+273/-62) — n/a: Adds a profile-scoped dispatcher lock with explicit cleanup and tests for [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448; it remains relevant as a closed predecessor superseded by the machine-global implementation shipped through fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068.
• fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #44919 [closed] fixes — (+106/-0) — n/a: Implements the machine-global cross-platform singleton lock at the shared Kanban root, directly addressing [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448; it remains relevant because its reviewed commits were salvaged into fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068 before this PR was closed.
• fix(gateway): avoid false kanban stuck alerts at concurrency cap #46892 best fix — (+256/-1) — n/a: Detects externally occupied concurrency capacity for fix: _kanban_dispatcher_watcher fires false "stuck" alert when hook fills concurrency cap #46800, but its keep_open review identifies cross-board aggregation that can suppress real warnings and requests per-physical-board accounting plus daemon-path handling.
• fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #49068 [merged] best fix — (+107/-0) — n/a: This merged reference implementation carries fix(kanban): machine-global singleton lock for the embedded dispatcher (#41448) #44919's machine-global lock, cross-platform acquisition, and explicit release into main, resolving the concurrent-dispatcher cause described by [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448.
• fix(kanban): promote auto-decompose failures from DEBUG to rate-limited WARNING #49861 best fix — (+90/-3) — n/a: Promotes non-successful auto-decompose outcomes from DEBUG-only logging to rate-limited WARNING and now includes controlled-time tests for the initial warning, suppression, and 300-second boundary requested by its keep_open verdict.
• fix(kanban): run auto-decompose tick under the profile secret scope #57837 related — (+170/-50) — n/a: Installs and resets a profile secret scope around the threaded auto-decompose tick, with focused scope-lifetime coverage and a field report supporting the keep_open salvage path for multiplexed credential resolution.
• fix(gateway): cast max_spawn config to int to prevent TypeError (#59499) #59588 related — (+98/-3) — n/a: Casts raw max_spawn values but accepts non-positive values and tests synthetic parsing rather than the production path; despite its keep_open review, fix(kanban): close parent log handle after spawn and validate max_spawn #65644 shows the same coercion with positive validation and a real watcher-path test.
• fix(kanban): log dispatcher tick summary on crash/timeout/reclaim/auto-block, not only on spawn #62893 related — (+80/-9) — n/a: Expands dispatcher summaries beyond spawn-only ticks to crashes, timeouts, reclaims, auto-blocks, and stale work, with the visible stale-only regression addressing its keep_open review.
• fix(kanban): report truthful gave-up causes #64605 related — (+48/-7) — n/a: Replaces the unsupported repeated-spawn-failure explanation for every gave_up event with the supplied sanitized error or a cause-neutral fallback, preserving the focused notifier salvage verified by its keep_open verdict.
• fix(kanban): prevent archived board watcher resurrection #65321 related — (+392/-32) — n/a: Makes stale named-board watcher and auto-decomposer opens existing-only so archive races cannot recreate a removed board or fall back to default, and current head 6bb0994 includes the propagation and race coverage requested by the keep_open verdict.
• fix(kanban): avoid false dispatcher stuck warnings #65581 best fix — (+485/-35) — n/a: Adds board-local gateway health accounting and daemon deferral handling for fix: _kanban_dispatcher_watcher fires false "stuck" alert when hook fills concurrency cap #46800, addressing its keep_open review's cross-board objection, but the diff also carries unrelated gateway-status changes and its explicit global-deferral flag covers max_in_progress rather than the externally occupied max_spawn repro.
• fix(kanban): close parent log handle after spawn and validate max_spawn #65644 related — (+80/-8) — n/a: Positively validates max_spawn through the real watcher path and closes the parent log handle after successful Popen, but its keep_open review still requires finally-based cleanup and a regression for a non-FileNotFoundError Popen failure.
• fix(gateway): respect explicit platform disable for HA and wrap kanban auto-decompose in profile scope #76864 [closed] related — (+62/-57) — n/a: Its Kanban half duplicates fix(kanban): run auto-decompose tick under the profile secret scope #57837's auto-decompose secret-scope repair and its Home Assistant half is unrelated; despite the earlier keep_open verdict, it remains relevant as a closed split source because the contributor identified the overlap and the author closed it in favor of fix(kanban): run auto-decompose tick under the profile secret scope #57837 and fix(gateway): respect explicit platform disable for Home Assistant in env overrides #76871.

Duplicates

#41679 and #44919 overlap on #41448, with #49068 as the merged machine-global consolidation; #46892 and #65581 overlap on #46800 but contain complementary health-accounting work; #59588 overlaps #65644 on max_spawn coercion; the Kanban half of #76864 duplicates #57837.

Suggested consolidation

Author action: rebase #65581 onto main or split out a focused #46800 change that preserves its board-local gateway and daemon health accounting, incorporates #46892's externally occupied max_spawn detection per physical board, and removes the unrelated gateway-status changes; this respects both keep_open reviews and the recorded best-fix status of #46892 and #65581. Keep #49861, #57837, #62893, #64605, and #65321 open on their documented salvage paths, require #65644's finally-based exceptional-Popen cleanup before closing #59588 as duplicate of #65644, and leave #41679, #44919, and #76864 closed in favor of #49068, #57837, and #76871 respectively.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I46800(["issue #46800 (open)"])
    subgraph Dup46892 ["PRs duplicating each other"]
        P46892["PR #46892 (open)"]
        P65581["PR #65581 (open)"]
    end
    P65581 -->|best fix| I46800
    class I46800 open
    class P46892 open
    class P65581 open
    class P46892 best
    class P65581 best
    class P65581 target
    click I46800 "https://github.com/NousResearch/hermes-agent/issues/46800"
    click P46892 "https://github.com/NousResearch/hermes-agent/pull/46892"
    click P65581 "https://github.com/NousResearch/hermes-agent/pull/65581"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 13 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 142 kB of PR diffs, 45 kB of issue/PR text, 29 kB of discussion (31 comments), 11 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.