fix(kanban): stop the false "check profile health" stall warning on a benign decline

[Kyzcreig]

fix(kanban): stop the false "check profile health" stall warning when the dispatcher is throttled

Symptom

The gateway kanban dispatcher logs, every 5 minutes:

kanban dispatcher stuck: ready queue non-empty for 6 consecutive ticks but 0
workers spawned. Check profile health (venv, PATH, credentials) and
`hermes kanban list --status ready`.

…for the entire duration of a provider rate-limit window, or of any large
fan-out where every eligible profile is at its concurrency cap. Nothing is
broken: the dispatcher resumes spawning normally the moment quota frees, and
there are zero spawn_failed events in the whole window. Users are told to
go audit their venv, PATH and credentials for a problem that does not exist.

Root cause

gateway/kanban_watchers.py, in _kanban_dispatcher_watcher:

ready_pending = await asyncio.to_thread(_ready_nonempty)
if ready_pending and not any_spawned:
    bad_ticks += 1

ready_pending and not any_spawned is not only the "stuck" condition — it is
also the healthy steady state whenever the dispatcher declines to spawn
for a benign, self-clearing reason:

decline	DispatchResult bucket
every eligible profile is at kanban.max_in_progress_per_profile	skipped_per_profile_capped
assignee bounced off a provider rate-limit / quota wall (released back to ready without counting a failure)	rate_limited
another dispatcher process holds this board's dispatch lock this tick	skipped_locked
a respawn guard is cooling a task down	respawn_guarded

DispatchResult already buckets each of those separately from real faults —
precisely so "busy/throttled" can be told apart from "genuinely stuck". Their
field docstrings say so verbatim (e.g. skipped_per_profile_capped: "NOT an
operator-actionable failure … Separate bucket so telemetry / dashboards can
show 'this profile is busy' vs 'task is genuinely stuck'"). The detector
simply never consulted them.

Second defect found while fixing the first: cross-board fault masking

The detector aggregates across all boards in one tick. The only "genuine
fault" signal available was DispatchResult.auto_blocked, which is populated
only when a board's spawn-failure circuit breaker trips — after
failure_limit consecutive failures.

So an early, pre-breaker spawn failure on board A (broken venv / PATH /
credentials; failures 1 .. failure_limit-1) was invisible as a fault. A
benign decline on board B in the same tick then reset the streak, masking A's
real stall until A's breaker finally tripped several ticks later. On a
multi-board gateway that alignment is routine, not theoretical — and it delays
the one warning a user actually needs.

Fix

1. hermes_cli/kanban_db.py — add DispatchResult.spawn_failed, populated
   at every _record_spawn_failure call site (all four: workspace-resolution
   and worker-launch failures, in both the ready and the review dispatch
   columns), whether or not that failure tripped the breaker. auto_blocked
   stays the breaker-trip subset; spawn_failed is the superset including
   early failures. Purely additive: no existing field or call signature changes.

2. gateway/kanban_watchers.py — extract the bad-tick decision into a pure,
   unit-testable _stall_streak_is_bad(ready_pending, any_spawned, results)
   that counts a zero-spawn tick only when it is not explained by a benign
   decline, or when a genuine fault (spawn_failed / auto_blocked) is
   present anywhere in the tick. The true positive the warning exists to catch
   still fires — and now fires on failure Terminal tool #1 instead of waiting for the
   breaker.

3. The warning text now states that no benign decline explains the streak, so a
   real fire is unambiguous rather than one more cry-wolf line.

Tests

tests/gateway/test_kanban_watchers_mixin.py — 11 behavior-contract tests
covering both directions (each of the four benign declines stays quiet; a bare
unexplained zero-spawn, an auto_blocked fault alongside a benign sibling
board, and an early spawn_failed alongside a benign sibling board all count),
plus an invariant test asserting every bucket name the detector reads actually
exists on DispatchResult — the detector reads them via
getattr(res, name, None), so a rename would otherwise degrade it silently to
"never fires".

The fake-result helper constructs a real DispatchResult, not a stand-in,
so the tests are pinned to the production dataclass.

tests/hermes_cli/test_kanban_db.py — 2 E2E tests through the real
dispatch_once with a failing spawn_fn: failure #1 of 5 lands in
spawn_failed and not in auto_blocked (the previously-invisible window),
and with failure_limit=1 the task lands in both, pinning
auto_blocked ⊆ spawn_failed.

RED proof (production logic reverted to the pre-fix rule, tests unchanged):
6 of the 13 new tests fail — the four benign-decline cases and both E2E
spawn_failed cases. See TEST-EVIDENCE.md.

GREEN: tests/gateway/test_kanban_watchers_mixin.py +
tests/hermes_cli/test_kanban_db.py = 247 passed (234 pre-existing on
clean main, +13 new). Zero pre-existing tests modified.

[alt-glitch]

This was generated by AI during triage.

Related: #46892, #65581, and #69010 address false Kanban stall telemetry. This PR adds a distinct fault-versus-benign-decline policy and spawn_failed reporting; please choose the intended aggregate semantics rather than treating the patches as duplicates.

[teknium1]

Thanks for isolating the current gateway false-positive and for adding real DispatchResult-based coverage. The premise is confirmed on current main: gateway/kanban_watchers.py:1456-1461 still counts every aggregate ready/no-spawn tick as stalled.

Problems

• The proposed aggregate predicate lets a benign decline on board A suppress an unexplained no-spawn stall on board B. The related fix(kanban): avoid false dispatcher stuck warnings #65581 discussion identifies the required board-local invariant; add that regression before using aggregate results.
• respawn_guarded includes blocker_auth: hermes_cli/kanban_db.py:7886-7893 says authentication needs human action, so treating every guard reason as benign removes the health signal for that case.
• hermes_cli/kanban_db.py:8222-8227 returns an empty result at the global max_in_progress cap, and hermes_cli/kanban.py:2611-2617 retains the same standalone health predicate. Both remain false-warning paths.

Suggested changes

• Make readiness/deferral matching board-local, distinguish actionable guard reasons, and surface global-cap deferral in DispatchResult for both gateway and daemon telemetry.

Automated hermes-sweeper review.

[Kyzcreig]

@teknium1 — your second point is correct and I am not going to defend it. _BENIGN_DECLINE_FIELDS lists respawn_guarded wholesale, and blocker_auth appears zero times in this diff. Since blocker_auth is documented at hermes_cli/kanban_db.py:8075 as a quota/auth fault that needs human action, this patch would classify an auth-broken profile as a benign decline and remove the exact health signal the warning exists to raise. That is a regression, not a refinement.

The fix: match on the respawn_guarded reason pairs rather than testing the list's truthiness, treat blocker_auth and any other actionable guard reason as a fault, and cover it with a test. I will also add the board-local regression #65581 asks for and surface the global max_in_progress deferral in DispatchResult. If you would rather consolidate with #46892, #65581 or #69010, I am happy to fold this in.

[Kyzcreig]

Rebased onto current main. Scope tightened per review: _BENIGN_DECLINE_FIELDS is now only capacity/lock-class declines (skipped_per_profile_capped, rate_limited, skipped_locked); blocker_auth (incl. Discord auth denial) stays a hard fault and is never treated as benign, and respawn_guarded is inspected per-reason rather than whitelisted. There is a parametrized case pinning ("blocker_auth", True) as a fault, and re-adding blocker_auth to the benign tuple turns the suite red.