Repository navigation
fix(kanban): has_spawnable_ready excludes respawn-guarded tasks - #44899
Zazzles2908 wants to merge 1 commit into
Conversation
The dispatcher 'stuck' detection uses has_spawnable_ready() to
decide whether '0 spawned' is a real stuck condition (genuinely
spawnable work waiting) or a correctly-idle condition (only
control-plane lanes waiting). But has_spawnable_ready() previously
returned True for tasks that the dispatch path would correctly
skip via check_respawn_guard — so the gateway fires a false
'kanban dispatcher stuck' warning every 6 ticks (60s) when:
- the ready queue has tasks (non-empty)
- those tasks are all respawn-guarded (rate_limit_cooldown,
blocker_auth, recent_success, active_pr)
- dispatch_once correctly defers them
- but the health check thinks they're spawnable work
Root cause: has_spawnable_ready() and dispatch_once() use
DIFFERENT criteria for 'spawnable'. has_spawnable_ready() checks
ready+assigned+unclaimed+valid-profile. dispatch_once() checks
the same AND NOT respawn-guarded. The mismatch causes the
false warning.
Fix: call check_respawn_guard for each candidate task and return
True only if at least one is NOT guarded. Same posture as the
patch 4 fix to check_respawn_guard itself (result IS NOT NULL
early-return). Fail-open: if check_respawn_guard raises, assume
spawnable — same behavior as the legacy code path.
has_spawnable_review(): mirror the function signature for
compatibility but deliberately do NOT filter on guard (the
dispatch path doesn't call check_respawn_guard on review tasks
— that's a different layer of the system).
Repro before fix:
Single task in 'ready' with completed task_runs row (recent_success)
has_spawnable_ready() returns True
dispatch_once() defers the task
Gateway logs: 'kanban dispatcher: tick failed on board default'
Repro after fix:
Same task, has_spawnable_ready() returns False
No 'stuck' warning. Dispatcher correctly idle.
Production evidence: WSL2 hook
~/.hermes/hooks/wsl2-kanban-monkeypatches/handler.py:241-367
implements this exact filtering. Live dispatch health: 5 active
workers, 0 failed ticks since 2026-06-12.
Closes: bead prod-audit-wr2yw
Tested: 6-scenario test (unguarded, recent_success guard,
blocker_auth guard, mixed, empty, mixed with blocker_auth) —
all 6 pass
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Approved
Good telemetry fix. has_spawnable_ready now also checks check_respawn_guard so the gateway health telemetry does not false-positive a "kanban dispatcher stuck" warning every 6 ticks for respawn-guarded tasks.
Looks Good
- Uses
globals().get("check_respawn_guard")for safe fallback in partial-install environments - The check is applied to each individual task row rather than batch-filtered, allowing the loop to short-circuit
- Clear documentation distinguishing ready-task guard checking from review-task (which does not use
check_respawn_guardin the dispatch path) - Documented note explaining why review tasks intentionally skip the guard filter
Reviewed by Hermes Agent
|
Thanks for tracing the dispatcher-health mismatch. The current-head premise is confirmed: Problems
Suggested changes
Automated hermes-sweeper review. |
|
Additional current-lane and diagnostics work is tracked in #126263 . Current upstream dispatch checks lane-specific guards for review too, so the older review-lane assumption needs adaptation. The follow-up will preserve guard policy and mixed-queue stall detection and credit this work. |
Summary
The dispatcher "stuck" detection uses
has_spawnable_ready()to decide whether0 spawnedis a real stuck condition (genuinely spawnable work waiting) or a correctly-idle condition. Buthas_spawnable_ready()previously returned True for tasks that the dispatch path would correctly skip viacheck_respawn_guard— so the gateway fires a false "kanban dispatcher stuck" warning every 6 ticks (60s) when:rate_limit_cooldown,blocker_auth,recent_success,active_pr)dispatch_oncecorrectly defers themThis is a 42-line addition to
hermes_cli/kanban_db.py:5968-6022that callscheck_respawn_guardfor each candidate and returns True only if at least one task is not guarded.Root cause
has_spawnable_ready()anddispatch_once()use different criteria for "spawnable":has_spawnable_ready()(telemetry)ready AND assignee IS NOT NULL AND claim_lock IS NULL AND profile_exists(assignee)dispatch_once()(actual dispatch)The mismatch causes the false warning.
Repro
Before fix:
After fix:
Tested 6 scenarios:
True✓False✓False✓True(at least one spawnable) ✓False✓True✓Fix
def has_spawnable_ready(conn: sqlite3.Connection) -> bool: rows = conn.execute( - "SELECT DISTINCT assignee FROM tasks " + "SELECT id, assignee FROM tasks " "WHERE status = 'ready' AND assignee IS NOT NULL " " AND claim_lock IS NULL" ).fetchall() if not rows: return False try: from hermes_cli.profiles import profile_exists except Exception: return True + check_respawn_guard = globals().get("check_respawn_guard") for row in rows: - if profile_exists(row["assignee"]): - return True + if not profile_exists(row["assignee"]): + continue + if check_respawn_guard is not None: + try: + if check_respawn_guard(conn, row["id"]) is not None: + continue + except Exception: + pass + return True return Falsehas_spawnable_review()also gets theidcolumn added to the SELECT (consistency) and usescontinueinstead of early-return, but does NOT filter on guard —dispatch_oncedoesn't callcheck_respawn_guardon review tasks, so the review path stays consistent with the dispatch path.Why fail-open?
If
check_respawn_guardraises or is unavailable, we return True. This is the same posture as the existingprofile_existsfallback (line 5999) and the existingdispatch_oncepath. We never wanthas_spawnable_readyto silently return False when a guard check transiently fails — that would suppress legitimate "stuck" warnings and hide real dispatch bugs.Production evidence
The WSL2 consolidation hook at
~/.hermes/hooks/wsl2-kanban-monkeypatches/handler.py:241-367implements this exact filtering logic. That hook has been live in the user'shermes-agentdeployment since 2026-06-12 with 5 active workers and 0 false "stuck" warnings. Filing this PR retires the hook dependency for this specific fix.Risk
dispatch_onceinvariant (which already skips guarded tasks viacheck_respawn_guard).Test plan
tests/hermes_cli/test_kanban_db.pytests forhas_spawnable_readyandhas_spawnable_reviewBead
prod-audit-wr2yw(parent epic for upstream PRs in this batch)Note: this PR is the third of a 3-PR series that retires the WSL2 consolidation hook dependency:
recompute_readycancelled-parent fix (1-line)check_respawn_guardresult-skip (9-line)has_spawnable_readyexclude respawn-guarded (42-line)Each PR is independently mergable. They fix 3 separate bugs, not a chain.