Repository navigation
Heartbeats stay quiet during hermes pause instead of resending the paused notice every poll, and /loop wakeups wait for resume (salvage #126905) - #130601
Merged
kshitijk4poor merged 6 commits intoOct 1, 2026
Conversation
… paused notice While ESTOP was engaged the heartbeat poller kept injecting the due tick every 5s; the estop gate answered each one with the user-facing paused notice and the refunded tick stayed due, so the notice was re-sent every poll (~10k Telegram messages in 5h, twice in prod). - _heartbeat_poll_watch: skip before claiming the tick while paused (logged once per engagement via check_paused); it fires normally after hermes resume. - _hm_estop_gate: a synthetic heartbeat event is dropped silently; real user messages still get the paused notice. (cherry picked from commit 74c0412)
…urns The poller now checks `hermes pause` before claiming a heartbeat tick, so the inbound gate only sees heartbeats that slipped past that check. Say so, and point at settle_heartbeat_attempt as what keeps the tick due for after resume.
/loop wakeups are injected with internal=True, and _hm_estop_gate returns early for internal events, so `hermes pause` did not stop loop ticks from starting agent turns. This is the same problem the heartbeat poller fix handles. Check the emergency stop before LoopManager.fire_tick. is_due() is read-only and fire_tick() is what moves next_due_at forward, so the tick stays due and fires on the first scan after `hermes resume`. The watcher sleeps for its interval (15s) after every scan, so being paused does not make it busy-spin. check_paused logs once per engagement.
b533839532 gates _loop_wakeup_fire_one on the emergency stop because loop wakeups are internal events that skip the inbound estop gate, but nothing exercised that leg. Fold a pause leg into the existing direct-harness loop wakeup test: while engaged, fire_tick is never called and no turn reaches the adapter; after disengage the same due tick fires exactly once. The estop sentinel is isolated under the test's own HERMES_HOME. Reverting run_goals.py to b533839532^ turns this red (fire_tick claimed + "tick" injected while paused).
This was referenced Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While
hermes pauseis engaged, a due heartbeat no longer sends the paused notice to the chat on every 5s poll (it fires once afterhermes resume), and scheduled /loop wakeups no longer start agent turns until resume.Salvages #126905; supersedes the duplicate #127525. No linked issue.
What happened
The heartbeat poller had no pause check, so it injected the due tick on every poll;
_hm_estop_gateanswered each one with the paused notice and the tick stayed due, so the notice repeated indefinitely (reported as ~10k Telegram messages in 6h and a flood ban). /loop wakeups are injected as internal events, and_hm_estop_gatereturns early for internal events, sohermes pausedid not stop them at all.Change
gateway/run_goals.py_heartbeat_poll_watch: skip while paused, before the tick is claimed, so it fires normally after resume.gateway/run_goals.py_loop_wakeup_fire_one:check_paused("loop", logger)afteris_dueand beforefire_tick.is_dueonly reads state, so the tick stays due for the first scan after resume; the watcher's 15s sleep per scan means no busy loop.gateway/run_inbound.py_hm_estop_gate: a heartbeat event that slips past the poller check is dropped silently (""). Real user messages still get the paused notice. Docstring explains why.tests/gateway/test_heartbeat_poller.py(contributor); a pause/resume leg folded into the existingtest_loop_wakeup_from_secondary_route_fires_through_its_own_botintests/gateway/test_multiplex_notice_egress_profile_adapter.py(no new test function).contributors/emails/: map entry for the contributor.Credit
_hm_estop_gatedocstring, and the contributor email map.Verification
test_heartbeat_poller.py,test_loop_command.py,test_multiplex_notice_egress_profile_adapter.py,cron/test_estop.py: 37 passed.test_heartbeat_acceptance.py,test_heartbeat_execution_ownership.py,test_heartbeat_watch_lifecycle.py,test_telegram_reply_mode.py: 33 passed.tui_gateway/test_notification_turn_release.py: 6 passed.assert ([1], ['tick']) == ([], [])./loop 5m /statusunderhermes pauseran on main (handled['/status'], 1 tick fired) and is held on head ([], 0 ticks), then fires once after resume.git diff --checkclean. Windows-footguns check: no new findings (one existingwrite_textwithoutencoding=at line 67 of the multiplex test file is on main and untouched here).Maintainer notes
hermes pausenot stopping /loop wakeups: they are internal events, which bypassed_hm_estop_gate. This was found in review and is not part of fix(gateway): heartbeats must not resend the paused notice every poll #126905.