Skip to content

Heartbeats stay quiet during hermes pause instead of resending the paused notice every poll, and /loop wakeups wait for resume (salvage #126905) - #130601

Merged
kshitijk4poor merged 6 commits into
NousResearch:mainfrom
kshitijk4poor:fix/p1-1001-heartbeat-paused-notice
Oct 1, 2026
Merged

kshitijk4poor merged 6 commits into
NousResearch:mainfrom
kshitijk4poor:fix/p1-1001-heartbeat-paused-notice

Conversation

@kshitijk4poor

Copy link
Copy Markdown

While hermes pause is engaged, a due heartbeat no longer sends the paused notice to the chat on every 5s poll (it fires once after hermes resume), and scheduled /loop wakeups no longer start agent turns until resume.

Salvages #126905; supersedes the duplicate #127525. No linked issue.

What happened

The heartbeat poller had no pause check, so it injected the due tick on every poll; _hm_estop_gate answered each one with the paused notice and the tick stayed due, so the notice repeated indefinitely (reported as ~10k Telegram messages in 6h and a flood ban). /loop wakeups are injected as internal events, and _hm_estop_gate returns early for internal events, so hermes pause did not stop them at all.

Change

  • gateway/run_goals.py _heartbeat_poll_watch: skip while paused, before the tick is claimed, so it fires normally after resume.
  • gateway/run_goals.py _loop_wakeup_fire_one: check_paused("loop", logger) after is_due and before fire_tick. is_due only reads state, so the tick stays due for the first scan after resume; the watcher's 15s sleep per scan means no busy loop.
  • gateway/run_inbound.py _hm_estop_gate: a heartbeat event that slips past the poller check is dropped silently (""). Real user messages still get the paused notice. Docstring explains why.
  • Tests: 2 new in tests/gateway/test_heartbeat_poller.py (contributor); a pause/resume leg folded into the existing test_loop_wakeup_from_secondary_route_fires_through_its_own_bot in tests/gateway/test_multiplex_notice_egress_profile_adapter.py (no new test function).
  • contributors/emails/: map entry for the contributor.

Credit

Verification

  • On the rebased head: test_heartbeat_poller.py, test_loop_command.py, test_multiplex_notice_egress_profile_adapter.py, cron/test_estop.py: 37 passed. test_heartbeat_acceptance.py, test_heartbeat_execution_ownership.py, test_heartbeat_watch_lifecycle.py, test_telegram_reply_mode.py: 33 passed. tui_gateway/test_notification_turn_release.py: 6 passed.
  • Red on base: both new heartbeat tests fail on main (a heartbeat MessageEvent is received and the paused notice is sent). With the /loop check reverted, the folded test fails with assert ([1], ['tick']) == ([], []).
  • Probe, base vs head: a due /loop 5m /status under hermes pause ran on main (handled ['/status'], 1 tick fired) and is held on head ([], 0 ticks), then fires once after resume.
  • Mutation: reverting the poller check, the gate's heartbeat drop, or the /loop check each turns a kept test red.
  • ruff F401/F811/F821: no new findings vs main. git diff --check clean. Windows-footguns check: no new findings (one existing write_text without encoding= at line 67 of the multiplex test file is on main and untouched here).
  • Reviewed over 2 rounds (regression, reuse, quality, efficiency, tests/probe/mutation).

Maintainer notes

kshitijk4poor and others added 5 commits October 1, 2026 20:04
… paused notice

While ESTOP was engaged the heartbeat poller kept injecting the due tick every
5s; the estop gate answered each one with the user-facing paused notice and the
refunded tick stayed due, so the notice was re-sent every poll (~10k Telegram
messages in 5h, twice in prod).

- _heartbeat_poll_watch: skip before claiming the tick while paused (logged once
  per engagement via check_paused); it fires normally after hermes resume.
- _hm_estop_gate: a synthetic heartbeat event is dropped silently; real user
  messages still get the paused notice.

(cherry picked from commit 74c0412)
…urns

The poller now checks `hermes pause` before claiming a heartbeat tick, so the
inbound gate only sees heartbeats that slipped past that check. Say so, and
point at settle_heartbeat_attempt as what keeps the tick due for after resume.
/loop wakeups are injected with internal=True, and _hm_estop_gate returns early
for internal events, so `hermes pause` did not stop loop ticks from starting
agent turns. This is the same problem the heartbeat poller fix handles.

Check the emergency stop before LoopManager.fire_tick. is_due() is read-only and
fire_tick() is what moves next_due_at forward, so the tick stays due and fires on
the first scan after `hermes resume`. The watcher sleeps for its interval (15s)
after every scan, so being paused does not make it busy-spin. check_paused logs
once per engagement.
b533839532 gates _loop_wakeup_fire_one on the emergency stop because loop
wakeups are internal events that skip the inbound estop gate, but nothing
exercised that leg. Fold a pause leg into the existing direct-harness loop
wakeup test: while engaged, fire_tick is never called and no turn reaches the
adapter; after disengage the same due tick fires exactly once. The estop
sentinel is isolated under the test's own HERMES_HOME.

Reverting run_goals.py to b533839532^ turns this red
(fire_tick claimed + "tick" injected while paused).
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants