Skip to content

fix(gateway): hermes pause holds msgraph, raft and plugin-injected turns - #131662

Open
EloquentBrush0x wants to merge 1 commit into
NousResearch:mainfrom
EloquentBrush0x:fix/pause-internal-turns
Open

EloquentBrush0x wants to merge 1 commit into
NousResearch:mainfrom
EloquentBrush0x:fix/pause-internal-turns

Conversation

@EloquentBrush0x

Copy link
Copy Markdown

Fixes #131661

Summary

#130601 (P1, merged) found that hermes pause still let heartbeat and loop wakeups start turns: they are internal=True, and _hm_estop_gate lets internal events through. That pass-through is deliberate, because internal events are normally in-flight work completions (test_gateway_internal_events_bypass_estop; "in-flight work is never killed"). So #130601 checked the pause at those two producers. Three more internal=True producers start new work and were not covered: Microsoft Graph notifications (msgraph_webhook), Raft bridge wakes (raft._handle_wake) and plugin message injections (_schedule_plugin_message_injection).

Impact

While the operator has paused the gateway, each Graph change notification, Raft wake or plugin injection starts an agent turn with tool access. That breaks the documented contract: no "gateway turns start until resumed". A busy Graph subscription keeps starting turns for the whole pause.

Fix

agent.estop.check_paused at each producer, mirroring #130601:

  • msgraph: a valid, authenticated notification answers 503 + Retry-After: 60, the same as the managed-cron webhook (api_server "refuse the fire and ask NAS to retry later"). The receipt is not recorded, so Graph's redelivery after resume is accepted rather than deduped.
  • raft: after the bridge-token check (pause state stays hidden from unauthenticated callers), it answers 503 + Retry-After with error: "paused", alongside the adapter's existing retryable not_ready 503.
  • plugin injection: returns False ("not injected"), the same as a stopped gateway.

When not paused, behavior is unchanged.

Scope note

Background-process completions/watch notifications, async-delegation wakes, Yuanbao recall interrupts and startup resumes stay ungated on purpose. They are in-flight work, which the pause design and its test keep running. #121054 (allowlisted identities during a pause) touches the same gate from a different angle; it does not overlap.

Test plan

  • Regression tests, extending each producer's existing test file:

    while paused main fix
    Graph notification (test_msgraph_webhook.py) 202, turn scheduled 503 + Retry-After, nothing scheduled; after resume the redelivery gets 202 and is scheduled
    Raft wake (test_raft_adapter.py) 202, handle_message called 503 paused + Retry-After; unauthenticated → 401
    plugin injection (test_plugin_message_injection.py) True, scheduled False, not scheduled
  • Mutation-verified per producer: reverting each of msgraph_webhook.py, raft/adapter.py, run_inbound.py fails only its own new test; restored → 36/36

  • Adjacent suites (29 files touching estop, msgraph, raft, plugin injection, run_inbound): 1414 passed, 1 skipped

  • ruff check clean; check-windows-footguns.py adds nothing (one existing read_text(encoding="utf-8") in test_raft_adapter.py, unchanged from main)

Competitor check

Searched "pause msgraph", "estop internal event", "hermes pause webhook", "pause plugin injection", "raft wake pause", check_paused: no open PR covers these producers. Adjacent: #127476 (kanban dispatch), #121054 (allowlisted identity during a pause).

The inbound estop gate lets every internal=True event through, by design:
those are in-flight work completions (background process and delegation
results), and a pause never kills in-flight work. NousResearch#130601 found that
heartbeat and loop wakeups are internal too but are new work, and gated
them at their producers. Three more internal producers start new work from
outside: Microsoft Graph notifications, Raft bridge wakes and plugin
message injections. All three still started agent turns during a pause.

Check the pause at each producer, as NousResearch#130601 did. The two webhooks answer
503 + Retry-After like the managed-cron webhook, so the sender redelivers
after resume (msgraph does not record the receipt, so the redelivery is not
deduped); plugin injection returns False ("not injected").
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Oct 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: hermes pause still starts turns from msgraph notifications, raft wakes and plugin injections

2 participants