Repository navigation
fix(gateway): hermes pause holds msgraph, raft and plugin-injected turns - #131662
Open
EloquentBrush0x wants to merge 1 commit into
Open
EloquentBrush0x wants to merge 1 commit into
EloquentBrush0x wants to merge 1 commit into
Conversation
The inbound estop gate lets every internal=True event through, by design: those are in-flight work completions (background process and delegation results), and a pause never kills in-flight work. NousResearch#130601 found that heartbeat and loop wakeups are internal too but are new work, and gated them at their producers. Three more internal producers start new work from outside: Microsoft Graph notifications, Raft bridge wakes and plugin message injections. All three still started agent turns during a pause. Check the pause at each producer, as NousResearch#130601 did. The two webhooks answer 503 + Retry-After like the managed-cron webhook, so the sender redelivers after resume (msgraph does not record the receipt, so the redelivery is not deduped); plugin injection returns False ("not injected").
13 of 20 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #131661
Summary
#130601 (P1, merged) found that
hermes pausestill let heartbeat and loop wakeups start turns: they areinternal=True, and_hm_estop_gatelets internal events through. That pass-through is deliberate, because internal events are normally in-flight work completions (test_gateway_internal_events_bypass_estop; "in-flight work is never killed"). So #130601 checked the pause at those two producers. Three moreinternal=Trueproducers start new work and were not covered: Microsoft Graph notifications (msgraph_webhook), Raft bridge wakes (raft._handle_wake) and plugin message injections (_schedule_plugin_message_injection).Impact
While the operator has paused the gateway, each Graph change notification, Raft wake or plugin injection starts an agent turn with tool access. That breaks the documented contract: no "gateway turns start until resumed". A busy Graph subscription keeps starting turns for the whole pause.
Fix
agent.estop.check_pausedat each producer, mirroring #130601:Retry-After: 60, the same as the managed-cron webhook (api_server"refuse the fire and ask NAS to retry later"). The receipt is not recorded, so Graph's redelivery after resume is accepted rather than deduped.Retry-Afterwitherror: "paused", alongside the adapter's existing retryablenot_ready503.False("not injected"), the same as a stopped gateway.When not paused, behavior is unchanged.
Scope note
Background-process completions/watch notifications, async-delegation wakes, Yuanbao recall interrupts and startup resumes stay ungated on purpose. They are in-flight work, which the pause design and its test keep running. #121054 (allowlisted identities during a pause) touches the same gate from a different angle; it does not overlap.
Test plan
Regression tests, extending each producer's existing test file:
test_msgraph_webhook.py)test_raft_adapter.py)handle_messagecalledpaused+ Retry-After; unauthenticated → 401test_plugin_message_injection.py)True, scheduledFalse, not scheduledMutation-verified per producer: reverting each of
msgraph_webhook.py,raft/adapter.py,run_inbound.pyfails only its own new test; restored → 36/36Adjacent suites (29 files touching estop, msgraph, raft, plugin injection, run_inbound): 1414 passed, 1 skipped
ruff checkclean;check-windows-footguns.pyadds nothing (one existingread_text(encoding="utf-8")intest_raft_adapter.py, unchanged from main)Competitor check
Searched "pause msgraph", "estop internal event", "hermes pause webhook", "pause plugin injection", "raft wake pause",
check_paused: no open PR covers these producers. Adjacent: #127476 (kanban dispatch), #121054 (allowlisted identity during a pause).