Skip to content

net: scope worker gateway topology watches - #918

Draft
Patrick W. Healy (phealy) wants to merge 4 commits into
phealy/net-api-assignmentsfrom
phealy/net-api-worker-watches
Draft

Patrick W. Healy (phealy) wants to merge 4 commits into
phealy/net-api-assignmentsfrom
phealy/net-api-worker-watches

Conversation

@phealy

@phealy Patrick W. Healy (phealy) commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stop worker agents from watching cluster-wide gateway topology they don't use.

  • Gateway informers. The agent starts the GatewayPool, GatewayPoolNode and SiteGatewayPoolAssignment informers only while its role or its Site's assignments need them. It stops them when they no longer do.
  • MTU.
    • MTU is resolved from actual forwarding paths rather than from undirected topology components.
    • SiteGatewayPoolAssignment MTUs no longer apply to ordinary worker mesh links. They still apply on the gateway path to an assigned Site.
  • Watch timeout. The randomized Node informer watch timeout is doubled, so watches are re-established half as often. LIST requests are unchanged.
  • Startup race. The gateway event callback is installed before the reconcile worker starts, and events go through a synchronized lookup. Events that arrive during startup are no longer lost.

Testing

  • Unit tests for:
    • informer start and stop across role and assignment changes
    • registering the callback after events start arriving (add, update and delete for all three resources)
    • MTU along forwarding paths
    • the watch timeout wrapper
  • Tested on an isolated AKS cluster with two gateways and two remote workers:
    • ICMP and HTTP succeeded in both directions, 10 of 10.
    • Removing the remote Site from the assignment and adding it back stopped and restarted exactly the three gateway informers per remote. No agents restarted.
    • Demoting a gateway and promoting it again replaced only that gateway's agent. The other gateway carried traffic with no loss.

Stack

This is one of five stacked PRs that reduce API server load from unbounded-net-controller and unbounded-net-node. Kubernetes remains the source of truth. Review them bottom to top:

  1. net: skip no-op managed kube-proxy reconciliation #914 Controller no-op reconciliation
  2. net: reconcile SiteNodeSlices per Site #915 Targeted SiteNodeSlice reconciliation
  3. net: publish node assignments and replace stale agents #916 Controller-published node assignments
  4. net: adopt and acknowledge node assignments in the agent #917 Node agent assignment adoption
  5. net: scope worker gateway topology watches #918 Scoped worker gateway watches

Motivation. A large cluster running v0.8.0 had the controller continuously writing objects that hadn't changed. Every node agent also watched the whole cluster's gateway topology. On a canary cluster, the full stack cut controller API calls by 54.4% and node agent API calls by 73.8%.

Use forwarding paths rather than undirected topology components for MTU resolution, and dynamically synchronize gateway-specific informers only while the pinned role or direct Site assignments require them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
Wrap dynamic watch requests so reflector-selected randomized timeouts are doubled for both watch-list initialization and normal watches without changing LIST requests or other watch options.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
Keep SiteGatewayPoolAssignment MTUs off ordinary worker mesh links while retaining them for gateway-to-assigned-Site forwarding paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
Install the update callback once before launching the reconcile worker and dispatch lifecycle events through a synchronized callback lookup. Cover late callback registration with immediate add, update, and delete events for all three gateway resources.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved startup concurrency, blocked reconciliation, MTU calculation, and status consistency issues prevent approval.

6 open findings
What changed in this PR

Reduces node-agent API server load by running gateway topology watches only when needed, completing the five-PR optimization stack.

Changes:

  • Adds gateway informer lifecycle management, synchronized callbacks, and readiness reporting.
  • Calculates MTU limits from forwarding paths rather than connected topology.
  • Doubles informer watch timeouts and adds tests and documentation.
File Description
docs/​net/​configuration.md Documents forwarding-path MTU limits.
docs/​net/​architecture.md Explains conditional gateway watches.
docs/​content/​reference/​networking/​configuration.md Updates public MTU guidance.
cmd/​unbounded-net-node/​status_server.go Adds lifecycle-aware readiness and status.
cmd/​unbounded-net-node/​status_server_http_test.go Tests gateway readiness responses.
cmd/​unbounded-net-node/​site_watch_reconcile.go Integrates informer lifecycle and transit MTUs.
cmd/​unbounded-net-node/​reconciliation_helpers.go Adds forwarding-path MTU calculations.
cmd/​unbounded-net-node/​reconciliation_helpers_test.go Tests revised MTU behavior.
cmd/​unbounded-net-node/​node_types.go Carries transit MTU limits.
cmd/​unbounded-net-node/​mtu.go Applies role-aware assignment limits.
cmd/​unbounded-net-node/​main.go Separates discovery and gateway informers.
cmd/​unbounded-net-node/​informer_factory.go Doubles watch timeouts.
cmd/​unbounded-net-node/​informer_factory_test.go Tests watch and LIST options.
cmd/​unbounded-net-node/​gateway_informer_lifecycle.go Manages conditional gateway caches.
cmd/​unbounded-net-node/​gateway_informer_lifecycle_test.go Tests lifecycle transitions and callbacks.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +308 to +309
for _, hop := range path {
if hop.Type == "GatewayPool" {
Comment on lines +319 to +323
for _, poolName := range peering.Spec.GatewayPools {
if pathPools[strings.TrimSpace(poolName)] {
matchedPools++
}
}
for range 3 {
gatewayRequired := gatewayCachesRequired(adoptedAssignment, mySiteName, assignmentInformer)

gatewaySnapshot, retireGatewayInformers, err = gatewayInformers.prepare(ctx, gatewayRequired)

transitTunnelMTU := learnedGatewayTransitTunnelMTU(pool.Name, learnedRoutes, poolPeerings)
if isGatewayNode {
transitTunnelMTU = gatewayPoolPeeringTunnelMTU(localGatewayPools, pool.Name, poolPeerings)
nodes: nodes,
poolPeerings: poolPeerings,
}
if !cache.WaitForCacheSync(runCtx.Done(), pools.HasSynced, nodes.HasSynced, poolPeerings.HasSynced) {
Comment on lines +3095 to +3097
gatewayPoolInformer := s.gatewayPoolInformer
if s.gatewayInformers != nil {
gatewayPoolInformer = s.gatewayInformers.snapshot().pools

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants