Skip to content

net: publish node assignments and replace stale agents - #916

Open
Patrick W. Healy (phealy) wants to merge 4 commits into
phealy/net-api-site-slicesfrom
phealy/net-api-assignment-controller
Open

Patrick W. Healy (phealy) wants to merge 4 commits into
phealy/net-api-site-slicesfrom
phealy/net-api-assignment-controller

Conversation

@phealy

@phealy Patrick W. Healy (phealy) commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Have the controller publish each Node's assignment and replace only agents that are running with an outdated one.

  • Pool label. The controller writes a preferred GatewayPool label containing the raw pool name. The patch is a JSON patch guarded by the Node's UID and resourceVersion.
  • Targeted replacement. The controller compares the Site and GatewayPool that each agent Pod has acknowledged against the Node's current assignment. It deletes only Pods whose acknowledgment is outdated, with a UID precondition. Pods that haven't acknowledged yet are skipped.
  • Per-Node errors. Assignments are reconciled Node by Node and failures are combined into one error. One invalid Node can't block the others.
  • Admission and RBAC:
    • Agents can update only their own Node, identified by the node name and UID bound to their token.
    • Agents can't change labels or pod CIDRs. The only annotations they can set are wg-pubkey, tunnel-mtu, discovered-public-ip and discovered-public-ip-expires-at.
    • Agents can write the assignment acknowledgment only on their own Pod.
    • A new policy rejects GatewayPool names longer than 63 characters, because the name is used as a label value.

The node agent side (adopting and acknowledging the assignment) is in #917. The controller is expected to be upgraded before the agent.

Testing

  • Unit tests for publishing assignments, targeted replacement, and per-Node error isolation. The isolation test lists an invalid Node first and a valid Node second.
  • Admission policy tests, including negative cases, and updated operator plan goldens.

Stack

This is one of five stacked PRs that reduce API server load from unbounded-net-controller and unbounded-net-node. Kubernetes remains the source of truth. Review them bottom to top:

  1. net: skip no-op managed kube-proxy reconciliation #914 Controller no-op reconciliation
  2. net: reconcile SiteNodeSlices per Site #915 Targeted SiteNodeSlice reconciliation
  3. net: publish node assignments and replace stale agents #916 Controller-published node assignments
  4. net: adopt and acknowledge node assignments in the agent #917 Node agent assignment adoption
  5. net: scope worker gateway topology watches #918 Scoped worker gateway watches

Motivation. A large cluster running v0.8.0 had the controller continuously writing objects that hadn't changed. Every node agent also watched the whole cluster's gateway topology. On a canary cluster, the full stack cut controller API calls by 54.4% and node agent API calls by 73.8%.

@phealy
Patrick W. Healy (phealy) added this pull request to stack #919 October 14, 2026 23:10
@phealy Patrick W. Healy (phealy) changed the title phealy/net api assignment controller net: publish node assignments and replace stale agents Oct 14, 2026
@phealy
Patrick W. Healy (phealy) requested a balanced review from Copilot October 14, 2026 23:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Stale caches can trigger unnecessary agent deletion, and the name policy blocks cleanup of legacy GatewayPools.

2 open findings
What changed in this PR

Adds controller-published Node assignments and targeted agent replacement, preparing for agent adoption in #917.

Changes:

  • Publishes GatewayPool labels with guarded patches and per-Node error isolation.
  • Replaces agents whose acknowledged assignments are outdated.
  • Adds admission restrictions, permissions, Pod identity fields, and tests.
File Description
internal/​operator/​components/​net/​vap_test.go Tests assignment ownership and admission restrictions.
internal/​operator/​components/​net/​net_test.go Updates operator plan expectations.
internal/​net/​controller/​node_assignment_controller.go Replaces agents with outdated acknowledgments.
internal/​net/​controller/​node_assignment_controller_test.go Tests replacement decisions and UID guards.
internal/​net/​controller/​gatewaypool_helpers_test.go Tests assignment publishing and error isolation.
internal/​net/​controller/​gatewaypool_controller.go Publishes guarded GatewayPool assignments.
deploy/​net/​node/​rbac_test.go Tests acknowledgment permissions and Pod identity.
deploy/​net/​node/​03-daemonset.yaml.tmpl Exposes Pod name and UID.
deploy/​net/​node/​02-rbac.yaml.tmpl Grants namespaced Pod acknowledgment permissions.
deploy/​net/​controller/​rbac_test.go Checks controller Pod deletion permission.
deploy/​net/​controller/​09-vap.yaml.tmpl Adds ownership, deletion, and pool-name restrictions.
deploy/​net/​controller/​02-rbac.yaml.tmpl Grants targeted agent deletion permission.
cmd/​unbounded-net-controller/​node_auth_informers.go Indexes agent Pods by Node.
cmd/​unbounded-net-controller/​node_auth_informers_test.go Verifies the Pod index.
cmd/​unbounded-net-controller/​main.go Starts the assignment controller.
api/​net/​v1alpha1/​types.go Defines assignment labels and acknowledgment annotations.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deploy/net/controller/09-vap.yaml.tmpl Outdated
Comment thread internal/net/controller/node_assignment_controller.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved admission safeguards and security-sensitive agent replacement require corrections and human rollout validation.

2 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread deploy/net/node/02-rbac.yaml.tmpl
Comment thread deploy/net/controller/09-vap.yaml.tmpl Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Disruptive network-agent replacement and admission-enforced permissions require final human validation of rollout safety.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Admission-enforced permission expansions and automatic networking-agent replacement warrant final human review.

0 open findings

🧠 Review effort: Balanced

@phealy
Patrick W. Healy (phealy) marked this pull request as ready for review October 16, 2026 23:50
@phealy
Patrick W. Healy (phealy) requested a review from a team October 16, 2026 23:50
@phealy
Patrick W. Healy (phealy) force-pushed the phealy/net-api-assignment-controller branch from 3958b4f to ce40b57 Compare October 21, 2026 18:52
stacked auto-merge was automatically disabled October 23, 2026 23:31

Auto-merge was disabled for another pull request in the stack

@phealy
Patrick W. Healy (phealy) added this pull request to the merge queue Oct 23, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue because a pull request earlier in the stack was removed Oct 23, 2026
Add preferred GatewayPool labels, UID-safe targeted agent replacement, and the admission/RBAC contract for effective assignment acknowledgments. Node-side adoption follows in the next layer.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
Keep agent updates token-bound and prevent agent label and pod CIDR changes, allow only agent-owned annotations, and cover negative admission cases. Validate raw GatewayPool name length at admission and reconcile unaffected Nodes despite per-Node errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
The Node and Pod informers are independent, so a cached Node can lag an
agent that already acknowledged the live assignment. Confirm the mismatch
against the live Node and Pod and delete with UID and resourceVersion
preconditions.

Check GatewayPool name length only on CREATE. Names are immutable, so
existing longer pools must remain updatable to drain and finalize.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
Preserve same-key taints unless they are the exact managed gateway taint. Apply restricting policies and Deny bindings before namespace Pod permission expansions; propagate installation failures to dependent grants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ab9b78ee-42b8-4841-8d74-cd78d002d48b
@phealy
Patrick W. Healy (phealy) force-pushed the phealy/net-api-assignment-controller branch from 2aa300b to d8ac5cc Compare October 23, 2026 23:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants