Repository navigation
operator: auto-discover API server endpoint from kube-public/cluster-info - #480
Merged
Philip Lombardi (plombardi89) merged 3 commits intoJul 14, 2026
Conversation
git describe can surface a nearest tag containing a slash (e.g.
agent-artifacts/v20260710), which is invalid in a Docker image reference and
breaks local/e2e image builds ("invalid reference format") and the image refs
in rendered manifests. Add VERSION_TAG (VERSION with '/' replaced by '-') and
use it for every image tag, while keeping VERSION intact for the embedded
version string (ldflags) and release artifact paths. Unifies the existing
one-off PLAYPEN_TAG sanitization.
…ter-info The operator no longer requires UNBOUNDED_API_SERVER_ENDPOINT to be configured. At startup it resolves the endpoint from the standard kube-public/cluster-info ConfigMap (the server field of the embedded kubeconfig); an explicit --api-server-endpoint / $UNBOUNDED_API_SERVER_ENDPOINT override still wins. With neither an override nor a discoverable value the operator fails hard, since machina and metalman cannot function without an endpoint. kubectl unbounded install stops force-populating the endpoint: it writes a value only when --api-server-endpoint is given, otherwise preserving any existing value and dropping the kubeconfig-host fallback so auto-discovery takes effect. Extract the cluster-info reader shared with metalman into a new internal/clusterinfo package and migrate internal/metalman/commands to it. The operator's existing cluster-wide configmaps RBAC already permits reading kube-public/cluster-info, so no RBAC change is required.
The layered DHCP-free HTTP smoke test (smoke-metalman-http.py) still resolved the metalman-controller ServiceAccount kubeconfig from the legacy unbounded-kube namespace, while the rendered metalman RBAC now creates that ServiceAccount in unbounded-system. The token request therefore failed with a non-zero exit before metalman could start, breaking the layered-http-smoke job whenever the metalman smoke workflow ran. Reuse the shared smoke.METALMAN_NAMESPACE constant (unbounded-system) so the HTTP smoke stays in lockstep with the PXE smoke and the namespace rename.
Jordan Olshevski (jveski)
approved these changes
Jul 14, 2026
Philip Lombardi (plombardi89)
merged commit Jul 14, 2026
c0d407e
into
feature/unbounded-system
11 checks passed
Philip Lombardi (plombardi89)
deleted the
feature/operator-apiserver-autodiscovery
branch
July 14, 2026 20:59
Israel Miller (imiller31)
pushed a commit
to imiller31/unbounded
that referenced
this pull request
Jul 17, 2026
…gration (Azure#388) * docs(designs): add unbounded-system integration plan + tracking Integration branch tracking doc for the namespace-consolidation effort: the problem, the integration-branch strategy (keep main releasable), the PR-A/PR-B/PR-C decomposition, the migration workflow, and a living status checklist updated as each chunk opens and merges. * docs(designs): track PR-B re-target and PR-A (Azure#386) open * docs(designs): record umbrella tracking PR Azure#388 + [unbounded-system] title convention * docs(designs): record Azure#372 review fixes + operator-only migration + release gate * docs(designs): finding-3 fixed via unbounded.SystemNamespace() helper * [unbounded-system] Unify components into a single configurable namespace (Azure#372) * build: add unified UNBOUNDED_NAMESPACE feeding all component namespaces Introduce a single UNBOUNDED_NAMESPACE (default unbounded-system) that every per-component *_NAMESPACE var derives from, so overriding it moves all components at once while individual components can still be overridden. Re-point MACHINA/MACHINE_OPS/ORCA/NET/GANTRY/UNBOUNDED_STORAGE_SUPERVISOR namespace vars and fix the hardcoded -n unbounded-kube in machina-run. * deploy: default machina/machine-ops/orca/storage-supervisor namespace to unbounded-system Flip the template default from {{ default "unbounded-kube" .Namespace }} to unbounded-system across the machina, machine-ops, orca, and storage-supervisor manifest templates, plus the orca cachestore/cluster service DNS defaults. Genuine kube-system/kube-public RBAC reads are unchanged. * net: default namespace to unbounded-system Add a default to the net manifest templates (previously bare {{ .Namespace }}) so they render into unbounded-system, and align the controller/webhook Go defaults (leader-election lease namespace, POD_NAMESPACE fallbacks) and the kubectl install constant to match. Genuine kube-system reads (kube-proxy, extension-apiserver-authentication) are unchanged. * gantry: templatize manifests onto unbounded-system namespace Convert the gantry deploy manifests from hardcoded gantry-system plain YAML to *.yaml.tmpl that default to unbounded-system, add a gantry-manifests render target (rendered/ gitignored), and update the gantry test fixtures, e2e references, and README. Runtime namespace was already sourced from the Downward API, so no Go constant change is needed. * inventory: templatize manifests, fix machina-system orphan, default to unbounded-system Convert the inventory manifests from plain .yaml (which already held template syntax but were never rendered) to proper *.yaml.tmpl, default their namespace to unbounded-system, and replace the non-Go-template {{ SSL_MODE }} / {{ PASSWORD }} tokens with real template keys. Fix the aggregator Service, which was pinned to a stray machina-system namespace, to co-locate with its Deployment. Add an inventory-manifests render target and rework hack/inventory-dev/local.sh to render-then-apply. * go: default hardcoded component namespaces to unbounded-system Update the remaining hardcoded namespace constants and flag defaults (machina SSH-secret namespace, machine-ops leader-election/credential secret, kubectl install + deploy-pxe namespaces, metalman bootstrap SA), fix site_init to use the machina namespace constant instead of a hardcoded literal, and resolve metalman's leader-election lease namespace from POD_NAMESPACE (injected via the Downward API by deploy-pxe) so the lease and its RBAC stay co-located. Product/repo-name references (Azure/unbounded-kube, 'unbounded-kube machines/sites') are unchanged. Update affected test fixtures. * hack: point scripts, smoke checks, and examples at unbounded-system Update orca install/smoke scripts, the storage-mesh config generator, the release core-namespaces-ready smoke check, the metalman lease cleanup, the orca config example DNS names, and the agent e2e-kind harness namespace references to the unified unbounded-system namespace. Resource/binary/label names (unbounded-net-controller, unbounded-net-gateway, ...) and product/repo references are left unchanged. * docs: document unbounded-system namespace + add migration runbook Update guides and reference docs to the unified unbounded-system namespace (machina/metalman SSH/PXE namespaces, net kubectl/exec namespace flags, net leader-election default), preserving the net.unbounded-kube.io API domain and product-name references. Add designs/namespace-unification.md describing the configuration model and a forward-only migration runbook for existing clusters. * test/ci: finalize unbounded-system unification and fix stragglers Point the orca stable garage template default and the gantry configmap regression test at unbounded-system / the renamed *.yaml.tmpl, update the from-source aks-ssh-quickstart workflow and component READMEs, and align the orca manifests render test. Verified via make fmt, lint (0 issues), test (green), and build. Note: .github/workflows/release-upgrade.yaml still references the old namespaces; it deploys released artifacts and implements cross-version upgrade semantics, so it must be updated when the first unbounded-system release is cut (see designs/namespace-unification.md). * refactor: centralize the default namespace in internal/unbounded Add internal/unbounded.SystemNamespace as the Go-side source of truth for the default install namespace, and point the production default/fallback sites and the descriptive local consts (machinaNamespace, unboundedCNINamespace, deployPXENamespace, SecretNamespaceUnboundedSystem, BootstrapSANamespace, orcadev defaultNamespace, net/machine-ops/metalman leader-election defaults) at it. Update the tests coupled to those defaults to reference the const instead of the literal. Add a drift-guard test that renders each component's namespace template with no Namespace override and asserts the template default matches unbounded.SystemNamespace, so the Go const and the manifest templates cannot silently diverge. The Makefile UNBOUNDED_NAMESPACE and the template literals remain the render/deploy source of truth (they cannot reference Go); the drift-guard keeps them aligned. * fix: catch stragglers still pointing at old namespaces Three call sites were missed by the unbounded-system unification: - aks-ssh-quickstart workflow: the ACR pull-secret step created the secret and patched the unbounded-net-node ServiceAccount in the unbounded-net namespace. site init now installs net into unbounded-system, so the patch (and create) hard-failed under set -euo pipefail. Point both at unbounded-system. - kubectl unbounded net: the namespace auto-detection probed unbounded-net and kube-system but not unbounded-system, so net resources could not be located without an explicit --namespace. Add unbounded.SystemNamespace as the primary candidate, keeping the old names as legacy fallbacks. - deploy/gantry/README: the Files table and apply order still referenced the raw *.yaml files and applied them directly, but they are now *.yaml.tmpl rendered via make gantry-manifests into rendered/. Update the docs and the broken overlay link. * docs: drop namespace-unification design doc The design doc is not needed in designs/. The one-time manual cluster cutover steps it carried are tracked in the follow-up issue instead, so no operational knowledge is lost. * hack: add namespace migration script for legacy clusters Existing clusters installed unbounded across split namespaces (machina/metalman in unbounded-kube, net in unbounded-net). Consolidating onto unbounded-system requires a one-time move because namespaces cannot be renamed in place, and we want every cluster off the old namespaces rather than lingering on the split layout. Add hack/scripts/migrate-namespace.sh, an idempotent, sequenced migrator for machina + metalman + net: - copies operator-owned state (Secrets, the machina-config ConfigMap, and metalman PXE Deployments) into unbounded-system, skipping auto-managed and regenerable resources (SA tokens, the net serving cert); - redeploys the components there from the released manifests, supplied either by --release TAG (downloaded from GitHub via curl) or an extracted --manifests-dir for air-gapped use; - rewrites the cluster-scoped CR secret references that enforce a namespace (MachineOperationCredential auth secretRef, Machine pxe.redfish passwordRef); - decommissions the old namespaces only after the new components are healthy, so a failure mid-run leaves a recoverable cluster and re-running completes. Cluster-scoped CRs/CRDs are left untouched; gantry/inventory are out of scope. There is a brief net data-plane interruption during the cutover (old and new net cannot run in parallel), which is inherent to the move. Add hack/smoke-namespace-migration.py, a kind-based e2e that stages a neutralized legacy install plus operator state and asserts the consolidated end-state across four scenarios (happy + idempotency, dry-run, resume, release-download), wired into a path-triggered smoke workflow. Also fix a rename straggler in the `site` command help text (unbounded-kube -> unbounded). * namespace: address review - operator-owned migration + machina ns + docs - Remove the standalone migrate-namespace.sh, its kind smoke test, and the smoke workflow: migration from the legacy split namespaces is now handled by unbounded-operator, so the shell path is no longer needed. - Resolve machina's SSH-secret namespace at runtime from POD_NAMESPACE (injected via the Downward API on the machina Deployment) instead of a compile-time constant, and use it for both getSecretValue and the informer cache scope. Add a --namespace flag to 'kubectl unbounded machine register' so the written Secret and its Machine ref land in the namespace machina runs in. This makes a non-default install namespace line up end-to-end. - Point the net day-2 operations docs (pods, node agents, leases, serving cert, endpoints, NetworkPolicies) at unbounded-system, where net now runs. make fmt/lint/test/build all green. * namespace: centralize namespace resolution in unbounded.SystemNamespace() Replace the SystemNamespace const with a POD_NAMESPACE-aware helper so every component resolves its runtime namespace the same way: - internal/unbounded: SystemNamespace() returns POD_NAMESPACE when set (injected via the Downward API), else the unexported systemNamespace default. This folds the scattered os.Getenv("POD_NAMESPACE")-with-fallback logic (net controller, metalman lease, machina secret lookup) into one helper. - Convert the compile-time namespace consts that derived from it to package vars (they now read POD_NAMESPACE at init: client tools see the default, in-cluster components see their pod namespace). - Drop machina's SecretNamespaceUnboundedSystem in favor of the helper for the SSH-secret read and informer cache scope. - Drift-guard test becomes an internal test comparing template defaults to the unexported systemNamespace. - Makefile documents that a non-default namespace lines up end to end and that 'machine register --namespace' is needed when installing to one. Deployments keep injecting POD_NAMESPACE, so a non-default install namespace works end to end. make fmt/lint/test/build all green. * namespace: inline unbounded.SystemNamespace() instead of alias vars Remove the local namespace aliases that just wrapped the helper (machinaNamespace, deployPXENamespace, unboundedCNINamespace, BootstrapSANamespace, orcadev defaultNamespace) and call unbounded.SystemNamespace() directly at each use site. Now that SystemNamespace() is itself the resolver, the aliases added indirection with no value. * docs(designs): PR-B merged; PR-A resynced clean with operator on unbounded-system; PR-C now migration-only * [unbounded-system] unbounded-operator: Site API, component reconcile, and released-cluster migration (Azure#386) * Introduce unbounded-operator on the unbounded-system namespace Add the top-level unbounded-operator and reshape Site into the object it reconciles, running on the unified unbounded-system namespace. Components install at the operator's own version into unbounded-system, so per-site image and namespace overrides are gone: net and machina are cluster singletons and metalman/storage are site-scoped. Site API: - Move Site from net.unbounded-cloud.io/v1alpha1 to the cluster-scoped unbounded-cloud.io/v1alpha3 (machina) group. spec.components now covers only the operator-managed workloads (machina, metalman, storage) plus per-component status; net is not a component. SiteComponentSpec is just {enabled} (image/namespace overrides removed). The CRD moves with it. - Retarget the net controller, validating webhook, unboundednet client, net-node agent, and the 'net create' path to the new group/version; split RBAC and the webhook rules accordingly. Operator: - cmd/unbounded-operator + internal/operator: a controller-runtime manager that reconciles Site into component workloads by server-side applying the embedded rendered manifests. net is deployed as a cluster singleton whenever any Site exists; machina as a singleton whenever any Site enables it; metalman is a per-site Deployment (with POD_NAMESPACE) plus its shared RBAC; storage is a per-site DaemonSet node-selected on net.unbounded-cloud.io/site. Config is reduced to the metalman image (version-matched at render) and the machina apiServerEndpoint. - Deploy assets under deploy/unbounded-operator and an image build. kubectl plugin: - Replace the imperative install paths with a single 'install' command that bootstraps CRDs + the operator; 'site init' writes declarative Site resources. Drop --enable-net and the per-component image-override install flags along with the net: block in the site template. Namespace: the operator, its defaults, and the kubectl install paths all resolve to unbounded.SystemNamespace(); operator manifests render into unbounded-system and the drift-guard covers the operator template. make fmt/lint/test/build all green. * Migrate existing clusters onto unbounded-system via the operator Add the operator-driven migration that moves a pre-redesign cluster (machina, metalman, and storage in unbounded-kube; net in unbounded-net; net-group Sites) onto the unified unbounded-system namespace and the machina-group Site API. It runs as an always-on, leader-elected manager Runnable gated by --reap-legacy-resources (default on); there is no one-shot subcommand and no standalone shell migrator. Each idempotent pass: 0. Translates every pre-redesign net-group Site (sites.net.unbounded-cloud.io) into the machina-group Site the operator reconciles, copying the networking spec verbatim and inferring spec.components from the running legacy workloads: storage on every translated Site, machina on the cluster Site, and metalman where a per-site metalman Deployment is detected. Existing machina-group Sites are never clobbered. 1. Copies non-regenerable state (Secrets, the machina-config and unbounded-storage-config ConfigMaps) into unbounded-system, stripping server-managed metadata, never overwriting existing target copies, and skipping regenerable/auto-managed secrets (net serving cert, SA tokens, Helm release secrets). 2. Rewrites the namespace embedded in cluster-scoped secret references (Machine.spec.pxe.redfish.passwordRef, MachineOperationCredential spec.auth.secretRef). 3. Reaps the operator-owned resources left in each legacy namespace once the new workloads are in place (label-scoped; net last; empty-footprint components skipped). Storage is gated on the per-site unbounded-storage-supervisor-<site> DaemonSets. Net is gated only on the new net workloads being CREATED (not Ready): the old and new net-node DaemonSets both use hostNetwork and cannot occupy the same node host ports, so the old net must be reaped to let the new net schedule (a brief net data-plane gap during cutover is expected and accepted). 4. Once drained, deletes the legacy net-group Site CRD and the legacy namespaces. Translating the Sites first gives the operator Sites to reconcile (bringing up the new components, including the net singleton that re-labels nodes for storage node-selection) before the reaper drains the old namespaces. Wire the reaper into the operator manager and broaden the operator ClusterRole to read legacy net-group Sites, translate/create machina-group Sites, rewrite Machine/MachineOperationCredential secret refs, install the component RBAC (escalate + bind, since the net/machina Roles grant permissions the operator does not itself hold), reap by label (deletecollection), and delete the drained legacy resources, namespaces, and Site CRD. Testing. Two complementary reaper e2e's: - e2e/operator (Go, //go:build e2e): drives the LegacyReaper in-process against a kind apiserver with staged inert legacy workloads and Ready targets; fast, per-PR, covers the reaper logic incl. storage/metalman reaping. - hack/operator-upgrade-e2e/e2e.py (standalone Python, run by the operator-upgrade-e2e workflow on dispatch + nightly): stands up kind with a local registry, installs the real released v0.1.19 multi-namespace version via that release's `kubectl unbounded site init --manage-cni-plugin=false`, then upgrades via this tree's `kubectl unbounded install` and asserts the reaper migrates net + machina for real (real datapath). Storage (RDMA) and metalman (PXE) can't run in vanilla kind and are excluded there. The driver is self-contained so it runs the same locally and in CI. To build the operator image with registry-hosted component images, images/unbounded-operator/ Containerfile gains NET_CONTROLLER_IMAGE/NET_NODE_IMAGE/MACHINA_IMAGE build-args (default empty; release builds unchanged). The faithful upgrade e2e surfaced three real fixes included here: the operator RBAC escalate/bind (component RBAC install), the net cutover host-port gating above, and deletecollection for the reaper's label-scoped deletes. Each now has a fast per-PR regression guard so it is caught without waiting for the nightly e2e: TestReapOnceReapsNetWhenNewNetNotReady + TestNetTargetsPresent (net gated on creation, not readiness), and TestOperatorClusterRoleAllowsComponentRBACInstall + TestOperatorClusterRoleGrantsReaperDeletes (operator RBAC has escalate/bind and deletecollection for every reapableKinds() type). make fmt/lint/test/build all green; both the in-process and the faithful released-version upgrade e2e pass on kind. * [unbounded-system] Address PR386 review, deprecate net site label, harden migration Address the review findings on PR386, deprecate the net-group site label, and fix the issues surfaced by running the faithful upgrade e2e. Review findings (operator / migration / kubectl / release): - Reconcile is declarative: disabling a component or deleting a Site tears down its workloads (singleton teardown, per-site cleanup, Site finalizer). - Per-site storage gets a per-site ConfigMap the DaemonSet mounts; metalman and storage node-select the canonical site label. - Reaper: gate storage reaping on every storage-enabled Site's DaemonSet; copy Machine cloud-init ConfigMaps and repoint their refs; fold legacy storage config into the translated Site; upsert machina-config so migrated config wins over reconcile defaults. SiteNodeSlice ownerRef uses the v1alpha3 GVK. - Operator namespace is authoritative (--namespace + POD_NAMESPACE); install wires it through. - kubectl: `site init --enable-storage` applies to the remote Site. - Release: build/sign the unbounded-operator image; storage supervisor tag defaults to $(VERSION) (no :latest) with a guard test for embedded :latest. Deprecate net.unbounded-cloud.io/site in favor of unbounded-cloud.io/site: - Net dual-writes both node labels and reads canonical-first with fallback. - Storage / metalman / kube-proxy node-select the canonical label; the managed kube-proxy DaemonSet is recreated on the immutable-selector change. - ValidatingAdmissionPolicy allows the net SA to write the canonical label. - Docs updated. Migration bugs found by the faithful upgrade e2e: - Operator ClusterRole grants update/patch on sites (for the Site finalizer) and on net-group sites (to clear the protection finalizer so the legacy CRD can delete). - Reaper clears finalizers on abandoned legacy net Sites before deleting the legacy Site CRD, avoiding a deletion deadlock on clusters with assigned nodes. - e2e: cluster Site nodeCidr is computed to cover the kind nodes so labeling is exercised; verify asserts the node-label dual-write. Make hostNetwork controller Deployments restart cleanly: - net-controller and per-site metalman use RollingUpdate maxSurge=0 so the old pod frees its host ports before the new one starts; e2e asserts a clean net-controller rollout restart. make build / lint / test pass; the faithful v0.1.19 -> operator upgrade e2e passes on kind. * [unbounded-system] Harden site lifecycle and label migration * [unbounded-system] Make storage config ConfigMap-owned * [unbounded-system] Address review nits: MachinaComponentSpec and Site status conditions Wrap the machina component spec in a dedicated MachinaComponentSpec type (mirroring MetalmanComponentSpec/StorageComponentSpec) so machina-specific fields can be added later without an API break. Replace Site.status.components (a map of custom SiteComponentStatus) with a standard []metav1.Condition. The operator now publishes one condition per component (NetReady, MachinaReady, MetalmanReady, StorageReady) so callers can `kubectl wait --for=condition=NetReady site/<name>`. Regenerated deepcopy and the Site CRD. * [unbounded-system] Fix migration sharp edges from review - Reaper self-deletion guard: the migration reaper filters the target namespace out of its legacy-namespace set so installing with --namespace unbounded-kube (or unbounded-net) can no longer drain and delete the namespace it just migrated into. `kubectl unbounded install` now also rejects a legacy namespace up front. The legacy namespace names and an IsLegacyNamespace helper move to internal/unbounded as the single source of truth shared by the CLI guard and the reaper. - release-upgrade workflow: wait on and summarize the operator-managed workloads in unbounded-system instead of the removed unbounded-kube / unbounded-net namespaces, and wait for each workload to exist before checking rollout status (they are now created asynchronously by the operator). - CLI docs: correct the install --namespace default to unbounded-system and drop the removed --net-namespace, image, and --enable-net flags from cli.md and the ssh / existing-cluster guides. - site init: fix swapped cluster node/pod CIDR validation error messages. Adds regression tests for each fix. * [unbounded-system] Operator owns CRD lifecycle and config; harden migration (Azure#464) * [unbounded-system] Operator owns CRD lifecycle and config; harden migration Make the unbounded-operator self-sufficient for CRD lifecycle and move environment-specific settings off Deployment args into a ConfigMap, then harden the legacy migration gates. Operator-owned CRDs: - Add operator.BootstrapCRDs: at startup the operator server-side applies every CRD embedded in the machina and net manifests and waits for each to become Established, before the manager starts (the typed Site informer cannot sync until the Site CRD is served). Idempotent on every start. - `kubectl unbounded install` no longer applies CRDs or takes --skip-crds; it bootstraps the operator and waits for its rollout, then for the CRDs. The reconcile loop skips CRD objects when applying net manifests. Config via ConfigMap: - Add deploy/unbounded-operator/04-configmap.yaml.tmpl (unbounded-operator-config), surfaced to the Deployment via envFrom. --api-server-endpoint and --reap-legacy-resources now default from UNBOUNDED_API_SERVER_ENDPOINT and UNBOUNDED_REAP_LEGACY_RESOURCES (an explicit flag still overrides). install writes api-server-endpoint into the ConfigMap instead of a Deployment arg. Migration hardening: - Detect legacy per-site metalman by Deployment name or canonical/deprecated site label, not a single label match. - Gate metalman reaping on the per-site replacement Deployment existing (presence gate, like net: hostNetwork host-port ownership). - Require storage DaemonSets to actually schedule (DesiredNumberScheduled >= 1 and observed generation current) before reaping the legacy supervisor, so a zero-desired DaemonSet no longer reads as ready. - Delete the legacy net-group Site CRD only after the new net controller is Available, so the GC does not transiently drop SiteNodeSlices mid-cutover. - Warn and emit an Event when a drained legacy namespace still holds non-operator workloads before the whole-namespace delete. - Reconcile Sites on generation changes only, so net/machina manifests are not re-applied on status-only Site updates. - Retarget the namespace as a substring (SA usernames, flag values), not just on exact-value equality. Tests: CRD bootstrap, env-var flag defaults, metalman detection/presence gate, storage readiness gates, net-controller CRD gate, foreign-workload detection; extend the upgrade e2e to assert the operator (not install) established and owns the Site CRD and that the operator-config ConfigMap is present. * [unbounded-system] Address install/operator review findings Fixes from PR review of the operator CRD-lifecycle/config work. 1. install no longer reports success against an unavailable operator. waitForOperator gated on AvailableReplicas>=desired, which the old ReplicaSet satisfies while a new crash-looping pod surges in during an upgrade. Replace it with a full rollout-complete check (ObservedGeneration current and Replicas == UpdatedReplicas == AvailableReplicas == desired), the same condition `kubectl rollout status` waits on. 2. Liveness no longer kills the operator mid-bootstrap. BootstrapCRDs runs before the manager starts the health server, so /healthz is unserved for up to crdEstablishedTimeout (~2m) while liveness (10s + 3x15s) would restart at ~40s. Add a startupProbe with a 150s budget (30 x 5s) that exceeds the bootstrap timeout and suppresses liveness until the health server binds. 3. Re-install with a changed endpoint now rolls the operator. install wrote the endpoint only into the ConfigMap, but envFrom is read once at pod start and the pod template was unchanged, so no rollout occurred. Stamp a config-hash annotation (sha256 of the endpoint) onto the Deployment pod template; a changed endpoint changes the template and triggers a rollout that the strengthened waitForOperator waits on. 4. The make/GitOps render path can set the endpoint again. Restore .APIServerEndpoint into the operator-config ConfigMap and re-add the UNBOUNDED_OPERATOR_API_SERVER_ENDPOINT Makefile variable / --set. The Deployment template stamps the same config-hash (sprig sha256sum) so raw `kubectl apply` of re-rendered manifests also rolls on an endpoint change. --skip-crds stays removed (CRDs are operator-owned); documented as a breaking change in the CLI reference. Tests: rollout-complete table + waitForOperator upgrade/complete cases; config-hash stamped and endpoint-sensitive; BootstrapCRDs applies all 12 embedded CRDs and blocks until Established; a render test asserting the ConfigMap endpoint and the matching Deployment hash annotation stay in sync. * [unbounded-system] Harden operator reconciliation and migration * net: harden SiteNodeSlice ownership and reconcile against delete/recreate races (Azure#473) Set SiteNodeSlice owner references to blockOwnerDeletion:false (and accept nil/false while repairing true) so the controller no longer needs sites/finalizers RBAC to write slices. Guard every slice, status, and finalizer write with a live GET plus an expected-UID check so a Site that is deleted and recreated under the same name is never mutated with stale data, and delete extra or orphaned slices with UID + ResourceVersion preconditions. Preserve foreign metadata by mutating a deep copy of the live slice instead of rebuilding it, propagate errors up the reconcile chain so a failed pass re-dirties and retries rather than silently dropping work, aggregate per-site status/finalizer errors, count gateway nodes in the Site status, convert node string slices to []interface{} for valid unstructured content, clean up orphaned slices whose Site is gone, and wire SiteNodeSlice informer events to mark slices dirty. * operator: preserve migrated config, hash full ConfigMap payloads, and harden the legacy reaper (Azure#474) Route every destructive migration read through an uncached APIReader and make namespaceExists fail closed, so the reaper never reaps based on stale cache data. createIfAbsent now Gets into a freshly zeroed object and compares the complete payload before treating an existing target as migrated, closing a silent data-loss path where the real client merges Get results into the desired object's maps. copyConfigMapByName refuses to leave a dangling cloud-init reference, translateSite requires spec-equivalence and is re-verified live before the legacy Site CRD is deleted, and the machina/net/ storage readiness gates re-check ResourceVersion + full-payload hash to avoid TOCTOU reaping during an in-flight rollout. Replace the config.yaml-only hash with configMapPayloadHash over the complete Data + BinaryData, gate per-site storage config, require UpdatedNumberScheduled on DaemonSet readiness, broaden the foreign-workload audit to pods, RCs, RSs, jobs, and cronjobs (fail closed, skipping controller-owned and terminating legacy descendants), copy the Secret Immutable field, and fix the inverted SiteNodeSlice owner-reference predicate that otherwise stalled migration. On the reconciler side, preserve a live unbounded-net-config instead of force-applying embedded defaults, stamp the config hash on both net workloads, watch all managed ConfigMaps and singleton workloads, reconcile retained net/machina without recreating them from nothing, and patch machina/storage config with optimistic locking. Adds the foreign-workload audit RBAC grants. The reaper migration e2e (reaper_e2e_test.go) travels with this change: it now stamps the full-payload config hash on the staged target workloads and drives a restricted-RBAC SiteController owner-repair, exercising the new gate behavior. * operator: own CRD lifecycle with periodic maintenance and idempotent install repair (Azure#475) Add a leader-elected CRDMaintainer that re-applies and re-establishes the complete owned CRD set on an interval, so a deleted or drifted CRD is repaired without waiting for a pod restart. Transient maintenance failures are logged and retried on the next interval; only a run of consecutive failures (MaxFailures, default 3) stops the manager, letting the pod restart after any terminating CRD has finished deleting. Introduce the exported RequiredCRDNames as the single source of truth (all 12 CRDs) and wait on the full set at bootstrap; reject CRDs carrying a deletionTimestamp as established in both the operator and the install client so a terminating CRD is never treated as healthy. kubectl unbounded install now preflights/waits on the complete RequiredCRDNames set, hashes the full operator ConfigMap payload (not just the endpoint) so a change to UNBOUNDED_REAP_LEGACY_RESOURCES rolls the operator, preserves the live reaper value via strconv.ParseBool and fails closed on an invalid value before applying, and repairs missing CRDs against an already-healthy operator via an idempotent repair token that only rolls when CRDs are unhealthy and the prior rollout is complete. Pin the operator Deployment to the Recreate strategy with rollingUpdate nulled so rollout completion is unambiguous, template the reap flag, and thread it through the Makefile and render tests. * test: real-kind reaper migration and faithful upgrade e2e coverage (Azure#476) test: faithful upgrade e2e coverage and net paths for the operator e2e workflow Extend the faithful upgrade e2e (hack/operator-upgrade-e2e/e2e.py) to assert byte-for-byte preservation of the migrated unbounded-net-config payload and its config hash on both net workloads, that a BinaryData-only change rolls both workloads via the managed ConfigMap watch, and that an identical same-release reinstall repairs a deleted CRD by rotating the repair token and restarting the operator without changing its image or config hash. Add the net controller and net CRD paths to the operator reaper e2e workflow triggers so future changes to internal/net/controller and deploy/net/crd exercise the migration e2e. The reaper migration e2e itself now lives with the reaper change (the operator config/migration PR); the upgrade e2e here runs on demand and nightly. * operator: auto-discover API server endpoint from kube-public/cluster-info (Azure#480) * build: sanitize the git-derived VERSION for Docker image tags git describe can surface a nearest tag containing a slash (e.g. agent-artifacts/v20260710), which is invalid in a Docker image reference and breaks local/e2e image builds ("invalid reference format") and the image refs in rendered manifests. Add VERSION_TAG (VERSION with '/' replaced by '-') and use it for every image tag, while keeping VERSION intact for the embedded version string (ldflags) and release artifact paths. Unifies the existing one-off PLAYPEN_TAG sanitization. * operator: auto-discover the API server endpoint from kube-public/cluster-info The operator no longer requires UNBOUNDED_API_SERVER_ENDPOINT to be configured. At startup it resolves the endpoint from the standard kube-public/cluster-info ConfigMap (the server field of the embedded kubeconfig); an explicit --api-server-endpoint / $UNBOUNDED_API_SERVER_ENDPOINT override still wins. With neither an override nor a discoverable value the operator fails hard, since machina and metalman cannot function without an endpoint. kubectl unbounded install stops force-populating the endpoint: it writes a value only when --api-server-endpoint is given, otherwise preserving any existing value and dropping the kubeconfig-host fallback so auto-discovery takes effect. Extract the cluster-info reader shared with metalman into a new internal/clusterinfo package and migrate internal/metalman/commands to it. The operator's existing cluster-wide configmaps RBAC already permits reading kube-public/cluster-info, so no RBAC change is required. * test: fix layered HTTP metalman smoke to use unbounded-system namespace The layered DHCP-free HTTP smoke test (smoke-metalman-http.py) still resolved the metalman-controller ServiceAccount kubeconfig from the legacy unbounded-kube namespace, while the rendered metalman RBAC now creates that ServiceAccount in unbounded-system. The token request therefore failed with a non-zero exit before metalman could start, breaking the layered-http-smoke job whenever the metalman smoke workflow ran. Reuse the shared smoke.METALMAN_NAMESPACE constant (unbounded-system) so the HTTP smoke stays in lockstep with the PXE smoke and the namespace rename. * operator, metalman: discover API server endpoint and CA on clusters without cluster-info (AKS) (Azure#484) * operator: fall back to KUBERNETES_SERVICE_HOST FQDN for the API server endpoint Managed control planes such as AKS do not publish the kube-public/cluster-info ConfigMap, so the operator could not auto-discover the API server endpoint it advertises to provisioned machines and had to be given --api-server-endpoint explicitly. Add a discovery fallback: resolve the endpoint as override -> cluster-info -> KUBERNETES_SERVICE_HOST FQDN -> fail hard. The operator pod now carries the kubernetes.azure.com/set-kube-service-host-fqdn label (as machina, net, and machine-ops already do), which makes KUBERNETES_SERVICE_HOST the public API FQDN on AKS and is inert elsewhere. The fallback is guarded: KUBERNETES_SERVICE_HOST is only accepted when it is an external FQDN, not an in-cluster ClusterIP or the kubernetes.default.svc DNS name. A joining node cannot reach those, so on clusters without cluster-info and without the FQDN env the operator keeps failing hard with an actionable error rather than silently advertising a dead endpoint. The shared logic lives in internal/clusterinfo (Discover, KubeServiceHostEndpoint, InClusterCA) so metalman can reuse it. * metalman: source the CA from the in-cluster SA when cluster-info is absent Metalman serves a kubeconfig to provisioning nodes and read both the API server URL and the CA from kube-public/cluster-info. On managed control planes (e.g. AKS) that ConfigMap does not exist, so metalman could not resolve a CA and the remote-node join path was broken even once the operator resolved an endpoint. Have the operator inject METALMAN_APISERVER_URL with the endpoint it already resolved, so metalman advertises exactly what machina writes and does not need to rediscover the URL. Make cluster-info best-effort in the ClusterInfoWatcher: the URL comes from the override (or cluster-info), and the CA comes from cluster-info when available and otherwise from the in-cluster service-account mount, whose cluster CA signs the API server serving cert (including the AKS FQDN SAN). * operator: resolve the API server endpoint with a URL-only discoverer The operator advertises only the endpoint URL, but resolution went through clusterinfo.Discover, which also read and returned the in-cluster CA. That made the operator fail on the FQDN-fallback path when the CA mount was unreadable, even though it discards the CA. Replace Discover with a URL-only DiscoverURL (cluster-info -> KUBERNETES_SERVICE_HOST FQDN) that never touches the CA, and point the operator at it. The CA-inclusive Discover had no other caller, so remove it (and its test); metalman still sources its CA via clusterinfo.Resolve and clusterinfo.InClusterCA directly. * metalman: re-resolve cluster-info on a provider-agnostic interval The ClusterInfoWatcher only refreshed on cluster-info ConfigMap events, so on clusters that publish no cluster-info (e.g. AKS) refresh ran once at startup and never again, and in-cluster CA rotation (a file change) was never picked up. The type comment also still claimed live API server URL rotation, which is stale now that the operator pins the URL via METALMAN_APISERVER_URL. Keep the informer for prompt pickup where cluster-info exists and add a periodic re-resolve (clusterInfoPollInterval, 5m) as a provider-agnostic floor that re-reads both the cluster-info CA and the in-cluster SA CA on every cluster. The informer resync is disabled so the ticker is the single owner of interval-based refresh, the stale doc comment is corrected, and the AKS "cluster-info unavailable" message drops to Debug so the poll does not log it forever. * nightly: deploy via operator install instead of blind manifest apply The nightly workflow predated the operator model: its upgrade path ran 'make machina-manifests net-manifests' and applied net/machina directly, so the operator (and its migration reaper) was never deployed. Rework it to mirror release-upgrade.yaml: - build the unbounded-operator, metalman, netboot, and unbounded-storage-supervisor images (all referenced by the operator when it reconciles a Site), pinning the operator's embedded net/machina/storage images and metalman's netboot image to the nightly tag. - bootstrap CRDs and the operator via 'kubectl unbounded install' for both init and upgrade; on init, 'site init --skip-install' creates the Site the operator reconciles. - key mode detection off sites.unbounded-cloud.io and fail if only the legacy sites.net.unbounded-cloud.io CRD is present. - wait for the operator plus the async component rollouts. - drop the direct render/apply of net/machina manifests and the Azure#235 machina-config merge workaround (now owned by the operator). * operator: set the AKS kube-service-host-fqdn as a pod annotation, not a label The unbounded-operator Deployment placed kubernetes.azure.com/set-kube-service-host-fqdn under the pod template labels. AKS only honors it as a pod annotation (per the AKS egress docs), so KUBERNETES_SERVICE_HOST stayed the in-cluster ClusterIP. On AKS, which does not publish kube-public/cluster-info, the operator's endpoint discovery then exhausted all tiers and crashlooped with 'no API server endpoint configured ... and no external KUBERNETES_SERVICE_HOST FQDN is available'. Every other component (net controller/node, machina, machine-ops, playpen, kube-proxy) already uses the annotation form; the operator was the lone outlier. Move it into the annotations block and update the render test to assert the annotation. Also correct the now-inaccurate 'pod label' wording in the operator and clusterinfo doc comments. * machina: restore Site networking print columns, move components to -o wide The machina-group Site kept every networking field from the old net-group Site (nodeCidrs, podCidrAssignments, status.sliceCount) but its default 'kubectl get site' columns were changed to the component booleans, hiding the networking data operators relied on. Make the default view match the old net Site again (Node CIDRs, Pod CIDR Assignments, Nodes, Slices, Age) and mark the Machina/Metalman/Storage columns priority=1 so they only appear under 'kubectl get site -o wide'. Display-only; regenerated the Site CRD's additionalPrinterColumns. * [unbounded-system] harden released-cluster migration and install (Azure#499) * net: gate SiteNodeSlice orphan cleanup on a healthy, non-empty Site source cleanupOrphanSiteNodeSlices concluded a slice was orphaned from a per-slice Site GET returning NotFound, and ran unconditionally before the len(sites)==0 guard in updateAllSiteSlices. When the machina Site informer is observed empty (the unbounded-system migration/upgrade window, before legacy net-group Sites are translated into the machina group, or a brief Site CRD outage), every SiteNodeSlice was deleted and not recreated until Sites reappeared, tearing down inter-node WireGuard peers and pod-CIDR routing cluster-wide. Cleanup now: - lists live machina Sites once (fixing the per-pass O(slices) Site GET amplification) and returns without deleting anything if the list fails; - skips deletion entirely when zero Sites are present while slices exist (legitimate whole-Site deletion is handled by owner-reference GC); - keeps a slice whose Site still exists in the legacy net group, so an untranslated Site's slices are never deleted mid-migration; - deletes only genuine orphans (empty siteName, or a Site absent from both groups) with the existing UID+ResourceVersion delete preconditions. Updates the existing cleanup tests to the corrected behavior and adds coverage for the empty-Site-source window and the untranslated-legacy-Site guard. * kubectl install: retarget manifest namespace from the build-time literal, not SystemNamespace() install rewrote the namespace baked into the rendered operator manifests using unbounded.SystemNamespace() as the rewrite source. SystemNamespace() returns \$POD_NAMESPACE when set, so when install runs in-cluster (a bootstrap Job/init container) or in any shell that exports POD_NAMESPACE, the "from" value no longer matched the build-time literal "unbounded-system". Cluster-scoped references that only rewriteNamespace can fix - notably the ClusterRoleBinding subject namespace - were then left at "unbounded-system" while the ServiceAccount was created in the target namespace, leaving the operator with no RBAC (forbidden on every call, never Ready, install times out). Export unbounded.DefaultSystemNamespace as the single build-default source of truth, use it as the rewrite source in install, and point the operator's buildDefaultNamespace at the same constant. Adds a regression test that sets POD_NAMESPACE and asserts the ClusterRoleBinding subject is retargeted. * operator: keep CRD maintenance retrying instead of stopping the manager CRDMaintainer.Start returned an error after MaxFailures (default 3) consecutive maintenance failures, which in controller-runtime stops the entire manager and exits the pod, taking the Site reconciler and the migration reaper down with it. The stop bought little: CRDs the operator has already established stay served by the apiserver regardless of operator liveness, and a terminating CRD already crashloops via the startup BootstrapCRDs path independent of the maintainer. A persistent-but-transient apiserver blip would therefore needlessly kill reconcilers that could have kept working (e.g. a migration stalls because the reaper was killed mid-drain). Maintenance failures are now logged with a consecutive-failure count and retried on the next interval; only context cancellation (manager shutdown) stops the loop. Removes the now-unused MaxFailures field and default, and replaces the crash-after-N test with one asserting persistent failures keep retrying without stopping. * operator: widen the reaper's pre-delete audit to PVCs and skipped Secrets The reaper deletes each drained legacy namespace wholesale, but the migration only copies Secrets (minus SkipSecretNames), the named/cloud-init/storage ConfigMaps out. warnOnForeignWorkloads only audited workload kinds, so data-bearing resources the migration never copies were destroyed with no mention: e.g. a PersistentVolumeClaim left in unbounded-kube, or the deliberately-skipped unbounded-net-serving-cert Secret, would vanish on namespace deletion with only a workload-scoped warning. The pre-delete audit now also surfaces every PersistentVolumeClaim and every Secret in SkipSecretNames, folded into the same warning log line and Event (reason unchanged) so operators see the full blast radius. Deletion still proceeds (warn-and-proceed); this only widens visibility. ConfigMaps are not enumerated per-name because the migration copies several dynamically-named ConfigMap sets (cloud-init, per-site storage), which would produce false positives. * net: grant read on legacy net-group Sites for orphan-cleanup migration check The SiteNodeSlice orphan-cleanup migration guard (legacySiteExists) issues a GET against net.unbounded-cloud.io/sites to tell an untranslated Site (still present in the legacy group) from one that is genuinely gone. The net controller ClusterRole granted only unbounded-cloud.io/sites, so during the partial-migration window that GET returned 403: the slice was still preserved (a lookup error is never read as 'gone'), but the intended legacy-present->keep path never executed and the controller logged errors every pass. Grant read-only get/list/watch on net.unbounded-cloud.io/sites to the net controller (and mirror it on the reaper e2e's restricted identity). Adds a unit test asserting a denied legacy lookup preserves the slice. * test(e2e): prove SiteController preserves slices through the migration window Adds TestSiteControllerPreservesSlicesDuringMigrationWindow, a kind-based e2e that runs the real net SiteController against a real API server and asserts the dataplane source of truth (SiteNodeSlices) is never torn down during the released-cluster migration/upgrade: * Phase 1 (empty machina window): legacy Sites + slices present, no machina Sites yet -> the slice stays continuously present (the pre-fix controller deleted every slice here). Verified this phase fails on the reverted fix. * Phase 2 (convergence): the machina Site is created and the slice is re-owned. * Phase 3 (partial migration): a legacy-only Site's slice survives (exercises the legacy-group check, needs the net RBAC grant). * Phase 4 (genuine orphan): a slice whose Site is absent from both groups is still deleted. Parametrizes createCluster into createClusterNamed so the new test uses its own kind cluster and cannot collide with the reaper e2e. * test(upgrade-e2e): monitor SiteNodeSlice (and optional WireGuard) continuity Adds a background ContinuityMonitor to the faithful upgrade e2e that samples the net dataplane's source of truth across the whole migration window (operator install -> reaper completion) and fails the run on a regression: * SiteNodeSlice presence (primary): net-node programs WireGuard peers and pod-CIDR routes from SiteNodeSlices, so a slice that is present and then vanishes means that site's dataplane was torn down. Disappearance is debounced over two samples to avoid transient-read false positives. * --probe-dataplane (best-effort): additionally samples WireGuard peer counts inside net-node pods and flags a collapse to zero; self-disables if peer data is unavailable so it never false-fails. A cross-node pod ping is intentionally NOT used: this harness runs CNI-free alongside kindnet, so pod traffic uses kindnet rather than the unbounded WireGuard mesh and would not reflect the unbounded dataplane.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the operator's API server endpoint configuration optional by auto-discovering it from the standard
kube-public/cluster-infoConfigMap, per reviewer feedback. It is now only required as an override or when the cluster does not publish cluster-info.Behavior
--api-server-endpoint/$UNBOUNDED_API_SERVER_ENDPOINTwins; otherwise it reads theserverfield fromkube-public/cluster-info. With neither, the operator fails hard (machina and metalman cannot function without an endpoint).kubectl unbounded install: only writesUNBOUNDED_API_SERVER_ENDPOINTwhen--api-server-endpointis passed; otherwise preserves any existing value and no longer falls back to the kubeconfig host, so auto-discovery takes effect on a plain install.configmaps get/list/watch.Changes
internal/clusterinfopackage (cluster-info reader);internal/metalman/commandsmigrated onto it (removes the duplicate).cmd/unbounded-operator/main.go:resolveAPIServerEndpoint(override/discover/fail-hard) wired into startup.cmd/kubectl-unbounded/app/install.go: preserve/override endpoint, drop kubeconfig fallback.git describecan yield a slash-containing tag (agent-artifacts/v20260710-...) that is invalid as a Docker tag and broke image builds and rendered manifest image refs. AddedVERSION_TAG(/->-) used for all image tags, keepingVERSIONfor ldflags/release paths. This unblocked the e2e image build.internal/metalman/**change trips thesmoke-metalmanworkflow path filter, which surfaced a pre-existing break on thefeature/unbounded-systembase: the layered DHCP-free HTTP smoke (hack/smoke-metalman-http.py) still resolved themetalman-controllerServiceAccount from the legacyunbounded-kubenamespace, while the rendered metalman RBAC now creates it inunbounded-system.kubectl -n unbounded-kube create token metalman-controllerfailed before metalman could start, reddening thelayered-http-smokejob (pxe-smokealready used the new namespace and passed). Fixed by reusing the sharedsmoke.METALMAN_NAMESPACEconstant so the HTTP smoke stays in lockstep with the PXE smoke and the namespace rename.Verification
make fmt/lint/build/testandgo test -raceon touched packages: green.TestOperatorReaperMigration: pass.--api-server-endpoint->UNBOUNDED_API_SERVER_ENDPOINTleft empty -> operator auto-discovers and becomes Ready (reconciler + reaper run).kube-public/cluster-infowith no override -> operator crash-loops withno API server endpoint configured (...) and cluster-info discovery failed: ... configmaps "cluster-info" not found(fail-hard confirmed).invalid reference format).smoke-metalmanworkflow: bothpxe-smokeandlayered-http-smokenow targetunbounded-system(previouslylayered-http-smokefailed on theunbounded-kubetoken request).Scope
Operator -> machina only.
machine-ops-controller(--api-server-endpoint) and net (apiserverURL) keep their parallel mechanisms and could adoptinternal/clusterinfolater.Note: the faithful upgrade e2e (
e2e.py, nightly/manual) now naturally exercises discovery since it installs without--api-server-endpoint.