feat(agent): validate prequalified NVIDIA DRA hosts

[Kevin Cho (chokevin)]

Agent-side NVIDIA installation and service mutation conflicted with the Hillsboro image-ownership model. DRA enrollment now validates one exact externally produced Azure Linux or Ubuntu host contract before host/rootfs mutation or Node registration, and fails closed without installing, repairing, or falling back.

• Removes Unbounded ownership of IMEX package installation, service mutation, helper staging, and channel-directory creation.
• Dispatches only on the exact configured profile, profile version, contract version, artifact path, and canonical document digest. There is no OS inference, alias, or cross-profile fallback.
• Uses separate closed decoders and semantic validators for the Azure Linux and Ubuntu contracts, including canonical JSON, exact sidecar, immutable image identity, runtime/package evidence, local NVLink evidence, and authenticity checks.
• Rejects unknown/duplicate fields, noncanonical bytes, blocked evidence, digest or attestation mismatch, cross-contract/path substitution, service/runtime mismatch, and missing runtime prerequisites.
• Reports NVIDIAIMEXPrepared=False with reason HostQualificationFailed before registration on failure; successful validation reports HostQualified without claiming ComputeDomain or physical multi-node fabric readiness.
• Preserves ownership: aks-flex-dev produces and qualifies the image, MetalMan applies a separately approved HostReplace, Unbounded validates and joins, and producer-owned GPU Operator/DRA manages ComputeDomains, IMEX, CDI claims, and workload injection.

Exact producer contracts

Source of truth: aks-flex-dev commit aad7463945848555d392b1c33c906ef4d4e37eeb in ADO PR 17532408.

OS profile	Profile/version	Contract	Host artifact
Azure Linux 3	hillsboro-gb200-gpu-operator-ready / v2026.10.14-1	hillsboro.nvidia-host-capabilities/v1	/etc/unbounded/host-capabilities/nvidia-gpu-operator.json
Ubuntu 24.04.4	hillsboro-gb200-ubuntu-gpu-operator-ready / v2026.10.14-1	hillsboro.ubuntu-nvidia-host-capabilities/v1	/etc/unbounded/host-capabilities/nvidia-gpu-operator-ubuntu.json

The committed Ubuntu blocked fixture is copied byte-for-byte from the producer: 4,325 bytes, no BOM or trailing newline, SHA-256 311893302546decb52cc9420e1b71603f22a48623a1686e17cd7eb2b0abe8596. It is decoder/canonicalization evidence only: acceptance.status=blocked is rejected for readiness. The Azure blocked fixture remains regression evidence only as well.

MachineConfiguration now carries all five immutable consumer inputs: profile, profile version, contract version, artifact path, and qualified artifact digest. Generated CRDs and kubectl unbounded machine config create/get expose the same fields.

Current Hillsboro scope

• gb200-3 and gb200-4 are the only NVLink/ComputeDomain/IMEX acceptance pair. Their read-only host evidence shows Ubuntu 24.04.4, kernel 6.14.0-1009-azure-nvidia, NVIDIA 580.178.04, Container Toolkit 1.19.1, containerd 2.1.8, and healthy local NVLink.
• gb200-2 is non-NVLink scope only. Its earlier degraded local NVLink evidence must not block gb200-3/4 and must not become an eligibility claim.
• gb200-1 is cordoned and excluded.
• No host currently has a producer-qualified immutable Ubuntu artifact/digest/attestation. No join, HostReplace, cluster-resource, package, service, workload, rack, or other live mutation is authorized.

Verification

Passed on Azure Linux 3 arm64:

• go test ./pkg/agent/config ./pkg/agent/goalstates ./internal/provision ./internal/machineops ./internal/metalman/netboot ./cmd/agent/internal/daemon
• go test ./cmd/kubectl-unbounded/app -run "TestRunConfigCreate_(NVIDIAIMEXDRA|RequiresNVIDIAHostQualification|RejectsUnknownNVIDIAIMEXMode)$"
• go vet ./pkg/agent/config ./pkg/agent/goalstates ./internal/provision ./cmd/kubectl-unbounded/app
• golangci-lint v2.13.1 across ./api/... ./cmd/... ./deploy/... ./e2e/... ./hack/... ./internal/... ./pkg/... with zero issues

Passed locally:

• go generate ./api/machina/v1alpha3
• exact Ubuntu fixture byte count, digest, sidecar, and producer-file comparison
• git diff --check

Unable to complete CI=1 make test on the arm64 validation host because the pre-existing TPM simulator dependency cannot find openssl/aes.h. The run completed all other packages, including every changed package; GitHub CI remains the authoritative full-suite gate.

End-to-end acceptance gate

• Producer published the final closed dual-contract schemas and canonical blocked regression fixtures.
• PR feat(agent): validate prequalified NVIDIA DRA hosts #913 implements exact Azure Linux and Ubuntu dispatch plus positive and negative dual-profile tests.
• Producer publishes a qualified, authenticated artifact and exact digest for the selected host image.
• A live canary/join is separately authorized and succeeds without consumer-side installation or repair.
• Record a simple 10-20 second Hillsboro video showing immutable profile/digest accepted, Machine HostQualified/NVIDIAIMEXPrepared=True, Node Ready with four GPUs, and GPU Operator/DRA/ComputeDomain readiness or the exact PR-specific terminal result.
• Attach detailed commands, YAML, events, and logs separately from the video.

A blocked or failing run cannot satisfy the video gate. The recording must distinguish the producer-owned GPU Operator deployment from PR #913's validation/join boundary.

Risk and non-goals

DRA mode is intentionally fail-closed. This PR does not install GPU Operator, modify NVIDIA packages or services, repair physical NVLink/NVSwitch/InfiniBand, manage ComputeDomains/channels, or authorize a live rollout. Rollback is to omit DRA qualification inputs or revert the MachineConfiguration before any separately approved host lifecycle operation.

Review guide

1. pkg/agent/goalstates/nvidia.go and nvidia_test.go: exact registry dispatch, closed decoders, canonical bytes, semantics, and dual-profile negative matrix.
2. pkg/agent/config/config.go, api/machina/v1alpha3/machineconfiguration_types.go, and generated CRDs: five-field public contract and fail-closed validation.
3. pkg/agent/goalstates/resolve.go: tuple rejection before artifact reads or host preparation.
4. docs/content/reference/gpu/nvidia-gpu-operator.md: producer ownership, current rollout blockers, and evidence gate.

AI assistance: GitHub Copilot helped implement, test, review, and draft documentation for this change; the author verified the final diff, commands, and PR text.

[Kevin Cho (chokevin)]

Closing for now. After clarifying ownership, NVIDIA package and image qualification belongs to the producer, and we do not currently have an established Unbounded requirement to block Node registration on a separate consumer-side qualification contract. If that admission boundary becomes a product requirement, we can revisit it as a focused change with an explicit owner and consumer.