Repository navigation
fix(nodeopupgrade): upgrade from the image's files, not the pod's - #314
Merged
Merged
Conversation
jimmykarily
added a commit
to kairos-io/kairos-docs
that referenced
this pull request
Nov 12, 2026
When the upgrade image has no /etc/resolv.conf, /etc/hosts or /etc/hostname, the container runtime creates an empty file there to mount the Pod's copy onto. kairos-io/kairos-operator#314 leaves any of the three that is missing or empty out of the new system, so the node matches a fresh install of the image. For /etc/resolv.conf that keeps images that rely on systemd to create the symlink at boot working. The page now explains this, shows the resulting --exclude-path in the example command, and says the kairos-agent v3.6.0 requirement also applies when the image lacks one of the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
After a NodeOpUpgrade and reboot, nodes could end up with a regular /etc/resolv.conf holding the upgrade pod's DNS settings: the cluster DNS service IP and the pod's search domains. The host then resolved names through cluster DNS. CoreDNS pods using `forward . /etc/resolv.conf` forwarded to themselves, reported a loop, and image pulls on the host failed. The upgrade runs kairos-agent inside a container of the target image and copies the container's root filesystem into a new host image. Kubernetes mounts pod-specific files over the image's /etc/resolv.conf, /etc/hosts and /etc/hostname, and kairos-agent's rsync follows those mounts, so the pod's files were copied instead of the image's. /etc/hosts and /etc/hostname were already left out with --exclude-path; /etc/resolv.conf was not. The script now bind-mounts / (non-recursively) into a temporary directory and passes that as the upgrade source. A plain bind mount shows the container's root filesystem without anything mounted on top of it, so the new host image gets the image's own copy of all three files, the same as a fresh install of that image. The pod's own resolver settings stay in place for the rest of the script. The operator no longer adds --exclude-path for /etc/hostname and /etc/hosts. Excluding a path leaves it out of the new image entirely; with the bind mount the image's files are the right ones to copy, and the Kairos boot stages that manage hostname and hosts see the same files as after a fresh install. A side effect is that NodeOpUpgrade no longer needs kairos-agent v3.6.0+ unless spec.excludePaths is set. The ExcludePaths doc comment also said excluded paths "on the host are preserved". kairos-agent builds the new image from scratch, so an excluded path is missing from it rather than kept from the host; the comment now says so. Fixes kairos-io/kairos#6109 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
No e2e test ran a NodeOpUpgrade. This one runs a recovery-only upgrade (which does not reboot the kind node) with a stand-in image: busybox plus marker /etc/resolv.conf, /etc/hosts and /etc/hostname, and a fake kairos-agent that records what it finds under its upgrade source onto the node. The test checks that the source holds the image's three files and none of the pod's other mounts, and that the upgrade container keeps its cluster DNS settings. Against the previous upgrade script it fails with the pod's resolver settings in the source, the same content reported in kairos-io/kairos#6109. The image ships /etc/resolv.conf as a regular file. With a symlink, the container runtime mounts the pod's file at the link target under /run, which the upgrade script hides by mounting the host's /run there, so the leak would not show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
When the upgrade image has no /etc/resolv.conf, /etc/hosts or
/etc/hostname, the container runtime creates an empty file at that path
in the container's writable layer, so it has something to mount the
pod's copy onto. The plain bind mount the upgrade copies from shows
those empty files, so the new system got them too, where a fresh
install of the image has none.
For /etc/resolv.conf that breaks DNS on images that rely on this
systemd tmpfiles rule to create the symlink at boot:
L! /etc/resolv.conf - - - - ../run/systemd/resolve/stub-resolv.conf
The rule only acts when the path does not exist. Hadron v4.5.0 ships
the rule and no /etc/resolv.conf. Its kairos-init boot step that relinks
/etc/resolv.conf to /run/systemd/resolve/resolv.conf does not run on
every boot: it checks `[ -f /etc/hosts ]` while another initramfs step
replaces /etc/hosts concurrently, and the immucore log on a Kairos VM
showed that check passing on some boots and failing on others, even when
the image ships /etc/hosts. Upgrading to a Hadron image with the relink
command disabled and the empty file copied left /etc/resolv.conf empty,
and the host could not resolve names.
The upgrade script now checks the three files in the bind mount and
passes --exclude-path for each one that is missing or empty. The new
system then matches a fresh install of the image, and on Hadron the
tmpfiles rule creates the stub-resolv.conf symlink when the relink step
does not run (verified on a Kairos VM). An image that ships one of the
files empty on purpose loses nothing, since an empty file and a missing
one mean the same for these three.
An --exclude-path flag needs kairos-agent v3.6.0+, so a NodeOpUpgrade to
an image that lacks one of the files needs that version, as it did
before the bind mount.
The e2e test gains a second stand-in image with none of the three files.
Before this change the upgrade source showed the runtime's empty files
and kairos-agent got no exclusions; now all three are excluded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
jimmykarily
force-pushed
the
fix/upgrade-pod-resolv-conf
branch
from
November 12, 2026 11:44
0e4828f to
8d209d4
Compare
jimmykarily
marked this pull request as ready for review
November 12, 2026 11:48
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The resolver-file check can omit a shipped /etc/resolv.conf symlink when its target is absent or empty, potentially leaving upgraded nodes without DNS; symlink preservation and a dangling-symlink regression test are needed.
1 open issue
What changed in this PR
Updates NodeOpUpgrade to use the target image’s filesystem rather than Kubernetes-mounted pod files, addressing DNS configuration leakage during upgrades.
Changes:
- Uses a non-recursive bind mount as the source for active and recovery upgrades.
- Replaces fixed exclusions with checks for missing or empty files, while preserving user-configured exclusions.
- Adds unit and end-to-end coverage and clarifies exclusion semantics.
| File | Description |
|---|---|
test/e2e/nodeopupgrade_test.go |
Adds upgrade-source checks for images with and without network configuration files. |
internal/controller/nodeopupgrade_controller.go |
Prepares the isolated source and calculates automatic exclusions. |
internal/controller/nodeopupgrade_controller_test.go |
Tests source selection, exclusions, ordering, and bind-mount failure handling. |
internal/controller/defaults.go |
Removes the fixed hosts and hostname exclusion list. |
api/v1alpha1/nodeopupgrade_types.go |
Clarifies what exclusions omit and when agent version requirements apply. |
Review effort: Balanced
The upgrade script leaves /etc/resolv.conf, /etc/hosts and /etc/hostname out of the new system when they are empty in the bind mount, using `[ ! -s ]`. That test follows symlinks, so an image that ships /etc/resolv.conf as a symlink could lose it: - A relative link, such as ../run/systemd/resolve/stub-resolv.conf, resolves inside the bind mount. When the container runtime mounts the pod's resolv.conf it follows the link and creates an empty file at the target, so the link looked empty and was excluded. - An absolute link resolves against the upgrade container's /run, which the script has replaced with the host's, so the result depended on the host. The check now skips symlinks (`[ ! -L ]`), so a symlink is always copied as is and only a missing file or an empty regular file is excluded. The e2e test gains a third stand-in image whose /etc/resolv.conf is a relative link to systemd-resolved's stub file. Before this change the upgrade passed --exclude-path /etc/resolv.conf for it; now nothing is excluded and the link reaches the upgrade source. Reported by Copilot review on #314. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
jimmykarily
added a commit
to kairos-io/kairos-docs
that referenced
this pull request
Nov 12, 2026
kairos-io/kairos-operator#314 only leaves /etc/resolv.conf, /etc/hosts and /etc/hostname out of the new system when they are missing or empty regular files. A symlink is copied as is, so an image that ships /etc/resolv.conf as a link to systemd-resolved's file keeps it. The page said "missing or empty", which reads as if an empty or dangling link target would also exclude the link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes kairos-io/kairos#6109
What happens today
After a NodeOpUpgrade and the reboot that follows, a node can come up with an
/etc/resolv.confthat belongs to the upgrade pod instead of the host:The host then sends its own DNS lookups to the cluster DNS service. CoreDNS pods that use
forward . /etc/resolv.confend up forwarding to themselves and report a loop, and image pulls on the host fail with DNS errors. Node readiness checks still pass, so this is easy to miss.Why
kairos-agent upgrade --source dir:/inside a container of the target image. kairos-agent builds the new system image by rsyncing that container's root filesystem./etc/resolv.conf,/etc/hostsand/etc/hostnamein every container.--exclude-pathfor/etc/hostsand/etc/hostname, but not for/etc/resolv.conf.This affects stock Kairos images, not only custom ones. Upgrading a Hadron v4.5.0 k3s node to the same Hadron image with the released v0.3.0 operator left the pod's file on the host after the reboot in all four runs I did (see the VM runs below).
Hadron's kairos-init config has an
initramfsstep that relinks/etc/resolv.confto/run/systemd/resolve/resolv.confunderif: '[ -f /etc/hosts ]'. That check races with the31_hosts.yamlstep that runsrm -f /etc/hostsand thenln -sf /usr/local/etc/hosts /etc/hosts: yip chains the files in name order, but30_ulimit.yamlhas noinitramfssteps and resets the chain, so31_hosts.yamlruns concurrently with05_network.yaml(dirOpsin yip'spkg/executor/default.go; the immucore log shows the two files' steps interleaved). With v0.3.0,/etc/hostsis not in the new system at all, so the check fails unless the hosts step has already finished, and the pod's file stays. That race is a separate bug, fixed in mudler/yip#355. With only the yip fix, the check runs before31_hosts.yamlevery time and v0.3.0 upgrades keep the pod's file on every boot (verified on a VM), so this PR is still needed.The fix
The upgrade script now bind-mounts
/without--rbindinto a temporary directory and passes that directory as the upgrade source. A non-recursive bind mount shows the container's root filesystem without anything mounted on top of it, so the new host image gets the image's own/etc/resolv.conf,/etc/hostsand/etc/hostname. The upgrade container keeps its own cluster DNS settings for the rest of the script.When the image has no copy of one of those files, the container runtime creates an empty file at that path in the container's writable layer, to have something to mount the pod's copy onto. The bind mount shows that empty file. The script leaves any of the three that is missing, or is an empty regular file, out of the new system with
--exclude-path, so the node matches a fresh install of the image. A symlink is always copied as is: following it would test whatever its target holds inside the upgrade container, such as the empty file the runtime creates at a relative link's target when it mounts the pod's copy there, or the host's/runfor an absolute link. Without that, an image that relies on systemd's tmpfiles ruleL! /etc/resolv.conf - - - - ../run/systemd/resolve/stub-resolv.confwould get an empty/etc/resolv.confand no symlink, because the rule only acts when the path does not exist. An image that ships one of the files empty on purpose loses nothing; an empty file and a missing one mean the same for these three.If
mktempor the bind mount fails, the script stops before kairos-agent runs.Adding
/etc/resolv.confto a fixed exclusion list would not have worked. kairos-agent builds the new image from scratch, so an excluded path is left out of it entirely, and images that ship their own/etc/resolv.confwould lose it.Behaviour changes
--exclude-path /etc/hostname --exclude-path /etc/hosts. Images that ship non-empty copies now get them installed. On images built with kairos-init, the boot stage in31_hosts.yamlrewrites both files on every boot, so the result after reboot is unchanged. A custom image that ships a fixed, non-empty/etc/hostnameand lacks that boot stage would now give upgraded nodes that hostname, as a fresh install of the same image already does.spec.excludePathsis passed through as before, after the automatic exclusions.--exclude-pathneeds kairos-agent v3.6.0+. That is now needed whenspec.excludePathsis set or when the image lacks one of the three files.mainpasses the flag on every upgrade, so this is no stricter than today.ExcludePathsdoc comment said excluded paths "on the host are preserved". kairos-agent leaves them out of the new image instead; the comment now says so. The CRD is generated withmaxDescLen=0, so no manifest changes.Tests
mountandkairos-agentstubbed; themount --bindstub fills the bind point with chosen image files. They check that the bind mount happens before kairos-agent and is the source of every kairos-agent call, that kairos-agent does not run if the bind mount fails, and which files are excluded for an image with all three files, with an empty one, with none, and with/etc/resolv.confas a symlink whose target is missing or empty.test/e2e/nodeopupgrade_test.go; no e2e test ran a NodeOpUpgrade before). It runs recovery-only upgrades, which do not reboot the kind node, with two busybox stand-in images and a fakekairos-agentthat records what it finds in its upgrade source./etc/resolv.conf,/etc/hostsand/etc/hostname: onmainthe source's/etc/resolv.confissearch default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5and the test fails. With this PR the source holds the image's three files, nothing under/proc, and no exclusions are passed.--exclude-path./etc/resolv.confis a relative link to../run/systemd/resolve/stub-resolv.conf: the link's target in the source is the runtime's empty file. Without the symlink check the upgrade passed--exclude-path /etc/resolv.conf; with it nothing is excluded and the link reaches the source.open /dev/loop0: no such file or directoryinside the kind node. They ran before the NodeOpUpgrade test and do not touch the changed code. I did not rerun them onmain.tests/provider_upgrade_k8s_test.go) deploys the operator image pinned inconfig/manager/kustomization.yaml, currently the released v0.3.0. I ran that test locally from the ISOkairos-hadron-v0.6.3-standard-amd64-generic-v4.5.0-k3sv1.37.2+k3s1.iso, upgrading toquay.io/kairos/hadron:v0.6.3-standard-amd64-generic-v4.5.0-k3s-v1.37.2-k3s1, with the operator switched to an image built from this branch. I added read-only checks after the reboot; they are not part of this PR.kairos-agent upgrade --source dir:/tmp/tmp.d1wfVZB22R --exclude-path /etc/resolv.conf --exclude-path /etc/hostname(the Hadron image has no/etc/resolv.confand an empty/etc/hostname), the node rebooted (boot ID changed), and k3s came backReady./etc/resolv.conf -> /run/systemd/resolve/resolv.conf, as before the upgrade, withnameserver 10.0.2.3and nocluster.local;resolvectlreportsresolv.conf mode: uplink;getent hosts quay.ioresolves. Hostname and/etc/hostsare unchanged.COS_ACTIVE) read-only after the reboot and read/etc/resolv.conffrom it, and grep/run/immucore/immucore.logfor the relink step.[ -f /etc/hosts ]: exit status 1) even though that image ships/etc/hosts, the new system had a 0-byte/etc/resolv.conf, systemd did not create its symlink, andgetent hosts quay.iofailed./etc/resolv.confis left out of the new system, systemd's tmpfiles rule creates/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf, resolved runs instubmode, and lookups work. So the branch works whether or not the relink step wins the race.search operator-system.svc.cluster.local svc.cluster.local cluster.local,nameserver 10.43.0.10,options ndots:5and has no/etc/hosts. The running node keeps that file,resolvectlreportsresolv.conf mode: foreignwith DNS server10.43.0.10, andgetent hosts quay.iofails. This is the reported bug. The kairos-io/kairos test still passes, since it only checks that the node isReady./etc/resolv.confas a relative symlink to../run/systemd/resolve/stub-resolv.conf: without the symlink check the new system image had no/etc/resolv.conf; with it the new system image holds the link as shipped. On Hadron the running node ends up the same either way, because the relink step rewrites/etc/resolv.confat boot (uplinkmode, lookups work)./etc/resolv.conf(the Hadron image plus a marker file, the reporter's situation): with v0.3.0 the new system image holds the pod's file, as above. With this branch it holds the image's own file (# resolv.conf shipped in the upgrade image,nameserver 9.9.9.9), and the running node ends up with the systemd-resolved symlink,uplinkmode, and working lookups.The operator docs get a matching update in kairos-io/kairos-docs#794.
Environment
Arch Linux (kernel 7.2.11), Go 1.27.3, kind v0.17.0, envtest Kubernetes 1.37, QEMU with KVM for the VM test. Unit tests with
make test, e2e withginkgo -v ./test/e2e, VM test withginkgo --label-filter provider-upgrade-k8sin kairos-io/kairos atb2697a28.🤖 Generated with Claude Code