Skip to content

fix(nodeopupgrade): upgrade from the image's files, not the pod's - #314

Merged
wrkode merged 4 commits into
mainfrom
fix/upgrade-pod-resolv-conf
Nov 17, 2026
Merged

wrkode merged 4 commits into
mainfrom
fix/upgrade-pod-resolv-conf

Conversation

@jimmykarily

@jimmykarily jimmykarily commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes kairos-io/kairos#6109

What happens today

After a NodeOpUpgrade and the reboot that follows, a node can come up with an /etc/resolv.conf that belongs to the upgrade pod instead of the host:

search <upgrade-namespace>.svc.cluster.local svc.cluster.local cluster.local
nameserver <cluster DNS service IP>
options ndots:5

The host then sends its own DNS lookups to the cluster DNS service. CoreDNS pods that use forward . /etc/resolv.conf end up forwarding to themselves and report a loop, and image pulls on the host fail with DNS errors. Node readiness checks still pass, so this is easy to miss.

Why

  1. A NodeOpUpgrade runs kairos-agent upgrade --source dir:/ inside a container of the target image. kairos-agent builds the new system image by rsyncing that container's root filesystem.
  2. Kubernetes mounts pod-specific files over the image's /etc/resolv.conf, /etc/hosts and /etc/hostname in every container.
  3. rsync follows those mounts, so the pod's files end up in the new host image. The operator already passed --exclude-path for /etc/hosts and /etc/hostname, but not for /etc/resolv.conf.

This affects stock Kairos images, not only custom ones. Upgrading a Hadron v4.5.0 k3s node to the same Hadron image with the released v0.3.0 operator left the pod's file on the host after the reboot in all four runs I did (see the VM runs below).

Hadron's kairos-init config has an initramfs step that relinks /etc/resolv.conf to /run/systemd/resolve/resolv.conf under if: '[ -f /etc/hosts ]'. That check races with the 31_hosts.yaml step that runs rm -f /etc/hosts and then ln -sf /usr/local/etc/hosts /etc/hosts: yip chains the files in name order, but 30_ulimit.yaml has no initramfs steps and resets the chain, so 31_hosts.yaml runs concurrently with 05_network.yaml (dirOps in yip's pkg/executor/default.go; the immucore log shows the two files' steps interleaved). With v0.3.0, /etc/hosts is not in the new system at all, so the check fails unless the hosts step has already finished, and the pod's file stays. That race is a separate bug, fixed in mudler/yip#355. With only the yip fix, the check runs before 31_hosts.yaml every time and v0.3.0 upgrades keep the pod's file on every boot (verified on a VM), so this PR is still needed.

The fix

The upgrade script now bind-mounts / without --rbind into a temporary directory and passes that directory as the upgrade source. A non-recursive bind mount shows the container's root filesystem without anything mounted on top of it, so the new host image gets the image's own /etc/resolv.conf, /etc/hosts and /etc/hostname. The upgrade container keeps its own cluster DNS settings for the rest of the script.

When the image has no copy of one of those files, the container runtime creates an empty file at that path in the container's writable layer, to have something to mount the pod's copy onto. The bind mount shows that empty file. The script leaves any of the three that is missing, or is an empty regular file, out of the new system with --exclude-path, so the node matches a fresh install of the image. A symlink is always copied as is: following it would test whatever its target holds inside the upgrade container, such as the empty file the runtime creates at a relative link's target when it mounts the pod's copy there, or the host's /run for an absolute link. Without that, an image that relies on systemd's tmpfiles rule L! /etc/resolv.conf - - - - ../run/systemd/resolve/stub-resolv.conf would get an empty /etc/resolv.conf and no symlink, because the rule only acts when the path does not exist. An image that ships one of the files empty on purpose loses nothing; an empty file and a missing one mean the same for these three.

UPGRADE_ROOTFS=$(mktemp -d)
mount --bind / "${UPGRADE_ROOTFS}"
UPGRADE_EXCLUDES=""
for f in /etc/resolv.conf /etc/hosts /etc/hostname; do
  if [ ! -L "${UPGRADE_ROOTFS}${f}" ] && [ ! -s "${UPGRADE_ROOTFS}${f}" ]; then
    UPGRADE_EXCLUDES="${UPGRADE_EXCLUDES} --exclude-path ${f}"
  fi
done
kairos-agent upgrade --source "dir:${UPGRADE_ROOTFS}"${UPGRADE_EXCLUDES} <user excludePaths>

If mktemp or the bind mount fails, the script stops before kairos-agent runs.

Adding /etc/resolv.conf to a fixed exclusion list would not have worked. kairos-agent builds the new image from scratch, so an excluded path is left out of it entirely, and images that ship their own /etc/resolv.conf would lose it.

Behaviour changes

  • The operator no longer always passes --exclude-path /etc/hostname --exclude-path /etc/hosts. Images that ship non-empty copies now get them installed. On images built with kairos-init, the boot stage in 31_hosts.yaml rewrites both files on every boot, so the result after reboot is unchanged. A custom image that ships a fixed, non-empty /etc/hostname and lacks that boot stage would now give upgraded nodes that hostname, as a fresh install of the same image already does.
  • spec.excludePaths is passed through as before, after the automatic exclusions.
  • --exclude-path needs kairos-agent v3.6.0+. That is now needed when spec.excludePaths is set or when the image lacks one of the three files. main passes the flag on every upgrade, so this is no stricter than today.
  • The ExcludePaths doc comment said excluded paths "on the host are preserved". kairos-agent leaves them out of the new image instead; the comment now says so. The CRD is generated with maxDescLen=0, so no manifest changes.

Tests

  • Unit tests run the generated script with mount and kairos-agent stubbed; the mount --bind stub fills the bind point with chosen image files. They check that the bind mount happens before kairos-agent and is the source of every kairos-agent call, that kairos-agent does not run if the bind mount fails, and which files are excluded for an image with all three files, with an empty one, with none, and with /etc/resolv.conf as a symlink whose target is missing or empty.
  • New e2e test (test/e2e/nodeopupgrade_test.go; no e2e test ran a NodeOpUpgrade before). It runs recovery-only upgrades, which do not reboot the kind node, with two busybox stand-in images and a fake kairos-agent that records what it finds in its upgrade source.
    • Image with marker /etc/resolv.conf, /etc/hosts and /etc/hostname: on main the source's /etc/resolv.conf is search default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5 and the test fails. With this PR the source holds the image's three files, nothing under /proc, and no exclusions are passed.
    • Image with none of the three files: the source shows the runtime's empty files, and all three are passed as --exclude-path.
    • Image whose /etc/resolv.conf is a relative link to ../run/systemd/resolve/stub-resolv.conf: the link's target in the source is the runtime's empty file. Without the symlink check the upgrade passed --exclude-path /etc/resolv.conf; with it nothing is excluded and the link reaches the source.
  • Full e2e suite locally: 31 of 37 passed (before the second e2e case was added). The 6 failures are OSArtifact raw, VHD and GCE disk builds failing with open /dev/loop0: no such file or directory inside the kind node. They ran before the NodeOpUpgrade test and do not touch the changed code. I did not rerun them on main.
  • Real upgrade on a Kairos VM. No CI job covers this before merge: the real upgrade test in kairos-io/kairos (tests/provider_upgrade_k8s_test.go) deploys the operator image pinned in config/manager/kustomization.yaml, currently the released v0.3.0. I ran that test locally from the ISO kairos-hadron-v0.6.3-standard-amd64-generic-v4.5.0-k3sv1.37.2+k3s1.iso, upgrading to quay.io/kairos/hadron:v0.6.3-standard-amd64-generic-v4.5.0-k3s-v1.37.2-k3s1, with the operator switched to an image built from this branch. I added read-only checks after the reboot; they are not part of this PR.
    • The upgrade ran kairos-agent upgrade --source dir:/tmp/tmp.d1wfVZB22R --exclude-path /etc/resolv.conf --exclude-path /etc/hostname (the Hadron image has no /etc/resolv.conf and an empty /etc/hostname), the node rebooted (boot ID changed), and k3s came back Ready.
    • After the reboot /etc/resolv.conf -> /run/systemd/resolve/resolv.conf, as before the upgrade, with nameserver 10.0.2.3 and no cluster.local; resolvectl reports resolv.conf mode: uplink; getent hosts quay.io resolves. Hostname and /etc/hosts are unchanged.
    • To see what the upgrade writes regardless of boot steps, later runs also mount the new system image (COS_ACTIVE) read-only after the reboot and read /etc/resolv.conf from it, and grep /run/immucore/immucore.log for the relink step.
    • Without the empty-file exclusions (operator built from the bind-mount commit alone): one run on stock Hadron ended with a working symlink because the relink step ran. With a Hadron image whose relink command is disabled, the immucore log showed the relink check failing ([ -f /etc/hosts ]: exit status 1) even though that image ships /etc/hosts, the new system had a 0-byte /etc/resolv.conf, systemd did not create its symlink, and getent hosts quay.io failed.
    • With this branch, same relink-disabled image: /etc/resolv.conf is left out of the new system, systemd's tmpfiles rule creates /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf, resolved runs in stub mode, and lookups work. So the branch works whether or not the relink step wins the race.
    • Released v0.3.0 operator, stock Hadron image: the new system image holds search operator-system.svc.cluster.local svc.cluster.local cluster.local, nameserver 10.43.0.10, options ndots:5 and has no /etc/hosts. The running node keeps that file, resolvectl reports resolv.conf mode: foreign with DNS server 10.43.0.10, and getent hosts quay.io fails. This is the reported bug. The kairos-io/kairos test still passes, since it only checks that the node is Ready.
    • Upgrade image with /etc/resolv.conf as a relative symlink to ../run/systemd/resolve/stub-resolv.conf: without the symlink check the new system image had no /etc/resolv.conf; with it the new system image holds the link as shipped. On Hadron the running node ends up the same either way, because the relink step rewrites /etc/resolv.conf at boot (uplink mode, lookups work).
    • Upgrade image with a regular /etc/resolv.conf (the Hadron image plus a marker file, the reporter's situation): with v0.3.0 the new system image holds the pod's file, as above. With this branch it holds the image's own file (# resolv.conf shipped in the upgrade image, nameserver 9.9.9.9), and the running node ends up with the systemd-resolved symlink, uplink mode, and working lookups.

The operator docs get a matching update in kairos-io/kairos-docs#794.

Environment

Arch Linux (kernel 7.2.11), Go 1.27.3, kind v0.17.0, envtest Kubernetes 1.37, QEMU with KVM for the VM test. Unit tests with make test, e2e with ginkgo -v ./test/e2e, VM test with ginkgo --label-filter provider-upgrade-k8s in kairos-io/kairos at b2697a28.

🤖 Generated with Claude Code

jimmykarily added a commit to kairos-io/kairos-docs that referenced this pull request Nov 12, 2026
When the upgrade image has no /etc/resolv.conf, /etc/hosts or
/etc/hostname, the container runtime creates an empty file there to
mount the Pod's copy onto. kairos-io/kairos-operator#314 leaves any of
the three that is missing or empty out of the new system, so the node
matches a fresh install of the image. For /etc/resolv.conf that keeps
images that rely on systemd to create the symlink at boot working.

The page now explains this, shows the resulting --exclude-path in the
example command, and says the kairos-agent v3.6.0 requirement also
applies when the image lacks one of the files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
jimmykarily and others added 3 commits November 12, 2026 13:44
After a NodeOpUpgrade and reboot, nodes could end up with a regular
/etc/resolv.conf holding the upgrade pod's DNS settings: the cluster DNS
service IP and the pod's search domains. The host then resolved names
through cluster DNS. CoreDNS pods using `forward . /etc/resolv.conf`
forwarded to themselves, reported a loop, and image pulls on the host
failed.

The upgrade runs kairos-agent inside a container of the target image and
copies the container's root filesystem into a new host image. Kubernetes
mounts pod-specific files over the image's /etc/resolv.conf, /etc/hosts
and /etc/hostname, and kairos-agent's rsync follows those mounts, so the
pod's files were copied instead of the image's. /etc/hosts and
/etc/hostname were already left out with --exclude-path;
/etc/resolv.conf was not.

The script now bind-mounts / (non-recursively) into a temporary
directory and passes that as the upgrade source. A plain bind mount shows
the container's root filesystem without anything mounted on top of it,
so the new host image gets the image's own copy of all three files, the
same as a fresh install of that image. The pod's own resolver settings
stay in place for the rest of the script.

The operator no longer adds --exclude-path for /etc/hostname and
/etc/hosts. Excluding a path leaves it out of the new image entirely;
with the bind mount the image's files are the right ones to copy, and
the Kairos boot stages that manage hostname and hosts see the same files
as after a fresh install. A side effect is that NodeOpUpgrade no longer
needs kairos-agent v3.6.0+ unless spec.excludePaths is set.

The ExcludePaths doc comment also said excluded paths "on the host are
preserved". kairos-agent builds the new image from scratch, so an
excluded path is missing from it rather than kept from the host; the
comment now says so.

Fixes kairos-io/kairos#6109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
No e2e test ran a NodeOpUpgrade. This one runs a recovery-only upgrade
(which does not reboot the kind node) with a stand-in image: busybox plus
marker /etc/resolv.conf, /etc/hosts and /etc/hostname, and a fake
kairos-agent that records what it finds under its upgrade source onto
the node.

The test checks that the source holds the image's three files and none
of the pod's other mounts, and that the upgrade container keeps its
cluster DNS settings. Against the previous upgrade script it fails with
the pod's resolver settings in the source, the same content reported in
kairos-io/kairos#6109.

The image ships /etc/resolv.conf as a regular file. With a symlink, the
container runtime mounts the pod's file at the link target under /run,
which the upgrade script hides by mounting the host's /run there, so the
leak would not show.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
When the upgrade image has no /etc/resolv.conf, /etc/hosts or
/etc/hostname, the container runtime creates an empty file at that path
in the container's writable layer, so it has something to mount the
pod's copy onto. The plain bind mount the upgrade copies from shows
those empty files, so the new system got them too, where a fresh
install of the image has none.

For /etc/resolv.conf that breaks DNS on images that rely on this
systemd tmpfiles rule to create the symlink at boot:

    L! /etc/resolv.conf - - - - ../run/systemd/resolve/stub-resolv.conf

The rule only acts when the path does not exist. Hadron v4.5.0 ships
the rule and no /etc/resolv.conf. Its kairos-init boot step that relinks
/etc/resolv.conf to /run/systemd/resolve/resolv.conf does not run on
every boot: it checks `[ -f /etc/hosts ]` while another initramfs step
replaces /etc/hosts concurrently, and the immucore log on a Kairos VM
showed that check passing on some boots and failing on others, even when
the image ships /etc/hosts. Upgrading to a Hadron image with the relink
command disabled and the empty file copied left /etc/resolv.conf empty,
and the host could not resolve names.

The upgrade script now checks the three files in the bind mount and
passes --exclude-path for each one that is missing or empty. The new
system then matches a fresh install of the image, and on Hadron the
tmpfiles rule creates the stub-resolv.conf symlink when the relink step
does not run (verified on a Kairos VM). An image that ships one of the
files empty on purpose loses nothing, since an empty file and a missing
one mean the same for these three.

An --exclude-path flag needs kairos-agent v3.6.0+, so a NodeOpUpgrade to
an image that lacks one of the files needs that version, as it did
before the bind mount.

The e2e test gains a second stand-in image with none of the three files.
Before this change the upgrade source showed the runtime's empty files
and kairos-agent got no exclusions; now all three are excluded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
@jimmykarily
jimmykarily force-pushed the fix/upgrade-pod-resolv-conf branch from 0e4828f to 8d209d4 Compare November 12, 2026 11:44
@jimmykarily
jimmykarily marked this pull request as ready for review November 12, 2026 11:48
@jimmykarily
jimmykarily requested a balanced review from Copilot November 12, 2026 11:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The resolver-file check can omit a shipped /etc/resolv.conf symlink when its target is absent or empty, potentially leaving upgraded nodes without DNS; symlink preservation and a dangling-symlink regression test are needed.

1 open issue
What changed in this PR

Updates NodeOpUpgrade to use the target image’s filesystem rather than Kubernetes-mounted pod files, addressing DNS configuration leakage during upgrades.

Changes:

  • Uses a non-recursive bind mount as the source for active and recovery upgrades.
  • Replaces fixed exclusions with checks for missing or empty files, while preserving user-configured exclusions.
  • Adds unit and end-to-end coverage and clarifies exclusion semantics.
File Description
test/​e2e/​nodeopupgrade_test.go Adds upgrade-source checks for images with and without network configuration files.
internal/​controller/​nodeopupgrade_controller.go Prepares the isolated source and calculates automatic exclusions.
internal/​controller/​nodeopupgrade_controller_test.go Tests source selection, exclusions, ordering, and bind-mount failure handling.
internal/​controller/​defaults.go Removes the fixed hosts and hostname exclusion list.
api/​v1alpha1/​nodeopupgrade_types.go Clarifies what exclusions omit and when agent version requirements apply.

Review effort: Balanced

The upgrade script leaves /etc/resolv.conf, /etc/hosts and /etc/hostname
out of the new system when they are empty in the bind mount, using
`[ ! -s ]`. That test follows symlinks, so an image that ships
/etc/resolv.conf as a symlink could lose it:

- A relative link, such as ../run/systemd/resolve/stub-resolv.conf,
  resolves inside the bind mount. When the container runtime mounts the
  pod's resolv.conf it follows the link and creates an empty file at
  the target, so the link looked empty and was excluded.
- An absolute link resolves against the upgrade container's /run, which
  the script has replaced with the host's, so the result depended on
  the host.

The check now skips symlinks (`[ ! -L ]`), so a symlink is always copied
as is and only a missing file or an empty regular file is excluded.

The e2e test gains a third stand-in image whose /etc/resolv.conf is a
relative link to systemd-resolved's stub file. Before this change the
upgrade passed --exclude-path /etc/resolv.conf for it; now nothing is
excluded and the link reaches the upgrade source.

Reported by Copilot review on #314.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
jimmykarily added a commit to kairos-io/kairos-docs that referenced this pull request Nov 12, 2026
kairos-io/kairos-operator#314 only leaves /etc/resolv.conf, /etc/hosts
and /etc/hostname out of the new system when they are missing or empty
regular files. A symlink is copied as is, so an image that ships
/etc/resolv.conf as a link to systemd-resolved's file keeps it. The page
said "missing or empty", which reads as if an empty or dangling link
target would also exclude the link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
@jimmykarily jimmykarily moved this from QA to Under review 🔍 in 🧙Issue tracking board Nov 13, 2026
@jimmykarily
jimmykarily requested a review from a team November 13, 2026 13:37

@wrkode wrkode left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wrkode
wrkode merged commit 4f89bb5 into main Nov 17, 2026
6 of 7 checks passed
@wrkode
wrkode deleted the fix/upgrade-pod-resolv-conf branch November 17, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NodeOpUpgrade copies Pod resolv.conf into host OS, causing CoreDNS loops and image-pull failures

3 participants