Skip to content

fix(executor): keep the file order across a config with no steps - #355

Merged
jimmykarily merged 1 commit into
masterfrom
fix/dirops-chain-across-empty-files
Nov 13, 2026
Merged

jimmykarily merged 1 commit into
masterfrom
fix/dirops-chain-across-empty-files

Conversation

@jimmykarily

Copy link
Copy Markdown
Collaborator

What happens

yip runs the configs in a directory in name order: each step of a config waits for the steps of the config before it. When a config in the middle has no steps for the stage being run, that ordering breaks. The next config waits for nothing and runs at the same time as every config before it.

On Kairos this makes the boot nondeterministic. /system/oem/30_ulimit.yaml has no initramfs steps, so during the immucore initramfs stage 31_hosts.yaml runs concurrently with 05_network.yaml:

  • 05_network.yaml relinks /etc/resolv.conf to /run/systemd/resolve/resolv.conf only if: '[ -f /etc/hosts ]'.
  • 31_hosts.yaml runs rm -f /etc/hosts and then ln -sf /usr/local/etc/hosts /etc/hosts.

Whether the relink runs depends on which of the two gets there first. On a Kairos VM the immucore log (/run/immucore/immucore.log) showed the two files' steps interleaved, and the relink check passing on some boots and failing on others, including boots where the system image ships /etc/hosts. A node can come up with a different /etc/resolv.conf from one boot to the next. This made kairos-io/kairos#6109 (an upgrade leaving the Kubernetes pod's /etc/resolv.conf on the host) show up on most boots instead of being masked by the relink.

Cause

In dirOps (pkg/executor/default.go), prev holds the steps of the previous config so the next config can depend on them. It was set to the current config's steps unconditionally, so a config with no steps for the stage set it to an empty list, and the next config got no dependencies.

Fix

A config with no steps for the stage now leaves prev alone, so the next config that has steps waits for the last one that had any.

Tests

New specs in pkg/executor/default_test.go, with 01_first.yaml (a test step that sleeps 0.5 s), 02_other_stage.yaml (steps only for another stage) and 03_last.yaml (a test step):

  • Graph puts 03_last.yaml's step in a later layer than 01_first.yaml's. Before the fix both were in layer 1.
  • Run writes first then last. Before the fix it wrote last then first.
go run github.com/onsi/ginkgo/v2/ginkgo --race --focus "no steps for the stage" ./pkg/executor/

On my machine the user-management specs in pkg/executor and pkg/plugins fail with open /run/lock/foo.lock: permission denied, with and without this change; they need a writable /run/lock, which the Ubuntu CI runners have.

On a Kairos VM

I built the kairos binary (which includes immucore) from kairos-io/kairos v4.5.0 twice, once as is and once with this change applied to yip v1.26.4 (the version v4.5.0 uses; the change applies by hand there because v1.26.4 predates #345). Each went into the Hadron v4.5.0 k3s image with a regenerated initrd, and a VM upgraded to it through kairos-operator.

  • Unpatched: 31_hosts.yaml steps start in the middle of 05_network.yaml's (log line 112, between lines 49 and 147).
  • Patched: every 05_network.yaml step (lines 56 to 129) logs before the first 31_hosts.yaml step (line 167), and the relink check result depended only on whether the system image had /etc/hosts.
  • immucore's initramfs stage took 310 ms unpatched and 327 ms and 330 ms in two patched boots (/run/immucore/boot-timeline.json, one boot each).

Who is affected

Anyone running yip on a directory where a config without steps for the stage sits between configs that have some.

Environment

Arch Linux (kernel 7.2.11), Go 1.27.3, QEMU with KVM. yip at b1c67be plus this commit (702ccb8); VM tests with kairos-io/kairos b2697a28 tests and the Hadron v4.5.0 k3s ISO.

🤖 Generated with Claude Code

yip runs the configs in a directory in name order: every step of a
config waits for the steps of the config before it. A config that has
no steps for the stage being run reset that chain, so the next config
waited for nothing and ran concurrently with every config before it.

Kairos ships /system/oem/30_ulimit.yaml with no initramfs steps, so
31_hosts.yaml ran concurrently with 05_network.yaml during the initramfs
stage. 05_network.yaml relinks /etc/resolv.conf only when this check
passes:

    if: '[ -f /etc/hosts ]'

while 31_hosts.yaml runs `rm -f /etc/hosts` and then recreates it as a
symlink. On a Kairos VM the immucore log showed the
two files' steps interleaved and the relink check passing on some boots
and failing on others, so the same node could come up with a different
/etc/resolv.conf from one boot to the next.

A config with no steps for the stage now leaves the chain alone, so the
next config that has steps waits for the last one that had any.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the fix preserving execution order across configs with no steps for the requested stage.

No open issues 🎉

What changed in this PR

Fixes yip’s directory execution ordering when an intervening config has no steps for the requested stage, preventing later configs from running concurrently with earlier work.

Changes:

  • Preserve dependencies on the last config with steps when skipping an inactive config.
  • Add regression tests for graph layer ordering and runtime execution order.
File Description
pkg/​executor/​default.go Preserve the dependency chain across configs with no steps for the selected stage.
pkg/​executor/​default_test.go Verify graph and execution ordering across an inactive middle config.

Review effort: Balanced

itxaka-agent added a commit to itxaka-agent/triage-assets that referenced this pull request Nov 12, 2026
Co-authored-by: Itxaka <itxakaserrano@gmail.com>
Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
@itxaka-agent

Copy link
Copy Markdown

QA: pass at 702ccb8

Built two core Hadron ISOs from kairos master b2697a28, one with yip at the base (b1c67be) and one at this head (702ccb8), checked with go version -m on the kairos binary. Installed each to disk in QEMU and booted the installed system 5 times each with rd.immucore.debug, reading /run/immucore/immucore.log. Line numbers are inside the initramfs stage.

boot first 05_network last 05_network first 31_hosts resolv.conf relink
base 1 875 1167 841 ran
base 2 855 1124 859 ran
base 3 848 1116 860 ran
base 4 874 1122 870 ran
base 5 853 1113 900 skipped
head 1 939 997 1293 ran
head 2 to 5 948 1006 1265 ran

Base interleaved on every boot. On base boot 5, 31_hosts ran its rm -f /etc/hosts right before the [ -f /etc/hosts ] check, the relink was skipped and the node came up with /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf instead of /run/systemd/resolve/resolv.conf. On head every 05_network step finished before the first 31_hosts step on all 5 boots, the relink always ran, and all boots reached running with no failed units.

Unit tests (golang:1.26 container): both new specs fail on base code (layers showed 01_first and 03_last both in layer 1) and pass on head. Full go test -race ./pkg/executor/... ./pkg/schema/... passes on head.

boots
unit

Signed-off-by: itxaka-agent itxaka-agent@users.noreply.github.com

@jimmykarily
jimmykarily merged commit 8aa3fcf into master Nov 13, 2026
7 checks passed
ci-robbot pushed a commit to ci-forks/kairos that referenced this pull request Nov 18, 2026
yip v1.27.0 runs the config files of a stage directory in name order
again when one of them has no steps for that stage. Before, such a file
let the next file start alongside the earlier ones. On Hadron,
30_ulimit.yaml has no initramfs steps, so 31_hosts.yaml ran alongside
05_network.yaml, and whether 05_network.yaml linked /etc/resolv.conf to
systemd-resolved depended on which finished first
(mudler/yip#355, kairos-io#6109).

The release also carries the other yip fixes since v1.26.4: a config
that cannot be read or parsed no longer drops the rest of the
directory, the dns plugin writes the file the stage names instead of a
symlink's target, and package manager commands are retried. The other
go.mod changes are indirect dependencies yip v1.27.0 requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants