Skip to content

fix(kairos-init): retry a package manager command that failed - #6070

Closed
ci-robbot wants to merge 1 commit into
kairos-io:masterfrom
ci-forks:ci-doctor/yip-package-retry
Closed

ci-robbot wants to merge 1 commit into
kairos-io:masterfrom
ci-forks:ci-doctor/yip-package-retry

Conversation

@ci-robbot

Copy link
Copy Markdown
Collaborator

Part of #5560

Summary

The master workflow is red on fa68014 (run https://github.com/kairos-io/kairos/actions/runs/42849163678), and the cause is not the commit. A distro mirror was mid-sync while kairos-init ran its package step, and that step runs each package manager command once:

ubuntu-advantage-tools : Depends: ubuntu-pro-client (>= 38ubuntu0~20.04) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

The build-flavors / ubuntu:20.04 (amd64) job of the same run installed the same package set without error. This is a flake that keeps coming back. Package mirror failures in kairos-init failed master 5 times across 4 runs in the last 50 commits:

Run Job Error
42849163678 build-iso amd64-core-ubuntu-20.04 focal ubuntu-pro-client unmet dependency
42840392751 build-flavors ubuntu:24.04 (amd64) +uki perl-base_5.38.2-3.2ubuntu0.7_amd64.deb 404 Not Found
42783634848 build-flavors ubuntu:22.04 (amd64) +k3s perl-base_5.34.0-3ubuntu1.10 404
42783634848 build-flavors almalinux:9 (arm64) repomd.xml parser error ... Document is empty
42596158930 build-flavors almalinux:9 (arm64) same appstream repomd.xml error

mudler/yip#350 (merged 2026-10-27) adds the retry: the packages plugin runs each refresh, upgrade, install and remove command up to three times, waiting 5 and 15 seconds in between. No yip tag contains it yet (latest is v1.26.4), so kairos still runs the old code. The logs confirm this: the failing apt-get install ran once.

What changed

Limits

  • A retry helps when the archive or mirror catches up within about 20 seconds: the 404 and empty repomd.xml cases.
  • It may not fix the focal case. That error comes from a stale package index, and the retry runs apt-get install again without a new apt-get update.

How it was tested

  • CGO_ENABLED=0 go build ./... passes.
  • After make kairos-init-embed-stubs, go test ./kairos-init/... ./sdk/... passes.
  • I checked that runPackageCommand with packageAttempts = 3 is in the module that Go resolves.
  • I could not reproduce a mirror failure here, because this machine has no access to the distro archives. The flavor builds of this PR exercise the new code path.

AI disclosure: the ci-doctor bot (Claude) diagnosed this, wrote the change and opened this PR. No human read the code before it was opened.

🤖 Generated with Claude Code

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Check Result
Environment PR head 99d6c2e14 (yip v1.26.5-0.20261027073353-b1c67bef913c) against master fa6801435 (yip v1.26.4). kairos-init built from each with CGO_ENABLED=0 after make kairos-init-embed-stubs, run in alpine:3.22 and ubuntu:24.04 containers on amd64
Reproduction Real kairos-init -s install with every step but installPackages skipped, -l debug. A shim at /usr/local/sbin/apk sends the first N apk add calls through an unreachable proxy (127.0.0.1:9), so the real apk fails to fetch, the way a dead mirror does. Later calls go to the real mirror. N = 1, 2, 3
Expected A failed install is retried up to 3 times, 5 s then 15 s apart, and the build succeeds once the mirror answers. 3 failures still fail the build. Master fails on the first error
Actual See below. Matches on every row
Proofs Log excerpts below
Binary Failed apk add calls Attempts Exit Packages after (16 before)
master 1 1 1 16
PR 1 2 (retry after 5 s) 0 299
PR 2 3 (5 s, then 15 s) 0 299
PR 3 3 1, Package install failed after 3 attempts 16

PR, 2 failures:

19:03:53 shim: apk add call #1
19:03:58 shim: apk add call #2
19:04:13 shim: apk add call #3
DBG [10] Running install (attempt 1 of 3)
WRN [10] Package install failed: failed to run apk add --no-cache file gawk iptables ...
DBG [10] Running install (attempt 2 of 3)
WRN [10] Package install failed: failed to run apk add --no-cache file gawk iptables ...
DBG [10] Running install (attempt 3 of 3)
EXIT=0
packages before=16 after=299

Master, 1 failure:

19:03:45 shim: apk add call #1
DBG [10] Running install
ERR Failed to run the install stage: 1 error occurred:
EXIT=1
packages before=16 after=16

On ubuntu:24.04 the refresh step hit real 403 Forbidden errors from the archive (this host's container egress proxy blocks Ubuntu mirrors), and the PR build retried it as intended: Running refresh (attempt 1 of 3), Retrying in 5s, attempt 2 of 3, Retrying in 15s, attempt 3 of 3, then exit 1.

CGO_ENABLED=0 go test ./kairos-init/... passes on the PR head.

Limits:

  • I could only show recovery with apk. The proxy blocks Ubuntu and Fedora mirrors, so a successful apt or dnf install after a retry was not run. All package managers go through the same runPackageCommand in yip.
  • The focal ubuntu-pro-client unmet-dependency case was not reproduced. As the PR body says, a retry runs apt-get install again without a new apt-get update, so it may not fix a stale index.
  • The bump also brings in yip Add entangle support to discovery #345 (executor), 📖 Update raspberry.md #339 (dns) and 📖 Update contributing icons #340 (docs). I did not exercise those on a booted node.
  • unit-tests / test was still running when this was posted. qa-gate is red only because the label was missing.

Summary: With this bump, kairos-init retries a failed package command for real: one or two mirror failures in a row no longer fail the build, three still do. Master fails the build on the first failure.

@ci-robbot ci-robbot moved this from QA to QA OK in 🧙Issue tracking board Nov 10, 2026
itxaka-agent
itxaka-agent previously approved these changes Nov 11, 2026

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Approving. This is a pure dependency bump: go.mod moves yip to v1.26.5-0.20261027073353-b1c67bef913c, which is the mudler/yip#350 merge commit (b1c67bef913c, committed 2026-10-27 07:33:53Z, matching the pseudo-version timestamp). go.sum swaps the yip sums and drops stale entries. No code changes, CI is green apart from qa-gate, which waits on the QA label.

Note it also pulls in yip#345, #339 and #340. Renovate can move it to a tag once one is cut.

@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch from 99d6c2e to 35b93fc Compare November 11, 2026 12:18
@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch 2 times, most recently from 9ef1207 to 182efb4 Compare November 11, 2026 15:33
@ci-robbot ci-robbot closed this Nov 11, 2026
@github-project-automation github-project-automation Bot moved this from QA OK to Done ✅ in 🧙Issue tracking board Nov 11, 2026
@ci-robbot ci-robbot reopened this Nov 11, 2026
@github-project-automation github-project-automation Bot moved this from Done ✅ to Under review 🔍 in 🧙Issue tracking board Nov 11, 2026
@ci-robbot ci-robbot moved this from Under review 🔍 to QA OK in 🧙Issue tracking board Nov 11, 2026
@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch from 182efb4 to c8b7e51 Compare November 11, 2026 21:19

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Approving at c8b7e51. The diff is only the yip bump in go.mod and go.sum. The pseudo-version points at the mudler/yip#350 merge (b1c67bef913c), and the go.sum hashes match the Go checksum database. CI is green.

@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch 3 times, most recently from e4f0e43 to 8b92a69 Compare November 16, 2026 06:26
@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch 4 times, most recently from c700ce7 to 7fa8bcd Compare November 17, 2026 15:26
Bump github.com/mudler/yip to b1c67bef913c, the merge of mudler/yip#350.
The packages plugin now runs each refresh, upgrade, install and remove
command up to three times, waiting 5 and 15 seconds, instead of failing
the whole kairos-init stage on one mirror that is mid-sync.

yip has no tag with that commit yet, so this is a pseudo-version.
Renovate moves it to the next tag. go mod tidy also drops stale sums.

Part of kairos-io#5560

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot
ci-robbot force-pushed the ci-doctor/yip-package-retry branch from 7fa8bcd to 48f6151 Compare November 18, 2026 09:19
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Superseded by d9a5c75 (#6122), which bumps yip to v1.27.0 and already contains the package manager retry from mudler/yip#350.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants