Skip to content

fix(ci): take the encryption deps off luet 0.33.0 and retry a failed pull - #5124

Open
ci-robbot wants to merge 5 commits into
kairos-io:masterfrom
ci-forks:bot/5123-encryption-deps-retry
Open

ci-robbot wants to merge 5 commits into
kairos-io:masterfrom
ci-forks:bot/5123-encryption-deps-retry

Conversation

@ci-robbot

@ci-robbot ci-robbot commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

encryption-* legs die in the Encryption deps step, in three different ways
that all come from the same six lines.

1. The step pinned luet 0.33.0, released in October 2022

That build does not survive a failed pull. When a blob from quay arrives
truncated it logs the download as a warning, leaves the artifact nil, and then
dereferences it:

 WARNING   failed downloading image quay.io/kairos/packages:kubectl-container-1.31.0-1: unexpected EOF
panic: runtime error: invalid memory address or nil pointer dereference
	artifact.(*PackageArtifact).Verify(0x0, ...)
	  pkg/api/core/types/artifact/artifact.go:105
	installer.(*LuetInstaller).getPackage(...)
	  pkg/installer/installer.go:963

That took master red on b3cd998 (run 37599588356) and again on 3dd0f1f
(run 37740630820), on a different encryption leg each time.

luet 0.37.0 returns the download error instead (mudler/luet#393), and
quay.io/kairos/packages is itself published with luet 0.36.0, so 0.33.0 was
the one version nothing else in the project used.

No built-in renovate manager scans a release binary fetched by URL, which is
how the pin stayed four years behind. The version moves into a LUET_VERSION
variable behind a # renovate: comment, and renovate.json gets a custom
manager that reads that shape in any workflow file. It is the same matcher the
repository already uses for .sh files, with the quoted name:tag form
replaced by a bare version.

2. A transient failure still has no retry

A clean error is still a red leg. Run 36871582383 on #5111 lost
encryption-remote-https-pinned while the other 13 encryption legs of the same
run, against the identical repository, all passed. The step now tries three
times. The repository index and the partial artifact are both cached, so the
cache is dropped between attempts or the next one reads the same bytes back.

The other half of that belongs in luet, not here: mudler/luet#404. luet was
reporting a layer it could not fetch as a successful unpack of an empty image,
because go-containerregistry closes the tar writer before closing the
extraction stream with the error, so the unpack stops at the end-of-archive
marker and never reads it.

3. Neither curl checked its status

Both calls get -f and --retry. Without -f, curl writes the body of an
HTTP error to the output file and still exits 0, so a bad gateway from the
release host installs an error page as /usr/bin/luet, or pipes one into
sudo bash for cinc-auditor.

How it was tested

There is no runner for a workflow change, so each piece was checked on its own:

  • Downloaded luet-0.37.0-linux-amd64 from the release URL this builds, ran
    luet --version, and confirmed repo add writes the same repository file as
    0.33.0 apart from an added empty auth: block, and that install takes the
    same flags.
  • Read quay.io/kairos/packages with 0.37.0. It reaches the registry and
    resolves the repository. The install itself cannot finish off a runner,
    because unpacking needs to lchown as root.
  • Confirmed the nil guard this depends on is in the 0.37.0 tag, not only on
    luet master.
  • Compiled every matchStrings entry in renovate.json with Go's regexp,
    which is the same RE2 engine renovate uses, and ran them over every workflow
    file. The new manager matches once, in this file, and reports
    datasource=github-releases depName=mudler/luet currentValue=0.37.0.
  • Parsed the workflow with yaml.v3 and ran bash -n over all 11 of its run
    blocks.

Fixes #5123

Written with AI assistance. No human has read it yet.

🤖 Generated with Claude Code

…the leg

luet pulls quay.io/kairos/packages:repository.yaml on first use. When that
pull returns an empty blob it resolves against an empty tree and reports
the packages as not found, so the step dies with a message that reads like
a repository content problem. Run 36871582383 lost
encryption-remote-https-pinned this way while the other 13 encryption legs
of the same run passed.

The repository is cached, so retrying on its own reads the same empty index
back. Drop /var/cache/luet/repos/kairos between attempts, three attempts
with a growing delay, and keep the non-zero exit when all three fail.

Give both curl calls -f and --retry as well. Without -f curl writes the
body of an HTTP error to the output file and still exits 0, so a bad
gateway from the release host installs an error page as /usr/bin/luet, or
pipes one into sudo bash for cinc-auditor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot
ci-robbot requested a review from a team as a code owner October 2, 2026 02:00
mudler/luet#404 makes luet report a pull that unpacked nothing as the download
failure it is, and pulls the image again before giving up, so the workflow has
no reason to loop over `luet install` and clear the repository cache itself.

What stays is the `curl` hardening, which is a separate fault: without `-f`
curl writes the body of an HTTP error to the output file and still exits 0, so
a bad gateway from the release host installs an error page as /usr/bin/luet,
or pipes one into `sudo bash` for cinc-auditor.

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ci-robbot ci-robbot changed the title fix(ci): retry the encryption test deps, an empty luet index reddens the leg fix(ci): make a failed download of the encryption test deps fail Oct 2, 2026
@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Taking a look at this PR now and will post a review shortly.

itxaka-agent
itxaka-agent previously approved these changes Oct 2, 2026

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Looks good. Adding -f stops an HTTP error page from being saved as /usr/bin/luet, and the retries cover short outages on the release host. The flags match what release-scan.yml already uses.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Audit summary: one review round on head c220fab. Checked the workflow change to the two curl calls; no findings, approved. Nothing was run locally (static review only).

… pull

c220fab dropped the retry on the premise that mudler/luet#404 had made it
unnecessary. That pull request is still open, and this step pins the luet
binary released in 2022, so merging it upstream would not change what runs
here either.

luet 0.33.0 leaves the artifact nil when a pull fails and then dereferences
it, so a truncated blob from quay is a SIGSEGV rather than an error:

  WARNING failed downloading image quay.io/kairos/packages:kubectl-container-1.31.0-1: unexpected EOF
  panic: runtime error: invalid memory address or nil pointer dereference
  artifact.(*PackageArtifact).Verify(0x0, ...)

That is the only failing job in run 37599588356 on master b3cd998.

Three attempts with a growing delay, dropping /var/cache/luet in between:
both the repository index and the partial artifact are cached, so a retry
that keeps the cache reads the same bytes back. The non-zero exit after the
third attempt stays, so a real outage still reddens the leg.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Restored the retry around luet install. The commit that dropped it assumed mudler/luet#404 had made it unnecessary, but that PR is still open and this step pins the luet binary released in 2022, so merging it upstream would not change what runs here.

Evidence it is still needed: run 37599588356 on master b3cd9981 has exactly one failing job, encryption-remote-ek-only, and it is this step crashing on a truncated pull from quay (unexpected EOF then artifact.(*PackageArtifact).Verify(0x0)). New head 8805b872.

mudler and others added 2 commits October 7, 2026 12:10
kairos-io#5277 renamed the cinc-auditor step to "for the InSpec profile tests" and
gated it on the new `cis` test as well, in the same region where this branch
replaces the single `luet install` line with the retry loop. Both sides kept:
master's step name and condition, this branch's loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
The encryption deps step downloaded luet 0.33.0, released in October 2022.
That build does not survive a failed pull: when a blob from quay arrives
truncated it logs the download as a warning, leaves the artifact nil, and
then dereferences it.

  WARNING failed downloading image quay.io/kairos/packages:kubectl-container-1.31.0-1: unexpected EOF
  panic: runtime error: invalid memory address or nil pointer dereference
  artifact.(*PackageArtifact).Verify(0x0, ...)
  installer.(*LuetInstaller).getPackage(...)

That is what took master red on b3cd998 (run 37599588356) and again on
3dd0f1f (run 37740630820), on a different encryption leg each time.

luet 0.37.0 returns the download error instead of panicking, and
quay.io/kairos/packages is published with luet 0.36.0, so the consumer
was the one version nothing else in the project used.

No built-in renovate manager scans a release binary fetched by URL, which
is how the pin stayed four years behind. The version now lives in a
LUET_VERSION variable behind a renovate comment, and a custom manager
reads that shape in any workflow file.

The retry loop stays. A clean error is still a red leg.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Round 1 at 236736c: approve. The exit 0 inside the retry loop ends the step cleanly, three failed attempts end in exit 1, and the new Renovate matcher captures 0.37.0 from the LUET_VERSION line without overlapping the existing manager.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Tested head 236736c5. It merges cleanly onto master 8c24a223, and master's Encryption deps step is still the old six lines, so the hunk applies as written.

The PR's own CI does not run this step: the encryption legs are not in the PR matrix, and reusable-qemu-test.yaml would come from master anyway. So I extracted the run: of Encryption deps from the PR and from master with yq, and ran each under bash --noprofile --norc -eo pipefail in a clean fedora:44 container as root, with a sudo shim. Real downloads from GitHub releases and quay.io/kairos/packages. To reproduce the failure from #5123, I put a CONNECT proxy in front that cuts a TLS tunnel after 2 MiB, either for quay hosts only or for githubusercontent only, for a chosen number of tunnels.

Check Result
Environment fedora:44 container, amd64, luet 0.33.0 (master) and 0.37.0 (PR), renovate/renovate local dry run
Clean network master and PR both install kubectl v1.31.0 and k3d v5.8.3, exit 0
A: master, 1 quay blob cut Reproduces the bug: WARNING failed downloading image ...k3d-utils-5.8.3: unexpected EOF, then panic: ... artifact.(*PackageArtifact).Verify(0x0, ...), exit 2
B: PR, 1 quay blob cut No panic. luet 0.37.0 logs the error, downloads the blob again and installs both packages on attempt 1, exit 0
C: PR, every quay blob cut luet install failed on attempt 1/2/3, dropping the luet cache, then luet install did not succeed in 3 attempts, exit 1. Clean error, no panic
D: PR, 4 cuts (attempt 1 fails) Attempt 1 fails, cache dropped, attempt 2 installs both packages, exit 0, kubectl and k3d run
E: luet binary download cut once master: curl: (56) ... unexpected eof, step exit 56. PR: curl retries after 2 s, gets luet 0.37.0, step exit 0
curl -f on a 404 old curl -L: exit 0, writes the 9-byte Not Found body to the output file. new curl -fL: exit 22, no file
Renovate manager --dry-run=lookup --enabled-managers=custom.regex over the PR tree: the new manager matches only mudler/luet currentValue=0.37.0 in reusable-qemu-test.yaml (latest, no update). With the pin set to 0.36.0 it proposes newValue 0.37.0, branch renovate/mudler-luet-0.x
Proofs https://gist.github.com/ci-robbot/d0b23b98e61ecc01fb3450ac9a29c5d7 (all run logs, the extracted steps, the proxy and the runner script, Renovate lookups)

Summary: Under the same truncated quay download, master's step panics exactly as in #5123 and the PR's step either recovers inside luet 0.37.0 or through the new 3-attempt loop. When the registry stays broken it fails with a clear message. The curl changes and the Renovate manager also work as described.

Not covered: a real GitHub runner run of an encryption-* leg. Out of scope, but noticed: when a download fails, luet 0.37.0 reports it as Error: file conflict found: Failed downloading package: .... The "file conflict" part is misleading.

@ci-robbot ci-robbot moved this from QA to QA OK in 🧙Issue tracking board Oct 26, 2026

This branch had an error being deployed

1 failed (outdated) deployment
fork-pr-builds — c220fab9 Deployed Oct 2, 2026 by ci-robbot via authorize #996
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: QA OK

Development

Successfully merging this pull request may close these issues.

CI: the encryption test deps step has no retry, so one empty luet index pull reddens the whole PR

3 participants