Reusable base of an OpenWrt package CI: build a package with the official SDK,
test it in QEMU and build firmware images with it using the official Image
Builder. This repository holds only the base (reusable workflows, scripts,
pinned tooling, the generic package test) and its own CI, which tests the base
with a sample package (chrony). The packages themselves, their
package-specific tests and the planned upstream pull request automation live in
wehagy/openwrt-ci-packages,
which calls the workflows below.
| Done | Pending |
|---|---|
SDK build (build.yml) on lines 25.12 and main, architectures given by the caller |
NetBird runtime tests (caller repository) |
| Own SDK image on GHCR, rebuilt when older than 1 day or on request | Scheduled SDK image refresh (none today: lazy only) |
shfmt v3.14.1 (release binary, sha256 pinned) installed in the SDK image; the job fails if the image has no shfmt |
docker/sdk-image/Dockerfile, step "Check that the SDK image has shfmt" of .github/workflows/build.yml |
QEMU test (test.yml) on lines 25.12 and main, sources feed and artifact |
Upstream PR automation and testing (caller repository) |
QEMU targets x86-64, armsr-armv8, malta-le, malta-be on both lines |
|
Image build (images.yml) on lines 25.12 and main, profiles and packages given by the caller |
|
ci.yml: build both lines, then test each build and build its images |
|
Callable from another repository: ci_ref, tests_path, test_secrets, sdk_container |
harness/lines.yml: per-line data.release_sourceselects how the image version is found:releases(25.12: latest25.12.xof the releases index) orsnapshot(main: theversion.buildinfoof the target directory, e.g.r37969-5799d2da76). Both lines also have the build keys (sdk_tag_suffix,image_grace_days,packages_branch,imagebuilder_tag_suffix).images.yml: the globs of the files kept as artifacts. The image choices of a caller (profiles, packages) are inputs ofimages.yml, not data of this repository; the base has no default profile.targets.yml: per-QEMU-target data (x86-64,armsr-armv8,malta-le,malta-be) and the image name per line.selftests/: pytest tests of the harness helpers (no VM).tests/: pytest tests run inside the VMs through aparcar/openwrt-tests.openwrt-tests.uv.lock: seed lock of the Python environment of those tests (see "Pinned test environment").
scripts/(POSIX shell, used by the workflows)- SDK and Image Builder images (the daily refresh writes them, the builds
read them):
image-refresh.sh,refresh-plan.sh,resolve-refresh.sh,resolve-sdk.sh,image-decide-selftest.sh,ghcr-versions-selftest.sh; the builds' side:sdk-image-info.sh,pull-imagebuilder.sh,image-stale-check.sh; self-tests with stub registry, docker, gh and curl (lib/image-stubs.sh):image-refresh-selftest.sh(the writer),image-read-selftest.sh(the readers),workflow-permissions-selftest.sh(onlysdkandibofrefresh-images.ymlhavepackages: write).lib/image-decide.sh(rebuild/reuse and stale rules),lib/image-upstream.sh(the official container's sha256sums). - Build:
resolve-build.sh,collect-apk.sh,build-summary.sh; feed source:resolve-feed.sh,feed-record.sh,feed-inputs-selftest.sh. - Test:
validate-inputs.sh,caller-inputs-selftest.sh,matrix.sh,resolve.sh,fetch-image.sh,fetch-package.sh,prepare-target.sh,run-tests.sh,summary.sh. - Image inputs:
resolve-profiles.sh(profiles againstprofiles.json),resolve-archs.sh(architecture matrix ofbuild.yml),image-inputs-selftest.sh,lib/image-inputs.sh(validators). - Images:
resolve-image.sh,pull-imagebuilder.sh,imagebuilder-make.sh(runs inside the Image Builder container) withib-setup.shandib-setup-selftest.sh,ib-pins-selftest.sh,check-image-outcome.sh,collect-images.sh,image-summary.sh. lib/common.sh: shared helpers.lib/selftest.sh(case bookkeeping of the self-tests),lib/ghcr.sh(registry reads),lib/image-decide.sh(rebuild/reuse and stale decisions),lib/sdk-labels.sh(image label keys),lib/ghcr-versions.sh(delete of a package version),lib/ib-pins.sh(install of the built.apkby path and the check).
- SDK and Image Builder images (the daily refresh writes them, the builds
read them):
docker/sdk-image/Dockerfile,docker/ib-image/Dockerfile: derive the SDK image and the snapshot Image Builder image from the official containers (setup.sh run once)..github/workflows/build.yml: reusable SDK build.test.yml: reusable QEMU test.images.yml: reusable Image Builder build.ci.yml: the base's own CI, calls the three workflows above for the sample packagechronyon push tofeat/**.
scripts/also has:report-revisions.sh(prints the harness and caller commits of a job),install-tests.sh(copies the pytest files),with-test-secrets.sh(exports the caller secrets to the tests only).harness/sample-tests/: caller tests of this repository, used byci.yml(see "Package tests").
- Untrusted feed code (the
openwrt/packagestree, or the repository offeed_repository) runs only in thesdkjob ofbuild.yml, inside the SDK container, withpermissions: contents: readand no secrets. The container receives only the variables listed in the--envflags of thedocker runofopenwrt/gh-action-sdk(none ofACTIONS_RUNTIME_TOKEN,ACTIONS_CACHE_URL,GITHUB_TOKEN). - The caller secrets (
test_secrets) exist only in thetest.ymljob, and reach only the pytest process. That process runs code nobody here reviewed at the time of the run: thetests/,conftest.pyand strategies of any push tomainofaparcar/openwrt-tests(the base follows that branch, see "Pinned test environment"), and labgrid and the other packages from PyPI (the seed lock limits the others, not labgrid within its major). A hostile push there, or a hostile release, could read the named secrets and writeGITHUB_ENVfor the later steps of that job. Accepted risk (decision 03/11); the commit and the labgrid version of each run are in its summary. Theuv lockanduv syncstep has no secrets, but sdists build code there. - No shared caches: no job of
build.yml,test.ymlorimages.ymlusesactions/cache.setup-uvintest.ymlruns withenable-cache: false, so the job that holds secrets neither restores nor saves an Actions cache. - The feed is mounted into the SDK container with its
.git, and the code of a PR runs in the container with write access to it. Accepted risk: the maintainer reviews the code of a PR before it is run (decision 03/11). - Known residue:
openwrt/gh-action-sdk(composite action, pinned) enables the caches ofdocker/setup-qemu-action,docker/setup-buildx-actionand of thedocker/build-push-actionbuild (type=gha) in thesdkjob. No job with secrets restores those keys; to remove them, replace the action by inline steps that builddocker/sdk-imagewithout a cache. The jobs withpackages: write(refresh-images.yml) use no cache at all.
jobs:
build:
permissions: {contents: read}
uses: wehagy/openwrt-ci/.github/workflows/build.yml@main
with: {line: "25.12", package: chrony, ci_ref: main}
test:
needs: build
uses: wehagy/openwrt-ci/.github/workflows/test.yml@main
with:
line: "25.12"
package: chrony
source: artifact
ci_ref: main
tests_path: packages/chrony/tests # optional
test_secrets: EXAMPLE_KEY # optional
secrets: inherit # only with test_secrets
image:
needs: build
uses: wehagy/openwrt-ci/.github/workflows/images.yml@main
with:
line: "25.12"
package: chrony
ci_ref: main
profiles: | # optional: see images.yml
x86/64/generic
ath79/generic/tplink_archer-c7-v4:too-big
packages: luci-ssl -dnsmasq dnsmasq-full(with the same profiles: given to build.yml.)
No tags are used: the callers follow a branch (@main).
A reusable workflow runs in the caller's context: a plain actions/checkout
fetches the calling repository, so scripts/, harness/ and docker/ would be
missing. Every job of the three workflows therefore starts with the same two
steps: check out wehagy/openwrt-ci at ci_ref (default main) into
ci-base/, then ci-base/scripts/report-revisions.sh, which prints
harness: wehagy/openwrt-ci ref <ci_ref> -> <sha of ci-base>
caller: <owner>/<repo> run commit <GITHUB_SHA>
(and appends it to the job summary). Every script then runs as
ci-base/scripts/...; the data files are found relative to the script.
ci_refis an input ofbuild.yml,test.ymlandimages.yml. A caller that uses@<ref>must passci_ref: <ref>; the workflow file comes from theuses:ref, the scripts fromci_ref, and nothing keeps them equal. A branch that moves between the two reads gets mixed versions (risk).ci.ymlof this repository passesci_ref: ${{ github.sha }}, so a branch tests its own scripts (the localuses: ./...is the same commit).- Why a step in each job and not a composite action: a composite action must
be named with a fixed
uses: wehagy/openwrt-ci/...@<ref>(no expression is allowed there) before any checkout exists, so it would pin its own ref apart fromci_refand a branch would run the action frommain. The cost is 11 copied lines in six jobs. - The harness repository is the
HARNESS_CI_REPOSITORYenv of each workflow (wehagy/openwrt-ci); a fork changes it.
tests_path is a directory of the calling repository (e.g.
packages/chrony/tests). When set, only that directory is checked out
(sparse-checkout, into caller/) and its top-level *.py files are copied
to openwrt-tests/tests/ext/ next to the generic test, so they run in the same
pytest session and VM (scripts/install-tests.sh) and can import
harness_util and use the fixtures of conftest.py (vm_ssh). A file named
like a harness file (conftest.py, harness_util.py, test_package.py) fails
the job. Empty input, a missing directory or no *.py file: only the generic
test runs (::notice:: in the log). The value is validated
(scripts/validate-inputs.sh: relative, [A-Za-z0-9._/-], at most 256
characters, no .. or . component, no leading - (git could read it as an
option) and no .git component in any letter case).
Tests of this logic:
scripts/caller-inputs-selftest.sh(step "Self-test the caller inputs" of thetargetsjob intest.yml, so every caller runs it too) runs, with fixed inputs,check_tests_path,check_secret_names,validate-inputs.sh,install-tests.sh(only top-level*.pycopied, name collisions fail and leave the harness file intact, missing directory) andwith-test-secrets.sh. It runs every case and fails at the end if one failed. It needsjqand/proc.ci.ymlcallstest.ymlwithtests_path: harness/sample-testsandtest_secrets: SAMPLE_SECRET(secrets: inherit), so the base tests its caller side without the calling repository.test_caller_contract.pychecks inside the QEMU job: the files sit next to the harness files, the listed secret is exported (empty here: the base defines none),GITHUB_TOKENand the secrets JSON are not in the environment of pytest, and no ancestor of pytest hasHARNESS_SECRETS_JSONin its/proc/<pid>/environ.
By default every package is tested on the four targets of
harness/targets.yml with 128M of VM memory. A caller whose package needs
something else says so in its inputs:
| Input | Meaning | Accepted |
|---|---|---|
targets |
targets to test, separated by spaces or newlines | 1 to 16 ids of [a-z0-9][a-z0-9-]*, each a key of targets.yml with an image for the line; empty: all (x86-64, armsr-armv8, malta-le, malta-be) |
memory |
VM memory of every tested target | NNNM from 64M to 1024M; empty: the memory of targets.yml |
- 128M is the default of the
targets/qemu_*.yamlfiles of aparcar/openwrt-tests (checked atmainas of 03/11/2026, the day the base began to follow it; no commit is named becausemainmoves:qemu_x86-64.yaml,qemu_armsr-armv8.yaml,qemu_malta-le.yaml,qemu_malta-be.yaml), and the value ofmemoryintargets.yml. A package that needs more passesmemory: netbird was OOM-killed at 128M onx86-64(run 41480369349), so its caller passes256M. - The
targetsjob validates both inputs and builds the matrix (scripts/matrix.sh); an unknown id fails there withunknown test target '<id>' on line <line> (known: ...)and a memory outside the range withinvalid VM memory (allowed: <megabytes>M from 64M to 1024M), before any VM starts. The memory is shown in the summary of thetargetsjob and of each VM job (target: ..., memory: ...). - Pass the same
targetstobuild.yml: its defaultarchs(@tests @profiles) take the architectures of the chosen targets, plus those of the image profiles, so a subset builds only what the tests and images need.scripts/image-inputs-selftest.shcovers the validators, the selection and the memory override.
The base names no secret. The caller passes secrets: inherit and lists the
names in test_secrets (spaces or commas, 1 to 32 names, [A-Z][A-Z0-9_]*).
Names that change how a process behaves are refused: GITHUB_*, RUNNER_*,
ACTIONS_*, HARNESS_*, LD_*, BASH_*, PYTHON*, UV_*, PYTEST_*,
LG_*, NODE_OPTIONS, PATH, HOME, SHELL, IFS, ENV, PS4, TMPDIR.
- The step
Run testsgetstoJSON(secrets)and the names as env and runsexec ci-base/scripts/with-test-secrets.sh ci-base/scripts/run-tests.sh. The wrapper exports each named secret as an environment variable (notGITHUB_ENV), removes the JSON andexecsrun-tests.sh(uv, pytest). Values are masked by GitHub; the log shows onlytest-secrets: NAME is setor... is absent or empty. - What this does and does not give: the list limits what the tests receive,
and no process above pytest keeps the JSON with all the caller secrets in
its environment (the step's bash is replaced by the wrapper, which is
replaced by
run-tests.sh; proved bycaller-inputs-selftest.shand bytest_no_ancestor_keeps_the_secrets_json, which read/proc/<pid>/environof every ancestor). It is hygiene against accidental exposure, not isolation: the tests are code of the calling repository, whose own workflow can read all its secrets anyway. The runner process (Runner.Worker) keeps the secrets in its memory, because it builds the step environment and masks the log; nothing here changes that, and a process withptraceor root can read them. - A name that is not a secret of the caller is exported empty: a test sees an
empty variable and can skip with a reason. Without
secrets: inherit,toJSON(secrets)holds onlygithub_token(measured: keys["github_token"]with and withoutinheritin a repository with no other secret), so every listed name is absent. - Why not an explicit
secrets:entry: the base would have to declare names, or one opaquetest_envsecret the caller composes.inheritplus a list keeps the base generic and the caller in control. It requires the caller and the base to be able to usesecrets: inherit(same owner or organization). - Limit: the env block of the
Run testsstep printstoJSON(secrets), so the log shows the names of all secrets the caller passed ("NAME": "***"), never their values.
sdk_container (default ghcr.io/wehagy/openwrt-sdk) and ib_container
(default ghcr.io/wehagy/openwrt-imagebuilder) name the GHCR repositories of
the SDK and Image Builder images. build.yml and images.yml only read them,
anonymously, so both packages must be public and no job of those workflows
has packages: write. The images are written by refresh-images.yml, which
the caller schedules daily; its jobs push with the caller's GITHUB_TOKEN,
which needs, for each of the two packages: GitHub > Packages > the package >
Package settings > Manage Actions access > Add Repository > the calling
repository, role Write (Admin to also delete the superseded -prev
version). A package that does not exist yet is created by the first push,
private: set it to public (Package settings > Change visibility) before the
builds read it. A denied push fails the job with
::error::push of ... denied. The GITHUB_TOKEN of <repo> needs write access to the package ... after printing the registry error.
Refresh by hand (all images; only those whose upstream changed are rebuilt;
-f force=true rebuilds the others too):
gh workflow run refresh-images.yml --repo <owner>/<caller-repository>
A build or image job that finds its image missing or private fails with this
command in the message (::error title=Image missing::).
The command names the repository that owns the refresh workflow, not
necessarily the one the job runs in: build.yml and images.yml take the
optional input refresh_repository (<owner>/<name>, default the calling
repository), which this repository's own ci.yml sets to
wehagy/openwrt-ci-packages. The refresh builds only what the package
configs of that repository name (lines, QEMU targets, image profiles), so the
message also says to add a missing architecture or profile to a
packages/<name>/config.yml first. The annotation is printed on stderr, so a
step that captures stdout does not swallow it.
A one-off image of a profile that no config names (the custom-image form of
the caller, other_profile) does not need the refresh: images.yml has the
input ib_fallback (default false). When it is true and the harness Image
Builder image is missing (HTTP 404 only; any other answer fails), the job takes
the official container instead (docker.io/openwrt/imagebuilder:<target>-<subtarget>-<release>
for a release line, ...-<imagebuilder_tag_suffix> for a snapshot line,
whose setup.sh the job runs, about 10 minutes more), says so in a
::notice title=Image Builder downloaded at run time:: and writes nothing to
GHCR. The SDK has no such fallback: a package built for an architecture that
no config names fails with the message above.
ci_refand theuses:ref are two inputs; a branch that moves between the read of the workflow file and the checkout mixes versions./orgs/<owner>/packages/...(owner an organization) is written but was not exercised: every run so far had a user owner.- A push denied by the registry is reported with the permission to grant; the message was not provoked in a run (a fresh image is always reused).
- A rebuild started by a caller needs the package role Write; deleting the old
-prevversion probably needs Admin (not exercised).
Inputs (line and package required):
| Input | Meaning | Accepted |
|---|---|---|
line |
OpenWrt line, a key of harness/lines.yml with build keys |
25.12 or main |
package |
package to build, a directory of openwrt/packages |
[a-z0-9][a-z0-9._+-]*, <= 128 chars |
Optional inputs: archs and profiles (see "Architectures");
sdk_container (see above);
feed_repository and feed_ref (see "Feed source"); ci_ref (see above).
Outputs: feed_repository, feed_ref and feed_sha (see "Feed source").
The workflow only reads the SDK images: it needs no packages: write
(refresh-images.yml writes them, see below). There is no refresh_sdk input.
- Two jobs:
feedonce andsdk(the build) per architecture of thearchsinput (fail-fast: false; by defaultx86_64,aarch64_generic,mipsel_24kc,mips_24kc, those of the QEMU test targets). - The SDK image repository is named once, by the workflow
envHARNESS_SDK_CONTAINER(thesdk_containerinput, defaultghcr.io/wehagy/openwrt-sdk): the daily refresh publishes it,sdkreads it and passes it to the action asCONTAINER.scripts/resolve-sdk.shhas no default. scripts/resolve-build.shvalidates the inputs and readssdk_tag_suffix,image_grace_daysandpackages_branchfromlines.yml.- Official SDK container:
docker.io/openwrt/sdk:<arch>-<sdk_tag_suffix>. Snapshot containers ship onlysetup.sh, which downloads the ~280 MB SDK from the OpenWrt mirror (13-14 min from the runners). - Harness SDK image:
<sdk_container>:<arch>-<sdk_tag_suffix>(defaultghcr.io/wehagy/openwrt-sdk), the official container withsetup.shalready run and removed (docker/sdk-image/Dockerfile).openwrt/gh-action-sdkbuilds from it (CONTAINERandARCH), so its entrypoint finds nosetup.shand skips the download. - Default feed ref of
openwrt/packages:openwrt-25.12for25.12,masterformain(thepackages_branchkey).
- Official SDK container:
- No job rebuilds an SDK image:
refresh-images.ymldoes (see "Reusable workflowrefresh-images.yml").feedstarts withscripts/image-decide-selftest.sh(the stale rule, the readers of the labels and of the official container's sha256sums, with a stubcurl). - The
sdkjob stepPin the SDK image(scripts/sdk-image-info.sh) reads the image anonymously and fails if it is missing, not public or its labels are invalid, with the refresh command in the message. It records<tag>@<digest>(HARNESS_SDK_PIN), digest, creation time, SDK archive and its sha256. A stale image (scripts/image-stale-check.sh: the sha256 of its archive differs from the one the sha256sums at the labelio.github.wehagy.openwrt-ci.upstream-urllists now, and it is older thanimage_grace_days) only raises::warning title=Image stale::; so does a check that cannot be made (labels missing, sha256sums not downloadable).openwrt/gh-action-sdkbuildsFROM $CONTAINER:$ARCHwithARCHset to the pin, so the build reads exactly the digest of the summary even if the refresh moves the tag meanwhile. - The full feed tree is checked out (Go packages include
lang/golangby relative path; by defaultopenwrt/packages, see "Feed source"), then built withopenwrt/gh-action-sdkpinned to commitf5813d30. scripts/collect-apk.shtakes the one<package>-<version>.apkof the SDK output (none or several fail the job).- The job summary (
scripts/build-summary.sh) lists line, arch, SDK image and digest and pin, creation date, SDK archive and sha256, feed repository, ref and commit, package version and build duration. - Artifact:
apk-<package>-<line>-<arch>(the.apkalone).
The architecture matrix of sdk (and, in refresh-images.yml, of the SDK images) is computed by the feed job
(scripts/resolve-archs.sh), so an invalid value fails before an SDK image is
rebuilt.
| Input | Meaning | Accepted |
|---|---|---|
archs |
package architectures to build, separated by spaces or newlines | @tests, @profiles, or a name [a-z0-9][a-z0-9_-]* (<= 64 chars); 1 to 16 after expansion; default @tests @profiles |
targets |
QEMU test targets whose architectures @tests stands for |
same value as the targets input of test.yml; empty: all |
profiles |
image profiles whose architectures @profiles adds |
same value as the profiles input of images.yml (see there); empty: none |
@testsis thearchof every target ofharness/targets.ymlselected bytargets(by default all:x86_64,aarch64_generic,mipsel_24kc,mips_24kc), so the architectures the QEMU jobs need are never listed twice.@profilesis thearch_packagesof theprofiles.jsonof each profile (see "Reusable workflowimages.yml"). Duplicates are dropped.- A caller passes the same
profilestobuild.ymlandimages.yml, and the architectures of its images are built without being named. A caller that wants only the architecture of one profile passesarchs: "@profiles". - A new architecture needs no change here: its SDK image
<sdk_container>:<arch>-<suffix>is missing, so thesdkjob fails with the refresh command; the caller ofrefresh-images.ymlmust list the architecture (through its targets or profiles). The official containerdocker.io/openwrt/sdk:<arch>-<suffix>must exist; a name that does not exist fails at the pull in the refresh. - Job names (
<line> <arch>) and the artifactapk-<package>-<line>-<arch>are unchanged; only the set of architectures moved from the workflow file to the inputs.
By default a build uses the branch of openwrt/packages named by the line
(packages_branch). A caller that builds another tree (a version bump pushed to
a test repository, an upstream pull request) passes:
| Input | Meaning | Accepted |
|---|---|---|
feed_repository |
public GitHub repository | <owner>/<name>; owner [A-Za-z0-9-] (<= 39, no leading -), name [A-Za-z0-9._-] (<= 100, no leading -, not ., .. or *.git); default openwrt/packages |
feed_ref |
what to build in that repository | a branch, a tag, a full commit SHA or refs/pull/<n>/head (<= 9 digits); empty: the line's packages_branch |
- Both values are validated by
scripts/resolve-feed.sh(check_feed_repositoryandcheck_feed_refoflib/common.sh) before anything is written to$GITHUB_ENV; nothing is echoed. A ref is[A-Za-z0-9._/+-], at most 200 characters, with no.., no leading-, no leading, trailing or doubled/, no component starting with., no.lockor.ending; any otherrefs/name is refused. - The job
feedruns first (it also gatessdk, so an invalid input fails before an architecture job starts). It checks the feed out, reads the commit (scripts/feed-record.sh) and exposes it as the workflow outputsfeed_repository,feed_refandfeed_sha; its job summary has the sectionfeed sourcewith the three values. Thesdkjobs check out that SHA, so the five architectures build one tree even if the branch moves during the run, and fail if the checkout is not at that SHA. - The summary of each
sdkjob haspackages feed: <repository> <ref> @ <sha>. - A caller reads the outputs with
needs.build.outputs.feed_shaetc. - Both summaries are also printed to the job log (
tee), so the values can be read without the web page. packageis passed to the SDK as the package to compile and must also be the name of the.apk: a source package whose apk has another name (e.g.udnsbuildslibudns) fails at "no .apk" or, for the binary name, at "No rule to make target package//download" (observed with PR 31056 ofopenwrt/packages).images.ymlandtest.ymldo not read the feed: they use the.apkof the build artifact and the release or snapshot feed of OpenWrt, so they need no change.- Tests:
scripts/feed-inputs-selftest.sh(step "Feed source self-test" of thefeedjob, run by every caller) covers the validators,resolve-feed.sh,resolve-build.shandfeed-record.shwith fixed inputs, including values that must be refused (.., leading-, otherrefs/names, a newline, a forged::error::line) and the defaults of both lines. It needsyqandgit.
The only workflow that writes the SDK and Image Builder images on GHCR. The
caller schedules it daily and by hand (wehagy/openwrt-ci-packages:
.github/workflows/refresh-images.yml) and passes the lists it derives from
its package configs. Inputs: lines (required, lines of harness/lines.yml,
separated by spaces), targets (QEMU test targets whose architectures get an
SDK image; empty: all), profiles (as in images.yml; their architectures
get an SDK image and their targets an Image Builder image), force (boolean,
default false), sdk_container, ib_container, ci_ref. The caller grants
packages: write to the calling job.
- Job
plan: self-tests (image-decide-selftest.sh,ghcr-versions-selftest.sh,ib-setup-selftest.sh,image-inputs-selftest.sh), thenscripts/refresh-plan.sh: one SDK image per (line, architecture) (scripts/resolve-archs.sh, asbuild.yml) and one Image Builder image per (line, target, subtarget) of the profiles (scripts/resolve-profiles.sh, checked againstprofiles.json). An unknown line or profile fails here, before any image job. - Jobs
sdkandib(packages: write, one job per image,fail-fast: false; the only jobs with that permission; they check out only the harness atci_ref, run no package, feed or pull request code and use no cache):scripts/resolve-refresh.shandscripts/image-refresh.sh. - SDK image
<sdk_container>:<arch>-<sdk_tag_suffix>and, for a snapshot line, Image Builder image<ib_container>:<target>-<subtarget>-<imagebuilder_tag_suffix>(modebuild): the official container is pulled; its environment (TARGET,DOWNLOAD_FILE,VERSION_PATH,FILE_HOST) names thesha256sumsthat itssetup.shreads; the one lineDOWNLOAD_FILEmatches there (thegrepofsetup.sh) is the upstream archive and its sha256 (lib/image-upstream.sh). The image is rebuilt only when it is missing, its recipe (io.github.wehagy.openwrt-ci.dockerfile-sha256, the sha256 ofdocker/sdk-image/Dockerfileordocker/ib-image/Dockerfile), its upstream address (...upstream-url) or the archive sha256 (...sdk-archive-sha256) differs from the label, or withforce. There is no age rule: an image whose upstream did not change is kept however old (lib/image-decide.sh,image_decide, proved byimage-decide-selftest.sh). - Image Builder image of a release line (mode
mirror,25.12): a copy ofdocker.io/openwrt/imagebuilder:<target>-<subtarget>-<latest release>, whose Image Builder is already set up (RUN_SETUP=1inopenwrt/docker); the image keeps the tag of the line (...-openwrt-25.12) and the labels...release,...source-imageand...source-digest; it is rebuilt when that container's digest changed (a new release or a rebuilt container). - The build runs
setup.shthroughscripts/ib-setup.sh(inside the Dockerfile): it is repeated only when the archive does not match thesha256sums(a snapshot directory is published file by file), at most 3 attempts within a time budget, with a::warning::for each retry; any other failure fails the job. After the build the archive name must equal the one looked up before, and the labels carry the sha256 of the archive the image holds. The decision is made again just before the push (no lock); a job that finds an identical image pushed meanwhile discards its build. - Push and retention are those of the former lazy refresh: one plain image
manifest (
--provenance=false --sbom=false), the superseded image kept as<tag>-prev, the version that was<tag>-prevdeleted, a 404 on the delete accepted only when a fresh listing no longer has the version (lib/ghcr-versions.sh). The labelorg.opencontainers.image.sourcenames the repository whose token pushed. - Not in GHCR: the Go build cache and the QEMU firmware images.
Inputs (line, package, source required; all validated by
scripts/validate-inputs.sh before use; an invalid value fails the run).
Optional: tests_path, test_secrets, targets, memory, apk_timeout,
services_disabled and ci_ref (see above and "Test targets and VM memory").
| Input | Meaning | Accepted |
|---|---|---|
line |
OpenWrt line, a key of harness/lines.yml |
main or NN.NN (25.12 and main have QEMU data) |
package |
package name to install and check | [a-z0-9][a-z0-9._+-]*, <= 128 chars |
source |
where the .apk comes from |
feed (apk feed of the line: release feed for 25.12, snapshot feed for main) or artifact (apk-<package>-<line>-<arch> from build.yml in the same workflow run) |
- One job per target of
harness/targets.ymlthat has an image for the line (x86-64,armsr-armv8,malta-le,malta-be; all four on both lines). Each job downloads the image (line25.12: latest stable release; linemain: current snapshot), verifies its sha256 against thesha256sumsof the same directory (downloaded first), boots it in QEMU, gets the.apk(scripts/fetch-package.sh) and runsharness/tests/test_package.py. - Snapshot images are rebuilt daily and each target directory is published on
its own, so two targets of one run can report different snapshot versions.
The job summary therefore lists, per job, the release or snapshot version
(
HARNESS_RELEASE), the image URL, the image sha256 and the image date (Last-Modifiedof the file). - For
main,scripts/fetch-image.shreadsversion.buildinfobefore thesha256sumsand the image and again after them; if it changed, the download is repeated (3 attempts), so the version in the summary is the one of the tested image. A sha256 mismatch with an unchanged version fails. - With
source: artifact, a target uses the artifact of itsarchintargets.yml(x86_64,aarch64_generic,mipsel_24kc,mips_24kc).aarch64_cortex-a53has no QEMU target and is never tested; it is built only when a profile ofimages.ymlneeds it. - Image formats (see
harness/targets.yml):x86-64is a gzipped combined image;armsr-armv8(both lines) andmalta-le/malta-beonmainboot*-initramfs-kernel.bin;malta-le/malta-beon25.12boot*-vmlinux-initramfs.elf;armsr-armv8andmalta-*run from RAM (with the default 128M, see "Test targets and VM memory"). - Network of the VM (
harness/tests/conftest.py): the session waits for a default route and an A record, then runsapk update, repeated up to 5 times, 5 s apart, only when stderr hasFailed to send request: Operation not permitted; any other failure (UNTRUSTED signature,404 Not Found, ...) fails on the first attempt.unavailablealone does not qualify: the1 unavailable, 0 staleline of the lost fetch is what a missing file prints too. Onmalta-leandmalta-be(where the test creates thewaninterface) one of the parallel fetches of the firstapk updatefailed withwget: Failed to send request: Operation not permittedwhile the other repositories succeeded, a different repository each time (runs 41427983225, 41360955048); route and DNS were already up, and the cause inside the VM was not proven. Each repeat is logged as aharnesswarning with the output ofapk updateand added as a line to the job summary; after the last attempt, or at a failure that is not the lost fetch, the session fails with the output of the attempts made. - Timeouts in the VM: queries are bounded at 30 s. The
apkcommands oftest_package.py(apk info -L,apk info -e,apk list -I) have no timeout of ours: labgrid's default applies (SSHDriver.runhastimeout=None, checked in labgrid 26.0.1). A caller whose package needs a bound setsapk_timeout(whole seconds, 30 to 600;scripts/validate-inputs.shrefuses anything else before a VM starts):apk info -Lright after the install took 0.5 to 15 s in healthy runs and once more than 30 s (TimeoutExpired, run 41437522326, job 125258840649,x86-64onmain, with netbird at 128 MB), so netbird passes 120. The summary of each VM job saysapk command timeout: N s (apk_timeout input of the caller)ornone (labgrid default; ...), and the test records the measured duration ofapk info -Las the junit propertyapk_info_L_seconds(also in the job log:harness: apk info -L <package> took N s), so the timeout can be dropped once the values stay far below 30 s. harness/selftests/(step "Harness helpers self-test" oftest.yml) tests the retry helpers with a stub in place of the VM.- Tests: install,
apkreports the expected version, and every/etc/init.dscript the package installs is a procd script. The service rule (harness_util.judge_instancesandservice_sampler, tested offline inharness/selftests/test_services.pyagainstubus call service listJSON inharness/selftests/fixtures/ubus-service-list/): afterstopandstartof the script, at least one instance is running, every running instance keeps its pid for 10 s (SERVICE_STABLE), and an instance that is not running passes only if it exited withexit_code0 (a one-shot such aschrony-hotplug). A daemon that exited 0 alone, a non-zero or missingexit_code, a changing pid or no instance at all fails after 60 s. The service test is skipped when the package installs no init script, and for a script the caller lists inservices_disabled(names under/etc/init.dseparated by spaces or commas, validated bycheck_service_names): the skip is logged (harness: service <name> skipped, declared disabled by default), recorded as the junit propertyservice_<name>and named in the job summary; when all scripts are disabled the test is reported as skipped with that reason. A listed name the package does not install (a typo, or a script a PR renamed or that exists on one line only) does not fail the test: it is logged, recorded as the junit propertyservices_disabled_unknown, named in the job summary and shown as a::warning::annotation of the job. The scripts the package does install are still checked. The job summary lists passed, failed and skipped per test.
Inputs (validated before use, like build.yml): line (a key of
harness/lines.yml) is required; optional package, profiles, packages
and ci_ref (see above). The base names no device: the image choices belong to
the caller (in wehagy/openwrt-ci-packages,
packages/<name>/config.yml).
| Input | Meaning | Accepted |
|---|---|---|
package |
package built by build.yml, installed in every image |
as in build.yml; empty: images without a package of their own (no apk-* artifact is read, no manifest line is checked) |
profiles |
the images to build, separated by spaces or newlines | <target>/<subtarget>/<profile>[:too-big]; target and subtarget [a-z0-9]+ (<= 32), profile [A-Za-z0-9_-]+ (<= 64); 1 to 16; empty: no image (the image jobs are skipped; the profiles job says images: no profile given, no image is built in its summary) |
packages |
packages added to every image next to the built one | <name> adds, -<name> removes (Image Builder PACKAGES syntax, e.g. -dnsmasq dnsmasq-full); name as package; at most 100 and 2048 characters; empty: none |
-
:too-bigafter a profile is the expectation described below (the image is known not to fit the flash). -
The job
profiles(scripts/resolve-profiles.sh) reads, for each profile, the officialprofiles.jsonof its target and subtarget: for25.12the directory of the latest25.12.xrelease (as the Image Builder tag), formainthe snapshot directory (image_base_urlofharness/lines.yml).- The file must exist (otherwise
target <t>/<s> does not exist on line <l>: no profiles.json at <url>) and itsprofilesobject must have the profile (otherwiseprofile <p> does not exist in <t>/<s> on line <l> (<n> profiles there; nearest: ...)). The package architecture is itsarch_packagesfield. An invalid input fails here, within about a minute, before an SDK or Image Builder job exists. - The matrix has one entry per profile with the id (job name
<line> <id>, artifact name), target, subtarget, profile, architecture and expectation. The id is the profile name in lower case with_as-(glinet_gl-mt1300isglinet-gl-mt1300); a profile calledgenericgets<target>-<subtarget>in front (x86-64-generic). Two entries with the same id fail. - The values are checked (charset, length, count) before anything is
written to
$GITHUB_OUTPUTor$GITHUB_ENV, and never echoed.
- The file must exist (otherwise
-
One job per entry (
fail-fast: false). The Image Builder package list is thepackagesinput followed by the built package. After the buildscripts/collect-images.shchecks the manifest:<package> - <version>is a line, every package added bypackagesis listed and every package removed with-is not (names are compared whole:-dnsmasqis satisfied bydnsmasq-full). -
scripts/image-inputs-selftest.sh(step "Image inputs self-test" of theprofilesjob here and of thefeedjob ofbuild.yml) proves the validators, the resolution against a fake mirror (file://URLs, no network) and the manifest check with fixed inputs. -
Image Builder: the harness image
<ib_container>:<target>-<subtarget>-<imagebuilder_tag_suffix>(defaultghcr.io/wehagy/openwrt-imagebuilder,scripts/resolve-image.sh), read only: the daily refresh (refresh-images.yml) copies the official release container (25.12) or runs thesetup.shof the snapshot container once (main). The recipe hash (the label that makes a changed recipe rebuild the image) covers theDockerfile,ib-setup.shandlib/common.sh; the digest of the official container the snapshot build starts from is recorded but does not decide (that container is republished with every snapshot, which would rebuild the image daily). A missing image fails the job with the refresh command; a stale one only warns (scripts/pull-imagebuilder.sh,scripts/image-stale-check.sh). The image jobs download no Image Builder. Of the text below,<tag>is the tag of the official container the image is made from.25.12(release_source: releases):<tag>is the latest25.12.xrelease, found astest.ymlfinds its release (latest_releaseinscripts/lib/common.sh). The release image contains the Image Builder.main(release_source: snapshot):<tag>ismain, a snapshot tag that ships onlysetup.sh; the container downloads the Image Builder fromdownloads.openwrt.orgwhen it starts (timed as "setup.sh" in the summary). Risk: that mirror gave ~330 KB/s from runners for the SDK.- The snapshot directory is published file by file. In runs 41427929552
and 41427983225 (
ath79/generic)setup.shfailed after an 8-15 minute download withsha256sum: WARNING: 1 computed checksum did NOT match: thesha256sumsit got was signed on the day before and listedc732955d..., while the archive had been republished at 10:33:36 and thesha256sumsonly at 10:38:33 (Last-Modifiedread afterwards; the same archive then matched in run 41446811960).scripts/ib-setup.sh(called byimagebuilder-make.sh) therefore runssetup.shagain, up to 3 times with 120 s between them, when and only when its output has the checksum mismatch; each retry printswarn: image builder archive did not match sha256sums (attempt N of 3), retrying in 120 sand a::warning title=Image Builder setup retried::annotation (shown in the run summary), and any other failure, or a mismatch in all attempts, fails the job. A retry starts only if the time used, the wait and the longest attempt so far fit inHARNESS_IB_SETUP_BUDGET(default 3000 s; theimagejob times out at 90 min = 5400 s and the pull,make imageand the upload need the rest; one attempt took 3235 s for bananapi); otherwise the job fails at once withnot retrying: <used> s used, a retry needs about <n> s, budget <b> sinstead of being killed by the timeout. A retry fetchessha256sumsagain and restores the keys thatsetup.shdeletes.scripts/ib-setup-selftest.sh(step "Image Builder setup self-test" of theprofilesjob) runs it against a fakesetup.sh.
-
The
.apkofapk-<package>-<line>-<arch>is copied to the Image Builderpackages/directory and signed there with the local key of the Image Builder (apk adbsign; the key is created bymake _check_keys, the target the Makefile runs itself). A file installed by path is not covered by the signature of thepackages/index, and the Image Builder (CONFIG_SIGNATURE_CHECK=y) refuses an unsigned one (UNTRUSTED signature, measured on 25.12.2). -
The built package is requested by the path of that
.apk(scripts/imagebuilder-make.sh,lib/ib-pins.sh), not by name: a name resolves against the local index and the official feeds together and takes the highest version, and at an equal version apk installs the feed's package even with the local repository listed first (measured on 25.12.2). apk installs a file given by path itself, at a lower, equal or higher version than the feed (measured on 25.12.2 and onmain), and the package version stays as built.<name>=<version>does not help at an equal version, and a tagged repository (@localinrepositories,<name>@local) leaves<name>@localin the world of the image, which makesapkon the device refuse every change (repository tag ... does not exist). The.apkmetadata (apk adbdump) is read and its name and version are checked against the file name. Packages of thepackagesinput stay as given (the feed chooses); a-<package>that removes the built package, and a version or a path in thepackagesinput, are refused. -
The world of the image (
/etc/apk/world) lists the package by name, as the world of an official image does, so the device later upgrades it from the feed. apk writes<name><><checksum>for a package installed from a file; that pin would keepapk upgradefrom ever replacing the package. The Image Builder copies itsFILESdirectory over the root after the packages are installed, soimagebuilder-make.shinstalls the packages once withmake manifest(an Image Builder target) to learn the world apk writes, and givesmake imageaFILESdirectory whoseetc/apk/worldis that world with only the pin of the built package replaced by its name (ib_plain_world; the=pins the Image Builder itself writes,base-files=...,kernel=...,libc=..., stay). The installed database is not touched: it still holds the record of the built.apk. On the device,apk upgradetakes a higher version of the package from the feed (Upgrading chrony (4.8-r3 -> 99.0-r0)); at an equal version it replaces the built.apkwith the feed package of that version (Replacing chrony (4.8-r3 -> 4.8-r3)), without an error, because a package installed from a file has no repository.apk addandapk delwork as on an official image. Measured on 25.12.2 and onmainwith a package of the same name built byapk mkpkg. -
scripts/collect-images.shrequires the line<package> - <version>in*.manifest(version of the built.apk, equal to the one in its metadata), then that the installed package is the built one: the checksum of its record in the apk database of the image root (C:Q1...) must be the one of the record apk writes when it installs the same.apkin a scratch root with the same repositories (local-expected.db, made byimagebuilder-make.sh). A package of the feeds has another checksum, at any version. It then requires that the world of the image (local-world) is the world apk wrote (local-world-pinned) with only that pin replaced by the name, and that no entry carries><,@or/(ib_check_world). Then it keeps the files matchingimage_globsplus the manifest andsha256sums.scripts/ib-pins-selftest.sh(step "Package pin self-test" of theprofilesjob) covers the package list, the metadata readers, the check and the world. -
An image that does not fit the flash: the Image Builder does not fail, it only prints
WARNING: Image file ... is too big: <size> > <limit>and deletes the image (Build/check-size);tplink-safeloaderrefuses earlier withfile-system partition too big (more than <limit> bytes)and the warning then has an empty size. That empty-size warning is printed whenever the image file does not exist, whatever removed it (include/image-commands.mk:... && mv $@.new $@ || rm -f $@), so it proves nothing.scripts/check-image-outcome.shaccepts "too big" only with numbers: acheck-sizemessage with<size>><limit>, or apartition too bigmessage with the size ofroot.squashfs(rootfs-bytes, thedd"bytes copied" line before the message) ><limit>. Any "too big" message that is not proven so (size not above the limit, root filesystem not above the partition, no root filesystem size) or only empty-size warnings: the job fails ("not proven by numbers"). The proven result is compared with the profile's:too-bigsuffix:suffix Image builds Image too big (proven) absent (default) job green job fails, message and sizes quoted :too-bigjob fails: "now fits", remove the suffix job green, no artifact The summary of a
:too-bigjob quotes the Image Builder messages with the actual and limit bytes (for a tplink-safeloader image, the root filesystem against its partition, because the warning has no size).- Failures: a
makeerror or a missing.apkfails the Image Builder step in both cases. A profile with:too-bigthat is proven too big skips the manifest check (scripts/collect-images.sh) and the artifact because no image exists, so a wrong manifest or a missing package in such a profile is not detected. scripts/image-outcome-selftest.sh(step "Outcome check self-test" of theprofilesjob inimages.yml) runscheck-image-outcome.shon fixedmake.logfixtures: the empty-size warning alone,10 > 20, apartition too bigwithout or with a smaller root filesystem must fail; the realtotolink-x5000randtplink-archer-c7-v4messages are a green too big; a fitting image and a fitting:too-bigprofile ("now fits") are checked too.
- Failures: a
-
The job summary (
scripts/image-summary.sh) lists profile, Image Builder image and digest, size and pull time,setup.shandmake imagedurations, the expectation and outcome, the manifest check, every artifact file with size and sha256 and, on failure or expected failure, the Image Builder messages ofmake.log. -
Artifact:
images-<package>-<line>-<profile id>(nopkgstands for the package when there is none; no artifact for a profile that is too big as expected).
Tests the base with the sample package chrony. Every call passes
ci_ref: ${{ github.sha }}, the commit under test. On push to feat/** and on
workflow_dispatch.
Dispatch inputs: lines
(both, 25.12 or main: the other build job is skipped); package
(default chrony), feed_repository and feed_ref (passed to build.yml,
see "Feed source"); stages (all, or build to skip the test and image
jobs); profiles and packages (the profiles and packages inputs; empty
profiles: x86/64/generic, passed to build.yml and images.yml like any caller; empty packages: none, so the image holds the Image Builder's default packages
and the package under test). No job has packages: write (the workflow
workflow-permissions-selftest.sh fails if one gets it). It passes
refresh_repository: wehagy/openwrt-ci-packages to every build.yml and
images.yml call, so a missing image names the repository whose
refresh-images.yml creates it. The job feed-report prints the
feed outputs (feed_repository, feed_ref, feed_sha) of the builds that
succeeded.
| Job | Calls | Inputs | Needs |
|---|---|---|---|
build-25-12 |
build.yml |
line: 25.12 |
|
build-main |
build.yml |
line: main |
|
test-25-12 |
test.yml |
line: 25.12, source: artifact |
build-25-12 |
test-main |
test.yml |
line: main, source: artifact |
build-main |
image-25-12 |
images.yml |
line: 25.12 |
build-25-12 |
image-main |
images.yml |
line: main |
build-main |
| Name | From | Content |
|---|---|---|
apk-<package>-<line>-<arch> |
build.yml |
the built .apk |
images-<package>-<line>-<profile> |
images.yml |
sysupgrade/factory/combined/sdcard images, *.manifest, sha256sums |
qemu-<package>-<line>-<target> |
test.yml |
results.xml (junit) |
qemu-<package>-<line>-<target>-logs |
test.yml |
labgrid logs, pip-freeze.txt |
Packages are added in the calling repository
(wehagy/openwrt-ci-packages,
packages/<name>/), not here. Only what the code supports today:
- Build and test a package of
openwrt/packages: callbuild.ymlwithlineandpackage, thentest.ymlwith the samelineandpackage,source: artifactandneeds:the build job; passci_ref. - Test a package of the official release feed without building it: call
test.ymlwithsource: feed. harness/tests/test_package.pyis generic: no per-package code or data. Package-specific runtime tests are files in the caller repository, given bytests_path; they do not exist yet.- Build images with the package: call
images.ymlwith the sameline,package,profilesandpackages,needs:the build job; pass the sameprofilestobuild.yml, so that the architectures of the images are built. - Package trees other than the line's branch of
openwrt/packages(a test repository, an upstream pull request):feed_repositoryandfeed_ref(see "Feed source"). Local feeds of the calling repository are not supported.
Nothing about the test environment is pinned to a commit or a release, except
the Python version (UV_PYTHON in test.yml) and the dependencies that are
neither of the two below (the seed lock).
aparcar/openwrt-testsfollows itsmainbranch. Thetargetsjob oftest.ymlreads the tip once (scripts/resolve-tests-ref.sh) and every target job of that run checks out that commit, so the targets of one run never test with different versions.- labgrid comes from PyPI, always the newest release of the 26 series
(
LABGRID_REQUIREMENTinscripts/lib/uv-lock.sh:labgrid>=26,<27, the only place). A new major is a decision of ours: change the requirement and the seed. uv ignoresconstraint-dependenciesandoverride-dependencieswritten in auv.toml, so the requirement is passed touv lockitself (--upgrade-package).scripts/prepare-tests-env.shsetsUV_NO_SOURCES=1, because thepyproject.tomlof openwrt-tests points labgrid at theaparcar/labgridfork branchstaging; that flag drops every[tool.uv.sources]entry, so the script fails if thepyproject.tomlhas one for another package. It also setsUV_EXCLUDE_NEWERto the instant of thetargetsjob, so a release published during a run cannot split its targets either. - Every other dependency stays where
harness/openwrt-tests.uv.lockhas it: the script copies the seed to the checkout, runsuv lock --upgrade-package 'labgrid>=26,<27'(only labgrid moves; uv adds or moves another package only when thepyproject.tomlof the new commit asks for something the seed cannot satisfy) anduv sync --locked. This costs one resolution per job (a few seconds) and keeps the rest of the environment from changing without a reason. - The resolved versions are in the summary of each VM job
(
openwrt-tests: <commit>,labgrid: <version>), in the junit propertiesopenwrt_tests_shaandlabgrid_versionand inpip-freeze.txt; a job fails if either is missing (scripts/summary.sh,test_package.py). scripts/uv-lock-selftest.shruns the real uv (HARNESS_UV_VERSIONintest.yml, the version the test jobs install too) against a fixture project and a local package index, with no network: the requirement yields the newest 26.x (not a 27, not an upload newer than the resolution instant),labgrid<26yields 25.x, an impossible requirement fails. Locally, in a container with uv 0.13.7 and Python 3.11 or newer:UV_SELFTEST_VERSION=0.13.7 scripts/uv-lock-selftest.sh.- To refresh the seed (not needed for a run): run
scripts/prepare-tests-env.shon a checkout ofmainand copy theuv.lockit leaves there toharness/openwrt-tests.uv.lock.
Every place where the base differs from the default of the project it builds
on: aparcar/openwrt-tests (QEMU
targets and fixtures, defaults checked at main as of 03/11/2026), labgrid,
openwrt/gh-action-sdk, the Image
Builder and the official OpenWrt images. "Package" says whether a calling
repository can change it with an input of the reusable workflows.
| What | Where | Why | Package |
|---|---|---|---|
KVM on x86-64: -enable-kvm added to extra_args, /dev/kvm opened to all users |
scripts/prepare-target.sh (the kvm branch), step "Enable KVM" of .github/workflows/test.yml, kvm: true of x86-64 in harness/targets.yml |
labgrid's QEMUDriver never adds -enable-kvm; the runner offers /dev/kvm for the x86 guest only (kvm: false for the others) |
no |
dnsmasq filter_aaaa=1 in the VM |
harness/tests/conftest.py (ci_network, the filter_aaaa setting) (F1) |
the runner has no IPv6 egress but downloads.openwrt.org has an AAAA record, so apk waited on an unreachable address |
no |
network.wan created on eth1 (dhcp) when the image has none |
harness/tests/conftest.py (ci_network, the network.wan check) (F2) |
malta-le on 25.12 and malta-be on both lines define only lan, so the VM had no default route |
no |
apk add --allow-untrusted --force-non-repository for the package under test |
harness/tests/test_package.py (install fixture) |
the CI-built .apk is unsigned; the initramfs targets have no persistent overlay, so apk refuses a local file otherwise |
no |
Timeouts above the 30 s default of run_check/wait_for: CMD_TIMEOUT 600 s, NETWORK_TIMEOUT 120 s, SERVICE_TIMEOUT 60 s (QUERY_TIMEOUT 30 s is the labgrid default) |
harness/tests/conftest.py (CMD_TIMEOUT, QUERY_TIMEOUT, NETWORK_TIMEOUT), harness/tests/test_package.py (CMD_TIMEOUT, QUERY_TIMEOUT, SERVICE_TIMEOUT) |
emulated targets without KVM are slow (install, apk update, service start) |
no |
Timeout of the apk commands of the test: none by default (labgrid SSHDriver.run has timeout=None); a caller can set 30 to 600 s |
harness/tests/harness_util.py (apk_timeout_kwargs), apk_timeout input of .github/workflows/test.yml |
apk info -L right after the install took 0.5 to 15 s and once more than 30 s (run 41437522326, job 125258840649, x86-64 on main, netbird at 128M); the measured duration is recorded as apk_info_L_seconds |
yes (apk_timeout) |
Service test: a procd instance that is not running passes if it exited with exit_code 0, but at least one instance must run and be stable; scripts a caller lists in services_disabled are skipped |
harness/tests/harness_util.py (judge_instances, service_sampler, split_disabled), services_disabled input of .github/workflows/test.yml |
chrony declares a second, one-shot instance (chrony-hotplug, exit_code 0, run 41552951052, job 125652090597) that an all-instances-running rule failed; a package whose service is disabled by default has no instance after start |
yes (services_disabled) |
| VM memory 128M for every target, as upstream; overridable | memory of each target in harness/targets.yml and memory input of .github/workflows/test.yml (applied by scripts/prepare-target.sh) |
not a deviation by default; netbird needs 256M (OOM-killed at 128M on x86-64, run 41480369349) |
yes (memory) |
apk update repeated up to 5 times, 5 s apart, only when stderr has Failed to send request: Operation not permitted |
harness/tests/harness_util.py (APK_UPDATE_*, apk_update), harness/tests/conftest.py (F3) |
one of the parallel fetches of the first apk update was lost on malta-le and malta-be (runs 41427983225, 41360955048); any other failure fails at once |
no |
openwrt-tests follows main (one commit per run) and labgrid comes from PyPI (>=26,<27) instead of the aparcar/labgrid fork, branch staging, that its pyproject.toml names ([tool.uv.sources], ignored with UV_NO_SOURCES=1); Python 3.15; the other dependencies stay at a seed lock of our own |
LABGRID_REQUIREMENT in scripts/lib/uv-lock.sh, harness/openwrt-tests.uv.lock, scripts/prepare-tests-env.sh, scripts/resolve-tests-ref.sh, UV_PYTHON and the steps "Resolve the openwrt-tests commit" and "Install the openwrt-tests environment" of .github/workflows/test.yml |
user decision: always the latest upstream tests and the latest stable labgrid, without a major jump; the fork staging branch was only a pin to a development commit, and the PyPI release of the 26 series is what upstream's own default (labgrid>=25.0) resolves to once the source is ignored. The price is that a run can change with no commit of ours; the resolved versions are shown in the summary |
no |
Own SDK and Image Builder images on GHCR (ghcr.io/wehagy/openwrt-sdk:<arch>-<suffix>, ghcr.io/wehagy/openwrt-imagebuilder:<target>-<subtarget>-<suffix>), refreshed by a separate daily job and only read by the runs (no packages: write, no rebuild by age, no refresh_sdk), instead of the SDK or Image Builder download that every job did |
docker/sdk-image/Dockerfile, docker/ib-image/Dockerfile, scripts/image-refresh.sh, .github/workflows/refresh-images.yml, sdk_tag_suffix, imagebuilder_tag_suffix and image_grace_days of each line in harness/lines.yml |
the snapshot SDK container ships only setup.sh, which downloads about 280 MB (13 to 14 minutes) and an Image Builder up to 8 to 15 minutes in every job; an image is rebuilt only when its upstream sha256 (or the release container digest) changed; the image is pinned by digest per run |
yes (sdk_container, ib_container) |
shfmt v3.14.1 (release binary, sha256 pinned) installed in the SDK image; the job fails if the image has no shfmt |
docker/sdk-image/Dockerfile, step "Check that the SDK image has shfmt" of .github/workflows/build.yml |
the upstream SDK image has no shfmt, so the init-script format check of make package/<pkg>/check is a no-op there (openwrt/packages run 41512251371, job 125513520170: find: 'shfmt': No such file or directory; gh-action-sdk issue #37; the check was added in gh-action-sdk da3664e, 2021-08-08); here it runs, and a package cannot override it |
no |
openwrt/gh-action-sdk pinned to commit f5813d30, called with CONTAINER = our image, ARCH = <tag>@<digest>, FEED_DIR = the full feed checkout |
the openwrt/gh-action-sdk step of job sdk in .github/workflows/build.yml |
no tags upstream; a reproducible build of the digest that was pinned; Go packages include lang/golang by relative path |
no |
The built .apk is copied to the Image Builder packages/ directory and signed there with the local key of the Image Builder (apk adbsign) |
scripts/imagebuilder-make.sh |
the package under test must be in the image; a file installed by path is not covered by the signature of the packages/ index (made by the Image Builder with apk mkndx --allow-untrusted), and the Image Builder (CONFIG_SIGNATURE_CHECK=y) refuses an unsigned one (UNTRUSTED signature, measured on 25.12.2) |
no |
The built package is requested by the path of its .apk, signed with the local key of the Image Builder, and the installed apk checksum is compared with the one apk gives the same .apk in a scratch root |
scripts/imagebuilder-make.sh, scripts/lib/ib-pins.sh (ib_local_packages, ib_check_installed_is_local), scripts/collect-images.sh |
a bare name or name=version installs a feed package of the same version (measured on 25.12.2), so the package under test would not be the one in the image; a path installs that file at any version; the checksum proves it |
no |
The /etc/apk/world of the image lists the built package by name (<name>), not as apk writes it for a file (<name><><checksum>): a FILES directory of the Image Builder with etc/apk/world, made from the world of an install-only run (make manifest) with only that pin replaced |
scripts/imagebuilder-make.sh, scripts/lib/ib-pins.sh (ib_plain_world, ib_check_world), scripts/collect-images.sh |
the pin ties the device to that very file: apk upgrade would never take the package from the feed. An official image lists the name alone, so the device upgrades the package from the feed as any other; the installed database still holds the built .apk (checksum check above). The image world must be the one of the install with the pin replaced, and no entry may carry ><, @ or / |
no |
setup-uv with enable-cache: false in the job that holds the secrets |
.github/workflows/test.yml (astral-sh/setup-uv step) |
astral-sh/setup-uv saves and restores an Actions cache by default on a GitHub-hosted runner; a cache saved by a run that built untrusted feed code must never be restored in the job that holds test_secrets (see Trust model) |
no |
Image Builder setup.sh repeated up to 3 times, 120 s apart, only on a sha256sum ... did NOT match, within a time budget (HARNESS_IB_SETUP_BUDGET, default 3000 s) |
scripts/ib-setup.sh (retry loop and HARNESS_IB_SETUP_BUDGET), scripts/imagebuilder-make.sh |
the snapshot directory is published file by file: the archive can be newer than its sha256sums (runs 41427929552, 41427983225); the budget keeps the retry inside the job timeout (timeout-minutes of job image in .github/workflows/images.yml, 90 min) |
no |
An image that is too big is a result, not a build failure: accepted only with numbers, compared with the :too-big expectation |
scripts/check-image-outcome.sh |
the Image Builder only prints WARNING: Image file ... is too big and deletes the image; the size-less warning proves nothing |
yes (:too-big on a profile) |
The released image and sha256sums are fetched and checked by us; for main the snapshot version is read before and after the download |
scripts/fetch-image.sh |
each snapshot target directory is published on its own, so the image and its checksum can change during the download | no |
| Job timeouts: 45 min per QEMU and per SDK job, 90 min per Image Builder job, 5 to 10 min for the setup jobs | timeout-minutes of the jobs of .github/workflows/test.yml (targets, qemu), .github/workflows/build.yml (feed, sdk), .github/workflows/refresh-images.yml (plan 15 min, sdk 60 min, ib 90 min) and .github/workflows/images.yml (profiles, image) |
a hung VM or download must not hold a runner for the 6 hour default | no |