Skip to main content

One flaky third-party download fails the entire image build

The observation​

Build UIS Container has failed twice in two days, both times inside provision-host-02-kubetools.sh, both times at "Installing k9s", on linux/arm64. kubectl and helm installed cleanly immediately before. A plain re-run passed both times.

Installing kubectl        ✓
Installing Helm           ✓
Installing k9s            ← fails here
Error executing provision-host-02-kubetools.sh. Continuing with next script.
...
provision-host-02-kubetools.sh: Failed (Exit code: 1)
ERROR: failed to build

Neither failure was caused by the commit being built — the first was a launcher-only change, the second touched uis and version.txt. Nothing in either goes near the provisioning path.

Third instance, 2026-09-08 — and it was NOT k9s​

Run 34243142437, linux/arm64, same script, same "Continuing with next script" followed by an overall exit 1. But the proximate cause was different:

E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-security/
   restricted/binary-arm64/Packages.gz
   File has unexpected size (7771348 != 7777009). Mirror sync in progress?
E: Some index files failed to download.

An Ubuntu mirror mid-sync — apt-get update, not a k9s release download.

🔴 And the re-run failed too, which breaks this plan's central assumption. Run 34243142437 re-run, ~9 minutes later: identical error, a different mirror IP (91.189.91.102 vs .103), so Canonical's sync was still propagating across the pool. Installing k9s succeeded on the re-run — it was three lines above the failure, which retires the k9s hypothesis outright.

So "a plain re-run passed both times" is no longer the observed behaviour, and any fix that assumes retry-eventually-works is unsafe: the outage lasted longer than a retry cycle and was not addressable from this repository at all.

🔴 That matters for the fix. Two instances pointing at k9s invited a narrow remedy: pin or vendor k9s. A third instance with an unrelated upstream — Canonical's own mirror — shows the class is any network fetch in the provisioning path, and there are many. Pinning k9s would have closed one door in a corridor.

⚠️ And it sharpens what the actual defect is. Dockerfile.uis-provision-host:81 says DOCKER_BUILD=true "allows provision scripts to continue on non-critical failures", and the log shows it working per-script — "Error executing provision-host-02-kubetools.sh. Continuing with next script." — and then the run still fails, because the provisioning summary exits 1 regardless. So the tolerate-failure mechanism is half-wired: it continues, and then the exit code undoes the continuing. Whatever ships here should decide one thing — either a failed tool install is fatal (and stop claiming otherwise) or it is not (and the summary must not exit 1) — because right now the comment and the behaviour disagree, which is the same class as three other defects found this week.

Context: found while merging five unrelated PRs (#304–#308). None touches the provisioning path; the commit being built was a PostgREST SQL fix. That is now three for three on "the failure had nothing to do with the change", which is the strongest argument for treating it as infrastructure rather than as a flaky test someone should watch.

⚠️ Operational consequence while this is red, and it is the reason this is not merely cosmetic: the image is never pushed, so ghcr.io/helpers-no/uis-provision-host:latest still serves the previous build. Every change merged in that window — including the PostgREST default-privileges fix — is on main and not in the image a user gets from ./uis pull. A green main with a red image build means the repository and the shipped artifact disagree, silently, and nothing surfaces that to a user.

Why it costs more than a re-run​

The build takes 11–13 minutes. A failure means noticing, diagnosing whether it is yours, re-running, and waiting again — roughly half an hour of elapsed time for a transient network error on someone else's CDN.

It also actively misleads. The failure lands on a commit, so the first question is always "did I break the build?" — and answering that honestly means reading the log rather than assuming. That cost is paid every time.

There is a subtler cost: a red build trains people to re-run rather than read. The next genuine failure will look like this one.

Two things worth separating​

The script already tolerates the failure — it prints "Error executing provision-host-02-kubetools.sh. Continuing with next script" and carries on — but the provisioning summary then reports the script as failed and the whole RUN layer exits 1. So the intent to continue is there and does not reach the exit code.

  1. Should a k9s download failure fail the image at all? k9s is an interactive TUI for humans debugging a cluster. Nothing in UIS's automation depends on it. An image without k9s is degraded, not broken.
  2. If it should, it needs a retry. One attempt against a third-party CDN, on a 12-minute build, is a coin flip that costs half an hour when it lands wrong.

Options, roughly in order of effort​

  • Retry the download — three attempts with a short backoff. Smallest change, addresses the immediate cost, does not change what the image contains.
  • Make non-essential tools non-fatal — install k9s (and similar) on a best-effort basis, and have the summary distinguish "a required tool failed" from "an optional tool was skipped". This is the one that matches the script's existing intent.
  • Pin and vendor — fetch from a mirror we control. Most robust, most work, and adds a thing to maintain.

My lean is retry plus the essential/optional distinction: the retry removes most occurrences, and the distinction means the ones that remain do not fail a build over a tool nothing depends on.

What this is not​

Not a UIS defect, and not caused by any change. It is a supply-chain fragility that UIS chose to treat as fatal.

  • Observed on 1fa5d36600cf (2026-08-25) and be7ac147ef21 (2026-08-26)
  • Same family as [[INVESTIGATE-system-launcher-image-version-drift]] in one respect only: both are about what it takes for a change to actually reach the machines that need it.