Multi-arch container image (amd64 + arm64) #5

Open
toasterson wants to merge 8 commits from claude/wi-019e534c-multi-arch-container-image-amd64-arm64 into main
Owner

Draft PR for Anima work item Multi-arch container image (amd64 + arm64) (019e534c-45a6-7331-a1e7-3a7187a1d5f6).

This PR is a scaffold opened by anima-triage. The implementation worker picks it up from here.

Close or mark ready manually once the implementation lands.

Draft PR for Anima work item **Multi-arch container image (amd64 + arm64)** (`019e534c-45a6-7331-a1e7-3a7187a1d5f6`). This PR is a scaffold opened by `anima-triage`. The implementation worker picks it up from here. - Marker: [`.anima/wip/019e534c.md`](.anima/wip/019e534c.md) - Current phase: `triaged` Close or mark ready manually once the implementation lands.
wip: triage Multi-arch container image (amd64 + arm64)
Some checks failed
CI / test (pull_request) Failing after 2h0m0s
CI / test (push) Failing after 2h0m1s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
344dc887c0
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 344dc887c0
Some checks failed
CI / test (pull_request) Failing after 2h0m0s
CI / test (push) Failing after 2h0m1s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
to b3152a6322
Some checks failed
CI / test (push) Failing after 58s
CI / test (pull_request) Failing after 1m10s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
2026-05-25 20:30:41 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from b3152a6322
Some checks failed
CI / test (push) Failing after 58s
CI / test (pull_request) Failing after 1m10s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
to 6003f14809
Some checks failed
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
CI / test (push) Has been cancelled
CI / docker (push) Has been cancelled
2026-05-31 22:46:41 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 6003f14809
Some checks failed
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
CI / test (push) Has been cancelled
CI / docker (push) Has been cancelled
to a4d406c982
Some checks are pending
CI / test (pull_request) Waiting to run
CI / test (push) Waiting to run
CI / docker (pull_request) Blocked by required conditions
CI / docker (push) Blocked by required conditions
2026-06-01 02:31:23 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from a4d406c982
Some checks are pending
CI / test (pull_request) Waiting to run
CI / test (push) Waiting to run
CI / docker (pull_request) Blocked by required conditions
CI / docker (push) Blocked by required conditions
to 8aa9736f0f
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 03:05:41 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 8aa9736f0f
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to 3315daa72d
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 03:46:21 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 3315daa72d
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to 72f72a47c8
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 05:16:43 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 72f72a47c8
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to f9642e5491
Some checks failed
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
CI / test (push) Failing after 35s
CI / docker (push) Has been cancelled
2026-06-01 06:35:41 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 6b005e4ecd
Some checks are pending
CI / test (push) Waiting to run
CI / test (pull_request) Waiting to run
CI / docker (push) Blocked by required conditions
CI / docker (pull_request) Blocked by required conditions
to 21e9c2076b
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 09:46:34 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 21e9c2076b
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to a0e4bdd9a5
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 10:36:06 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from a0e4bdd9a5
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to 86cca74533
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 13:16:19 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-multi-arch-container-image-amd64-arm64 from 50cef7dc23
Some checks are pending
CI / test (push) Waiting to run
CI / test (pull_request) Waiting to run
CI / docker (push) Blocked by required conditions
CI / docker (pull_request) Blocked by required conditions
to 5f0e98ab44
Some checks failed
CI / test (push) Failing after 2h0m8s
CI / test (pull_request) Failing after 2h0m9s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
2026-06-01 16:16:39 +00:00
Compare
Native CI: add .solstice/job.sh and share one check script with Forgejo
Some checks failed
CI / docker (push) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / test (push) Has been cancelled
CI / test (pull_request) Has been cancelled
e5ead22bbf
`get_project_ci_status` reported `resolvable: false` for Sia — no
`default_ci_steps` and no `.solstice/job.sh` — so every native dispatch
failed at config resolution before running anything.

That failure is not neutral. anima-server's solstice.rs routes it to
`handle_config_failure`, which parks the work item with
`ParkedReason::CiConfigUnresolvable`. The ABSENCE of this file reads as
"this work item's CI is broken" and parks real work. akh-medu learned this
the expensive way (8/8 native dispatches failed, the WI 1/2/3 series parked)
and its job.sh states the rule: a missing file must not read as "the code is
red". Sia had CI only on the Forgejo-Actions lane, so the native lane — the
one Anima's exact-head gate reads — had never produced a verdict at all.

One definition of the checks, two lanes
---------------------------------------
scripts/ci-checks.sh is now the only place the cargo commands are written.
Both .solstice/job.sh and .forgejo/workflows/ci.yml call it; neither re-lists
them. akh-medu's rule is that a native verdict must never be red for work the
Actions lane would call green, because the disagreement parks work items on
phantom failures. Two inline copies drift the first time someone edits one;
sharing the script makes that structurally impossible rather than a reviewer's
responsibility.

job.sh only provisions. It deliberately differs from the Anima/akh-medu
scripts in two places:

- It does not assume passwordless sudo. Those scripts call `sudo apt-get`
  unconditionally because their image grants it; a runner without sudo would
  fail with a bare "sudo: command not found" that classifies as a code
  failure and parks the item. Privilege is resolved once, only missing
  packages are installed, and the no-privilege case exits with a message
  naming the packages and the reason.
- It does not `rm -rf ../anima`. The workflow did, and on a persistent or
  shared runner that deletes a checkout the job does not own. An existing
  sibling is reused; a non-git path there is refused rather than removed.
  The resolved revision is always logged so an upstream anima-proto change
  is attributable. ANIMA_SIBLING_REV pins a run without freezing CI to a
  revision the shipped image does not use.

protoc keeps ci.yml's pinned 25.3 release, since Ubuntu's 3.12 rejects
anima-proto's proto3 `optional` fields, and installs into the checkout so it
needs no privileges. The build is debug: release codegen is what OOM-killed
rustc on the Solstice VMs for akh-medu, and an OOM reads as red on healthy
code. Sia's release build is still covered by the `docker` job on push to
main.

Validation
----------
scripts/test-ci-job.sh replaces every external command with a logging stub and
asserts on the control flow — the branches that only execute on a runner in an
unusual state, i.e. never until the day they matter. 10 scenarios, 36
assertions, all passing; it needs no toolchain and runs in seconds, so it is
wired into ci-checks.sh and guards the CI definition on both lanes.

It earned its keep immediately, catching three defects before any runner saw
the script: `"${profile_args[@]}"` on an empty array is an unbound-variable
error under `set -u` on bash < 4.4, so the script would have worked on the
runner but not locally, defeating the point of sharing it; the openssl probe
hardcoded /usr/include/openssl/ssl.h and reported a false "missing" on any
other layout, triggering a pointless privileged install; and the harness's own
no-sudo case was inheriting the host's cached sudo credentials, leaving the
no-privilege branch untested exactly where it was most likely to be wrong.

No application change: no Cargo.toml edit, no dependency version change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native CI: fix four blockers found in review of e5ead22
Some checks failed
CI / test (pull_request) Failing after 5m45s
CI / docker (pull_request) Has been skipped
CI / test (push) Failing after 7m1s
CI / docker (push) Has been skipped
255b3fc3b2
All four were real. Reproduced each before fixing.

1. The install died as root. `$SUDO DEBIAN_FRONTEND=noninteractive apt-get`
   looks like an assignment prefix but is not one: bash decides that from the
   text BEFORE expansion, and the first word is `$SUDO`. Once that expands to
   nothing — i.e. running as root, the case that should always work — the
   assignment becomes the command name:

       $ SUDO=""; $SUDO DEMO_ONLY=1 true
       bash: DEMO_ONLY=1: command not found   (rc=127)

   Routed through `env`, which is a real command in both shapes. The harness
   now exercises root explicitly via an `id` stub rather than only covering
   sudo/non-root, so the escalation-free path has real coverage.

2. A git worktree's `.git` is a FILE holding a gitdir pointer, so `[ -d .git ]`
   rejected a perfectly valid shared worktree — and the next branch then
   refused to run at all. This checkout is itself a worktree, so the script
   would have rejected the very tree it was written in. Now asks git
   (`rev-parse --is-inside-work-tree`) and confirms the path is that
   worktree's ROOT by comparing realpaths, so a subdirectory of an unrelated
   repo above it is still refused — non-destructively.

3. ANIMA_SIBLING_REV claimed a pin it never enforced: it was only applied on
   the clone path, so with an existing checkout the requested revision was
   silently ignored and the verdict was attributed to a revision that was
   never built. Now validated against the existing checkout and REFUSED on
   mismatch, without touching a checkout this job does not own. An unknown rev
   is likewise refused rather than fetched into it.

   ANIMA_SIBLING_DIR is gone. It was misleading: cargo reads the override path
   from the workspace [patch] table, so pointing the variable elsewhere
   provisioned a directory cargo would never look at while the build still
   failed on the real path. The sibling location is now derived from
   Cargo.toml, which is the only thing that actually decides it, and a missing
   [patch] path fails loudly instead of guessing.

4. `clippy --workspace --all-targets -- -D warnings` was stricter than the
   ci.yml it replaced (libs/bins only). Pulling tests, benches and examples
   into a -D warnings gate for the first time risks an unrelated baseline
   failure landing as a red verdict on unrelated work — the phantom-failure
   park this work item exists to stop. Restored to the previously-verified
   scope. Widening is a separate change that needs a clean baseline run and
   then moves both lanes together.

Also fixes unbounded recursion introduced when the harness was wired into
ci-checks.sh: the harness runs job.sh, which execs ci-checks.sh, which ran the
harness. Guarded with CI_SKIP_JOB_HARNESS, which the harness sets for its
children, and the guard itself is now covered.

Harness: 18 scenarios, 62 assertions, all passing. New fixtures are real, not
mocked around — an actual shared worktree created with `git worktree add`, an
actual non-root checkout, actual commit SHAs for the pin cases, and a real
Cargo.toml with a [patch] table so the derivation runs the same code path the
runner takes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Dockerfile needed no changes. It compiles the binary from source and
downloads nothing architecture-specific, so it is already arch-agnostic —
buildx emulation produces a correct arm64 image without a cross-toolchain.
The triage plan proposed TARGETARCH cross-compilation; that is an
optimisation, and it would introduce exactly the linker troubleshooting the
plan itself flagged as the risk. Not done, on purpose.

What was actually missing was the build invocation and the chart's pin.

  - `scripts/build-image.sh` and the CI `docker` job now use
    `docker buildx build --platform linux/amd64,linux/arm64 --push`, which
    emits a manifest list. The separate `docker push` steps are gone: a
    manifest list cannot be loaded into the local daemon, so buildx pushes
    directly. CI registers binfmt handlers first (idempotent) or the runner
    cannot execute aarch64 binaries.
  - `--build-only` is now explicitly single-platform and says so, because
    `--load` cannot take a manifest list. Overridable via `PLATFORMS`.
  - The chart's `nodeSelector: {kubernetes.io/arch: amd64}` is gone. Its own
    comment said "remove the nodeSelector once we have multi-arch". The
    template guards with `{{- with .Values.nodeSelector }}`, and an empty map
    is falsy in Go templates, so nothing renders.

This does NOT by itself let pods schedule on arm64. The homelab HelmRelease
also sets `nodeSelector: {kubernetes.io/arch: amd64}`, and a HelmRelease
value overrides the chart default. That file is in the Talos GitOps repo and
owned by another worker, so it is not touched here; removing it is recorded
in the handback as the follow-up that makes this effective. The ordering is
safe either way — until it is removed the workload simply stays amd64, which
is today's behaviour.

Verified statically, no image built: `bash -n` on build-image.sh; ci.yml
parses and the docker job is asserted to contain `buildx build`, `--push`,
`linux/arm64` and binfmt registration with no stale `docker push` command;
values.yaml parses with `nodeSelector: {}`. The first real multi-arch build
is the remaining verification and is reported as such — it is a heavy build
and was deliberately not run here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Puts this PR's original head into the branch's ancestry so the canonical
work-item branch can fast-forward without a force push and without losing
the PR's history.

The final tree is deliberately the reconciled one. The original branch is 44
commits behind main; much of its diff is .anima-exec/ runner telemetry that
was never part of the change, and the packaging surface it targeted has since
moved (the homelab HelmRelease now pins an immutable digest and disables
Keel). The work is reimplemented against main's current shape rather than
merged forward.

Net diff against current main is unchanged by this merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge the native CI job script so this head can produce a CI verdict
Some checks failed
CI / test (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / test (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
fce4031183
Native dispatch reads .solstice/job.sh from the tested commit; without it
every dispatch against this head fails at config resolution and re-parks the
work item. Additive; no packaging change on this branch is modified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
toasterson changed title from WIP: Multi-arch container image (amd64 + arm64) to Multi-arch container image (amd64 + arm64) 2026-09-22 11:17:52 +00:00
CI: make the job-script harness hermetic in command visibility and identity
All checks were successful
CI / test (push) Successful in 31m30s
CI / docker (push) Has been skipped
CI / test (pull_request) Successful in 29m37s
CI / docker (pull_request) Has been skipped
91070144c8
Native Linux CI on 255b3fc failed 56 passed / 6 failed
(request 982fb0cb-eb22-46d6-8dc1-2076e0e4e19c). The harness was not hermetic,
so it asserted against the host instead of the fixture, and the same suite
passed on macOS and failed on the CI container.

Two leaks, both real.

PATH fell through to /usr/bin:/bin. "Missing pkg-config" was therefore not
missing on any image that ships it, so scenarios 4, 5 and 11 silently
exercised the already-provisioned path rather than the install branches they
name. macOS has no /usr/bin/pkg-config, Debian does — hence the split.

Identity was inherited. job.sh branches on `id -u`, and an `id` stub was
installed only for the root scenario. On a container running as root every
"non-root" scenario took the root path, leaving the no-privilege branch
untested exactly where it was most likely to be wrong.

Fixes, in the harness only — job.sh's provisioning semantics are unchanged:

  - $BIN is now the ENTIRE PATH under test. Every legitimately-needed real
    tool is linked into it explicitly (bash sh env git sed awk grep cut head
    sort uniq mkdir rm cp ln chmod cat true false dirname basename touch wc),
    and nothing else is reachable, so absence is real absence.
  - `id` is stubbed for EVERY mode from the scenario's declared identity
    (root -> 0, otherwise 1000), never inherited from the host.

No assertion was dropped or weakened; the six that were failing now exercise
what their names claim. A new scenario 19 guards the leak itself — that PATH
under test is exactly the sandbox bin, that an unprovided command is absent,
that host pkg-config is unreachable, that identity is the declared one, and
that real tools are still reachable — so this cannot regress silently into
"passes on the developer's machine".

Proof, same container both ways (debian:bookworm-slim, pkg-config installed):

  pre-fix  (255b3fc harness), uid 0 : passed: 56   failed: 6
                                      exactly the six CI reported
  post-fix (this commit),     uid 0 : passed: 68   failed: 0
  post-fix (this commit),  uid 1000 : passed: 68   failed: 0
  post-fix, macOS host              : passed: 68   failed: 0

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge the hermetic-harness CI fix
All checks were successful
CI / test (push) Successful in 31m38s
CI / docker (push) Has been skipped
CI / test (pull_request) Successful in 31m9s
CI / docker (pull_request) Has been skipped
28ee30a5ed
Native Linux CI on the CI branch failed 56/6 because the job-script harness
was not hermetic: PATH fell through to /usr/bin so "missing pkg-config" was
not missing on Debian, and `id` was inherited so non-root scenarios ran as
root. Both are fixed in the harness only; job.sh's provisioning semantics are
unchanged.

Additive: no application or packaging change on this branch is modified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All checks were successful
CI / test (push) Successful in 31m38s
CI / docker (push) Has been skipped
CI / test (pull_request) Successful in 31m9s
CI / docker (pull_request) Has been skipped
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019e534c-multi-arch-container-image-amd64-arm64:claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch claude/wi-019e534c-multi-arch-container-image-amd64-arm64

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git rebase main
git switch main
git merge --ff-only claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git rebase main
git switch main
git merge --no-ff claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch main
git merge --squash claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch main
git merge --ff-only claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git switch main
git merge claude/wi-019e534c-multi-arch-container-image-amd64-arm64
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/sia!5
No description provided.