Remove Keel auto-update mechanism #9

Open
toasterson wants to merge 8 commits from claude/wi-019e534c-remove-keel-auto-update-mechanism into main
Owner

Draft PR for Anima work item Remove Keel auto-update mechanism (019e534c-4908-76e1-9091-87f39e4659eb).

This PR is a scaffold opened by anima-triage. The implementation worker picks it up from here.

Close or mark ready manually once the implementation lands.

Draft PR for Anima work item **Remove Keel auto-update mechanism** (`019e534c-4908-76e1-9091-87f39e4659eb`). This PR is a scaffold opened by `anima-triage`. The implementation worker picks it up from here. - Marker: [`.anima/wip/019e534c.md`](.anima/wip/019e534c.md) - Current phase: `triaged` Close or mark ready manually once the implementation lands.
wip: triage Remove Keel auto-update mechanism
Some checks failed
CI / test (push) Failing after 2h0m2s
CI / test (pull_request) Failing after 2h0m1s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
9bc89263f1
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from 9bc89263f1
Some checks failed
CI / test (push) Failing after 2h0m2s
CI / test (pull_request) Failing after 2h0m1s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
to 1cd988a072
Some checks failed
CI / test (push) Failing after 37s
CI / test (pull_request) Failing after 37s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
2026-05-25 20:17:28 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from 1cd988a072
Some checks failed
CI / test (push) Failing after 37s
CI / test (pull_request) Failing after 37s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
to e2b7f9b85d
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 08:50:07 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from e2b7f9b85d
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to 435794e426
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 09:46:11 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from 435794e426
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to d199dca3b0
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 10:47:54 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from d199dca3b0
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to ec6271f5b7
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
2026-06-01 13:06:44 +00:00
Compare
toasterson force-pushed claude/wi-019e534c-remove-keel-auto-update-mechanism from ec6271f5b7
Some checks are pending
CI / test (push) Waiting to run
CI / docker (push) Blocked by required conditions
CI / test (pull_request) Waiting to run
CI / docker (pull_request) Blocked by required conditions
to 0273145ba1
Some checks failed
CI / test (push) Failing after 2h0m2s
CI / test (pull_request) Failing after 2h0m1s
CI / docker (push) Has been skipped
CI / docker (pull_request) Has been skipped
2026-06-01 16:02:11 +00:00
Compare
Native CI: add .solstice/job.sh and share one check script with Forgejo
Some checks failed
CI / docker (push) Has been cancelled
CI / docker (pull_request) Has been cancelled
CI / test (push) Has been cancelled
CI / test (pull_request) Has been cancelled
e5ead22bbf
`get_project_ci_status` reported `resolvable: false` for Sia — no
`default_ci_steps` and no `.solstice/job.sh` — so every native dispatch
failed at config resolution before running anything.

That failure is not neutral. anima-server's solstice.rs routes it to
`handle_config_failure`, which parks the work item with
`ParkedReason::CiConfigUnresolvable`. The ABSENCE of this file reads as
"this work item's CI is broken" and parks real work. akh-medu learned this
the expensive way (8/8 native dispatches failed, the WI 1/2/3 series parked)
and its job.sh states the rule: a missing file must not read as "the code is
red". Sia had CI only on the Forgejo-Actions lane, so the native lane — the
one Anima's exact-head gate reads — had never produced a verdict at all.

One definition of the checks, two lanes
---------------------------------------
scripts/ci-checks.sh is now the only place the cargo commands are written.
Both .solstice/job.sh and .forgejo/workflows/ci.yml call it; neither re-lists
them. akh-medu's rule is that a native verdict must never be red for work the
Actions lane would call green, because the disagreement parks work items on
phantom failures. Two inline copies drift the first time someone edits one;
sharing the script makes that structurally impossible rather than a reviewer's
responsibility.

job.sh only provisions. It deliberately differs from the Anima/akh-medu
scripts in two places:

- It does not assume passwordless sudo. Those scripts call `sudo apt-get`
  unconditionally because their image grants it; a runner without sudo would
  fail with a bare "sudo: command not found" that classifies as a code
  failure and parks the item. Privilege is resolved once, only missing
  packages are installed, and the no-privilege case exits with a message
  naming the packages and the reason.
- It does not `rm -rf ../anima`. The workflow did, and on a persistent or
  shared runner that deletes a checkout the job does not own. An existing
  sibling is reused; a non-git path there is refused rather than removed.
  The resolved revision is always logged so an upstream anima-proto change
  is attributable. ANIMA_SIBLING_REV pins a run without freezing CI to a
  revision the shipped image does not use.

protoc keeps ci.yml's pinned 25.3 release, since Ubuntu's 3.12 rejects
anima-proto's proto3 `optional` fields, and installs into the checkout so it
needs no privileges. The build is debug: release codegen is what OOM-killed
rustc on the Solstice VMs for akh-medu, and an OOM reads as red on healthy
code. Sia's release build is still covered by the `docker` job on push to
main.

Validation
----------
scripts/test-ci-job.sh replaces every external command with a logging stub and
asserts on the control flow — the branches that only execute on a runner in an
unusual state, i.e. never until the day they matter. 10 scenarios, 36
assertions, all passing; it needs no toolchain and runs in seconds, so it is
wired into ci-checks.sh and guards the CI definition on both lanes.

It earned its keep immediately, catching three defects before any runner saw
the script: `"${profile_args[@]}"` on an empty array is an unbound-variable
error under `set -u` on bash < 4.4, so the script would have worked on the
runner but not locally, defeating the point of sharing it; the openssl probe
hardcoded /usr/include/openssl/ssl.h and reported a false "missing" on any
other layout, triggering a pointless privileged install; and the harness's own
no-sudo case was inheriting the host's cached sudo credentials, leaving the
no-privilege branch untested exactly where it was most likely to be wrong.

No application change: no Cargo.toml edit, no dependency version change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native CI: fix four blockers found in review of e5ead22
Some checks failed
CI / test (pull_request) Failing after 5m45s
CI / docker (pull_request) Has been skipped
CI / test (push) Failing after 7m1s
CI / docker (push) Has been skipped
255b3fc3b2
All four were real. Reproduced each before fixing.

1. The install died as root. `$SUDO DEBIAN_FRONTEND=noninteractive apt-get`
   looks like an assignment prefix but is not one: bash decides that from the
   text BEFORE expansion, and the first word is `$SUDO`. Once that expands to
   nothing — i.e. running as root, the case that should always work — the
   assignment becomes the command name:

       $ SUDO=""; $SUDO DEMO_ONLY=1 true
       bash: DEMO_ONLY=1: command not found   (rc=127)

   Routed through `env`, which is a real command in both shapes. The harness
   now exercises root explicitly via an `id` stub rather than only covering
   sudo/non-root, so the escalation-free path has real coverage.

2. A git worktree's `.git` is a FILE holding a gitdir pointer, so `[ -d .git ]`
   rejected a perfectly valid shared worktree — and the next branch then
   refused to run at all. This checkout is itself a worktree, so the script
   would have rejected the very tree it was written in. Now asks git
   (`rev-parse --is-inside-work-tree`) and confirms the path is that
   worktree's ROOT by comparing realpaths, so a subdirectory of an unrelated
   repo above it is still refused — non-destructively.

3. ANIMA_SIBLING_REV claimed a pin it never enforced: it was only applied on
   the clone path, so with an existing checkout the requested revision was
   silently ignored and the verdict was attributed to a revision that was
   never built. Now validated against the existing checkout and REFUSED on
   mismatch, without touching a checkout this job does not own. An unknown rev
   is likewise refused rather than fetched into it.

   ANIMA_SIBLING_DIR is gone. It was misleading: cargo reads the override path
   from the workspace [patch] table, so pointing the variable elsewhere
   provisioned a directory cargo would never look at while the build still
   failed on the real path. The sibling location is now derived from
   Cargo.toml, which is the only thing that actually decides it, and a missing
   [patch] path fails loudly instead of guessing.

4. `clippy --workspace --all-targets -- -D warnings` was stricter than the
   ci.yml it replaced (libs/bins only). Pulling tests, benches and examples
   into a -D warnings gate for the first time risks an unrelated baseline
   failure landing as a red verdict on unrelated work — the phantom-failure
   park this work item exists to stop. Restored to the previously-verified
   scope. Widening is a separate change that needs a clean baseline run and
   then moves both lanes together.

Also fixes unbounded recursion introduced when the harness was wired into
ci-checks.sh: the harness runs job.sh, which execs ci-checks.sh, which ran the
harness. Guarded with CI_SKIP_JOB_HARNESS, which the harness sets for its
children, and the guard itself is now covered.

Harness: 18 scenarios, 62 assertions, all passing. New fixtures are real, not
mocked around — an actual shared worktree created with `git worktree add`, an
actual non-root checkout, actual commit SHAs for the pin cases, and a real
Cargo.toml with a [patch] table so the derivation runs the same code path the
runner takes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keel polled the registry for `:latest` digest changes and force-restarted
the Deployment. The cluster stopped relying on that some time ago — the
HelmRelease in the homelab repo already sets `keel.enabled: false` with the
note "GitOps selects the tested release; disable mutable-tag rollouts", and
pins `image.tag` to an immutable `<sha>@sha256:<digest>`. What remained was
the chart still carrying the mechanism.

Removed:
  - the `keel.sh/{policy,match-tag,trigger,pollSchedule}` annotation block
    from the Deployment template, along with its `{{- if .Values.keel.enabled }}`
    guard
  - the `keel:` values block

Ordering is safe in both directions. Removing `.Values.keel` from the chart
while the HelmRelease still passes a `keel:` block is a no-op — Helm ignores
values a chart does not consume — so this can land before or after the
homelab manifest is cleaned up, with no window where the Deployment renders
wrong.

The homelab HelmRelease's now-inert `keel:` block is NOT touched here: that
file is in the Talos GitOps repo, owned by another worker. It is recorded in
the handback as follow-up.

`docs/deployment.md` said "Keel auto-restarts the Deployment when `:latest`
advances", which has been untrue since the release was pinned. It now
describes what actually happens.

Verified: values.yaml parses; all five templates have balanced
if/range/with vs end directives (deployment.yaml 4/4 after the removal);
no `keel` reference remains anywhere under deploy/ or docs/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Puts this PR's original head into the branch's ancestry so the canonical
work-item branch can fast-forward without a force push and without losing
the PR's history.

The final tree is deliberately the reconciled one. The original branch is 44
commits behind main; much of its diff is .anima-exec/ runner telemetry that
was never part of the change, and the packaging surface it targeted has since
moved (the homelab HelmRelease now pins an immutable digest and disables
Keel). The work is reimplemented against main's current shape rather than
merged forward.

Net diff against current main is unchanged by this merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge the native CI job script so this head can produce a CI verdict
Some checks failed
CI / test (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / test (pull_request) Has been cancelled
CI / docker (pull_request) Has been cancelled
ba56e8280a
Native dispatch reads .solstice/job.sh from the tested commit; without it
every dispatch against this head fails at config resolution and re-parks the
work item. Additive; no packaging change on this branch is modified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
toasterson changed title from WIP: Remove Keel auto-update mechanism to Remove Keel auto-update mechanism 2026-09-22 11:17:54 +00:00
CI: make the job-script harness hermetic in command visibility and identity
All checks were successful
CI / test (push) Successful in 31m30s
CI / docker (push) Has been skipped
CI / test (pull_request) Successful in 29m37s
CI / docker (pull_request) Has been skipped
91070144c8
Native Linux CI on 255b3fc failed 56 passed / 6 failed
(request 982fb0cb-eb22-46d6-8dc1-2076e0e4e19c). The harness was not hermetic,
so it asserted against the host instead of the fixture, and the same suite
passed on macOS and failed on the CI container.

Two leaks, both real.

PATH fell through to /usr/bin:/bin. "Missing pkg-config" was therefore not
missing on any image that ships it, so scenarios 4, 5 and 11 silently
exercised the already-provisioned path rather than the install branches they
name. macOS has no /usr/bin/pkg-config, Debian does — hence the split.

Identity was inherited. job.sh branches on `id -u`, and an `id` stub was
installed only for the root scenario. On a container running as root every
"non-root" scenario took the root path, leaving the no-privilege branch
untested exactly where it was most likely to be wrong.

Fixes, in the harness only — job.sh's provisioning semantics are unchanged:

  - $BIN is now the ENTIRE PATH under test. Every legitimately-needed real
    tool is linked into it explicitly (bash sh env git sed awk grep cut head
    sort uniq mkdir rm cp ln chmod cat true false dirname basename touch wc),
    and nothing else is reachable, so absence is real absence.
  - `id` is stubbed for EVERY mode from the scenario's declared identity
    (root -> 0, otherwise 1000), never inherited from the host.

No assertion was dropped or weakened; the six that were failing now exercise
what their names claim. A new scenario 19 guards the leak itself — that PATH
under test is exactly the sandbox bin, that an unprovided command is absent,
that host pkg-config is unreachable, that identity is the declared one, and
that real tools are still reachable — so this cannot regress silently into
"passes on the developer's machine".

Proof, same container both ways (debian:bookworm-slim, pkg-config installed):

  pre-fix  (255b3fc harness), uid 0 : passed: 56   failed: 6
                                      exactly the six CI reported
  post-fix (this commit),     uid 0 : passed: 68   failed: 0
  post-fix (this commit),  uid 1000 : passed: 68   failed: 0
  post-fix, macOS host              : passed: 68   failed: 0

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge the hermetic-harness CI fix
Some checks failed
CI / test (pull_request) Failing after 30m58s
CI / docker (pull_request) Has been skipped
CI / test (push) Failing after 35m0s
CI / docker (push) Has been skipped
00dcbbce8e
Native Linux CI on the CI branch failed 56/6 because the job-script harness
was not hermetic: PATH fell through to /usr/bin so "missing pkg-config" was
not missing on Debian, and `id` was inherited so non-root scenarios ran as
root. Both are fixed in the harness only; job.sh's provisioning semantics are
unchanged.

Additive: no application or packaging change on this branch is modified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Some checks failed
CI / test (pull_request) Failing after 30m58s
CI / docker (pull_request) Has been skipped
CI / test (push) Failing after 35m0s
CI / docker (push) Has been skipped
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019e534c-remove-keel-auto-update-mechanism:claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch claude/wi-019e534c-remove-keel-auto-update-mechanism

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch claude/wi-019e534c-remove-keel-auto-update-mechanism
git rebase main
git switch main
git merge --ff-only claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch claude/wi-019e534c-remove-keel-auto-update-mechanism
git rebase main
git switch main
git merge --no-ff claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch main
git merge --squash claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch main
git merge --ff-only claude/wi-019e534c-remove-keel-auto-update-mechanism
git switch main
git merge claude/wi-019e534c-remove-keel-auto-update-mechanism
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/sia!9
No description provided.