toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 12 Pull requests 37 Projects Releases 3 Packages 8 Wiki Activity Actions 3

CI: build through sccache storage outages (SCCACHE_IGNORE_SERVER_IO_ERROR=1 in ci.yml and .solstice/job.sh) #639

Open
toasterson wants to merge 2 commits from anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e into main
pull from: anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
merge into: toasterson:main
Conversation 0 Commits 2 Files changed 2 +128 -11
toasterson commented 2026-10-12 17:20:43 +00:00
Owner
Copy link

Proposed changes for work item 01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e.

Anima tracks the implementation, review and validation for this work item.

<!-- anima-work-item:01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e --> Proposed changes for work item `01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e`. Anima tracks the implementation, review and validation for this work item.
WI-556 CI: build through sccache storage outages
All checks were successful
CI / check-seshat (pull_request) Successful in 2m50s
Details
CI / test-sokar (pull_request) Successful in 10m7s
Details
CI / docker-seshd (pull_request) Successful in 9m13s
Details
CI / docker-akh-transcripts (pull_request) Successful in 11m54s
Details
CI / test-session-platform (pull_request) Successful in 25m37s
Details
CI / test-akh-medu (pull_request) Successful in 19m47s
Details
CI / docker-sokard (pull_request) Successful in 18m32s
Details
CI / publish-chart (pull_request) Successful in 17m15s
Details
CI / check-akh-medu (pull_request) Successful in 25m31s
Details
CI / platform-images-home (pull_request) Successful in 56s
Details
CI / platform-images-core (pull_request) Successful in 1m47s
Details
CI / platform-images-daemons (pull_request) Successful in 2m26s
Details
CI / platform-images-fleet (pull_request) Successful in 1m21s
Details
CI / platform-images-collect (pull_request) Successful in 56s
Details
631f98f162 
Main run 09ed501d lost three jobs to a transient HTTP 503 from the CI
build-cache storage backend, which kept the sokard:main-<sha> promotion
blocked (promotion needs all five test jobs green on one main run).

Two layers, both required for "cache outage => local compile, job green"
to be true rather than aspirational:

1. SCCACHE_IGNORE_SERVER_IO_ERROR=1, set as workflow-level env in
   .forgejo/workflows/ci.yml and at the top of .solstice/job.sh (the
   WI's prescribed fix): per-request cache-server I/O errors fall back
   to direct compilation instead of failing the rustc invocation.
   Covers mid-build storage errors on sccache versions/backends where
   those propagate to the client.

2. Probe the cache server once, before the first cargo invocation, and
   drop the wrapper when it cannot start. Measured on sccache 0.17.0
   (the version docker/fleet-support/workspace.Dockerfile pins): a dead
   cache backend fails the server's .sccache_check storage read at
   startup and every rustc probe through the wrapper then exits 2 —
   with or without the flag above. The existing start-server guard only
   covered check-akh-medu/test-akh-medu; it is now also applied to
   test-sokar, check-seshat and test-session-platform, and in job.sh it
   moved ahead of the first cargo call (check-seshat ran before it).
   The fallback unsets SCCACHE_BUCKET alongside RUSTC_WRAPPER because
   the build hosts wrap rustc via a shim that re-arms sccache whenever
   the bucket var is set (docker/fleet-support/rust-tool) — unsetting
   RUSTC_WRAPPER alone demonstrably re-wraps.

Verified by injecting cache outages against sccache 0.17.0 with a local
fake S3 whose request log proves the outage was exercised:
 - 503 mid-build (server healthy at start, then 503s): non-fatal cache
   miss / write error, build green with and without the flag.
 - 503 from job start, unguarded shape (the 09ed501d mode): exit 101,
   "Server startup failed: cache storage failed to read".
 - 503 from job start, guarded shape of this patch: server probe trips,
   wrapper + bucket dropped, build compiles natively, exit 0.

A cache outage is now an optimisation loss, never a job failure.
WI-556 CI: surface sccache startup diagnostics in cache-fallback guards
All checks were successful
CI / check-seshat (pull_request) Successful in 3m4s
Details
CI / test-sokar (pull_request) Successful in 12m19s
Details
CI / test-session-platform (pull_request) Successful in 19m30s
Details
CI / publish-chart (pull_request) Successful in 1m6s
Details
CI / docker-seshd (pull_request) Successful in 20m33s
Details
CI / test-akh-medu (pull_request) Successful in 20m49s
Details
CI / docker-akh-transcripts (pull_request) Successful in 17m6s
Details
CI / docker-sokard (pull_request) Successful in 31m14s
Details
CI / check-akh-medu (pull_request) Successful in 36m18s
Details
CI / platform-images-core (pull_request) Successful in 1m21s
Details
CI / platform-images-daemons (pull_request) Successful in 1m24s
Details
CI / platform-images-home (pull_request) Successful in 1m36s
Details
CI / platform-images-fleet (pull_request) Successful in 1m4s
Details
CI / platform-images-collect (pull_request) Successful in 1m6s
Details
53e0e71373 
Review round 1 error-handling finding (P2): every cache-startup probe sent
both output streams to /dev/null and then dropped RUSTC_WRAPPER and
SCCACHE_BUCKET with no warning, so a persistent cache outage (the main run
09ed501d HTTP 503 shape) was indistinguishable from healthy uncached
execution, and the cause was erased from the job log.

All six sccache guards (five in .forgejo/workflows/ci.yml, one in
.solstice/job.sh) now capture the 'sccache --start-server' output instead
of discarding it, and on failure emit a concise WARNING naming the exit
status and the local-compilation fallback, followed by sccache's own
startup diagnostics (one 'sccache-startup: '-prefixed line per output
line). The failure stays nonfatal: wrapper and bucket are still unset so
the rust-tool shim cannot re-arm the wrapper, and subsequent build or test
failures still propagate their diagnostics and exit status. The job.sh
combined condition is split so the 'sccache not installed' path stays
quiet while retaining the inherited-env cleanup.

Verified by executing the exact guard text extracted from both files under
bash -euo pipefail and dash -eu with a mocked sccache: healthy cache arms
the wrapper silently; injected HTTP 503 startup failure exits 0 with the
WARNING plus the 503 diagnostics on stderr and drops both variables; a
subsequent compiler error still exits 42 with its diagnostic; absent
sccache stays quiet. 48/48 probes pass. test_native_builders.py passes.
All checks were successful
CI / check-seshat (pull_request) Successful in 3m4s
Details
CI / test-sokar (pull_request) Successful in 12m19s
Details
CI / test-session-platform (pull_request) Successful in 19m30s
Details
CI / publish-chart (pull_request) Successful in 1m6s
Details
CI / docker-seshd (pull_request) Successful in 20m33s
Details
CI / test-akh-medu (pull_request) Successful in 20m49s
Details
CI / docker-akh-transcripts (pull_request) Successful in 17m6s
Details
CI / docker-sokard (pull_request) Successful in 31m14s
Details
CI / check-akh-medu (pull_request) Successful in 36m18s
Details
CI / platform-images-core (pull_request) Successful in 1m21s
Details
CI / platform-images-daemons (pull_request) Successful in 1m24s
Details
CI / platform-images-home (pull_request) Successful in 1m36s
Details
CI / platform-images-fleet (pull_request) Successful in 1m4s
Details
CI / platform-images-collect (pull_request) Successful in 1m6s
Details
This pull request has changes conflicting with the target branch.
  • .forgejo/workflows/ci.yml
  • .solstice/job.sh
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e:anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git rebase main
git switch main
git merge --ff-only anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git rebase main
git switch main
git merge --no-ff anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch main
git merge --squash anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch main
git merge --ff-only anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git switch main
git merge anima/01a130d9-cd0c-7bb1-ab8b-b9de509b5b3e
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
follow-up
Documented operational follow-up

No labels
follow-up
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!639
No description provided.