Make Rust CI cache startup optional in every compilation step #672

Merged

toasterson merged 1 commit from fix/ci-sccache-20261018 into main 2026-10-18 17:30:19 +00:00

Conversation 0 Commits 1 Files changed 4 +119 -42

toasterson commented 2026-10-18 16:57:42 +00:00

Owner

Copy link

Rust CI can fail before compilation when Solstice's injected sccache wrapper times out starting its daemon. Main be2ad983 request 805ee630-5313-4469-976a-bc8c11c774f0 hit this on rustc -vV; the native client tests never executed. Existing startup checks were absent from native jobs and, elsewhere, only changed an earlier shell's environment.

Source a shared cache guard in every compilation shell, including both native Linux acceptance jobs and the native .solstice/job.sh path. It probes sccache rustc -vV with a 15-second timeout and 2-second kill grace, sets an empty RUSTC_WRAPPER on failure (also overriding Cargo config), and allows local compilation for subsequent server I/O errors. Missing sccache or timeout also selects direct compilation. Cache diagnostics are suppressed to avoid exposing credential-bearing URLs. Product/test commands and required receipt checks are unchanged.

Validation: eight fault-injection tests cover startup failure, timeout, missing tools, separate-shell reinjection, successful probing and preservation of compiler failure. All 17 workflow jobs parse and every run script passes sh -n; .solstice/job.sh passes bash -n. A real sccache 0.18.0 probe against an unreachable S3 address fell back in 11 seconds, compiled a tiny offline crate directly, and preserved its deliberate syntax error. This local fault test does not establish Linux acceptance for this PR.

Diagnosis: the deployed executor uses Hetzner S3 hel1.your-objectstorage.com, bucket toasty-rust-cache, over Archibald's direct Hetzner route. It does not use home SeaweedFS or the Forgejo ingress fixed by Talos #222. Current authenticated bucket HEAD returned 200; only one matching startup timeout was found among 18 failed Akh jobs since October 17. The failed guest was already removed, so the exact historical daemon/backend cause remains unproven. The pinned 0.8.2 source confirms SCCACHE_IGNORE_SERVER_IO_ERROR alone does not cover startup failures.

Parent merge required. After merge, dispatch CI push-free on the resulting new main SHA and collect authoritative Solstice terminal results plus both native receipts (executed tests, zero ignored). This PR does not make the old be2ad983 run green and does not deploy Ptah.

Rust CI can fail before compilation when Solstice's injected `sccache` wrapper times out starting its daemon. Main `be2ad983` request `805ee630-5313-4469-976a-bc8c11c774f0` hit this on `rustc -vV`; the native client tests never executed. Existing startup checks were absent from native jobs and, elsewhere, only changed an earlier shell's environment. Source a shared cache guard in every compilation shell, including both native Linux acceptance jobs and the native `.solstice/job.sh` path. It probes `sccache rustc -vV` with a 15-second timeout and 2-second kill grace, sets an empty `RUSTC_WRAPPER` on failure (also overriding Cargo config), and allows local compilation for subsequent server I/O errors. Missing sccache or timeout also selects direct compilation. Cache diagnostics are suppressed to avoid exposing credential-bearing URLs. Product/test commands and required receipt checks are unchanged. Validation: eight fault-injection tests cover startup failure, timeout, missing tools, separate-shell reinjection, successful probing and preservation of compiler failure. All 17 workflow jobs parse and every run script passes `sh -n`; `.solstice/job.sh` passes `bash -n`. A real sccache 0.18.0 probe against an unreachable S3 address fell back in 11 seconds, compiled a tiny offline crate directly, and preserved its deliberate syntax error. This local fault test does not establish Linux acceptance for this PR. Diagnosis: the deployed executor uses Hetzner S3 `hel1.your-objectstorage.com`, bucket `toasty-rust-cache`, over Archibald's direct Hetzner route. It does not use home SeaweedFS or the Forgejo ingress fixed by Talos #222. Current authenticated bucket HEAD returned 200; only one matching startup timeout was found among 18 failed Akh jobs since October 17. The failed guest was already removed, so the exact historical daemon/backend cause remains unproven. The pinned 0.8.2 source confirms `SCCACHE_IGNORE_SERVER_IO_ERROR` alone does not cover startup failures. Parent merge required. After merge, dispatch CI push-free on the resulting **new main SHA** and collect authoritative Solstice terminal results plus both native receipts (executed tests, zero ignored). This PR does not make the old `be2ad983` run green and does not deploy Ptah.

toasterson added 1 commit 2026-10-18 16:57:43 +00:00

Make Rust CI cache startup optional in every compilation step e07b23eb48

toasterson merged commit ffdf032acb into main 2026-10-18 17:30:19 +00:00

toasterson referenced this pull request from a commit 2026-10-18 17:30:21 +00:00

Make Rust CI cache startup optional in every compilation step (#672)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!672

No description provided.