Retry complete source fetches and reject incomplete CI checkouts #23

Merged

toasterson merged 2 commits from fix/checkout-refusal-20261018 into main

2026-10-18 10:26:15 +00:00

toasterson commented

2026-10-18 10:06:03 +00:00

Owner

Fetch a complete shallow tree, retry failed fetches four times with 2/4/8-second backoff, then require the exact requested HEAD, a complete object graph and an unchanged tracked worktree before running any step. Git failures now fail closed instead of overlaying an archive. The no-Git archive path retries downloads and checks downloader status, gzip integrity and tar listing before extraction; TLS verification stays enabled. Both paths have a 600-second deadline that kills the checkout process group.

Validation: cargo check --locked -p workflow-runner; cargo test --locked -p workflow-runner (4 passed); ten real-Git/archive regressions including SHA-256 repositories (complete tree, transient refusals, retry exhaustion, missing tracked file despite successful checkout, corrupt objects, wrong commit, valid archive, downloader failure after valid tar output, truncated archive); cargo fmt and diff checks. The Python regressions run in native Linux CI and the legacy Linux hook.

Operational mitigation is separate: Talos PR #222 routes Archibald CI DNS and host consumers to 192.168.1.11 over wg-home. The router remains suspected and unverified. This code PR is for parent review/merge; no runner binary from this feature branch was deployed. After merge, publish/install both merged-main runner binaries through the normal Solstice release path. Rollback is the prior runner binaries/commit; this removes the resilience gate, so retain the independent mesh mitigation.

CI's blob:none fetch leaves Git to download blobs during checkout. During the 2026-10-18 code.aopc.cloud incident, Git reported hundreds of unreadable files yet returned success; workflow steps then failed on missing Cargo.toml and CI scripts. Fetch a complete shallow tree, retry failed fetches four times with 2/4/8-second backoff, then require the exact requested HEAD, a complete object graph and an unchanged tracked worktree before running any step. Git failures now fail closed instead of overlaying an archive. The no-Git archive path retries downloads and checks downloader status, gzip integrity and tar listing before extraction; TLS verification stays enabled. Both paths have a 600-second deadline that kills the checkout process group. Validation: cargo check --locked -p workflow-runner; cargo test --locked -p workflow-runner (4 passed); ten real-Git/archive regressions including SHA-256 repositories (complete tree, transient refusals, retry exhaustion, missing tracked file despite successful checkout, corrupt objects, wrong commit, valid archive, downloader failure after valid tar output, truncated archive); cargo fmt and diff checks. The Python regressions run in native Linux CI and the legacy Linux hook. Operational mitigation is separate: Talos PR #222 routes Archibald CI DNS and host consumers to 192.168.1.11 over wg-home. The router remains suspected and unverified. This code PR is for parent review/merge; no runner binary from this feature branch was deployed. After merge, publish/install both merged-main runner binaries through the normal Solstice release path. Rollback is the prior runner binaries/commit; this removes the resilience gate, so retain the independent mesh mitigation.

toasterson added 1 commit

2026-10-18 10:06:05 +00:00