Synchronize sandbox allocation regression without request timing races #682

Merged
toasterson merged 1 commit from fix/sandbox-allocation-test-synchronization into main 2026-10-20 00:34:12 +00:00
Owner

The required sandbox suite on combined main failed in slow_executor_does_not_serialize_another_allocation before it even attempted the fast allocation: the mock had not received the first request within one second. That includes durable allocation writes and HTTP scheduling, which have no one-second latency contract.

The regression now holds the slow executor response behind a semaphore. It starts the second allocation only after the first reaches the mock, asserts the fast allocation is Ready while the slow one is still Attaching, then opens the gate and verifies the slow allocation also completes. A single 30s watchdog detects a deadlock; correctness comes from the gate ordering rather than relative request/sleep timing. Production code is unchanged.

Diagnosis and provenance

Head: df1b49751f114a8e19bdcfc89b3dc887a902ad14. Base: eb4c13cdfd9c6107b94de8e9b65bfdc122a49310 (combined main with #673, ACP fix and #674–#678 via #681). Solstice request 6fd11ab4-b385-4c6f-a9a9-48fdd4cafa48, category step:verify-independent-daemon-profiles-and-platform-contracts, records sandbox.rs:362 Elapsed(()); suite 5 passed/1 failed, exit101. Main run 4377.

Authoritative platform category logs passed on all reviewed PR head refs: #675 (80464684, 6 sandbox tests), #676 (c8a1ea49), #677 (77cf2afa), #678 (0cb59f25) and #681 (ab01bab3, 2 sandbox tests each before the #675 integration). The slow test body is byte-identical at all these heads and failing main. Main's complete sandbox production file is byte-identical to #675; the tested path never calls release-proof or process-capture GC. Every sandbox test has its own service instance/database. The unchanged combined-main sandbox suite also passed locally. These observations identify a timing-sensitive test; no release-proof/GC interaction or production deadlock was demonstrated. The exact resource/scheduler delay of the CI miss was not traced.

Validation

  • Baseline combined-main sandbox suite: 6 passed.
  • Gated regression/full sandbox suite with sandbox-only feature selection: 6 passed.
  • Negative control: temporarily replacing per-session lock keys with a global key makes the revised regression fail at its watchdog; original production source restored byte-for-byte. The intentional failure is excluded from green results.
  • Repeated default-feature sandbox suite: 20 complete runs, 120 passed, 0 failed, 0 ignored, using cargo test --locked -p akh-session-services --features sandbox --test sandbox -j 4 -- --nocapture with dev/test debug0 and incremental0. This includes all six release-proof/retention/concurrency sandbox cases, not only the failing test.

Local receipts and exact-head comparisons are retained under analysis-anima-3w/ci4377/; source/log hashes bind the results. rustfmt and diff checks pass. The build guard stayed above 38.42 GiB free (31 GiB minimum required); own finished target was verified without open files and deleted, leaving 41.45 GiB free. Other targets were untouched. No PR merge, deployment, live recovery or CI rerun.

Exact-head CI is pending in run4378; no new Solstice success receipt has been claimed.

The required sandbox suite on combined main failed in `slow_executor_does_not_serialize_another_allocation` before it even attempted the fast allocation: the mock had not received the first request within one second. That includes durable allocation writes and HTTP scheduling, which have no one-second latency contract. The regression now holds the slow executor response behind a semaphore. It starts the second allocation only after the first reaches the mock, asserts the fast allocation is Ready while the slow one is still Attaching, then opens the gate and verifies the slow allocation also completes. A single 30s watchdog detects a deadlock; correctness comes from the gate ordering rather than relative request/sleep timing. Production code is unchanged. ## Diagnosis and provenance Head: `df1b49751f114a8e19bdcfc89b3dc887a902ad14`. Base: `eb4c13cdfd9c6107b94de8e9b65bfdc122a49310` (combined main with #673, ACP fix and #674–#678 via #681). Solstice request `6fd11ab4-b385-4c6f-a9a9-48fdd4cafa48`, category `step:verify-independent-daemon-profiles-and-platform-contracts`, records sandbox.rs:362 `Elapsed(())`; suite 5 passed/1 failed, exit101. [Main run 4377](https://code.aopc.cloud/toasterson/akh-medu/actions/runs/4377/jobs/0). Authoritative platform category logs passed on all reviewed PR head refs: #675 (`80464684`, 6 sandbox tests), #676 (`c8a1ea49`), #677 (`77cf2afa`), #678 (`0cb59f25`) and #681 (`ab01bab3`, 2 sandbox tests each before the #675 integration). The slow test body is byte-identical at all these heads and failing main. Main's complete sandbox production file is byte-identical to #675; the tested path never calls release-proof or process-capture GC. Every sandbox test has its own service instance/database. The unchanged combined-main sandbox suite also passed locally. These observations identify a timing-sensitive test; no release-proof/GC interaction or production deadlock was demonstrated. The exact resource/scheduler delay of the CI miss was not traced. ## Validation - Baseline combined-main sandbox suite: 6 passed. - Gated regression/full sandbox suite with sandbox-only feature selection: 6 passed. - Negative control: temporarily replacing per-session lock keys with a global key makes the revised regression fail at its watchdog; original production source restored byte-for-byte. The intentional failure is excluded from green results. - Repeated default-feature sandbox suite: **20 complete runs, 120 passed, 0 failed, 0 ignored**, using `cargo test --locked -p akh-session-services --features sandbox --test sandbox -j 4 -- --nocapture` with dev/test debug0 and incremental0. This includes all six release-proof/retention/concurrency sandbox cases, not only the failing test. Local receipts and exact-head comparisons are retained under `analysis-anima-3w/ci4377/`; source/log hashes bind the results. rustfmt and diff checks pass. The build guard stayed above 38.42 GiB free (31 GiB minimum required); own finished target was verified without open files and deleted, leaving 41.45 GiB free. Other targets were untouched. No PR merge, deployment, live recovery or CI rerun. Exact-head CI is pending in run4378; no new Solstice success receipt has been claimed.
test(sandbox): gate concurrent allocations instead of timing requests
All checks were successful
CI / test-native-client-computer (pull_request) Successful in 6m33s
CI / check-seshat (pull_request) Successful in 3m24s
CI / test-sokar (pull_request) Successful in 10m18s
CI / test-native-receiver-isolation (pull_request) Successful in 20m53s
CI / docker-seshd (pull_request) Successful in 8m17s
CI / test-sokar-sharing (pull_request) Successful in 31m2s
CI / test-session-platform (pull_request) Successful in 39m10s
CI / test-akh-medu (pull_request) Successful in 22m31s
CI / docker-akh-transcripts (pull_request) Successful in 8m51s
CI / publish-chart (pull_request) Successful in 3m39s
CI / check-akh-medu (pull_request) Successful in 31m2s
CI / platform-images-core (pull_request) Successful in 1m4s
CI / platform-images-home (pull_request) Successful in 1m26s
CI / platform-images-daemons (pull_request) Successful in 1m52s
CI / docker-sokard (pull_request) Successful in 23m2s
CI / platform-images-fleet (pull_request) Successful in 1m4s
CI / platform-images-collect (pull_request) Successful in 59s
df1b49751f
Sign in to join this conversation.
No reviewers
No labels
follow-up
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!682
No description provided.