Require exact workspace release proofs and durable cleanup backoff #675

Merged

toasterson merged 10 commits from fix/anima-a2-release-receipts into main 2026-10-19 20:00:44 +00:00

Conversation 0 Commits 10 Files changed 10 +1771 -124

toasterson commented 2026-10-19 08:12:26 +00:00

Owner

Copy link

Exact-owner release proof now covers never-bound manager-owned allocations, while cleanup outages/fences remain retryable and partial-work retention has a resource bound.

Targets main independently. Supplies Anima S1; no dependency on output/telemetry. No PR merge, deployment, live recovery or canary.

Review fixes

Manager-owned unbound resources receive first-party deletion/absence receipts; uncertain creates/commands still need explicit reconciliation. Fence waits use durable wait_attempts rather than the12 real-failure escalation counter. An audited admin-only re-arm endpoint resets cleanup without releasing ownership. Temporary partial-work retention on non-retain profiles expires after24h; a30s manager sweep performs UID-fenced deletion, with admin inventory and serialized duplicate expiry. Small owner/audit proof records are permanent by design.

Exact consumer contract

Workspace release v1 (A2; Anima S1)

An authenticated session manager calls POST /v1/allocations/{session}/release-proof:

{"session_id":"s","principal":"owner","profile":"workspace-profile","workspace":{"id":"w","sandbox_id":"sb","executor":"e","generation":7,"site":"site","root":"/work","platform":"linux","confinement":"sandbox"},"preserve_files":true}

workspace may be null before binding. The manager validates the stored principal,
profile and exact binding when supplied. It returns a durable AllocationReleaseProof:

{"schema_version":1,"manager_id":"sandboxes","request":{"session_id":"s","principal":"owner","profile":"workspace-profile","workspace":null,"preserve_files":false},"allocation_id":null,"workspace":null,"state":"absent","executor_receipt":null,"reconciliation_request_id":null,"recorded_at":1}

State is released | retained | absent. A non-absent proof names allocation_id
and actual workspace. It requires executor_receipt or the request ID of an
existing authenticated explicit operator absence reconciliation. The executor
receipt is {schema_version:1,session_id,workspace_id,generation,receipt_id, released:true,files_retained:bool}; the receipt is saved after fencing new effects,
observing no unresolved operations and stopping helpers. The manager saves that
exact receipt before resource cleanup, allowing restart/lost-response recovery
without asking a replacement executor to certify its predecessor's effects.

Explicit operator accepted loss uses the existing nullable field
reconciliation_request_id (a nonempty string), with state="released".
The authenticated manager copies this ID from its durable
allocation_release_attestation/<session> record. That record is written only by
admin allocation reconciliation with observed_absent=true, status completed,
quiescent owners, an exact allocation-request fingerprint and bounded evidence;
the operator/evidence audit remains durable under that request ID. Lost-executor
recovery also verifies resource absence through Kubernetes.

This attestation accepts file loss even if request.preserve_files=true, since
retention cannot be claimed after absence. It does not change the requested
preservation intent or fabricate files_retained=true/a retention deadline.
The proof retains the original bound workspace when known. Any requested binding
must still match exactly, including generation; proof.workspace=null is valid
on this path when request.workspace=null (with a nonnull allocation ID).
An empty/missing reconciliation ID cannot authorize this exception. An attached
executor receipt must still identify the exact released session/workspace and
generation; contradictory release/identity receipts are rejected. Receipt-only
preservation still requires the existing retained/files-retained proof.

For a cancelled bound turn whose pod is absent, the supported sequence is admin
allocation reconcile (explicit accepted-loss evidence), admin session cleanup
rearm (exact expected_first_blocked_at), then the normal cleanup tick. The
manager persists the attested proof and the session lifecycle exposes it with
workspace_released=true. Replays return the same proof, including its audit ID;
no replacement executor is asked to certify prior effects. Deploy this #675
follow-up before Anima #801's matching validator. No live recovery is authorized
or performed by this change.

absent is only returned with a null requested workspace, after the manager
atomically records a permanent tombstone under the allocation lock. A delayed
Allocate for that session is refused. A bare HTTP 404, missing/Completed pod,
transport failure, old terminal allocation or age is never a new proof. Legacy
manager routes cannot provide the new proof and fail closed during mixed rollout.

The session lifecycle response retains workspace_released. New optional fields:

• workspace_release_proof: the validated proof above, durably committed with
  workspace_released=true.
• cleanup: {schema_version:1,first_blocked_at,last_attempt_at,attempts,cause, next_retry_at:u64|null,operator_required:bool}. Times are epoch milliseconds.

Cause is allowlisted: publication_pending, turn_pending, transport,
proof_unavailable, unknown_outcome, owner_or_quiescence_conflict,
cleanup_busy, cleanup_unavailable. No raw endpoint/error text is included.
Cleanup uses a separate attempt counter, exponential 1s..300s durable backoff and
operator-required escalation after 12 failed automated attempts. Repeated ticks
before the saved deadline do not send another release or refresh first-blocked.
The owner fence remains on exhaustion. Existing operator reconciliation remains
explicit; no automatic recovery, claim release or effect replay is introduced.

Pending/unknown repository publications and nonterminal runtime turns prevent the
release call. Failed/cancelled turns request preserve_files: after exact executor
quiescence, the allocation becomes retained with files intact. A resource whose
deletion already started cannot attest later file retention. Existing archive and
core acceptance obligations remain. Consumers must not finalize on
workspace_released=false; historical bool-only receipts remain historical.

The brief's WI-568 pointer was checked: current Akh WI-568 is unrelated RSI-1
(01a14a7c-820c-7d20-b37d-4855572117ad), so no duplicate snapshot path was created.
Source synthesis E01/E04/E06 does not establish deployment or live recovery.
Deployment and an authorized canary with original session UUIDs remain pending.

Review additions: non-absent proofs may use manager_release {schema_version:1,session_id,allocation_id,request_id,resource_uid?,absent_observed:true} only for an unbound allocation the manager itself deleted/observed absent under its allocation lock. Ambiguous outstanding creates/commands still require explicit reconciliation. Absence without any manager-owned allocation remains a tombstone proof.

Cleanup now separates wait_attempts (durable poll backoff) from attempts (real failures). turn_pending/publication_pending/transport/busy waits never exhaust the escalation budget. Admin-only POST /v1/sessions/{id}/cleanup/rearm accepts {request_id,expected_first_blocked_at}; idempotent audit and clock guard reset retry counters without releasing ownership.

Temporary preserve_files on a non-retain profile expires after 24h (retained_until in proof). Manager sweeps up to64 expired allocations every30s after exact executor quiescence, with UID-fenced resource deletion; failed expiry remains visible via admin GET /v1/allocations/retained and retries. Explicit retain_on_release profiles keep their existing policy. Receipt/proof/tombstone records are permanent small audit/owner-fence records, not bulk captures; expiry does not erase the historical proof and its stated retention deadline.

Validation

Current follow-up head: 80464684ecc2b9afb7c9978aab8841d14dc55971.

Fable review of Anima #801 reopened this cross-repo case after the earlier #675 approval: cancelled turns request preservation, but operator-attested absence produced a released bound proof which the old validator rejected. The existing nonempty reconciliation_request_id is now the explicit accepted-loss exception for state=released, with the same exact expected binding checks. It is durably copied from admin reconciliation; no schema field or version is added. A null actual workspace is valid when the request workspace is null. Any supplied executor receipt must still agree on released/session/workspace/generation. Receipt-only retention requirements are unchanged.

Anima #801's implementer received the exact field semantics and full dependency head. Roll out this #675 head before the matching Anima validator; no deployment or live recovery was performed.

Prior local validation at operator-attestation head 0363aa65518210a22ca0eb220d269f832ed85350 (predates the CI fixture follow-up below):

• All36 akh-session-types library tests passed, including bound/null attestation, empty/missing audit ID, stale binding and contradictory executor-receipt cases.
• Session+sandbox service library:65 passed,1 existing environment-dependent test ignored. The real HTTP operator sequence creates a cancelled bound turn, preserves the 404 fence, rejects reconciliation while the pod exists, accepts explicit absence, calls cleanup/rearm, restarts the session app and exposes the durable released proof with the original workspace and attestation ID. The turn remains cancelled; no replacement executor is contacted and no retention is invented.
• All6 sandbox integration tests passed, including the defining lost-response/restart proof test.
• Negative control: substituting the old validator made the new full sequence fail exactly at lifecycle workspace_released=false after reconcile/rearm. The fixed validator was restored before all final passing runs.
• rustfmt and diff checks passed. Finished target verified without open files and deleted;36.42GiB free at10:52:56Z, peer target untouched. Builds used a31.5GiB stop/32GiB preflight guard to preserve the required31GiB.

Local receipts: attestation-types-final.log, attestation-services-final.log, attestation-sandbox-final.log, attestation-negative-control.log. Exact-head Solstice CI remains a separate merge gate; no blind rerun or shared CI incident investigation. No merge, force-push, deployment, Talos/Ptah edit or provider canary.

CI4363 follow-up

Normal follow-up head: 80464684ecc2b9afb7c9978aab8841d14dc55971. Main 67786deb93a2fe1bc421cbba2477c9c021aea2b0 (merged round16 #673) was integrated via ordinary merge 7960cbcd046076d70a26b466e3cec10e88fac0cd; round16 paths match main exactly. No force push.

Authoritative Solstice request 468c4fe6-e438-4225-a987-ef60b4c69703, category step:verify-independent-daemon-profiles-and-platform-contracts, tied to exact old head 0363aa65518210a22ca0eb220d269f832ed85350, identifies archived_cleanup_cannot_occupy_all_dispatch_workers at tests/sessions.rs:1817: timeout waiting for cleanup notification. The service library passed189 tests/7 ignored, sandbox integration passed6, then sessions integration passed28/failed1. The runner executed the tests; this is a stale mock route, not infrastructure or the new cancelled-bound operator sequence. Reported failing job.

The mock now handles /v1/allocations/{id}/release-proof, consumes the typed request, checks exact session/workspace identity and constructs a validating executor proof before signaling. The regression also asserts no slow cleanup RPC completed before active work dispatches. Original3s/4s timeouts and30s blocked RPC remain unchanged. Production release/attestation contract is unchanged. #676–#678 still use legacy /release with matching mocks and do not have this mismatch.

Validation: rustfmt and git diff checks passed. Both local attempts stopped during compilation under the disk guard, before any test executed; no local pass is claimed. The focused retry used only -p akh-session-services --no-default-features --features session,sandbox --test sessions archived_cleanup_cannot_occupy_all_dispatch_workers -j4 -- --exact --nocapture, dev/test debug0 and incremental0. CoS authorized pushing this ordinary commit and using exact-head Solstice CI when it could not fit. CI remains required; no blind rerun. Logs/hashes are recorded in ci4363-source-evidence.json and ci4363-validation.json.

Own target was verified without open files and deleted;32.83GiB free after cleanup, peer targets untouched. Guard stop31.30GiB preserved the31.25GiB floor. Anima #801 implementer was sent the new full head for its dependency pin; reconciliation_request_id semantics are unchanged. No PR merge, deployment, live recovery, Talos change or Ptah implementation edit.

Exact-owner release proof now covers never-bound manager-owned allocations, while cleanup outages/fences remain retryable and partial-work retention has a resource bound. Targets main independently. Supplies Anima S1; no dependency on output/telemetry. No PR merge, deployment, live recovery or canary. ## Review fixes Manager-owned unbound resources receive first-party deletion/absence receipts; uncertain creates/commands still need explicit reconciliation. Fence waits use durable wait_attempts rather than the12 real-failure escalation counter. An audited admin-only re-arm endpoint resets cleanup without releasing ownership. Temporary partial-work retention on non-retain profiles expires after24h; a30s manager sweep performs UID-fenced deletion, with admin inventory and serialized duplicate expiry. Small owner/audit proof records are permanent by design. ## Exact consumer contract # Workspace release v1 (A2; Anima S1) An authenticated session manager calls `POST /v1/allocations/{session}/release-proof`: ```json {"session_id":"s","principal":"owner","profile":"workspace-profile","workspace":{"id":"w","sandbox_id":"sb","executor":"e","generation":7,"site":"site","root":"/work","platform":"linux","confinement":"sandbox"},"preserve_files":true} ``` `workspace` may be null before binding. The manager validates the stored principal, profile and exact binding when supplied. It returns a durable `AllocationReleaseProof`: ```json {"schema_version":1,"manager_id":"sandboxes","request":{"session_id":"s","principal":"owner","profile":"workspace-profile","workspace":null,"preserve_files":false},"allocation_id":null,"workspace":null,"state":"absent","executor_receipt":null,"reconciliation_request_id":null,"recorded_at":1} ``` State is `released | retained | absent`. A non-absent proof names `allocation_id` and actual workspace. It requires `executor_receipt` or the request ID of an existing authenticated explicit operator absence reconciliation. The executor receipt is `{schema_version:1,session_id,workspace_id,generation,receipt_id, released:true,files_retained:bool}`; the receipt is saved after fencing new effects, observing no unresolved operations and stopping helpers. The manager saves that exact receipt before resource cleanup, allowing restart/lost-response recovery without asking a replacement executor to certify its predecessor's effects. Explicit operator accepted loss uses the existing nullable field `reconciliation_request_id` (a nonempty string), with `state="released"`. The authenticated manager copies this ID from its durable `allocation_release_attestation/<session>` record. That record is written only by admin allocation reconciliation with `observed_absent=true`, status `completed`, quiescent owners, an exact allocation-request fingerprint and bounded evidence; the operator/evidence audit remains durable under that request ID. Lost-executor recovery also verifies resource absence through Kubernetes. This attestation accepts file loss even if `request.preserve_files=true`, since retention cannot be claimed after absence. It does not change the requested preservation intent or fabricate `files_retained=true`/a retention deadline. The proof retains the original bound workspace when known. Any requested binding must still match exactly, including generation; `proof.workspace=null` is valid on this path when `request.workspace=null` (with a nonnull allocation ID). An empty/missing reconciliation ID cannot authorize this exception. An attached executor receipt must still identify the exact released session/workspace and generation; contradictory release/identity receipts are rejected. Receipt-only preservation still requires the existing retained/files-retained proof. For a cancelled bound turn whose pod is absent, the supported sequence is admin allocation reconcile (explicit accepted-loss evidence), admin session cleanup rearm (exact `expected_first_blocked_at`), then the normal cleanup tick. The manager persists the attested proof and the session lifecycle exposes it with `workspace_released=true`. Replays return the same proof, including its audit ID; no replacement executor is asked to certify prior effects. Deploy this #675 follow-up before Anima #801's matching validator. No live recovery is authorized or performed by this change. `absent` is only returned with a null requested workspace, after the manager atomically records a permanent tombstone under the allocation lock. A delayed Allocate for that session is refused. A bare HTTP 404, missing/Completed pod, transport failure, old terminal allocation or age is never a new proof. Legacy manager routes cannot provide the new proof and fail closed during mixed rollout. The session lifecycle response retains `workspace_released`. New optional fields: - `workspace_release_proof`: the validated proof above, durably committed with `workspace_released=true`. - `cleanup`: `{schema_version:1,first_blocked_at,last_attempt_at,attempts,cause, next_retry_at:u64|null,operator_required:bool}`. Times are epoch milliseconds. Cause is allowlisted: `publication_pending`, `turn_pending`, `transport`, `proof_unavailable`, `unknown_outcome`, `owner_or_quiescence_conflict`, `cleanup_busy`, `cleanup_unavailable`. No raw endpoint/error text is included. Cleanup uses a separate attempt counter, exponential 1s..300s durable backoff and operator-required escalation after 12 failed automated attempts. Repeated ticks before the saved deadline do not send another release or refresh first-blocked. The owner fence remains on exhaustion. Existing operator reconciliation remains explicit; no automatic recovery, claim release or effect replay is introduced. Pending/unknown repository publications and nonterminal runtime turns prevent the release call. Failed/cancelled turns request `preserve_files`: after exact executor quiescence, the allocation becomes retained with files intact. A resource whose deletion already started cannot attest later file retention. Existing archive and core acceptance obligations remain. Consumers must not finalize on `workspace_released=false`; historical bool-only receipts remain historical. The brief's WI-568 pointer was checked: current Akh WI-568 is unrelated RSI-1 (`01a14a7c-820c-7d20-b37d-4855572117ad`), so no duplicate snapshot path was created. Source synthesis E01/E04/E06 does not establish deployment or live recovery. Deployment and an authorized canary with original session UUIDs remain pending. Review additions: non-absent proofs may use `manager_release {schema_version:1,session_id,allocation_id,request_id,resource_uid?,absent_observed:true}` only for an unbound allocation the manager itself deleted/observed absent under its allocation lock. Ambiguous outstanding creates/commands still require explicit reconciliation. Absence without any manager-owned allocation remains a tombstone proof. Cleanup now separates `wait_attempts` (durable poll backoff) from `attempts` (real failures). turn_pending/publication_pending/transport/busy waits never exhaust the escalation budget. Admin-only POST `/v1/sessions/{id}/cleanup/rearm` accepts `{request_id,expected_first_blocked_at}`; idempotent audit and clock guard reset retry counters without releasing ownership. Temporary preserve_files on a non-retain profile expires after 24h (`retained_until` in proof). Manager sweeps up to64 expired allocations every30s after exact executor quiescence, with UID-fenced resource deletion; failed expiry remains visible via admin GET `/v1/allocations/retained` and retries. Explicit retain_on_release profiles keep their existing policy. Receipt/proof/tombstone records are permanent small audit/owner-fence records, not bulk captures; expiry does not erase the historical proof and its stated retention deadline. ## Validation Current follow-up head: `80464684ecc2b9afb7c9978aab8841d14dc55971`. Fable review of Anima #801 reopened this cross-repo case after the earlier #675 approval: cancelled turns request preservation, but operator-attested absence produced a released bound proof which the old validator rejected. The existing nonempty `reconciliation_request_id` is now the explicit accepted-loss exception for state=released, with the same exact expected binding checks. It is durably copied from admin reconciliation; no schema field or version is added. A null actual workspace is valid when the request workspace is null. Any supplied executor receipt must still agree on released/session/workspace/generation. Receipt-only retention requirements are unchanged. Anima #801's implementer received the exact field semantics and full dependency head. Roll out this #675 head before the matching Anima validator; no deployment or live recovery was performed. Prior local validation at operator-attestation head `0363aa65518210a22ca0eb220d269f832ed85350` (predates the CI fixture follow-up below): - All36 akh-session-types library tests passed, including bound/null attestation, empty/missing audit ID, stale binding and contradictory executor-receipt cases. - Session+sandbox service library:65 passed,1 existing environment-dependent test ignored. The real HTTP operator sequence creates a cancelled bound turn, preserves the 404 fence, rejects reconciliation while the pod exists, accepts explicit absence, calls cleanup/rearm, restarts the session app and exposes the durable released proof with the original workspace and attestation ID. The turn remains cancelled; no replacement executor is contacted and no retention is invented. - All6 sandbox integration tests passed, including the defining lost-response/restart proof test. - Negative control: substituting the old validator made the new full sequence fail exactly at lifecycle workspace_released=false after reconcile/rearm. The fixed validator was restored before all final passing runs. - rustfmt and diff checks passed. Finished target verified without open files and deleted;36.42GiB free at10:52:56Z, peer target untouched. Builds used a31.5GiB stop/32GiB preflight guard to preserve the required31GiB. Local receipts: attestation-types-final.log, attestation-services-final.log, attestation-sandbox-final.log, attestation-negative-control.log. Exact-head Solstice CI remains a separate merge gate; no blind rerun or shared CI incident investigation. No merge, force-push, deployment, Talos/Ptah edit or provider canary. ## CI4363 follow-up Normal follow-up head: `80464684ecc2b9afb7c9978aab8841d14dc55971`. Main `67786deb93a2fe1bc421cbba2477c9c021aea2b0` (merged round16 #673) was integrated via ordinary merge `7960cbcd046076d70a26b466e3cec10e88fac0cd`; round16 paths match main exactly. No force push. Authoritative Solstice request `468c4fe6-e438-4225-a987-ef60b4c69703`, category `step:verify-independent-daemon-profiles-and-platform-contracts`, tied to exact old head `0363aa65518210a22ca0eb220d269f832ed85350`, identifies `archived_cleanup_cannot_occupy_all_dispatch_workers` at tests/sessions.rs:1817: timeout waiting for cleanup notification. The service library passed189 tests/7 ignored, sandbox integration passed6, then sessions integration passed28/failed1. The runner executed the tests; this is a stale mock route, not infrastructure or the new cancelled-bound operator sequence. [Reported failing job](https://code.aopc.cloud/toasterson/akh-medu/actions/runs/4363/jobs/0). The mock now handles `/v1/allocations/{id}/release-proof`, consumes the typed request, checks exact session/workspace identity and constructs a validating executor proof before signaling. The regression also asserts no slow cleanup RPC completed before active work dispatches. Original3s/4s timeouts and30s blocked RPC remain unchanged. Production release/attestation contract is unchanged. #676–#678 still use legacy `/release` with matching mocks and do not have this mismatch. Validation: rustfmt and git diff checks passed. Both local attempts stopped during compilation under the disk guard, before any test executed; no local pass is claimed. The focused retry used only `-p akh-session-services --no-default-features --features session,sandbox --test sessions archived_cleanup_cannot_occupy_all_dispatch_workers -j4 -- --exact --nocapture`, dev/test debug0 and incremental0. CoS authorized pushing this ordinary commit and using exact-head Solstice CI when it could not fit. CI remains required; no blind rerun. Logs/hashes are recorded in ci4363-source-evidence.json and ci4363-validation.json. Own target was verified without open files and deleted;32.83GiB free after cleanup, peer targets untouched. Guard stop31.30GiB preserved the31.25GiB floor. Anima #801 implementer was sent the new full head for its dependency pin; `reconciliation_request_id` semantics are unchanged. No PR merge, deployment, live recovery, Talos change or Ptah implementation edit.

toasterson added 1 commit 2026-10-19 08:12:27 +00:00

Require exact durable workspace release proofs and back off cleanup 446c4e415e

toasterson added 2 commits 2026-10-19 08:33:45 +00:00

Validate all release owner coordinates and concurrent duplicate proof requests b6f44cbff2

Validate release request keys before proof lookup 3237a4d75a

toasterson added 3 commits 2026-10-19 09:38:23 +00:00

Repair unbound release proofs and bound temporary cleanup retention 4130048b0d

Recheck retention generation under allocation expiry lock 38e6629cd0

Delete unbound allocations even on explicit retain profiles 4d563f3673

toasterson added 1 commit 2026-10-19 10:13:52 +00:00

fix(sandbox): skip stale allocation during retention expiry a6d559dfe6

toasterson added 1 commit 2026-10-19 10:52:53 +00:00

fix(sandbox): honor audited absence for preserved workspace release 0363aa6551

toasterson added 2 commits 2026-10-19 12:44:03 +00:00

Integrate merged round16 main before cleanup CI follow-up 7960cbcd04

test(session): mock proof release in cleanup fairness regression 80464684ec

toasterson merged commit 75558325ea into main 2026-10-19 20:00:44 +00:00

toasterson referenced this pull request from a commit 2026-10-19 20:00:46 +00:00

Merge PR #675: Require exact workspace release proofs and durable cleanup backoff

toasterson referenced this pull request 2026-10-19 21:50:19 +00:00

Integrate stacked #676–#678 (A3 envelopes, telemetry, D1) into main #681

toasterson referenced this pull request 2026-10-19 23:38:35 +00:00

Synchronize sandbox allocation regression without request timing races #682

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!675

No description provided.