feat(sessions): relocatable private capabilities via OpenBao (PR 2/9) #653

Merged
toasterson merged 2 commits from feat/session-private-openbao into main 2026-10-15 23:40:51 +00:00
Owner

Session rows can survive a Postgres-backed cold restart while their MCP tokens, delegation grants and Sokar capabilities remain stranded in a pod's private files. This PR adds an opt-in OpenBao KV v2 backend and reconstructs verified private projections on a new data directory. Existing configurations continue to use local files.

This is PR 2/9, based on merged #652 and the design in #651. It is code/CI preparation only: no production enrollment, import, deployment, manifest change or volume switch. PRs 7–9 still require the disposition ledger and a separate operator GO. Please review and merge; no automatic merge is requested.

Implementation

  • akh-session-services: typed private_slots configuration, scoped opaque KV paths, CAS, exact-intent reconciliation after a lost write acknowledgement, retained revocation tombstones, bounded verified-TLS requests and fixed secret-free failures. Remote mode requires exclusive Postgres service ownership. Bootstrap credentials are reread from a private mounted token file for rotation.
  • MCP capabilities and delegation grants: private bytes precede PG receipts. A durable create intent preserves the session ID and original grant receipt across PG rollback. Cold startup validates receipts/hashes before recreating 0700 directories / 0600 files. Missing committed authority, conflicts and revocation fail closed; cached files are not a fallback.
  • Sokar: journals issuance, broker use, child publication and ordered child/parent retirement. Ambiguous issuance/use stays unresolved rather than minting replacements. Projection I/O and phase changes are serialized. Expired capabilities are not projected; retained expired-child bytes survive subsequent calls without recreating plaintext files. Child receipt loss is refused.
  • akh-private-slots: offline per-session plan/apply importer. Requires quiescence, exclusive PG ownership and a verified migration receipt; validates source receipts/hash ledger, checks the complete reviewed manifest before CAS writes, and verifies exact retries. No archive extraction or PG row mutation. Import identity is independent of the destination pod's projection directory.
  • scripts/platform-storage/PRIVATE-SLOTS.md: enrollment/configuration, crash ordering, private-source staging, rollback and remaining cutover gates. CI now runs a disposable OpenBao/Postgres contract harness.

Validation

  • cargo test --locked -p akh-session-services --lib --test sessions: 139 unit tests and 29 session integration tests passed; the real-service test is intentionally ignored in this command and run by the harness below.
  • AKH_STORE_TEST_DOCKER_CONTEXT=colima python3 scripts/platform-storage/test-private-slots.py: all 11 tests passed, including real OpenBao + exclusively owned Postgres cold restart/revocation, immutable retry/lost acknowledgement, conflicting bytes, private-write/PG-rollback recovery, missing/tampered/revoked authority, Sokar child retention/retirement/unknown outcomes/expiry, and hash-guarded import/repeat on another projection directory.
  • Default --all-targets check and independent session/runtime profile checks passed. Formatting and whitespace checks passed. Clippy completed; existing warnings remain in unrelated code, with no diagnostics in the new private-slot modules.
  • Observed local Rust/Cargo 1.97.1. The harness pulls latest, without version pins: OpenBao 2.7.1 (openbao/openbao@sha256:6d2b93856e3fcf7b18ad855a0b51eaba474dc8b79cf554379ea32034797d2acf) and PostgreSQL 18.6 (postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336). These digests record test provenance, not deployment pins. No provider tooling/image build was performed.
  • Host CI is separate evidence and will run on this PR; local checks do not assert production readiness.

Boundaries for the later cutover

PR 6 must reuse the houses migration #208/#212 SecretStore and Postgres-delivery patterns. It must deliver a narrowly scoped dynamic OpenBao writer; the daemon must not receive the ESO reader or operator identity. This PR creates no store, database role or production policy.

Sokar retains its existing file-path consumer contract. This PR reconstructs session-side files and journals broker-mediated child publication; it does not provide cross-pod file sharing or instrument direct CLI writes. Before enabling Sokar in the relocated platform, the core/workspace and manifest PRs must establish consumer-side delivery and audit every child-issuance path. Ambiguous external issuance/use still needs issuer/core evidence and operator disposition; the issuer lacks request-ID deduplication. The inspected platform configuration does not enable Sokar today.

The private importer does not claim to consume an entire backup. Unmatched files/uncommitted intents and the separate sessions-private backup must be accounted for in the disposition ledger. The protected wk01 archive was not accessed. No Board use, production credentials, cluster writes or deployment were involved.

Review fixes

  • Startup now skips valid non-active Sokar journals without projecting, issuing or advancing them, and logs an aggregate deferred-session count. Full rehydrate_private_slots() coverage exercises issuing, using, retiring and revoked, verifies healthy sessions still hydrate, and keeps ambiguous dispatch fenced.
  • Every definitive core outcome, including invalid_request, attempts private completion. transport and unknown_outcome still retain using without completing or replaying the effect.
  • A private-completion failure no longer returns past the result handling. The PG tool record keeps the original core response and a separate, defaulted private_sokar_finish_error code. Duplicate calls return that saved response without a second core request, even during an OpenBao outage; pending private reconciliation remains explicit.
  • PR 6/8 delivery decision: retain strict owner-only token checks. The current root-owned 0440/fsGroup: 10001 projected Secret must be copied using the existing private-state init pattern to a UID/GID 10001-owned 0600 file in a 0700 private directory. The runbook also requires restart-on-rotation or a trusted atomic-copy mechanism; an init-only copy cannot follow ESO rotation.
  • Fix validation: 139 unit tests and 29 session integration tests passed; all 11 disposable OpenBao/Postgres harness tests passed. All-targets build check, formatting and whitespace checks passed. Clippy completed with existing unrelated warnings. Host CI is separate and pending on the updated head.

Optional minor observations (stale-CAS diagnostic classification, create-intent retention and tombstones for never-provisioned MCP slots) are unchanged in this focused correction. No merge, production import, deployment or manifest change.

Session rows can survive a Postgres-backed cold restart while their MCP tokens, delegation grants and Sokar capabilities remain stranded in a pod's private files. This PR adds an opt-in OpenBao KV v2 backend and reconstructs verified private projections on a new data directory. Existing configurations continue to use local files. This is **PR 2/9**, based on merged #652 and the design in #651. It is code/CI preparation only: no production enrollment, import, deployment, manifest change or volume switch. PRs 7–9 still require the disposition ledger and a separate operator GO. Please review and merge; no automatic merge is requested. ### Implementation - `akh-session-services`: typed `private_slots` configuration, scoped opaque KV paths, CAS, exact-intent reconciliation after a lost write acknowledgement, retained revocation tombstones, bounded verified-TLS requests and fixed secret-free failures. Remote mode requires exclusive Postgres service ownership. Bootstrap credentials are reread from a private mounted token file for rotation. - MCP capabilities and delegation grants: private bytes precede PG receipts. A durable create intent preserves the session ID and original grant receipt across PG rollback. Cold startup validates receipts/hashes before recreating 0700 directories / 0600 files. Missing committed authority, conflicts and revocation fail closed; cached files are not a fallback. - Sokar: journals issuance, broker use, child publication and ordered child/parent retirement. Ambiguous issuance/use stays unresolved rather than minting replacements. Projection I/O and phase changes are serialized. Expired capabilities are not projected; retained expired-child bytes survive subsequent calls without recreating plaintext files. Child receipt loss is refused. - `akh-private-slots`: offline per-session plan/apply importer. Requires quiescence, exclusive PG ownership and a verified migration receipt; validates source receipts/hash ledger, checks the complete reviewed manifest before CAS writes, and verifies exact retries. No archive extraction or PG row mutation. Import identity is independent of the destination pod's projection directory. - `scripts/platform-storage/PRIVATE-SLOTS.md`: enrollment/configuration, crash ordering, private-source staging, rollback and remaining cutover gates. CI now runs a disposable OpenBao/Postgres contract harness. ### Validation - `cargo test --locked -p akh-session-services --lib --test sessions`: 139 unit tests and 29 session integration tests passed; the real-service test is intentionally ignored in this command and run by the harness below. - `AKH_STORE_TEST_DOCKER_CONTEXT=colima python3 scripts/platform-storage/test-private-slots.py`: all 11 tests passed, including real OpenBao + exclusively owned Postgres cold restart/revocation, immutable retry/lost acknowledgement, conflicting bytes, private-write/PG-rollback recovery, missing/tampered/revoked authority, Sokar child retention/retirement/unknown outcomes/expiry, and hash-guarded import/repeat on another projection directory. - Default `--all-targets` check and independent session/runtime profile checks passed. Formatting and whitespace checks passed. Clippy completed; existing warnings remain in unrelated code, with no diagnostics in the new private-slot modules. - Observed local Rust/Cargo 1.97.1. The harness pulls **latest**, without version pins: OpenBao 2.7.1 (`openbao/openbao@sha256:6d2b93856e3fcf7b18ad855a0b51eaba474dc8b79cf554379ea32034797d2acf`) and PostgreSQL 18.6 (`postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336`). These digests record test provenance, not deployment pins. No provider tooling/image build was performed. - Host CI is separate evidence and will run on this PR; local checks do not assert production readiness. ### Boundaries for the later cutover PR 6 must reuse the houses migration #208/#212 SecretStore and Postgres-delivery patterns. It must deliver a narrowly scoped dynamic OpenBao writer; the daemon must not receive the ESO reader or operator identity. This PR creates no store, database role or production policy. Sokar retains its existing file-path consumer contract. This PR reconstructs session-side files and journals broker-mediated child publication; it does not provide cross-pod file sharing or instrument direct CLI writes. Before enabling Sokar in the relocated platform, the core/workspace and manifest PRs must establish consumer-side delivery and audit every child-issuance path. Ambiguous external issuance/use still needs issuer/core evidence and operator disposition; the issuer lacks request-ID deduplication. The inspected platform configuration does not enable Sokar today. The private importer does not claim to consume an entire backup. Unmatched files/uncommitted intents and the separate `sessions-private` backup must be accounted for in the disposition ledger. The protected wk01 archive was not accessed. No Board use, production credentials, cluster writes or deployment were involved. ### Review fixes - Startup now skips valid non-active Sokar journals without projecting, issuing or advancing them, and logs an aggregate deferred-session count. Full `rehydrate_private_slots()` coverage exercises `issuing`, `using`, `retiring` and `revoked`, verifies healthy sessions still hydrate, and keeps ambiguous dispatch fenced. - Every definitive core outcome, including `invalid_request`, attempts private completion. `transport` and `unknown_outcome` still retain `using` without completing or replaying the effect. - A private-completion failure no longer returns past the result handling. The PG tool record keeps the original core response and a separate, defaulted `private_sokar_finish_error` code. Duplicate calls return that saved response without a second core request, even during an OpenBao outage; pending private reconciliation remains explicit. - PR 6/8 delivery decision: retain strict owner-only token checks. The current root-owned `0440`/`fsGroup: 10001` projected Secret must be copied using the existing private-state init pattern to a UID/GID 10001-owned `0600` file in a `0700` private directory. The runbook also requires restart-on-rotation or a trusted atomic-copy mechanism; an init-only copy cannot follow ESO rotation. - Fix validation: 139 unit tests and 29 session integration tests passed; all 11 disposable OpenBao/Postgres harness tests passed. All-targets build check, formatting and whitespace checks passed. Clippy completed with existing unrelated warnings. Host CI is separate and pending on the updated head. Optional minor observations (stale-CAS diagnostic classification, create-intent retention and tombstones for never-provisioned MCP slots) are unchanged in this focused correction. No merge, production import, deployment or manifest change.
feat(sessions): add relocatable private capabilities via OpenBao
Some checks failed
CI / check-seshat (pull_request) Successful in 6m51s
CI / publish-chart (pull_request) Successful in 57s
CI / test-sokar (pull_request) Successful in 11m57s
CI / check-akh-medu (pull_request) Successful in 11m59s
CI / docker-akh-transcripts (pull_request) Successful in 7m28s
CI / platform-images-home (pull_request) Has been cancelled
CI / platform-images-core (pull_request) Has been cancelled
CI / platform-images-daemons (pull_request) Has been cancelled
CI / platform-images-fleet (pull_request) Has been cancelled
CI / platform-images-collect (pull_request) Has been cancelled
CI / docker-seshd (pull_request) Has been cancelled
CI / docker-sokard (pull_request) Has been cancelled
CI / test-akh-medu (pull_request) Has been cancelled
CI / test-session-platform (pull_request) Has been cancelled
e15592a600
fix(sessions): defer Sokar recovery and preserve core results
All checks were successful
CI / test-sokar (pull_request) Successful in 6m20s
CI / check-seshat (pull_request) Successful in 2m33s
CI / test-akh-medu (pull_request) Successful in 12m33s
CI / test-session-platform (pull_request) Successful in 24m57s
CI / docker-seshd (pull_request) Successful in 17m56s
CI / docker-sokard (pull_request) Successful in 12m30s
CI / check-akh-medu (pull_request) Successful in 26m28s
CI / publish-chart (pull_request) Successful in 14m27s
CI / platform-images-core (pull_request) Successful in 52s
CI / platform-images-daemons (pull_request) Successful in 56s
CI / platform-images-home (pull_request) Successful in 6m47s
CI / docker-akh-transcripts (pull_request) Successful in 29m48s
CI / platform-images-fleet (pull_request) Successful in 56s
CI / platform-images-collect (pull_request) Successful in 5m19s
f1796d01a7
toasterson deleted branch feat/session-private-openbao 2026-10-15 23:40:55 +00:00
Sign in to join this conversation.
No reviewers
No labels
follow-up
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!653
No description provided.