feat(store): verified migration and exclusive ownership (PR 1/9) #652

Merged

toasterson merged 2 commits from feat/platform-store-safety into main 2026-10-15 21:55:05 +00:00

Conversation 0 Commits 2 Files changed 21 +2037 -117

toasterson commented 2026-10-15 20:48:25 +00:00

Owner

Copy link

Platform service-store startup could mark a failed redb import complete, silently ignore conflicting values, and let multiple runtimes share cached state without an exclusive owner. This PR adds verified offline migration and opt-in ownership fencing before any platform storage cutover. It implements PR 1/9 of the merged design in #651.

Changes

• Add akh-store-migrate with private, byte-preserving redb manifests, plan-only Postgres import, atomic verified import receipts, exact-manifest retry reconciliation, current-Postgres export and no-clobber redb restore. Refuse corrupt/missing sources, partial/conflicting targets, legacy markers masquerading as receipts, and active exclusive owners. Apply requires explicit --apply --quiescent; credentials are read from a named environment variable and never printed.
• Add explicit legacy, disabled and require_receipt import policies plus configurable cache capacity. Existing config/backend defaults remain compatible. Legacy imports now fail closed on unreadable sources or unequal conflicts instead of committing a misleading marker, and count only actual inserts.
• Add optional exclusive (schema, kind) ownership on one physical PostgreSQL connection. Reads, cache hits, writes and commits use that owner; reconnect is refused. Database errors and five-second query deadlines fence the instance. Independent owners use separate workers and transaction locks.
• Wire options and ownership checks through akh-session-services (akhrtd, akhsessd, akheventd and common startup) and the core operations API. Guard readiness/admission and dispatch, and cancel local service work on owner loss. Fix transaction rollback during caller unwind so it runs on the database worker.
• Add the offline runbook and a disposable PostgreSQL test harness to CI. No deployment manifests, ESO stores, database roles or houses placement/registry entries change.

Validation

• Disposable PostgreSQL harness: 47 service-store tests and 2 real akhrtd process tests passed, including all normally ignored PostgreSQL tests. Covers source preservation, exact-byte rollback, tampering, failed import rollback, receipt/retry checks, independent and competing owners, killed connections, pre-commit loss, blocked statements and caller unwind. The process test terminates the original owner connection and verifies readiness disappears and the daemon exits.
• cargo test --locked -p akh-session-services --lib: 129 passed.
• cargo test --locked --lib --features server core_service::tests: 25 passed, rerun after rebasing on current main.
• cargo check --locked -p akh-session-services --all-targets and cargo check --locked --lib --features server: passed.
• cargo clippy --locked -p akh-service-store --all-targets -- -D warnings: passed. The broader session-services strict-Clippy check reports 28 existing warnings in unchanged code (mostly collapsible if statements); it is not claimed green.
• cargo fmt --all -- --check and git diff --check: passed.
• Harness pulled postgres:latest; observed PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), digest sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336. This records the tested image, not a new pin. No provider image/tooling build or deployment was performed.

Review fixes

• Preserve the discarded sqlx error in tracing::error! at initialization and operation failures, including legacy mode. API faults stay generic; the connection URL is not logged. A corrupt-legacy-source process test verifies the specific cause appears without the URL, credential or source contents.
• Print migration Fault.message in the CLI. Integration tests distinguish nonempty targets, destinations changed after import, uncertain database outcomes, and refused no-clobber restores, without printing fixture payloads or credentials.
• Return store_owner_lost after cancellation requests, HTTP drain and telemetry cleanup. The real daemon test now requires exit code 1 and verifies the operation diagnostic and readiness loss.
• Document the non-reentrant transaction gate: use the transaction handle inside its closure, not Store reads/ownership checks.

Fix validation: 178 tests passed (47 store tests, 2 daemon process tests, 129 session-service unit tests); strict store Clippy, formatting and diff checks passed. The core test/check evidence above is retained from the initial PR validation; these fixes do not change core code. Tests use disposable local PostgreSQL only; no deployment, production import or volume change. The operator retains merge authority.

Operational boundary

This does not activate any production option, import production data, touch Pending pods, read/extract the protected wk01 archive, or switch/remove volumes. Tests use synthetic records and disposable local PostgreSQL only. No Board use.

Legacy writers must be drained before an eventual import/exclusive-owner cutover: they do not honor the new owner lock. Database ownership cannot prove an orphan provider stopped or undo an external effect; uncertain jobs retain their identity and require same-request reconciliation. Generic raw-record tooling does not split the existing shared runtime namespace or establish the disposition ledger. Core cognitive residency remains separate.

PRs 7–9 remain gated on that ledger and a separate operator GO. PR 6 must reuse the Talos #208/#212 houses artifacts and stop if registry/placement conflicts arise. Nothing is deployed here; future runtime artifacts must come from merged main. Operator reviews and merges this PR; PR 2 starts only after that merge.

Platform service-store startup could mark a failed redb import complete, silently ignore conflicting values, and let multiple runtimes share cached state without an exclusive owner. This PR adds verified offline migration and opt-in ownership fencing before any platform storage cutover. It implements **PR 1/9** of the merged design in #651. ### Changes - Add `akh-store-migrate` with private, byte-preserving redb manifests, plan-only Postgres import, atomic verified import receipts, exact-manifest retry reconciliation, current-Postgres export and no-clobber redb restore. Refuse corrupt/missing sources, partial/conflicting targets, legacy markers masquerading as receipts, and active exclusive owners. Apply requires explicit `--apply --quiescent`; credentials are read from a named environment variable and never printed. - Add explicit `legacy`, `disabled` and `require_receipt` import policies plus configurable cache capacity. Existing config/backend defaults remain compatible. Legacy imports now fail closed on unreadable sources or unequal conflicts instead of committing a misleading marker, and count only actual inserts. - Add optional exclusive `(schema, kind)` ownership on one physical PostgreSQL connection. Reads, cache hits, writes and commits use that owner; reconnect is refused. Database errors and five-second query deadlines fence the instance. Independent owners use separate workers and transaction locks. - Wire options and ownership checks through `akh-session-services` (`akhrtd`, `akhsessd`, `akheventd` and common startup) and the core operations API. Guard readiness/admission and dispatch, and cancel local service work on owner loss. Fix transaction rollback during caller unwind so it runs on the database worker. - Add the offline runbook and a disposable PostgreSQL test harness to CI. No deployment manifests, ESO stores, database roles or houses placement/registry entries change. ### Validation - Disposable PostgreSQL harness: **47 service-store tests and 2 real akhrtd process tests passed**, including all normally ignored PostgreSQL tests. Covers source preservation, exact-byte rollback, tampering, failed import rollback, receipt/retry checks, independent and competing owners, killed connections, pre-commit loss, blocked statements and caller unwind. The process test terminates the original owner connection and verifies readiness disappears and the daemon exits. - `cargo test --locked -p akh-session-services --lib`: **129 passed**. - `cargo test --locked --lib --features server core_service::tests`: **25 passed**, rerun after rebasing on current main. - `cargo check --locked -p akh-session-services --all-targets` and `cargo check --locked --lib --features server`: passed. - `cargo clippy --locked -p akh-service-store --all-targets -- -D warnings`: passed. The broader session-services strict-Clippy check reports 28 existing warnings in unchanged code (mostly collapsible `if` statements); it is not claimed green. - `cargo fmt --all -- --check` and `git diff --check`: passed. - Harness pulled `postgres:latest`; observed **PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2)**, digest `sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336`. This records the tested image, not a new pin. No provider image/tooling build or deployment was performed. ### Review fixes - Preserve the discarded sqlx error in `tracing::error!` at initialization and operation failures, including legacy mode. API faults stay generic; the connection URL is not logged. A corrupt-legacy-source process test verifies the specific cause appears without the URL, credential or source contents. - Print migration `Fault.message` in the CLI. Integration tests distinguish nonempty targets, destinations changed after import, uncertain database outcomes, and refused no-clobber restores, without printing fixture payloads or credentials. - Return `store_owner_lost` after cancellation requests, HTTP drain and telemetry cleanup. The real daemon test now requires exit code **1** and verifies the operation diagnostic and readiness loss. - Document the non-reentrant transaction gate: use the transaction handle inside its closure, not Store reads/ownership checks. Fix validation: **178 tests passed** (47 store tests, 2 daemon process tests, 129 session-service unit tests); strict store Clippy, formatting and diff checks passed. The core test/check evidence above is retained from the initial PR validation; these fixes do not change core code. Tests use disposable local PostgreSQL only; no deployment, production import or volume change. The operator retains merge authority. ### Operational boundary This does not activate any production option, import production data, touch Pending pods, read/extract the protected wk01 archive, or switch/remove volumes. Tests use synthetic records and disposable local PostgreSQL only. No Board use. Legacy writers must be drained before an eventual import/exclusive-owner cutover: they do not honor the new owner lock. Database ownership cannot prove an orphan provider stopped or undo an external effect; uncertain jobs retain their identity and require same-request reconciliation. Generic raw-record tooling does not split the existing shared runtime namespace or establish the disposition ledger. Core cognitive residency remains separate. PRs 7–9 remain gated on that ledger and a separate operator GO. PR 6 must reuse the Talos #208/#212 houses artifacts and stop if registry/placement conflicts arise. Nothing is deployed here; future runtime artifacts must come from merged main. **Operator reviews and merges this PR; PR 2 starts only after that merge.**

toasterson added 1 commit 2026-10-15 20:48:27 +00:00

feat(store): add verified offline migration and opt-in ownership fencing 3e6729680e

toasterson added 1 commit 2026-10-15 21:06:01 +00:00

fix(store): preserve diagnostics and fail daemon exit on owner loss e093a54ed7

toasterson merged commit 302d6ef799 into main 2026-10-15 21:55:05 +00:00

toasterson referenced this pull request from a commit 2026-10-15 21:55:09 +00:00

feat(store): verified migration and exclusive ownership (PR 1/9) (#652)

toasterson deleted branch feat/platform-store-safety 2026-10-15 21:55:12 +00:00

toasterson referenced this pull request 2026-10-15 22:25:46 +00:00

feat(sessions): relocatable private capabilities via OpenBao (PR 2/9) #653

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!652

No description provided.