feat(sharing): Postgres companion and verified link migration (PR 5/9) #661

Merged

toasterson merged 3 commits from feat/sokar-sharing-postgres into main 2026-10-16 08:54:56 +00:00

Conversation 0 Commits 3 Files changed 15 +1487 -22

toasterson commented 2026-10-16 07:27:18 +00:00

Owner

Copy link

The Talos Sokar sharing companion currently stores mutable link authorization state only in a node-local SQLite file. This PR vendors that companion and its tests into services/sokar-sharing/ and adds an opt-in Postgres backend, retaining SQLite as the default and preserving the HTTP/security contract. It is PR 5/9, following merged #655; merging does not change production behavior.

Implementation

• Source attribution records Talos main e061ea36a85d1915072b26b9c8f7ae34f14a9147. Existing fragment capabilities, hashed token storage, expiry/revocation, content pinning, post-fetch recheck, CSP, isolated origin and credential boundary remain intact.
• LINK_STORE=postgres selects typed link rows in an explicit owned schema, with unique token hashes and expiry index. LINK_DATABASE_URL_FILE supplies the ESO-delivered connection material. Production requires sslmode=verify-full and an explicit network hostname; driver text/URLs/row values never reach responses or logs. Bounded DB operations fail closed with no SQLite fallback.
• Offline importer validates a private, checkpointed source read-only, refuses unknown objects/malformed values, verifies every destination row and commits an aggregate receipt atomically. Exact retry requires a matching receipt and unchanged destination. Nonempty/conflicting targets refuse. Lost commit ACK remains unknown until inspection/reconciliation.
• Reverse exporter copies all current rows, including PG-era links and revocations, into a new verified/fsynced 0600 SQLite file without replacing an existing path. Inspect/export do not bootstrap absent schemas. Migration takes an exclusive advisory lock against all runtime transactions; operators must also stop/drain/fence old and new services. No automatic startup import.
• Latest unpinned Python/psycopg image build records observed dependency versions. PR CI builds only a disposable local test stage. The separate runtime build script requires a clean merged-main ancestor and emits a main-<sha> image/OCI revision; it does not push or deploy. No startup dependency installation, bytecode cache rebuild or PG-mode local DB writes.

Validation

python3 services/sokar-sharing/test-postgres.py: 57 tests passed, no skips, under each of umask 0002 and 0077. Includes the original HTTP/upstream suite on both SQLite and disposable Postgres, restart/URL continuity, outages, expiry/revocation races, strict import refusal, atomic rollback, concurrent import/write exclusion, lost-ack reconciliation, and reverse export after new writes. Fixture directories/files use explicit private modes. The container runs as UID/GID 10001 with read-only root and tmpfs; no fleet credentials.

Python syntax, shell syntax and git diff --check pass. The follow-up CI fixture fix compiled only akh-service-store; no workspace-wide Cargo or unrelated Docker sweep. Dedicated test-sokar-sharing CI job runs the same harness. Observed latest resolution: Python 3.15.0, psycopg/binary 3.3.6, libpq 18.6, SQLite 3.46.1, disposable Postgres 18.6. These are recorded observations, not pins. Details: services/sokar-sharing/VALIDATION.md.

Remaining operator gates

No Talos manifest/volume/backend change, production import, secret mutation, archive extraction, image publication or deployment. PR 6 must reuse houses #208/#212 CNPG/ESO/role artifacts and check the delivered TLS paths; this PR creates no competing store, role or registry/placement change. PR 9 still requires the disposition ledger and separate operator GO. The runbook documents preserved API limitations: create has no idempotency key, so a lost create response cannot recover plaintext capability from its hash; revocation is idempotent. Rollback after PG writes must use a fresh reverse export, never the stale archive/PVC.

Please review and merge; the agent will not merge.

Review fixes and CI diagnosis

• SQLite offline reads explicitly pass uri=True; the regression test verifies URI opt-in, rejects a real write through that connection, and checks unchanged source bytes/directory contents.
• Postgres ordered readback explicitly uses COLLATE "C".
• The failed e0831dfe run was not a sweep or sharing regression. Its broad platform suite passed. Original Solstice logs identified sqlite_independent_connections_serialize_admission unwrapping storage_error: database is locked in the later storage harness. The old test raced 60 durable commits against the existing five-second busy timeout.
• That fixture now uses channel-coordinated ownership/release, an immediate test-only lock probe that must refuse before invoking a contending callback, and read/modify/write after an observed commit. Production SQLite behavior/timeouts are unchanged. Local validation: all 24 service-store library tests, plus 20/20 targeted trials across four concurrent test processes, passed. No assertion was disabled or timeout widened.

Issue #662 records all three reported incidents with corrected attribution (sweep, analogy, SQLite contention), original assertions and runner evidence. It proposes a deterministic/injected-clock sweep fix in a separate small PR after PR 6; the unrelated analogy needs separate investigation.

Final exact-head CI evidence

All 15/15 jobs and commit-status checks passed in Forgejo run 4304 / API 9973 at c62371a0037bd22fd549e482c8836ee41f285ce9. This includes sharing parity/migration, the full platform job and disposable storage/private-slot harness, Akh-Medu and Sokar tests, Seshat/core checks, Docker checks, chart and all five platform-image jobs. The revised SQLite fixture explicitly passed in the original runner log, followed by all 24 service-store unit tests and the integration groups.

The PR remains open for operator review/merge. No merge, deployment, production import, volume switch or manifest change was performed.

The Talos Sokar sharing companion currently stores mutable link authorization state only in a node-local SQLite file. This PR vendors that companion and its tests into `services/sokar-sharing/` and adds an opt-in Postgres backend, retaining SQLite as the default and preserving the HTTP/security contract. It is PR **5/9**, following merged #655; merging does not change production behavior. ### Implementation - Source attribution records Talos main `e061ea36a85d1915072b26b9c8f7ae34f14a9147`. Existing fragment capabilities, hashed token storage, expiry/revocation, content pinning, post-fetch recheck, CSP, isolated origin and credential boundary remain intact. - `LINK_STORE=postgres` selects typed link rows in an explicit owned schema, with unique token hashes and expiry index. `LINK_DATABASE_URL_FILE` supplies the ESO-delivered connection material. Production requires `sslmode=verify-full` and an explicit network hostname; driver text/URLs/row values never reach responses or logs. Bounded DB operations fail closed with no SQLite fallback. - Offline importer validates a private, checkpointed source read-only, refuses unknown objects/malformed values, verifies every destination row and commits an aggregate receipt atomically. Exact retry requires a matching receipt **and** unchanged destination. Nonempty/conflicting targets refuse. Lost commit ACK remains unknown until inspection/reconciliation. - Reverse exporter copies all current rows, including PG-era links and revocations, into a new verified/fsynced 0600 SQLite file without replacing an existing path. Inspect/export do not bootstrap absent schemas. Migration takes an exclusive advisory lock against all runtime transactions; operators must also stop/drain/fence old and new services. No automatic startup import. - Latest unpinned Python/psycopg image build records observed dependency versions. PR CI builds only a disposable local test stage. The separate runtime build script requires a clean merged-main ancestor and emits a `main-<sha>` image/OCI revision; it does not push or deploy. No startup dependency installation, bytecode cache rebuild or PG-mode local DB writes. ### Validation `python3 services/sokar-sharing/test-postgres.py`: **57 tests passed, no skips, under each of umask 0002 and 0077**. Includes the original HTTP/upstream suite on both SQLite and disposable Postgres, restart/URL continuity, outages, expiry/revocation races, strict import refusal, atomic rollback, concurrent import/write exclusion, lost-ack reconciliation, and reverse export after new writes. Fixture directories/files use explicit private modes. The container runs as UID/GID 10001 with read-only root and tmpfs; no fleet credentials. Python syntax, shell syntax and `git diff --check` pass. The follow-up CI fixture fix compiled only akh-service-store; no workspace-wide Cargo or unrelated Docker sweep. Dedicated `test-sokar-sharing` CI job runs the same harness. Observed latest resolution: Python **3.15.0**, psycopg/binary **3.3.6**, libpq **18.6**, SQLite **3.46.1**, disposable Postgres **18.6**. These are recorded observations, not pins. Details: `services/sokar-sharing/VALIDATION.md`. ### Remaining operator gates No Talos manifest/volume/backend change, production import, secret mutation, archive extraction, image publication or deployment. PR 6 must reuse houses #208/#212 CNPG/ESO/role artifacts and check the delivered TLS paths; this PR creates no competing store, role or registry/placement change. PR 9 still requires the disposition ledger and separate operator GO. The runbook documents preserved API limitations: create has no idempotency key, so a lost create response cannot recover plaintext capability from its hash; revocation is idempotent. Rollback after PG writes must use a fresh reverse export, never the stale archive/PVC. Please review and merge; the agent will not merge. ### Review fixes and CI diagnosis - SQLite offline reads explicitly pass `uri=True`; the regression test verifies URI opt-in, rejects a real write through that connection, and checks unchanged source bytes/directory contents. - Postgres ordered readback explicitly uses `COLLATE "C"`. - The failed `e0831dfe` run was **not a sweep or sharing regression**. Its broad platform suite passed. Original Solstice logs identified `sqlite_independent_connections_serialize_admission` unwrapping `storage_error: database is locked` in the later storage harness. The old test raced 60 durable commits against the existing five-second busy timeout. - That fixture now uses channel-coordinated ownership/release, an immediate test-only lock probe that must refuse before invoking a contending callback, and read/modify/write after an observed commit. Production SQLite behavior/timeouts are unchanged. Local validation: all **24 service-store library tests**, plus **20/20 targeted trials across four concurrent test processes**, passed. No assertion was disabled or timeout widened. [Issue #662](https://code.aopc.cloud/toasterson/akh-medu/issues/662) records all three reported incidents with corrected attribution (sweep, analogy, SQLite contention), original assertions and runner evidence. It proposes a deterministic/injected-clock sweep fix in a separate small PR after PR 6; the unrelated analogy needs separate investigation. ### Final exact-head CI evidence All **15/15 jobs and commit-status checks passed** in [Forgejo run 4304 / API 9973](https://code.aopc.cloud/toasterson/akh-medu/actions/runs/4304) at `c62371a0037bd22fd549e482c8836ee41f285ce9`. This includes sharing parity/migration, the full platform job and disposable storage/private-slot harness, Akh-Medu and Sokar tests, Seshat/core checks, Docker checks, chart and all five platform-image jobs. The revised SQLite fixture explicitly passed in the original runner log, followed by all 24 service-store unit tests and the integration groups. The PR remains open for operator review/merge. No merge, deployment, production import, volume switch or manifest change was performed.

toasterson added 1 commit 2026-10-16 07:27:19 +00:00

feat(sharing): configurable Postgres companion and verified link migration aeffadbd81

toasterson added 1 commit 2026-10-16 07:42:51 +00:00

fix(sharing): explicitly open immutable SQLite URIs and order PG readback e0831dfeb6

toasterson referenced this pull request 2026-10-16 08:21:57 +00:00

Make unknown-job sweep tests deterministic after platform-storage PR 6 (CI incident evidence) #662

toasterson added 1 commit 2026-10-16 08:23:06 +00:00

test(store): coordinate SQLite admission contention without a timing race c62371a003

toasterson merged commit b28fa30e09 into main 2026-10-16 08:54:56 +00:00

toasterson referenced this pull request from a commit 2026-10-16 08:54:58 +00:00

feat(sharing): Postgres companion and verified link migration (PR 5/9) (#661)

toasterson deleted branch feat/sokar-sharing-postgres 2026-10-16 08:54:58 +00:00

toasterson referenced this pull request 2026-10-16 09:49:07 +00:00

fix(runtime): make unknown-job sweep tests deterministic #663

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!661

No description provided.