Fix authoritative house schema upgrades before hydration #671

Merged
toasterson merged 1 commit from fix/house-schema-upgrade-round12a into main 2026-10-18 16:37:30 +00:00
Owner

Upgrading an existing house from deployed 45a1b9bedb0d43aaec7eafec6cd58dfeed045347 to main 62456e4c26059728d0f24f2f514e4703b89d2a09 failed cold hydration with column akhomed_symbols.created_at does not exist. Ptah never became Ready and Talos #226 restored its prior image. This repair completes the schema upgrade before an authoritative house can publish a usable store pool.

Root cause and complete store audit

3c46cfe4445944a9ff7b1dcb4f05b010d082bd96 (#655) added akhomed_symbols.created_at, akhomed_symbols.source, and akhomed_workspace_manifest. Its matching m20261016_000001_workspace_manifest migration is correctly registered and ordered. The deployed house config has require_authoritative=true and migrate_on_start=false; startup/reconnect consequently skipped that migration and hydration selected the new columns from the old table.

The deployed commit's 13 migrations were run with its unmodified migration binary on disposable PostgreSQL 17.11. The checked-in fixture combines that schema-only dump with only the migration-history rows; synthetic test state is inserted separately. Every one of the 30 current SeaORM entities is audited against it. The only drift is the two symbol columns and the absent six-column manifest table (akh_id, workspace_id, generation, residency_epoch, digest, manifest). Every entity query succeeds after upgrade. No other missing entity columns were found. #670's inaccurate six-column description is corrected: the deployed symbol entity has five columns and main has seven.

Resulting behavior

  • Authoritative startup and background reconnect always await the existing advisory-locked migration runner before publishing a pool, even with migrate_on_start=false. Non-authoritative hosts retain the optional migration setting. A migration error yields no usable pool and cannot proceed to authoritative hydration.
  • New registered forward migration m20261018_000001_house_schema_compatibility, after the original workspace-manifest migration, repairs absent symbol columns or the manifest table if the earlier migration is already recorded. Existing values are preserved; historical migrations remain unchanged. Repair rollback leaves removal to the original migration's down step.
  • Legacy created_at=0 explicitly means unknown, and source=NULL means unavailable. The old symbol rows contain no reliable creation time: neither snapshot time, house awakening nor an adjacent fact's assertion time establishes symbol birth. We retain the existing documented sentinel without inventing timestamp or source evidence.

Regression and validation

The existing test-session-platform CI job runs both regression targets through its disposable PostgreSQL runner before its other contracts. The runner removes ambient DATABASE_URL, uses its own loopback-only container and uniquely named test databases, and cleans up afterward.

  • Frozen deployed-schema audit/upgrade and forward-repair tests: 2 passed, 0 ignored, on PostgreSQL 17.11 and again through the CI runner on 18.6.
  • Real daemon init_postgres plus fenced ServerState::get_engine cold hydration with the deployed flags: 1 passed, 0 ignored on both versions. Checks denied migration, original symbol IDs/fact, unknown metadata, runtime state and monotonic residency epoch; no agent/provider loop is started.
  • Store library: 23 passed, 1 pre-existing PostgreSQL mailbox test ignored. Local SQLite: 6 passed. Per-Akh layout: 11 passed. Formatting, whitespace, Python compilation and shell syntax pass.
  • Fresh-context independent source review: approved, no blocking findings. Reviewer did not regenerate the fixture or run the full daemon target; implementer test evidence is separate. The proposed minimal native-lane Docker invocation was removed following review; the existing Docker/full-server CI lane owns the regressions.

Diagnosis, fixture provenance, review and validation receipt: docs/ops/house-schema-upgrade-20261018/; operator evidence is recorded under round 12a. Both owned disposable database containers were removed. No production DB reads/writes, merges, deployment or live daemon probe flags were used. The authoritative database role must own its store schema and permit migration DDL. Exact-head CI and a future reviewed main-image rollout/cold-start acceptance remain pending.

Upgrading an existing house from deployed `45a1b9bedb0d43aaec7eafec6cd58dfeed045347` to main `62456e4c26059728d0f24f2f514e4703b89d2a09` failed cold hydration with `column akhomed_symbols.created_at does not exist`. Ptah never became Ready and Talos #226 restored its prior image. This repair completes the schema upgrade before an authoritative house can publish a usable store pool. ## Root cause and complete store audit `3c46cfe4445944a9ff7b1dcb4f05b010d082bd96` (#655) added `akhomed_symbols.created_at`, `akhomed_symbols.source`, and `akhomed_workspace_manifest`. Its matching `m20261016_000001_workspace_manifest` migration is correctly registered and ordered. The deployed house config has `require_authoritative=true` and `migrate_on_start=false`; startup/reconnect consequently skipped that migration and hydration selected the new columns from the old table. The deployed commit's 13 migrations were run with its unmodified migration binary on disposable PostgreSQL 17.11. The checked-in fixture combines that schema-only dump with only the migration-history rows; synthetic test state is inserted separately. Every one of the 30 current SeaORM entities is audited against it. The only drift is the two symbol columns and the absent six-column manifest table (`akh_id`, `workspace_id`, `generation`, `residency_epoch`, `digest`, `manifest`). Every entity query succeeds after upgrade. No other missing entity columns were found. #670's inaccurate six-column description is corrected: the deployed symbol entity has five columns and main has seven. ## Resulting behavior - Authoritative startup and background reconnect always await the existing advisory-locked migration runner before publishing a pool, even with `migrate_on_start=false`. Non-authoritative hosts retain the optional migration setting. A migration error yields no usable pool and cannot proceed to authoritative hydration. - New registered forward migration `m20261018_000001_house_schema_compatibility`, after the original workspace-manifest migration, repairs absent symbol columns or the manifest table if the earlier migration is already recorded. Existing values are preserved; historical migrations remain unchanged. Repair rollback leaves removal to the original migration's down step. - Legacy `created_at=0` explicitly means unknown, and `source=NULL` means unavailable. The old symbol rows contain no reliable creation time: neither snapshot time, house awakening nor an adjacent fact's assertion time establishes symbol birth. We retain the existing documented sentinel without inventing timestamp or source evidence. ## Regression and validation The existing `test-session-platform` CI job runs both regression targets through its disposable PostgreSQL runner before its other contracts. The runner removes ambient `DATABASE_URL`, uses its own loopback-only container and uniquely named test databases, and cleans up afterward. - Frozen deployed-schema audit/upgrade and forward-repair tests: **2 passed, 0 ignored**, on PostgreSQL 17.11 and again through the CI runner on 18.6. - Real daemon `init_postgres` plus fenced `ServerState::get_engine` cold hydration with the deployed flags: **1 passed, 0 ignored** on both versions. Checks denied migration, original symbol IDs/fact, unknown metadata, runtime state and monotonic residency epoch; no agent/provider loop is started. - Store library: **23 passed, 1 pre-existing PostgreSQL mailbox test ignored**. Local SQLite: **6 passed**. Per-Akh layout: **11 passed**. Formatting, whitespace, Python compilation and shell syntax pass. - Fresh-context independent source review: **approved, no blocking findings**. Reviewer did not regenerate the fixture or run the full daemon target; implementer test evidence is separate. The proposed minimal native-lane Docker invocation was removed following review; the existing Docker/full-server CI lane owns the regressions. Diagnosis, fixture provenance, review and validation receipt: `docs/ops/house-schema-upgrade-20261018/`; operator evidence is recorded under round 12a. Both owned disposable database containers were removed. No production DB reads/writes, merges, deployment or live daemon probe flags were used. The authoritative database role must own its store schema and permit migration DDL. Exact-head CI and a future reviewed main-image rollout/cold-start acceptance remain pending.
Fix authoritative house schema upgrades before hydration
All checks were successful
CI / test-sokar-sharing (pull_request) Successful in 2m15s
CI / test-native-receiver-isolation (pull_request) Successful in 3m50s
CI / check-seshat (pull_request) Successful in 4m12s
CI / test-native-client-computer (pull_request) Successful in 5m48s
CI / publish-chart (pull_request) Successful in 48s
CI / test-sokar (pull_request) Successful in 7m8s
CI / check-akh-medu (pull_request) Successful in 10m35s
CI / docker-akh-transcripts (pull_request) Successful in 8m43s
CI / docker-seshd (pull_request) Successful in 12m51s
CI / docker-sokard (pull_request) Successful in 17m18s
CI / test-session-platform (pull_request) Successful in 26m58s
CI / test-akh-medu (pull_request) Successful in 28m12s
CI / platform-images-daemons (pull_request) Successful in 1m2s
CI / platform-images-home (pull_request) Successful in 1m6s
CI / platform-images-core (pull_request) Successful in 1m11s
CI / platform-images-fleet (pull_request) Successful in 54s
CI / platform-images-collect (pull_request) Successful in 54s
4bf7e635a8
Sign in to join this conversation.
No reviewers
No labels
follow-up
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!671
No description provided.