D1: Persist turn context diagnostics with scoped occupancy queries

toasterson commented

2026-10-19 09:05:31 +00:00

Owner

D1 persists content-free typed model_observation, turn_usage and context_transform receipts alongside the central projection cursor. Exact immutable replay is a no-op; changed content/lineage records an invalid observation with unknown usage and projection continues. It preserves actual token/cache counts, unknown/zero distinctions, actual context occupancy/window/source, finish/terminal reasons, continuation/final-message byte census and compaction/elision provenance. SDK snapshots remain non-additive; this diagnostic path does not mutate canonical session cost totals.

WI detail and the scoped get_context_diagnostics MCP read expose bounded observations, an exact session filter, stable cursor and turn filters over_80_percent / length_or_incomplete. Session-WI identity and PAT project WorkItemRead permission are enforced. Query failures are errors/unavailable. Dispatch staging records the final rendered prompt hash and exact UTF-8 section bytes when component spans are unambiguous. Section token counts and runtime tool/history section breakdown remain explicitly unknown where no tokenizer/census exists. Dispatch-only sections are never presented as resumed model input.

Migration157 adds immutable receipts, dispatch census and central_turn_context_diagnostics query view. Wire DTOs are additive consumers of Akh telemetry #677 and D1 #678. Runtime dependency: compatible readers upgraded first, live jobs drained, AKH_RUNTIME_MODEL_EVENT_CONSUMER=model-events-v1 and AKH_RUNTIME_LEGACY_EVENTS=false; old runtimes/legacy projection yield no rows and historical absence stays unknown. Related Q2 #796 final-budget and Q3 #797 cost-view fixes remain separate concerns. Full contract: docs/ai/contracts/context-diagnostics-v1.md.

Validation: 558 server lib tests passed/6 environment-dependent ignored; full-schema PostgreSQL replay/unknown/zero/filter/cursor/ACL fixture passed. Existing production central projection fixture passes with new telemetry, changed immutable content retained as invalid while rows/cursor/replies continue, and exact duplicate summary accounting once. Final immutable prompt-census regression passed. Nightly CI runs the focused diagnostic database fixture. Scoped formatting, diff/shell checks and CI dry-run pass.

Solstice exact-head CI and independent review remain pending under the shared dispatch incident owned by round16. Local tests are not CI or runtime acceptance. Do not merge: operator human gate. No deployment, polling, compaction-policy change or live recovery action.

Fable follow-up: invalid/unsupported telemetry and changed immutable IDs now produce bounded invalid_observation receipts with unknown usage, static cause and SHA256; raw invalid payload is not persisted. Original immutable rows are preserved. Projection uses a savepoint so diagnostic failures cannot block the durable cursor, later replies or completion. Legacy post-claim census is best-effort and non-fatal. Regression coverage includes malformed usage, future source, malformed usage and unsupported sources, changed IDs, missing census storage and continued terminal replies.

ACL disclosure: get_work_item_detail now requires an exact-WI session anchor or PAT project WorkItemRead. This affects the whole detail response, including legacy fields; plan/companion session tokens without a WI anchor are denied. The dedicated diagnostic tool has the same boundary. Q4 separately preserves plan/companion project binding for its seven artifact tools.

Final Akh#678 d322e08bbcc8112bbe4291089a620dd5996833aa is consumed additively: event.usage.failure_cause and event.observation.failure_cause, normalized terminal_code (incomplete -> model_incomplete), nullable usage_known_calls, Codex provider-reported modelContextWindow/occupancy. Unknown historical fields remain null. Compatible readers must be upgraded first, jobs drained, then the runtime model-events-v1 consumer acknowledgment and modern events enabled. This PR performs no deployment.

Follow-up validation:559server unit tests passed/6ignored; full-schema diagnostics/query/census regression passed; production journal projection regression passed changed immutable IDs, unsupported source, valid Qwen display label, malformed usage and a subsequent final thread reply. Exact replay leaves one receipt; cursor reaches17. No modern transport activation performed.

Round2 follow-up: display labels reported_model/requested_model/provider/finish_reason/runtime accept bounded printable UTF8 (<=256bytes, no controls), separately from strict IDs/hashes. Exact Qwen3 235B (Q4_K_M) label now persists with actual usage. After rollout, watch session-token MCP denials containing permission_denied: exact work item required on get_work_item_detail, especially plan/companion callers without a WI anchor. No rollout is performed here. Modern runtime drain enforcement additionally requires AKH_DEPLOYMENT=deployed, compatible readers first, jobs drained, model-events-v1 acknowledgment and legacy events disabled.

Integration refresh (2026-10-19): normal main/dependency merge commits, no force push. Main is 8d3eb5855f. This head also includes refreshed #796 so the agreed merge order cannot repeat the same insertion conflict; its diff narrows as predecessors land. Shell/Rust syntax, diff and predecessor-ancestry checks pass. No fresh local cargo check: available disk headroom above the 31.25 GiB floor was smaller than the last measured owned compile target. Exact new-head CI is the gate; prior receipts remain prior-head evidence.

D1 persists content-free typed model_observation, turn_usage and context_transform receipts alongside the central projection cursor. Exact immutable replay is a no-op; changed content/lineage records an invalid observation with unknown usage and projection continues. It preserves actual token/cache counts, unknown/zero distinctions, actual context occupancy/window/source, finish/terminal reasons, continuation/final-message byte census and compaction/elision provenance. SDK snapshots remain non-additive; this diagnostic path does not mutate canonical session cost totals. WI detail and the scoped get_context_diagnostics MCP read expose bounded observations, an exact session filter, stable cursor and turn filters over_80_percent / length_or_incomplete. Session-WI identity and PAT project WorkItemRead permission are enforced. Query failures are errors/unavailable. Dispatch staging records the final rendered prompt hash and exact UTF-8 section bytes when component spans are unambiguous. Section token counts and runtime tool/history section breakdown remain explicitly unknown where no tokenizer/census exists. Dispatch-only sections are never presented as resumed model input. Migration157 adds immutable receipts, dispatch census and central_turn_context_diagnostics query view. Wire DTOs are additive consumers of Akh telemetry #677 and D1 #678. Runtime dependency: compatible readers upgraded first, live jobs drained, AKH_RUNTIME_MODEL_EVENT_CONSUMER=model-events-v1 and AKH_RUNTIME_LEGACY_EVENTS=false; old runtimes/legacy projection yield no rows and historical absence stays unknown. Related Q2 #796 final-budget and Q3 #797 cost-view fixes remain separate concerns. Full contract: docs/ai/contracts/context-diagnostics-v1.md. Validation: 558 server lib tests passed/6 environment-dependent ignored; full-schema PostgreSQL replay/unknown/zero/filter/cursor/ACL fixture passed. Existing production central projection fixture passes with new telemetry, changed immutable content retained as invalid while rows/cursor/replies continue, and exact duplicate summary accounting once. Final immutable prompt-census regression passed. Nightly CI runs the focused diagnostic database fixture. Scoped formatting, diff/shell checks and CI dry-run pass. Solstice exact-head CI and independent review remain pending under the shared dispatch incident owned by round16. Local tests are not CI or runtime acceptance. **Do not merge: operator human gate.** No deployment, polling, compaction-policy change or live recovery action. Fable follow-up: invalid/unsupported telemetry and changed immutable IDs now produce bounded invalid_observation receipts with unknown usage, static cause and SHA256; raw invalid payload is not persisted. Original immutable rows are preserved. Projection uses a savepoint so diagnostic failures cannot block the durable cursor, later replies or completion. Legacy post-claim census is best-effort and non-fatal. Regression coverage includes malformed usage, future source, malformed usage and unsupported sources, changed IDs, missing census storage and continued terminal replies. ACL disclosure: get_work_item_detail now requires an exact-WI session anchor or PAT project WorkItemRead. This affects the whole detail response, including legacy fields; plan/companion session tokens without a WI anchor are denied. The dedicated diagnostic tool has the same boundary. Q4 separately preserves plan/companion project binding for its seven artifact tools. Final Akh#678 d322e08bbcc8112bbe4291089a620dd5996833aa is consumed additively: event.usage.failure_cause and event.observation.failure_cause, normalized terminal_code (incomplete -> model_incomplete), nullable usage_known_calls, Codex provider-reported modelContextWindow/occupancy. Unknown historical fields remain null. Compatible readers must be upgraded first, jobs drained, then the runtime model-events-v1 consumer acknowledgment and modern events enabled. This PR performs no deployment. Follow-up validation:559server unit tests passed/6ignored; full-schema diagnostics/query/census regression passed; production journal projection regression passed changed immutable IDs, unsupported source, valid Qwen display label, malformed usage and a subsequent final thread reply. Exact replay leaves one receipt; cursor reaches17. No modern transport activation performed. Round2 follow-up: display labels reported_model/requested_model/provider/finish_reason/runtime accept bounded printable UTF8 (<=256bytes, no controls), separately from strict IDs/hashes. Exact Qwen3 235B (Q4_K_M) label now persists with actual usage. After rollout, watch session-token MCP denials containing permission_denied: exact work item required on get_work_item_detail, especially plan/companion callers without a WI anchor. No rollout is performed here. Modern runtime drain enforcement additionally requires AKH_DEPLOYMENT=deployed, compatible readers first, jobs drained, model-events-v1 acknowledgment and legacy events disabled.  Integration refresh (2026-10-19): normal main/dependency merge commits, no force push. Main is 8d3eb5855f74c616d176b788e2ca881d2f933e3f. This head also includes refreshed #796 so the agreed merge order cannot repeat the same insertion conflict; its diff narrows as predecessors land. Shell/Rust syntax, diff and predecessor-ancestry checks pass. No fresh local cargo check: available disk headroom above the 31.25 GiB floor was smaller than the last measured owned compile target. Exact new-head CI is the gate; prior receipts remain prior-head evidence.

toasterson added 1 commit

2026-10-19 09:05:33 +00:00