Persist typed model usage identity and context provenance #677

Merged

toasterson merged 17 commits from fix/anima-a3-model-observations into fix/anima-a3-model-envelopes 2026-10-19 20:01:18 +00:00

Conversation 0 Commits 17 Files changed 11 +867 -27

toasterson commented 2026-10-19 08:32:28 +00:00

Owner

Copy link

Persist immutable model observations without converting a successful model boundary into Unknown when diagnostics refuse an append, and enforce a safe event rollout/rollback order.

Targets #676 branch; supplies Anima S3/Q3 #797. No PR merge, deployment, live recovery or canary.

Review fixes

Observation refusal now goes through model_boundary and retains any original typed rejection. The real worker-path capacity test ends Failed/model_incomplete. Immutable side keys are collected at archive; archived replay checks the retained immutable event. Reader compatibility is backported to #674 so that floor skips unsupported content-free metadata while unknown controls fail closed. Session readers must upgrade first; exposing metadata requires the consumer acknowledgment and legacy flag disabled. Deployed startup refuses projection changes with live jobs, preserving event-prefix/cursor grammar. Pre-floor rollback requires restoring its store/config snapshot.

Exact consumer contract

Model observations v1 (A3 telemetry; Anima S3/Q3)

New RuntimeEvent kinds:

• model_observation: {observation:ModelObservation}.
• context_transform: {transform:ContextTransform}.

ModelObservation has schema_version:1, immutable id, nullable call_id,
source:chat_completions|native_candle|codex|claude,
accounting:per_call|provider_snapshot, runtime_job_id, session_id, turn_id,
generation, runtime, profile_revision, and these nullable fields:
provider, requested_model, reported_model, model_revision, provider_request_id,
http_status, attempts, finish_reason, terminal_code, cumulative_usage,
request_bytes, estimated_input_tokens, estimation_method,
context_limit_tokens, output_limit_tokens.

Required usage and optional cumulative_usage have nullable fields:
input_includes_cache:bool, input_tokens:u64, output_tokens:u64,
cache_read_input_tokens:u64, cache_write_input_tokens:u64,
reasoning_output_tokens:u64. A measured zero is zero. Missing/negative/malformed
counts remain unknown. No estimate is a billing count. OpenAI/native input counts
include cache; Claude SDK input_tokens exclude separately reported cache read/write.

Internal logical model calls have id=job/model/iteration, equal call_id, and
per-call accounting. Rejected HTTP retries stay one call with bounded attempts;
only provider-reported usage is recorded. Final failed calls retain available
metadata and safe terminal code. Native Candle reports its verified loaded model
hash, actual tokenizer prompt/generated/reused counts and declared limits. HTTP provider is the actual request host, excluding credentials and URL paths. Native SDK backend identity remains unknown when absent. HTTP
reports actual SSE model/system_fingerprint, never the configured alias as fact.
requested_model remains the declared request. Missing metadata stays unknown.

Native Codex/Claude SDK usage is an explicitly non-additive provider snapshot
when the SDK does not supply a trustworthy immutable model-call identity. The ID
fingerprints the sanitized counts/model snapshot within its job. Identical repeated
snapshots deduplicate, but equal counts do not prove distinct calls or a complete
turn ledger. Codex cumulative totals are separate from last-call counts. Consumers
must not sum snapshots, pretend they are per-call charges, infer historical model
versions from executor aliases, or combine cumulative and per-call counts.

A3 records immutable observations in the runtime transaction and rejects changed
payload/lineage reuse. Replayed identical events are no-ops. Consumers should
upsert by runtime/job/observation ID, retaining source/accounting and nulls. Anima
S3 must reconcile its central turn lineage and use Q3's truthful unknown/error
presentation. A per-turn total from per-call events is known only with a complete
call census and all required known counts; A1's typed failure remains separate.

ContextTransform is {schema_version:1,id,source,before_bytes,after_bytes, dropped_messages,artifact_sha256:string|null}. It records cognitive-core and
native-window decisions with byte counts and no prompt text. Tool output elision
references and exact complete byte/hash provenance are the separate A3 output
contract; metadata does not replace complete authorized evidence.

request_bytes is measured serialized HTTP request or native rendered prompt.
estimated_input_tokens is labelled serialized_utf8_bytes_ceil_div_4, not an
actual provider tokenizer count. Native tokenizer observations identify their
source and keep unknown reasoning/cache-write fields null. No provider body,
prompt content or credential enters the typed fields or metric labels.

New event kinds require compatible central consumers. Mixed old typed readers may
refuse unknown kinds and retain their existing fences; no live rollout is attempted.
This PR establishes source/code contracts, not historical deployment proof,
observability acceptance, or an authorized provider canary.

The existing AKH_RUNTIME_LEGACY_EVENTS=true projection omits these new variants as well as Reasoning, retaining complete observations in the durable job log. Modern central consumers require this compatibility flag to be disabled. The standalone runtime bridge records content-free diagnostics rather than appending observations to user text.

Review rollout/rollback gate prerequisite: set AKH_DEPLOYMENT=deployed (or
deployment: "deployed" in runtime service config) before relying on the persisted
projection drain gate. The default is development mode, where that gate is inert.
Establish deployed mode while keeping AKH_RUNTIME_LEGACY_EVENTS=true; an unchanged
legacy projection permits startup with existing jobs. Manifest changes belong to
the separately authorized deployment step, not this PR.

Review rollout/rollback gate: (1) install #674 compatibility readers in session service and runtime, which skip only unsupported content-free model metadata while rejecting unknown control events; this is the rollback floor. (2) upgrade session service/central readers to #677/#678 support. (3) upgrade runtimes with AKH_RUNTIME_LEGACY_EVENTS=true. (4) only after consumer upgrade, set AKH_RUNTIME_MODEL_EVENT_CONSUMER=model-events-v1 and disable AKH_RUNTIME_LEGACY_EVENTS. Both conditions are checked at wire projection; without the consumer acknowledgment new metadata stays hidden. Rollback to #674 is safe for live job decoding; pre-#674 binaries require pre-upgrade store/config restoration. Profile config with new deny_unknown_fields fields must also roll back. No runtime/deployment was changed by this PR.

Observation refusals at model boundaries use model_boundary and retain the original typed model error if present: successful completion plus observation capacity failure is Failed, never Unknown. Immutable dedupe side keys are deleted at archive, while event bytes remain; repeated archived observation replay checks the immutable event itself. SDK content-fingerprint snapshots remain non-additive even when distinct calls have identical counts.

Deployed runtime startup persists the selected legacy/model-event projection and refuses a change while that runtime owns any nonterminal live job. Upgrade session readers first, drain jobs, then flip the flags and restart. An unchanged projection permits ordinary restart/reconciliation. Rollback likewise drains before reverting projection/config; the compatibility reader floor allows stored newer metadata to be skipped safely. This prevents changing an existing session's event-prefix/cursor grammar mid-job.

Prior validation

Fable Round2 approved reviewed head 73944bebaaa2. Previous follow-up head: 82e110eed9ab358b9296d35b4b1492e7be53a0ed.

Rollout prerequisite: set AKH_DEPLOYMENT=deployed (or deployment: "deployed" in runtime config) before relying on the persisted projection drain gate. Development mode bypasses that gate. Establish deployed mode while retaining the legacy projection, upgrade session readers, drain live runtime jobs, then acknowledge the consumer and flip legacy projection. An isolated-process real Runtime::new test proves development bypass, deployed live-change refusal, and deployed drained success. No Talos manifests are changed; the separate deploy step owns them.

Current-head actual-startup regression:1 passed with three isolated child environments after cleaning local crates (r2-followup-telemetry-final.log). An earlier zero-selected shared-target run is invalid and excluded. Final integrated #678 suite115 passed/1 existing real-backend ignored. Full exact-head Solstice CI remains a merge gate; local results do not substitute for it. No merge, force-push, deployment or canary.

Finished task target was reclaimed after verifying no open files;37.08GiB free at10:16Z. Peer target preserved.

CI4360 follow-up

Current head: 77cf2afadfd4d29071b9948a3b3e6b5b9ba3f2bc.

Solstice exact-head evidence for #676 dae86ef17b identifies failed request e257c6a0-5f0a-4a9e-8b58-d372ffb7ad0f, category step:cargo-test-features-postgres-lib. The root suite had3961 passed,22 ignored and1 failure: agent::runtime::tests::codex_tools_use_typed_akh_registry_and_record_evidence, src/agent/runtime/tests.rs:117, expecting plain "7" while the fixture echoed the complete ToolResult JSON. This was a missed regression expectation from the reviewed complete-envelope contract, not a refused connection or runner fault. It reproduced locally before the fix. Reported Forgejo job: /toasterson/akh-medu/actions/runs/4360/jobs/7.

The root regression now checks all decoded fields {success:true,text:"7",data:null,receipt:null} and keeps its tool-evidence trace assertion. New Codex envelope coverage preserves both text-only and structured data/receipt results alongside the existing64KiB escaped-envelope refusal. The production envelope implementation hash is unchanged; no wire contract change. The identical stale root test and implementation existed on #677/#678, so normal stack integration carries this fix through both. No force-push or PR merge.

Current-head local receipts:

• ci4360-677-root.log: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3984 filtered out; finished in 0.20s
• ci4360-677-envelope.log: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 14 filtered out; finished in 0.01s

Local workspace packages were cleaned between worktrees to exclude stale Cargo reuse. Final full Linux CI is still a separate gate. No blind rerun was requested; the normal follow-up commits publish the diagnosed fix. No deployment, Talos/Ptah change or live provider probe. Builds used31.5GiB stop/32GiB preflight guards; the finished task target is reclaimed on handoff.

Own finished target was verified without open files and deleted; 37.269GiB free at 2026-10-19T11:58:41.253438+00:00. Peer targets were untouched.

Persist immutable model observations without converting a successful model boundary into Unknown when diagnostics refuse an append, and enforce a safe event rollout/rollback order. Targets #676 branch; supplies Anima S3/Q3 #797. No PR merge, deployment, live recovery or canary. ## Review fixes Observation refusal now goes through model_boundary and retains any original typed rejection. The real worker-path capacity test ends Failed/model_incomplete. Immutable side keys are collected at archive; archived replay checks the retained immutable event. Reader compatibility is backported to #674 so that floor skips unsupported content-free metadata while unknown controls fail closed. Session readers must upgrade first; exposing metadata requires the consumer acknowledgment and legacy flag disabled. Deployed startup refuses projection changes with live jobs, preserving event-prefix/cursor grammar. Pre-floor rollback requires restoring its store/config snapshot. ## Exact consumer contract # Model observations v1 (A3 telemetry; Anima S3/Q3) New RuntimeEvent kinds: - `model_observation`: `{observation:ModelObservation}`. - `context_transform`: `{transform:ContextTransform}`. ModelObservation has `schema_version:1`, immutable `id`, nullable `call_id`, `source:chat_completions|native_candle|codex|claude`, `accounting:per_call|provider_snapshot`, `runtime_job_id`, `session_id`, `turn_id`, `generation`, `runtime`, `profile_revision`, and these nullable fields: `provider`, `requested_model`, `reported_model`, `model_revision`, `provider_request_id`, `http_status`, `attempts`, `finish_reason`, `terminal_code`, `cumulative_usage`, `request_bytes`, `estimated_input_tokens`, `estimation_method`, `context_limit_tokens`, `output_limit_tokens`. Required `usage` and optional `cumulative_usage` have nullable fields: `input_includes_cache:bool`, `input_tokens:u64`, `output_tokens:u64`, `cache_read_input_tokens:u64`, `cache_write_input_tokens:u64`, `reasoning_output_tokens:u64`. A measured zero is zero. Missing/negative/malformed counts remain unknown. No estimate is a billing count. OpenAI/native input counts include cache; Claude SDK input_tokens exclude separately reported cache read/write. Internal logical model calls have `id=job/model/iteration`, equal `call_id`, and per-call accounting. Rejected HTTP retries stay one call with bounded `attempts`; only provider-reported usage is recorded. Final failed calls retain available metadata and safe terminal code. Native Candle reports its verified loaded model hash, actual tokenizer prompt/generated/reused counts and declared limits. HTTP `provider` is the actual request host, excluding credentials and URL paths. Native SDK backend identity remains unknown when absent. HTTP reports actual SSE `model`/`system_fingerprint`, never the configured alias as fact. `requested_model` remains the declared request. Missing metadata stays unknown. Native Codex/Claude SDK usage is an explicitly **non-additive provider snapshot** when the SDK does not supply a trustworthy immutable model-call identity. The ID fingerprints the sanitized counts/model snapshot within its job. Identical repeated snapshots deduplicate, but equal counts do not prove distinct calls or a complete turn ledger. Codex cumulative totals are separate from last-call counts. Consumers must not sum snapshots, pretend they are per-call charges, infer historical model versions from executor aliases, or combine cumulative and per-call counts. A3 records immutable observations in the runtime transaction and rejects changed payload/lineage reuse. Replayed identical events are no-ops. Consumers should upsert by runtime/job/observation ID, retaining source/accounting and nulls. Anima S3 must reconcile its central turn lineage and use Q3's truthful unknown/error presentation. A per-turn total from per-call events is known only with a complete call census and all required known counts; A1's typed failure remains separate. ContextTransform is `{schema_version:1,id,source,before_bytes,after_bytes, dropped_messages,artifact_sha256:string|null}`. It records cognitive-core and native-window decisions with byte counts and no prompt text. Tool output elision references and exact complete byte/hash provenance are the separate A3 output contract; metadata does not replace complete authorized evidence. `request_bytes` is measured serialized HTTP request or native rendered prompt. `estimated_input_tokens` is labelled `serialized_utf8_bytes_ceil_div_4`, not an actual provider tokenizer count. Native tokenizer observations identify their source and keep unknown reasoning/cache-write fields null. No provider body, prompt content or credential enters the typed fields or metric labels. New event kinds require compatible central consumers. Mixed old typed readers may refuse unknown kinds and retain their existing fences; no live rollout is attempted. This PR establishes source/code contracts, not historical deployment proof, observability acceptance, or an authorized provider canary. The existing AKH_RUNTIME_LEGACY_EVENTS=true projection omits these new variants as well as Reasoning, retaining complete observations in the durable job log. Modern central consumers require this compatibility flag to be disabled. The standalone runtime bridge records content-free diagnostics rather than appending observations to user text. Review rollout/rollback gate prerequisite: set `AKH_DEPLOYMENT=deployed` (or `deployment: "deployed"` in runtime service config) before relying on the persisted projection drain gate. The default is development mode, where that gate is inert. Establish deployed mode while keeping `AKH_RUNTIME_LEGACY_EVENTS=true`; an unchanged legacy projection permits startup with existing jobs. Manifest changes belong to the separately authorized deployment step, not this PR. Review rollout/rollback gate: (1) install #674 compatibility readers in session service and runtime, which skip only unsupported content-free model metadata while rejecting unknown control events; this is the rollback floor. (2) upgrade session service/central readers to #677/#678 support. (3) upgrade runtimes with AKH_RUNTIME_LEGACY_EVENTS=true. (4) only after consumer upgrade, set AKH_RUNTIME_MODEL_EVENT_CONSUMER=model-events-v1 and disable AKH_RUNTIME_LEGACY_EVENTS. Both conditions are checked at wire projection; without the consumer acknowledgment new metadata stays hidden. Rollback to #674 is safe for live job decoding; pre-#674 binaries require pre-upgrade store/config restoration. Profile config with new deny_unknown_fields fields must also roll back. No runtime/deployment was changed by this PR. Observation refusals at model boundaries use model_boundary and retain the original typed model error if present: successful completion plus observation capacity failure is Failed, never Unknown. Immutable dedupe side keys are deleted at archive, while event bytes remain; repeated archived observation replay checks the immutable event itself. SDK content-fingerprint snapshots remain non-additive even when distinct calls have identical counts. Deployed runtime startup persists the selected legacy/model-event projection and refuses a change while that runtime owns any nonterminal live job. Upgrade session readers first, drain jobs, then flip the flags and restart. An unchanged projection permits ordinary restart/reconciliation. Rollback likewise drains before reverting projection/config; the compatibility reader floor allows stored newer metadata to be skipped safely. This prevents changing an existing session's event-prefix/cursor grammar mid-job. ## Prior validation Fable Round2 approved reviewed head `73944bebaaa2`. Previous follow-up head: `82e110eed9ab358b9296d35b4b1492e7be53a0ed`. Rollout prerequisite: set AKH_DEPLOYMENT=deployed (or deployment: "deployed" in runtime config) before relying on the persisted projection drain gate. Development mode bypasses that gate. Establish deployed mode while retaining the legacy projection, upgrade session readers, drain live runtime jobs, then acknowledge the consumer and flip legacy projection. An isolated-process real Runtime::new test proves development bypass, deployed live-change refusal, and deployed drained success. No Talos manifests are changed; the separate deploy step owns them. Current-head actual-startup regression:1 passed with three isolated child environments after cleaning local crates (r2-followup-telemetry-final.log). An earlier zero-selected shared-target run is invalid and excluded. Final integrated #678 suite115 passed/1 existing real-backend ignored. Full exact-head Solstice CI remains a merge gate; local results do not substitute for it. No merge, force-push, deployment or canary. Finished task target was reclaimed after verifying no open files;37.08GiB free at10:16Z. Peer target preserved. ## CI4360 follow-up Current head: `77cf2afadfd4d29071b9948a3b3e6b5b9ba3f2bc`. Solstice exact-head evidence for #676 dae86ef17b02dc41b2123842dc19c0595471de70 identifies failed request e257c6a0-5f0a-4a9e-8b58-d372ffb7ad0f, category `step:cargo-test-features-postgres-lib`. The root suite had3961 passed,22 ignored and1 failure: `agent::runtime::tests::codex_tools_use_typed_akh_registry_and_record_evidence`, src/agent/runtime/tests.rs:117, expecting plain "7" while the fixture echoed the complete ToolResult JSON. This was a missed regression expectation from the reviewed complete-envelope contract, not a refused connection or runner fault. It reproduced locally before the fix. Reported Forgejo job: /toasterson/akh-medu/actions/runs/4360/jobs/7. The root regression now checks all decoded fields `{success:true,text:"7",data:null,receipt:null}` and keeps its tool-evidence trace assertion. New Codex envelope coverage preserves both text-only and structured data/receipt results alongside the existing64KiB escaped-envelope refusal. The production envelope implementation hash is unchanged; no wire contract change. The identical stale root test and implementation existed on #677/#678, so normal stack integration carries this fix through both. No force-push or PR merge. Current-head local receipts: - `ci4360-677-root.log`: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 3984 filtered out; finished in 0.20s - `ci4360-677-envelope.log`: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 14 filtered out; finished in 0.01s Local workspace packages were cleaned between worktrees to exclude stale Cargo reuse. Final full Linux CI is still a separate gate. No blind rerun was requested; the normal follow-up commits publish the diagnosed fix. No deployment, Talos/Ptah change or live provider probe. Builds used31.5GiB stop/32GiB preflight guards; the finished task target is reclaimed on handoff. Own finished target was verified without open files and deleted; 37.269GiB free at 2026-10-19T11:58:41.253438+00:00. Peer targets were untouched.

toasterson added 5 commits 2026-10-19 08:32:29 +00:00

Persist typed model usage identity and context provenance events eadd680a4b

Merge branch 'fix/anima-a3-model-envelopes' into fix/anima-a3-model-observations ... 4e5d2ac31a

# Conflicts:
#	crates/akh-session-services/src/internal.rs

Record provider origin without credential-bearing endpoint paths dda018957a

Merge branch 'fix/anima-a3-model-envelopes' into fix/anima-a3-model-observations 8072ef6961

Clarify provider origin and nullable accounting contract 51f2f7ce4b

toasterson added 2 commits 2026-10-19 08:45:51 +00:00

Integrate typed limit reset handling into observation stack 11161f7c59

Preserve provider binding after observation integration 4c830a3fc1

toasterson referenced this pull request 2026-10-19 08:46:04 +00:00

Emit per-turn usage and explicit context diagnostics #678

toasterson added 1 commit 2026-10-19 08:47:44 +00:00

Handle model diagnostics in standalone runtime bridge d523a37661

toasterson added 2 commits 2026-10-19 08:50:05 +00:00

Preserve legacy event projection with new observations 2ccf07de6f

Integrate explicit monthly limit classification 46f30cfe07

toasterson added 4 commits 2026-10-19 09:38:26 +00:00

Integrate reviewed output and provider fixes 6367961ec9

Integrate rollback-compatible runtime event readers 880cf6fbd4

Fence observation failures and enforce metadata rollout compatibility 0f08991787

Require drained deployed jobs before changing event projection 73944bebaa

toasterson added 2 commits 2026-10-19 10:15:00 +00:00

docs(runtime): require deployed mode for projection drain gate d3e226b1a0

Integrate A1 and artifact collector follow-ups 82e110eed9

toasterson added 1 commit 2026-10-19 11:56:45 +00:00

Integrate Codex tool envelope CI regression fix 77cf2afadf

toasterson merged commit 37b83b5e6c into fix/anima-a3-model-envelopes 2026-10-19 20:01:18 +00:00

toasterson referenced this pull request from a commit 2026-10-19 20:01:18 +00:00

Merge PR #677: Persist typed model usage identity and context provenance

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!677

No description provided.