Emit per-turn usage and explicit context diagnostics #678
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/anima-d1-context-diagnostics"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Expose known Codex context occupancy and typed failure categories in per-turn diagnostics while retaining unknown SDK totals and historical observations.
Targets #677 branch; supplies Anima D1 #798/revised Q1 #795/S3. No PR merge, deployment, live recovery or canary.
Review fixes
Codex tokenUsage.modelContextWindow now supplies the provider-reported window and occupancy from last input counts; Claude's unreported window remains unknown. event.usage.failure_cause(TurnUsage) and event.observation.failure_cause are nullable ModelFailureCause. terminal_code lifts the cause except incomplete maps to model_incomplete; lifecycle Fault.code is unchanged. usage_known_calls makes partial per-call census visible without inventing totals. Elision-event capacity refusal warns/continues with the canonical receipt; conflicts remain fatal. Session-first rollout, consumer acknowledgment and drained projection switch are enforced/documented.
Exact consumer contract
D1 turn and context diagnostics v1 (Anima Q1/S3)
Additive
model_observation.observationfields (nullable):history_messages:u64,history_bytes:u64are the HTTP continuation historybefore core assembly, excluding the mandatory system instruction;
model_messages:u64,model_messages_bytes:u64measure the final serializedmessages array including that instruction;
context_occupancy_percent:f64isactual input tokens including cache divided by
context_limit_tokenstimes 100.context_window_source:stringisconfigured_model_endpointornative_pinned_config. HTTP window size requires explicit positiveModelEndpoint.context_window_tokens; no alias-based model-spec guess. Theexisting byte/4 estimate stays labeled and never supplies actual occupancy.
Input that excludes cache requires both read/write counts to compute occupancy.
A null window/input/cache convention yields null occupancy; over-100 is retained.
A new event is
{"kind":"turn_usage","usage":{...}}. Required members:schema_version:1,id(job-id/turn-usage),runtime_job_id,session_id,turn_id,generation,accounting:per_call_sum|unknown,call_count:u64|null,usage:ModelUsage, and nullablefinish_reason,terminal_code,context_limit_tokens,context_occupancy_percent. ModelUsage retains the A3nullable
input_includes_cache,input_tokens,output_tokens,cache_read_input_tokens,cache_write_input_tokens,reasoning_output_tokens.Call count is null without reliable per-call accounting. Each count sums distinct per-call observations only if every observation supplies
that count. Missing counts and empty turns stay null. A mixed cache convention
makes input total unknown. Any SDK provider snapshot or uncertain worker outcome
makes the turn accounting unknown, rather than adding potentially cumulative
snapshots. Context/finish fields describe the last observed call, never a sum.
The summary is saved in the same transaction as the worker's durable outcome.
Restart/retransmission does not add another summary; old jobs are not backfilled.
context_transformcontinues the A3 schema. In addition to cognitive-core/nativewindow events, artifact-backed tool output elision emits source
tool_output_artifact, exact serialized before/after byte counts, zero droppedmessages and the complete artifact SHA-256. No content enters these events.
Provider-side transformations with no trustworthy observation remain unknown.
No usage/limit endpoint is polled. These events convey observations, not owner,
admission, replay or cleanup authority. Deployment and canary are pending.
AKH_RUNTIME_LEGACY_EVENTS=true projects out turn_usage for old typed readers; the full durable job log still retains it. Modern central readers need the flag disabled. The standalone bridge explicitly records new event diagnostics without adding them to reply text.
Review additions: Codex thread/tokenUsage/updated.tokenUsage.modelContextWindow supplies context_limit_tokens, source provider_reported, occupancy from last inputTokens. Window joins snapshot fingerprint so changed metadata cannot mutate an immutable event. Claude's observed SDK usage has no context-window field and remains unknown.
Both event.usage.failure_cause (TurnUsage) and event.observation.failure_cause (ModelObservation) are nullable typed ModelFailureCause. terminal_code uses the cause's snake_case category, except incomplete maps to model_incomplete for Anima's existing filter; absent cause keeps the original Fault.code. RuntimeJob.error.code lifecycle semantics stay unchanged. TurnUsage.usage_known_calls is nullable count of distinct per-call observations with known input and output counts; totals still remain unknown when any call omits them, SDK snapshots stay unknown.
Artifact-elision event capacity refusal warns and continues with the canonical tool receipt/result; identity/conflict errors remain fatal. Model observation refusal at a model boundary is Failed via model_boundary. Rollout requires compatibility readers/session service first, then AKH_RUNTIME_MODEL_EVENT_CONSUMER=model-events-v1 plus legacy flag disabled; rollback floor is #674's metadata-tolerant readers.
Prior validation
Fable Round2 approved reviewed head
d322e08bbcc8. Previous follow-up head:d632619d8a07b8044e0de99abf620683db46a764.Ordinary stack integration carries the A1 reset regression, output collection eligibility fix/test, and deployed-mode rollout prerequisite/startup test. D1 wire contracts are unchanged. The final integrated runtime service suite re-exercises the new tests plus failed-worker cause/filter code and observation/transform boundary handling; previously reviewed Codex provider coverage is unchanged.
Current-head runtime service library suite:115 passed,1 existing real OpenBao/Postgres test ignored (r2-followup-d1-final.log). Local workspace crates were cleaned before compilation to prevent cross-worktree Cargo artifact reuse. Final integrated #678 suite115 passed/1 existing real-backend ignored. Full exact-head Solstice CI remains a merge gate; local results do not substitute for it. No merge, force-push, deployment or canary.
Finished task target was reclaimed after verifying no open files;37.08GiB free at10:16Z. Peer target preserved.
CI4360 follow-up
Current head:
0cb59f25821d73df2dcc09e938c5075c6a6c3e1e.Solstice exact-head evidence for #676
dae86ef17bidentifies failed request e257c6a0-5f0a-4a9e-8b58-d372ffb7ad0f, categorystep:cargo-test-features-postgres-lib. The root suite had3961 passed,22 ignored and1 failure:agent::runtime::tests::codex_tools_use_typed_akh_registry_and_record_evidence, src/agent/runtime/tests.rs:117, expecting plain "7" while the fixture echoed the complete ToolResult JSON. This was a missed regression expectation from the reviewed complete-envelope contract, not a refused connection or runner fault. It reproduced locally before the fix. Reported Forgejo job: /toasterson/akh-medu/actions/runs/4360/jobs/7.The root regression now checks all decoded fields
{success:true,text:"7",data:null,receipt:null}and keeps its tool-evidence trace assertion. New Codex envelope coverage preserves both text-only and structured data/receipt results alongside the existing64KiB escaped-envelope refusal. The production envelope implementation hash is unchanged; no wire contract change. The identical stale root test and implementation existed on #677/#678, so normal stack integration carries this fix through both. No force-push or PR merge.Current-head local receipts:
ci4360-678-runtime-root.log: test result: ok. 10 passed; 0 failed; 2 ignored; 0 measured; 3973 filtered out; finished in 1.39sci4360-678-envelope.log: test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 14 filtered out; finished in 0.01sci4360-678-provider.log: test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 13 filtered out; finished in 0.46sLocal workspace packages were cleaned between worktrees to exclude stale Cargo reuse. Final full Linux CI is still a separate gate. No blind rerun was requested; the normal follow-up commits publish the diagnosed fix. No deployment, Talos/Ptah change or live provider probe. Builds used31.5GiB stop/32GiB preflight guards; the finished task target is reclaimed on handoff.
Own finished target was verified without open files and deleted; 37.269GiB free at 2026-10-19T11:58:41.253438+00:00. Peer targets were untouched.
toasterson referenced this pull request2026-10-19 10:16:42 +00:00