Preserve typed provider failures for central admission and provenance #674

Merged

toasterson merged 6 commits from fix/anima-a1-provider-outcomes into main 2026-10-19 20:00:26 +00:00

Conversation 0 Commits 6 Files changed 7 +1268 -143

toasterson commented 2026-10-19 08:03:00 +00:00

Owner

Copy link

Preserve typed provider failures/reset provenance through model boundaries so central admission can use exact account/model windows without polling providers.

Targets main. Supplies Q1 #795 and the compatibility floor for the stack. No PR merge, deployment, live recovery or canary.

Review fixes

Reset tests now reach the actual backoff branch: no Retry-After plus named5s reset waits at least5s before a second send; named2h reset with1s deadline rejects after one send. Configured identities match Anima's [A-Za-z0-9_.:/-]{1,128} grammar. Usage-limit rejection never retries locally; missing identity decodes as unknown. This PR also supplies the metadata-tolerant reader rollback floor required by #677/#678; unknown control events still fail closed.

Exact consumer contract

Model failure v1 (A1; Anima Q1/S3)

RuntimeJob.error and TurnRecord.error retain their existing Fault.code and
message. An optional model_failure field carries sanitized model-boundary
observations. It is omitted for legacy/non-model faults; missing values are
unknown, never zero, and this contract grants no authority to replay work.

{"code":"model_incomplete","message":"model_error: model returned HTTP 429; code=rate_limit_exceeded; type=rate_limit; request_id=req-123; attempts=3","model_failure":{"schema_version":1,"cause":"rate_limit","http_status":429,"retry_after_secs":0,"provider_code":"rate_limit_exceeded","provider_type":"rate_limit","provider_request_id":"req-123","attempts":3}}

Required metadata fields: schema_version:1 and cause. The cause vocabulary is
rate_limit, usage_limit, context_limit, output_limit, deadline, cancelled, transport,
provider_error, incomplete. Optional fields are http_status:u16,
retry_after_secs:u64, provider_code:string, provider_type:string,
provider_request_id:string, finish_reason:string, attempts:u32.

Provider identifiers and finish reasons allow ASCII alphanumeric, _, -, .
and at most 128 characters. No raw provider body or prompt enters metadata.
retry_after_secs is present only for numeric Retry-After in 0..=30. It preserves
the final rejection's header even at attempt three; it does not request another
retry. No header means absent metadata, despite the runtime's local 1s/2s default
backoff. Invalid, HTTP-date or >30s headers stop local retry and remain absent.

Only explicit HTTP 429 responses can retry, at most three sends. Cancellation
and the enclosing turn deadline still interrupt sends, reads and backoff. Partial
streams, transport ambiguity and tools are not retried. HTTP length finish maps
to output_limit; explicit context-refusal codes map to context_limit; missing
finish/DONE stays incomplete. A native finish reason never invents HTTP status,
request identity or a provider attempt count. Legacy model-boundary lifecycle
codes remain model_incomplete / model_cancelled so continuation reconciliation
and operator fences retain their semantics.

Consumers should first parse the optional typed metadata, retain their exact
legacy structured-error parser, and preserve unknown fields/versions as unknown.
A1 supplies classification for Q1's durable admission cooldown; per-call usage,
actual model revision and context/elision provenance are separate A3/S3 work.
Source evidence is analysis-anima-3w/synthesis.md E02/E04/E06, not a deployment
or historical provider-model attestation. Deployment and authorized canary pending.

Durable usage-limit windows (operator extension)

Rate/usage rejections also carry optional model_failure.limit:

{"schema_version":1,"kind":"tokens","reset_at_ms":1792567680000,"reset_source":{"kind":"header","name":"x-ratelimit-reset-tokens"},"identity":{"provider":"api.example.com","account":null,"model":"requested-model"}}

All five members are present. kind is requests|tokens|monthly_usage|null;
reset_at_ms is nullable absolute UTC Unix milliseconds, including long waits;
reset_source is nullable {kind:header|body,name:string}. identity has nullable
provider, account, model. Provider uses explicit ModelEndpoint.provider_identity
or endpoint DNS host, account only explicit opaque account_identity, model the
exact request binding. Configured identities are validated, never sanitized into
another value. No credential, token digest, default account or model alias is
substituted. An endpoint proxy host identifies that endpoint, not its hidden backend.

Supported sources: Retry-After integer seconds or HTTP-date; named
x-ratelimit-reset[-requests|-tokens|-project-requests|-project-tokens] in compound
durations (including ms), Unix epoch seconds/milliseconds or RFC3339; Anthropic
request/token/input-token/output-token reset RFC3339 headers. Header names retain
the selected canonical name from the HTTP library. Positive matching remaining
counts exclude unexhausted windows. Among exhausted or unknown windows and body
fields, select the latest valid reset conservatively, with its exact source/kind.
Body allowlist: /error/ or root reset_at_ms, reset_at, resets_at,
retry_after, retry_after_seconds, retry_after_ms; source name is its JSON
pointer. Only explicit limit_kind/limit_type, monthly_usage_limit code, or literal
monthly usage limit wording establish body kind. An unrelated named
request/token window is excluded when the body names a different limit kind.
Malformed/ambiguous values remain null. insufficient_quota yields usage_limit
but does not establish a monthly quota. Historical records remain unknown.

The legacy retry_after_secs remains independently bounded to numeric <=30.
Local sends retain the explicit-429 three-attempt policy. A known reset extends
backoff (never earlier than reset); if this exceeds 30s or remaining turn deadline,
return the original typed rejection immediately, rather than consuming the deadline.
Invalid/date/>30 Retry-After still stops local retry, while its parsed reset is
available to durable central admission. No rate/usage endpoint is queried.

Header format references: OpenAI rate limits,
Claude rate limits.

Review refinements: configured identities use [A-Za-z0-9_.:/-]{1,128}, matching Anima (no @). Missing limit identity defaults to all unknown. UsageLimit stops on the first rejection even without a reset; transient rate limits retain the three-send bound. ModelEndpoint uses deny_unknown_fields: rollback must also restore the older profile configuration before starting the older runtime.

Runtime/session readers in this compatibility release skip future model_observation/context_transform/turn_usage metadata when unsupported, while unknown control events still fail closed. This is the minimum rollback floor for the telemetry stack; binaries predating this floor require a pre-upgrade store/config snapshot, not an in-place rollback.

Validation

Fable Round2 approved reviewed head b79e50adb8d3. Current follow-up head: b2452e081e25604a5ce76da103075d7bfeef5b15.

The beyond-deadline case uses a20s named-header reset and10s deadline: ordinary1s retry spacing <10s deadline <20s reset <30s local cap. It asserts one send, typed requests limit, exact header provenance and immediate return before the deadline. This isolates the remaining-deadline check; the5s in-deadline reset still waits and retries successfully.

Current-head focused reset test:1 passed (r2-followup-a1.log). Final integrated #678 suite115 passed/1 existing real-backend ignored. Full exact-head Solstice CI remains a merge gate; local results do not substitute for it. No merge, force-push, deployment or canary.

Finished task target was reclaimed after verifying no open files;37.08GiB free at10:16Z. Peer target preserved.

Preserve typed provider failures/reset provenance through model boundaries so central admission can use exact account/model windows without polling providers. Targets main. Supplies Q1 #795 and the compatibility floor for the stack. No PR merge, deployment, live recovery or canary. ## Review fixes Reset tests now reach the actual backoff branch: no Retry-After plus named5s reset waits at least5s before a second send; named2h reset with1s deadline rejects after one send. Configured identities match Anima's `[A-Za-z0-9_.:/-]{1,128}` grammar. Usage-limit rejection never retries locally; missing identity decodes as unknown. This PR also supplies the metadata-tolerant reader rollback floor required by #677/#678; unknown control events still fail closed. ## Exact consumer contract # Model failure v1 (A1; Anima Q1/S3) `RuntimeJob.error` and `TurnRecord.error` retain their existing `Fault.code` and `message`. An optional `model_failure` field carries sanitized model-boundary observations. It is omitted for legacy/non-model faults; missing values are unknown, never zero, and this contract grants no authority to replay work. ```json {"code":"model_incomplete","message":"model_error: model returned HTTP 429; code=rate_limit_exceeded; type=rate_limit; request_id=req-123; attempts=3","model_failure":{"schema_version":1,"cause":"rate_limit","http_status":429,"retry_after_secs":0,"provider_code":"rate_limit_exceeded","provider_type":"rate_limit","provider_request_id":"req-123","attempts":3}} ``` Required metadata fields: `schema_version:1` and `cause`. The cause vocabulary is `rate_limit`, `usage_limit`, `context_limit`, `output_limit`, `deadline`, `cancelled`, `transport`, `provider_error`, `incomplete`. Optional fields are `http_status:u16`, `retry_after_secs:u64`, `provider_code:string`, `provider_type:string`, `provider_request_id:string`, `finish_reason:string`, `attempts:u32`. Provider identifiers and finish reasons allow ASCII alphanumeric, `_`, `-`, `.` and at most 128 characters. No raw provider body or prompt enters metadata. `retry_after_secs` is present only for numeric Retry-After in 0..=30. It preserves the final rejection's header even at attempt three; it does not request another retry. No header means absent metadata, despite the runtime's local 1s/2s default backoff. Invalid, HTTP-date or >30s headers stop local retry and remain absent. Only explicit HTTP 429 responses can retry, at most three sends. Cancellation and the enclosing turn deadline still interrupt sends, reads and backoff. Partial streams, transport ambiguity and tools are not retried. HTTP `length` finish maps to `output_limit`; explicit context-refusal codes map to `context_limit`; missing finish/DONE stays `incomplete`. A native finish reason never invents HTTP status, request identity or a provider attempt count. Legacy model-boundary lifecycle codes remain `model_incomplete` / `model_cancelled` so continuation reconciliation and operator fences retain their semantics. Consumers should first parse the optional typed metadata, retain their exact legacy structured-error parser, and preserve unknown fields/versions as unknown. A1 supplies classification for Q1's durable admission cooldown; per-call usage, actual model revision and context/elision provenance are separate A3/S3 work. Source evidence is `analysis-anima-3w/synthesis.md` E02/E04/E06, not a deployment or historical provider-model attestation. Deployment and authorized canary pending. ## Durable usage-limit windows (operator extension) Rate/usage rejections also carry optional `model_failure.limit`: ```json {"schema_version":1,"kind":"tokens","reset_at_ms":1792567680000,"reset_source":{"kind":"header","name":"x-ratelimit-reset-tokens"},"identity":{"provider":"api.example.com","account":null,"model":"requested-model"}} ``` All five members are present. `kind` is `requests|tokens|monthly_usage|null`; `reset_at_ms` is nullable absolute UTC Unix milliseconds, including long waits; `reset_source` is nullable `{kind:header|body,name:string}`. `identity` has nullable `provider`, `account`, `model`. Provider uses explicit `ModelEndpoint.provider_identity` or endpoint DNS host, account only explicit opaque `account_identity`, model the exact request binding. Configured identities are validated, never sanitized into another value. No credential, token digest, default account or model alias is substituted. An endpoint proxy host identifies that endpoint, not its hidden backend. Supported sources: Retry-After integer seconds or HTTP-date; named x-ratelimit-reset[-requests|-tokens|-project-requests|-project-tokens] in compound durations (including ms), Unix epoch seconds/milliseconds or RFC3339; Anthropic request/token/input-token/output-token reset RFC3339 headers. Header names retain the selected canonical name from the HTTP library. Positive matching remaining counts exclude unexhausted windows. Among exhausted or unknown windows and body fields, select the latest valid reset conservatively, with its exact source/kind. Body allowlist: `/error/` or root `reset_at_ms`, `reset_at`, `resets_at`, `retry_after`, `retry_after_seconds`, `retry_after_ms`; source name is its JSON pointer. Only explicit `limit_kind`/`limit_type`, `monthly_usage_limit` code, or literal `monthly usage limit` wording establish body kind. An unrelated named request/token window is excluded when the body names a different limit kind. Malformed/ambiguous values remain null. `insufficient_quota` yields `usage_limit` but does not establish a monthly quota. Historical records remain unknown. The legacy `retry_after_secs` remains independently bounded to numeric <=30. Local sends retain the explicit-429 three-attempt policy. A known reset extends backoff (never earlier than reset); if this exceeds 30s or remaining turn deadline, return the original typed rejection immediately, rather than consuming the deadline. Invalid/date/>30 Retry-After still stops local retry, while its parsed reset is available to durable central admission. No rate/usage endpoint is queried. Header format references: [OpenAI rate limits](https://developers.openai.com/api/docs/guides/rate-limits), [Claude rate limits](https://platform.claude.com/docs/en/api/rate-limits). Review refinements: configured identities use `[A-Za-z0-9_.:/-]{1,128}`, matching Anima (no @). Missing limit identity defaults to all unknown. UsageLimit stops on the first rejection even without a reset; transient rate limits retain the three-send bound. ModelEndpoint uses deny_unknown_fields: rollback must also restore the older profile configuration before starting the older runtime. Runtime/session readers in this compatibility release skip future model_observation/context_transform/turn_usage metadata when unsupported, while unknown control events still fail closed. This is the minimum rollback floor for the telemetry stack; binaries predating this floor require a pre-upgrade store/config snapshot, not an in-place rollback. ## Validation Fable Round2 approved reviewed head `b79e50adb8d3`. Current follow-up head: `b2452e081e25604a5ce76da103075d7bfeef5b15`. The beyond-deadline case uses a20s named-header reset and10s deadline: ordinary1s retry spacing <10s deadline <20s reset <30s local cap. It asserts one send, typed requests limit, exact header provenance and immediate return before the deadline. This isolates the remaining-deadline check; the5s in-deadline reset still waits and retries successfully. Current-head focused reset test:1 passed (r2-followup-a1.log). Final integrated #678 suite115 passed/1 existing real-backend ignored. Full exact-head Solstice CI remains a merge gate; local results do not substitute for it. No merge, force-push, deployment or canary. Finished task target was reclaimed after verifying no open files;37.08GiB free at10:16Z. Peer target preserved.

toasterson added 1 commit 2026-10-19 08:03:00 +00:00

Preserve typed model failure provenance at internal boundary 935bd735e1

toasterson referenced this pull request from toasterson/Anima 2026-10-19 08:08:36 +00:00

Q1: Queue usage-limited providers behind durable reset windows #795

toasterson referenced this pull request 2026-10-19 08:27:13 +00:00

Bound model tool envelopes with ranged artifacts and process cursors #676

toasterson referenced this pull request 2026-10-19 08:32:28 +00:00

Persist typed model usage identity and context provenance #677

toasterson added 1 commit 2026-10-19 08:43:34 +00:00

Carry durable provider limit resets and exact call identity 2f670d5d88

toasterson referenced this pull request 2026-10-19 08:46:04 +00:00

Emit per-turn usage and explicit context diagnostics #678

toasterson added 1 commit 2026-10-19 08:50:03 +00:00

Distinguish explicit monthly limits from unrelated rate windows 785ff07c52

toasterson added 2 commits 2026-10-19 09:38:20 +00:00

Address provider reset branch coverage and identity compatibility 7e39be5db5

Add rollback-safe reader floor for future model metadata b79e50adb8

toasterson added 1 commit 2026-10-19 10:13:51 +00:00

test(runtime): isolate reset beyond remaining deadline b2452e081e

toasterson merged commit 5cc23e838d into main 2026-10-19 20:00:26 +00:00

toasterson referenced this pull request from a commit 2026-10-19 20:00:29 +00:00

Merge PR #674: Preserve typed provider failures for central admission and provenance

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

follow-up

Documented operational follow-up

No labels

follow-up

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!674

No description provided.