Q1: Queue usage-limited providers behind durable reset windows

toasterson commented

2026-10-19 08:08:36 +00:00

Owner

Q1 records durable provider/account/model limit windows when an authenticated central or runner turn reports typed rate_limit/usage_limit, or the exact legacy structured 429. Queued work stays queued until the reported reset. Then one queued assignment claims a durable single-flight probe in its canonical allocation transaction. Other sessions/lanes wait; restart or an unknown probe outcome cannot authorize another probe. A successful authenticated turn closes the window; a new rejection extends it with its known reset or capped exponential fallback. No provider limit/usage endpoint is polled.

Grouping uses explicit lane provider/account configuration or A1's explicit opaque account plus provider identity, never tokens or guessed alias ownership. Unknown model fences the group; read-only preflights conservatively fence all models of a limited group. Configure unobserved shared-account aliases explicitly before first dispatch. Existing runtime bounded request policy remains inside the single probe assignment. WI detail and scheduling state show reset, state and probe outcome; structured logs record transitions.

Migration156 imports earlier cooldown state into the authoritative window table. Completion observations and close decisions share a transaction; legacy warm-turn receipts are keyed by sequence. Central implementation/review and legacy claims check admission before committing ownership/publication. Replays are no-ops. Missing reset fallback defaults300s, doubles only for a re-limited probe, cap86400s. Pending effects, workspace proof, held items and budgets preserve their gates. Live unknown outcomes require operator investigation; authenticated non-limit terminal outcomes reopen after30seconds, including failed/cancelled/retired sessions reconciled after restart.

Dependency: optional Akh-Medu #674 head b79e50adb8d3ec030edd22a786d965cb53958432 supplies reset/identity metadata. Exact historical structured failures use fallback. Contract: docs/ai/contracts/provider-admission-cooldown.md.

Validation: 558 server lib tests passed (6 environment-dependent ignored); disposable PostgreSQL18 with all migrations passed concurrent rejection coalescing, zero admission before reset, exactly one post-reset claimant, restart persistence and bounded missing-reset backoff. Existing central completion (2) and staging (1) regressions passed, including implementation/reviewer final admission after staging, aliases, replay, retained cleanup proof and noncharging progress. Final sequence-key schema rerun passed for the window fixture and all three central regressions. Scoped rustfmt, diff/shell checks and CI dry-run recorded in progress artifact. New database regression is wired into nightly CI.

Solstice exact-head CI and independent review are pending. Shared Forgejo-to-Solstice dispatch incident belongs to MausBot round16; no blind reruns. Local tests are distinct from CI and runtime acceptance. Do not merge: operator human gate. No deployment/live canary or recovery action.

Fable follow-up: terminal non-limit probes reopen after a bounded30s delay; the idempotent clear_provider_limit MCP verb requires operator.write PAT and user identity, records receipt+activity audit atomically, and never releases owners/replays work. Provider resets are clamped to ANIMA_PROVIDER_LIMIT_MAX_RESET_SECS (default35days, bounded1s–366days), preserving raw reported_reset_at and raising dashboard Attention plus warning. Migration159 also caps pre-existing extreme fences. Legacy review refusal now names ProviderLimited.

Validation at follow-up:560lib tests passed/6ignored; expanded disposable PostgreSQL regression passed failed/cancelled/reconciled probe endings, live non-limit terminal turn,30s bounded delay, persistent single-flight behavior, audited clear replay and existing concurrency/reset/backoff. Unit tests cover cap boundaries and missing/smuggled operator scope denial before DB access. CI remains pending under the round16 dispatch incident; no rerun/merge/deploy.

Round2 follow-up: cancelled settled turn sets now produce an admission-only CompletionRead::Cancelled observation through the completion path, reopening the probe after30s while preserving the resumable session/owner and leaving closing_intent unset. An explicit typed rejection arriving before cancellation retains its reported reset instead. Client and production completion PostgreSQL cases cover both paths.

Migration decision: retain153/156/159 deliberately as an additive preview-upgrade path. This implementation applied only disposable fixtures and claims no production application. We cannot establish that no operator applied an earlier published preview; retaining checksums and the one-time import permits that upgrade without rewriting applied history. The old table is intentionally dormant after import; provider_limit_windows is authoritative.

Integration refresh (2026-10-19): normal main/dependency merge commits, no force push. Main is 8d3eb5855f. This head also includes refreshed #799 so the agreed merge order cannot repeat the same insertion conflict; its diff narrows as predecessors land. Shell/Rust syntax, diff and predecessor-ancestry checks pass. No fresh local cargo check: available disk headroom above the 31.25 GiB floor was smaller than the last measured owned compile target. Exact new-head CI is the gate; prior receipts remain prior-head evidence.

Q1 records durable provider/account/model limit windows when an authenticated central or runner turn reports typed rate_limit/usage_limit, or the exact legacy structured 429. Queued work stays queued until the reported reset. Then one queued assignment claims a durable single-flight probe in its canonical allocation transaction. Other sessions/lanes wait; restart or an unknown probe outcome cannot authorize another probe. A successful authenticated turn closes the window; a new rejection extends it with its known reset or capped exponential fallback. No provider limit/usage endpoint is polled. Grouping uses explicit lane provider/account configuration or A1's explicit opaque account plus provider identity, never tokens or guessed alias ownership. Unknown model fences the group; read-only preflights conservatively fence all models of a limited group. Configure unobserved shared-account aliases explicitly before first dispatch. Existing runtime bounded request policy remains inside the single probe assignment. WI detail and scheduling state show reset, state and probe outcome; structured logs record transitions. Migration156 imports earlier cooldown state into the authoritative window table. Completion observations and close decisions share a transaction; legacy warm-turn receipts are keyed by sequence. Central implementation/review and legacy claims check admission before committing ownership/publication. Replays are no-ops. Missing reset fallback defaults300s, doubles only for a re-limited probe, cap86400s. Pending effects, workspace proof, held items and budgets preserve their gates. Live unknown outcomes require operator investigation; authenticated non-limit terminal outcomes reopen after30seconds, including failed/cancelled/retired sessions reconciled after restart. Dependency: optional Akh-Medu [#674](https://code.aopc.cloud/toasterson/akh-medu/pulls/674) head `b79e50adb8d3ec030edd22a786d965cb53958432` supplies reset/identity metadata. Exact historical structured failures use fallback. Contract: `docs/ai/contracts/provider-admission-cooldown.md`. Validation: 558 server lib tests passed (6 environment-dependent ignored); disposable PostgreSQL18 with all migrations passed concurrent rejection coalescing, zero admission before reset, exactly one post-reset claimant, restart persistence and bounded missing-reset backoff. Existing central completion (2) and staging (1) regressions passed, including implementation/reviewer final admission after staging, aliases, replay, retained cleanup proof and noncharging progress. Final sequence-key schema rerun passed for the window fixture and all three central regressions. Scoped rustfmt, diff/shell checks and CI dry-run recorded in progress artifact. New database regression is wired into nightly CI. Solstice exact-head CI and independent review are pending. Shared Forgejo-to-Solstice dispatch incident belongs to MausBot round16; no blind reruns. Local tests are distinct from CI and runtime acceptance. **Do not merge: operator human gate.** No deployment/live canary or recovery action. Fable follow-up: terminal non-limit probes reopen after a bounded30s delay; the idempotent clear_provider_limit MCP verb requires operator.write PAT and user identity, records receipt+activity audit atomically, and never releases owners/replays work. Provider resets are clamped to ANIMA_PROVIDER_LIMIT_MAX_RESET_SECS (default35days, bounded1s–366days), preserving raw reported_reset_at and raising dashboard Attention plus warning. Migration159 also caps pre-existing extreme fences. Legacy review refusal now names ProviderLimited. Validation at follow-up:560lib tests passed/6ignored; expanded disposable PostgreSQL regression passed failed/cancelled/reconciled probe endings, live non-limit terminal turn,30s bounded delay, persistent single-flight behavior, audited clear replay and existing concurrency/reset/backoff. Unit tests cover cap boundaries and missing/smuggled operator scope denial before DB access. CI remains pending under the round16 dispatch incident; no rerun/merge/deploy. Round2 follow-up: cancelled settled turn sets now produce an admission-only CompletionRead::Cancelled observation through the completion path, reopening the probe after30s while preserving the resumable session/owner and leaving closing_intent unset. An explicit typed rejection arriving before cancellation retains its reported reset instead. Client and production completion PostgreSQL cases cover both paths. Migration decision: retain153/156/159 deliberately as an additive preview-upgrade path. This implementation applied only disposable fixtures and claims no production application. We cannot establish that no operator applied an earlier published preview; retaining checksums and the one-time import permits that upgrade without rewriting applied history. The old table is intentionally dormant after import; provider_limit_windows is authoritative.  Integration refresh (2026-10-19): normal main/dependency merge commits, no force push. Main is 8d3eb5855f74c616d176b788e2ca881d2f933e3f. This head also includes refreshed #799 so the agreed merge order cannot repeat the same insertion conflict; its diff narrows as predecessors land. Shell/Rust syntax, diff and predecessor-ancestry checks pass. No fresh local cargo check: available disk headroom above the 31.25 GiB floor was smaller than the last measured owned compile target. Exact new-head CI is the gate; prior receipts remain prior-head evidence.

toasterson added 1 commit

2026-10-19 08:08:37 +00:00

Persist central provider cooldowns and gate fresh admission

CI / conflict-check (push) Successful in 50s

Details

CI / test (push) Has been cancelled

Details

eeaa095cf3

~~toasterson referenced this pull request from toasterson/akh-medu 2026-10-19 08:13:41 +00:00~~

Preserve typed provider failures for central admission and provenance #674

toasterson referenced this pull request

2026-10-19 08:20:56 +00:00

Q2: Refuse oversized final briefs without dropping mandatory input #796

toasterson changed title from ~~Q1: Persist provider throttling cooldowns for central admission~~ to Q1: Queue usage-limited providers behind durable reset windows

2026-10-19 08:54:49 +00:00

toasterson added 1 commit

2026-10-19 08:54:56 +00:00

Queue usage-limited providers behind durable reset windows and one probe

CI / conflict-check (push) Successful in 37s

Details

CI / test (push) Has been cancelled

Details

256b2df4a5

toasterson added 1 commit

2026-10-19 09:43:22 +00:00

fix: recover ended limit probes and audit bounded operator clearing