toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 12 Pull requests 38 Projects Releases 3 Packages 8 Wiki Activity Actions 3

Bound native inference lost-ack recovery and honor interactive deadlines #645

Open
toasterson wants to merge 2 commits from fix/odin-inference-lost-ack-20261015 into main
pull from: fix/odin-inference-lost-ack-20261015
merge into: toasterson:main
Conversation 0 Commits 2 Files changed 12 +1274 -51
toasterson commented 2026-10-15 07:14:34 +00:00
Owner
Copy link

Odin's interactive native turn waited 1,800 seconds after an I/O failure and repeated request lookup 404s, then surfaced an operational failure as a completion-contract refusal. Native submission now reconciles the exact ID and retries only the identical payload, at most three times within a 30s recovery window, using the existing runtime's durable ID deduplication. Missing/unreachable jobs fail promptly and retain receipts.

Pass the resolved turn/profile deadline into blocking native calls, use a 300s native default, and permit a rolling 30s extension only for newly observed runtime text/reasoning. Local polling/dots and replayed events do not extend it. Native failures and timeouts return proper ACP JSON-RPC errors; explicit cancellation and genuine contract refusal remain distinct. Add a bounded identity-aware _akh/inference/health probe and verified private-CA TLS support.

Includes regression tests for missing and admitted lost acknowledgements, retry payload identity/bounds, deadlines, streaming extensions, RPC error mapping, and TLS trust/SAN verification. Includes a read-only-by-default armini cutover script with planner tests and main-binary/probe/quiescence guards. The incident report documents evidence, rollout, rollback and remaining gates.

Companion Talos PR: https://github.com/Toasterson/talos/pull/207 replaces inference port-forwards with per-Akh private Cilium TLS LoadBalancers using existing credentials. Neither PR is merged; no host/cluster rollout or historical-request replay was performed. Deploy only a verified merged-main binary.

Validation: seven native backend tests; 62 agent-loop tests (one ignored); ACP operational-error regression; two archived-request dedup/conflict regressions; two Python cutover planner tests; production cargo check --locked --lib --features server; cargo fmt and git diff --check. A broader native test filter hit an unrelated native-audio fixture socket WouldBlock; its isolated rerun passed. Exact-head CI/review and post-merge live pilot acceptance remain separate gates.

Odin's interactive native turn waited 1,800 seconds after an I/O failure and repeated request lookup 404s, then surfaced an operational failure as a completion-contract refusal. Native submission now reconciles the exact ID and retries only the identical payload, at most three times within a 30s recovery window, using the existing runtime's durable ID deduplication. Missing/unreachable jobs fail promptly and retain receipts. Pass the resolved turn/profile deadline into blocking native calls, use a 300s native default, and permit a rolling 30s extension only for newly observed runtime text/reasoning. Local polling/dots and replayed events do not extend it. Native failures and timeouts return proper ACP JSON-RPC errors; explicit cancellation and genuine contract refusal remain distinct. Add a bounded identity-aware _akh/inference/health probe and verified private-CA TLS support. Includes regression tests for missing and admitted lost acknowledgements, retry payload identity/bounds, deadlines, streaming extensions, RPC error mapping, and TLS trust/SAN verification. Includes a read-only-by-default armini cutover script with planner tests and main-binary/probe/quiescence guards. The incident report documents evidence, rollout, rollback and remaining gates. Companion Talos PR: https://github.com/Toasterson/talos/pull/207 replaces inference port-forwards with per-Akh private Cilium TLS LoadBalancers using existing credentials. Neither PR is merged; no host/cluster rollout or historical-request replay was performed. Deploy only a verified merged-main binary. Validation: seven native backend tests; 62 agent-loop tests (one ignored); ACP operational-error regression; two archived-request dedup/conflict regressions; two Python cutover planner tests; production cargo check --locked --lib --features server; cargo fmt and git diff --check. A broader native test filter hit an unrelated native-audio fixture socket WouldBlock; its isolated rerun passed. Exact-head CI/review and post-merge live pilot acceptance remain separate gates.
Bound native inference recovery and interactive deadlines
Some checks failed
CI / docker-seshd (pull_request) Has been cancelled
Details
CI / docker-sokard (pull_request) Has been cancelled
Details
CI / docker-akh-transcripts (pull_request) Has been cancelled
Details
CI / publish-chart (pull_request) Has been cancelled
Details
CI / platform-images-home (pull_request) Has been cancelled
Details
CI / platform-images-core (pull_request) Has been cancelled
Details
CI / platform-images-daemons (pull_request) Has been cancelled
Details
CI / platform-images-fleet (pull_request) Has been cancelled
Details
CI / platform-images-collect (pull_request) Has been cancelled
Details
CI / test-akh-medu (pull_request) Has been cancelled
Details
CI / check-akh-medu (pull_request) Has been cancelled
Details
CI / check-seshat (pull_request) Has been cancelled
Details
CI / test-session-platform (pull_request) Has been cancelled
Details
CI / test-sokar (pull_request) Has been cancelled
Details
c4575997e9
Record companion incident pull request links
All checks were successful
CI / check-akh-medu (pull_request) Successful in 6m38s
Details
CI / check-seshat (pull_request) Successful in 10m15s
Details
CI / docker-seshd (pull_request) Successful in 6m51s
Details
CI / publish-chart (pull_request) Successful in 51s
Details
CI / test-sokar (pull_request) Successful in 20m20s
Details
CI / docker-sokard (pull_request) Successful in 13m25s
Details
CI / docker-akh-transcripts (pull_request) Successful in 13m33s
Details
CI / test-akh-medu (pull_request) Successful in 25m11s
Details
CI / test-session-platform (pull_request) Successful in 36m34s
Details
CI / platform-images-home (pull_request) Successful in 1m11s
Details
CI / platform-images-daemons (pull_request) Successful in 1m22s
Details
CI / platform-images-core (pull_request) Successful in 1m30s
Details
CI / platform-images-fleet (pull_request) Successful in 1m1s
Details
CI / platform-images-collect (pull_request) Successful in 1m1s
Details
58d0ca7d5d
All checks were successful
CI / check-akh-medu (pull_request) Successful in 6m38s
Details
CI / check-seshat (pull_request) Successful in 10m15s
Details
CI / docker-seshd (pull_request) Successful in 6m51s
Details
CI / publish-chart (pull_request) Successful in 51s
Details
CI / test-sokar (pull_request) Successful in 20m20s
Details
CI / docker-sokard (pull_request) Successful in 13m25s
Details
CI / docker-akh-transcripts (pull_request) Successful in 13m33s
Details
CI / test-akh-medu (pull_request) Successful in 25m11s
Details
CI / test-session-platform (pull_request) Successful in 36m34s
Details
CI / platform-images-home (pull_request) Successful in 1m11s
Details
CI / platform-images-daemons (pull_request) Successful in 1m22s
Details
CI / platform-images-core (pull_request) Successful in 1m30s
Details
CI / platform-images-fleet (pull_request) Successful in 1m1s
Details
CI / platform-images-collect (pull_request) Successful in 1m1s
Details
This pull request has changes conflicting with the target branch.
  • .forgejo/workflows/ci.yml
  • src/agent/chat_backend.rs
  • src/agent/native_backend.rs
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/odin-inference-lost-ack-20261015:fix/odin-inference-lost-ack-20261015
git switch fix/odin-inference-lost-ack-20261015

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff fix/odin-inference-lost-ack-20261015
git switch fix/odin-inference-lost-ack-20261015
git rebase main
git switch main
git merge --ff-only fix/odin-inference-lost-ack-20261015
git switch fix/odin-inference-lost-ack-20261015
git rebase main
git switch main
git merge --no-ff fix/odin-inference-lost-ack-20261015
git switch main
git merge --squash fix/odin-inference-lost-ack-20261015
git switch main
git merge --ff-only fix/odin-inference-lost-ack-20261015
git switch main
git merge fix/odin-inference-lost-ack-20261015
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
follow-up
Documented operational follow-up

No labels
follow-up
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!645
No description provided.