Fable: bound installed-conformance readiness latency during active inference #660

New issue

Open

opened 2026-10-16 06:19:07 +00:00 by toasterson · 0 comments

toasterson commented

2026-10-16 06:19:07 +00:00

Owner

Follow-up from the accepted native-house migration, Talos #218. This issue records a remaining limitation; the migration and cutover were accepted. No Board access is needed.

Reproduction evidence

On wk03, runtime-fable-review-1-5b58bc79c5-c6b8f used the main-45 runtime image sha256:23046e07a9beff86a2f6898813b6b1cb4aa8224a72f7586ee3d323e0e5854dc2. Image-native conformance init succeeded. Claude Code 2.1.301 was observed in the image (an observation, not a required version).

During a real Odin memory-recall turn, readiness became false at approximately 2026-10-16 06:04:34 UTC because this exec probe exceeded its 20-second timeout (period 30 seconds, failure threshold 2):

/usr/local/bin/akh-runtime-profile --config /config/service.json --profile fable-review-1 verify

A subsequent manual verification completed with fable-review-1: installed runtime conformance verified; Odin completed the turn and recalled the durable marker at 06:06:41 UTC, and the runtime returned Ready. Thus this observation is not evidence of an incorrect model or stale bridge. SD-card/CPU contention and verifier cost need measurement; root cause is not yet proven.

Final health, image identity and explicit limitation.
Successful final MausBot/provider recall, cutover report.
Private timing/event receipts: /Users/toasty/.local/share/akh-house-migration/20261016-go/cutover/fable-readiness-private.json and fable-final-progress.log; publish only sanitized timing/phase information.

Reproduce by measuring the same verification command and kubelet probe latencies during a representative Claude turn on wk03. Record CPU, I/O wait and verification-phase timings without provider credentials or prompt contents. Do not introduce parallel image pulls/rollouts on that node.

Impact

A healthy in-flight worker temporarily loses readiness and blocks Flux health progression; new traffic can lose its service endpoint despite successful inference. Increasing the timeout alone would mask the unmeasured verifier cost.

Proposed fix

Profile akh-runtime-profile verify in crates/akh-session-services/src/bin/akh-runtime-profile.rs and runtime::validate_conformance. Separate expensive immutable image/runtime/probe certification at init from bounded readiness checks. Cache only evidence bound to exact artifacts, invalidate it on drift, and retain fail-closed rejection of missing/stale/mismatched conformance. Avoid repeatedly spawning expensive CLI/version work or hashing unchanged large artifacts if profiling identifies it. Coordinate the resulting probe command/budget with the Talos runtime renderer.

Acceptance: measure readiness latency under an active turn on wk03; valid workers remain Ready within the declared probe budget, while corrupted/mismatched receipts and runtime/probe drift still fail closed. Record installed provider versions from the image; never pin or assert a CLI version. Keep wk03 placement, operator-held runtimes, full protocol behavior and sequential rollout constraints unchanged.

Follow-up from the accepted native-house migration, [Talos #218](https://github.com/Toasterson/talos/pull/218). This issue records a remaining limitation; the migration and cutover were accepted. No Board access is needed. ## Reproduction evidence On wk03, `runtime-fable-review-1-5b58bc79c5-c6b8f` used the main-45 runtime image `sha256:23046e07a9beff86a2f6898813b6b1cb4aa8224a72f7586ee3d323e0e5854dc2`. Image-native conformance init succeeded. Claude Code 2.1.301 was observed in the image (an observation, not a required version). During a real Odin memory-recall turn, readiness became false at approximately 2026-10-16 06:04:34 UTC because this exec probe exceeded its 20-second timeout (period 30 seconds, failure threshold 2): ```sh /usr/local/bin/akh-runtime-profile --config /config/service.json --profile fable-review-1 verify ``` A subsequent manual verification completed with `fable-review-1: installed runtime conformance verified`; Odin completed the turn and recalled the durable marker at 06:06:41 UTC, and the runtime returned Ready. Thus this observation is not evidence of an incorrect model or stale bridge. SD-card/CPU contention and verifier cost need measurement; root cause is not yet proven. - [Final health, image identity and explicit limitation](https://github.com/Toasterson/talos/blob/e061ea36a85d1915072b26b9c8f7ae34f14a9147/evidence/akh-houses-20261016/cutover-final-health.json). - [Successful final MausBot/provider recall](https://github.com/Toasterson/talos/blob/e061ea36a85d1915072b26b9c8f7ae34f14a9147/evidence/akh-houses-20261016/mausbot-final-cutover-acceptance.json), [cutover report](https://github.com/Toasterson/talos/blob/e061ea36a85d1915072b26b9c8f7ae34f14a9147/CUTOVER.md). - Private timing/event receipts: `/Users/toasty/.local/share/akh-house-migration/20261016-go/cutover/fable-readiness-private.json` and `fable-final-progress.log`; publish only sanitized timing/phase information. Reproduce by measuring the same verification command and kubelet probe latencies during a representative Claude turn on wk03. Record CPU, I/O wait and verification-phase timings without provider credentials or prompt contents. Do not introduce parallel image pulls/rollouts on that node. ## Impact A healthy in-flight worker temporarily loses readiness and blocks Flux health progression; new traffic can lose its service endpoint despite successful inference. Increasing the timeout alone would mask the unmeasured verifier cost. ## Proposed fix Profile `akh-runtime-profile verify` in `crates/akh-session-services/src/bin/akh-runtime-profile.rs` and `runtime::validate_conformance`. Separate expensive immutable image/runtime/probe certification at init from bounded readiness checks. Cache only evidence bound to exact artifacts, invalidate it on drift, and retain fail-closed rejection of missing/stale/mismatched conformance. Avoid repeatedly spawning expensive CLI/version work or hashing unchanged large artifacts if profiling identifies it. Coordinate the resulting probe command/budget with the Talos runtime renderer. Acceptance: measure readiness latency under an active turn on wk03; valid workers remain Ready within the declared probe budget, while corrupted/mismatched receipts and runtime/probe drift still fail closed. Record installed provider versions from the image; never pin or assert a CLI version. Keep wk03 placement, operator-held runtimes, full protocol behavior and sequential rollout constraints unchanged.

toasterson added the

follow-up

label

2026-10-16 06:19:07 +00:00

No labels

follow-up

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu#660

No description provided.

Rows
Columns