Bound transient stream recovery so completed reviews can retire #786

Merged

toasterson merged 5 commits from codex/plan14-stream-recovery-20261007 into main

2026-10-12 16:54:06 +00:00

toasterson commented

2026-10-07 19:31:27 +00:00

Owner

A RabbitMQ disconnect could permanently occupy every central relay worker: publish_chunk held the channel mutex while its reconnect helper tried to acquire the same mutex. Completed review turns then retained their Anima owners, preventing new reviews. The live incident showed successful reconnect followed by no further stream publications; a rolling restart released the owners and admitted pending reviews.

Pass the owned channel into recovery, recover channel-only failures as well as connection failures, and bound the full transient publish/reap operation to two seconds, including mutex waits and reconnect RPCs. Durable journal/transcript storage remains authoritative when the derived live feed times out.

Validation: 20 library tests passed; strict all-target anima-mq Clippy passed. Two explicitly executed real RabbitMQ tests concurrently publish after connection loss or channel-only close and consume all three original sequence IDs and payloads. Both tests fail against the original implementation (deadlock and missing publishes). A paused-time test covers contended mutex deadlines and guard release after cancellation. No new binaries or dependencies.

Anima WI-918 (01a117d1-1e3a-78b3-ae29-6a9d32a7d9bb), Plan 14. The operational restart is temporary recovery; exact-head review/CI, main release and permanent deployment remain required.

A RabbitMQ disconnect could permanently occupy every central relay worker: `publish_chunk` held the channel mutex while its reconnect helper tried to acquire the same mutex. Completed review turns then retained their Anima owners, preventing new reviews. The live incident showed successful reconnect followed by no further stream publications; a rolling restart released the owners and admitted pending reviews. Pass the owned channel into recovery, recover channel-only failures as well as connection failures, and bound the full transient publish/reap operation to two seconds, including mutex waits and reconnect RPCs. Durable journal/transcript storage remains authoritative when the derived live feed times out. Validation: 20 library tests passed; strict all-target anima-mq Clippy passed. Two explicitly executed real RabbitMQ tests concurrently publish after connection loss or channel-only close and consume all three original sequence IDs and payloads. Both tests fail against the original implementation (deadlock and missing publishes). A paused-time test covers contended mutex deadlines and guard release after cancellation. No new binaries or dependencies. Anima WI-918 (`01a117d1-1e3a-78b3-ae29-6a9d32a7d9bb`), Plan 14. The operational restart is temporary recovery; exact-head review/CI, main release and permanent deployment remain required.

toasterson added 1 commit

2026-10-07 19:31:27 +00:00