ci(nightly): bound every database-integration test binary #732

Merged

toasterson merged 1 commit from fix/nightly-pool-binary-timeout into main

2026-09-20 21:11:11 +00:00

toasterson commented

2026-09-20 19:21:52 +00:00

Owner

This runs each pool binary under timeout (default 1800 s, SOLSTICE_TEST_BIN_TIMEOUT_SECS to override), reports a timeout as a named failure, and makes the pool heartbeat list the binaries still running, so the hung test is visible from the streamed log rather than only from the per-binary log files.

🤖 Generated with Claude Code

The nightlies of 2026-09-15..09-20 each hung at "1 test binaries still running" for up to 137 hours. The Solstice executor's wall-clock abort does not kill the VM (CloudNebulaProject/solstice-ci#9), so five zombie VMs held every executor slot for days and all rust-labelled CI (akh-medu, Anima) queued behind them. Recovered today by killing the VMs by hand. This runs each pool binary under `timeout` (default 1800 s, `SOLSTICE_TEST_BIN_TIMEOUT_SECS` to override), reports a timeout as a named failure, and makes the pool heartbeat list the binaries still running, so the hung test is visible from the streamed log rather than only from the per-binary log files. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

toasterson force-pushed fix/nightly-pool-binary-timeout from 25aa89e0a5

CI / conflict-check (push) Successful in 10m29s

Details

CI / test (push) Failing after 35m0s

Details

to 37881ea432

CI / conflict-check (push) Successful in 27m5s

Details

CI / test (push) Successful in 26m12s

Details

2026-09-20 20:34:49 +00:00

Compare