Executor: wall-clock timeout abort never kills the VM; hung jobs hold slots until the qemu process dies #9

New issue

Open

opened 2026-09-20 19:20:49 +00:00 by toasterson · 0 comments

toasterson commented

2026-09-20 19:20:49 +00:00

Owner

Observed on Archibald 2026-09-20 (claude-code-toasty, Djehuty board thread "Homelab ops — Solstice rust lane wedged by phantom running jobs — 2026-09-20").

Five Anima nightly rust runs (977c8b85 started 09-15, 0fc88c1c 09-16, 0068bee7 09-17, 36387da5 09-18, b44f4e8c 09-20) each hung on one test binary (pool heartbeat "1 test binaries still running" forever). For every one the orchestrator logged, at exactly 10800 s:

WARN job exceeded wall-clock timeout; aborting request_id=... secs=10800

but the qemu process kept running (up to 137 h) and the job stayed running in jobs. With MAX_CONCURRENCY=5 the five zombies consumed every slot, so from 2026-09-18 nothing ran: solstice.jobs.v1 backed up (62 ready), Forgejo runs sat "waiting", and all eight proxies held in-flight tasks.

Recovery was kill -TERM of the five qemu pids. The moment each VM exited the orchestrator logged WARN ssh failed error=wall-clock timeout request_id=..., recorded the job as failed (ssh error: wall-clock timeout, exit 1) and started five queued jobs within seconds. So the abort path evidently awaits the SSH session and only unwinds once the guest goes away; it never issues the ACPI shutdown / SIGTERM / SIGKILL sequence that the normal teardown uses ("ACPI shutdown timed out, sending SIGTERM" appears for other jobs, never for these five).

Expected: on wall-clock timeout, tear the VM down with the same escalation as normal teardown (ACPI → SIGTERM → SIGKILL), independent of whether the SSH command future has returned, then mark the job failed and release the slot.

Related, not this repo's bug: the Anima nightly full suite itself hangs on one test binary every night; the same 2af564a9 commit hung on 09-20 and 8a4ee8aa on 09-15..09-18. Anima's .solstice/job.sh pool should probably enforce a per-binary timeout so a single hung test cannot eat the run.

Observed on Archibald 2026-09-20 (claude-code-toasty, Djehuty board thread "Homelab ops — Solstice rust lane wedged by phantom running jobs — 2026-09-20"). Five Anima nightly `rust` runs (977c8b85 started 09-15, 0fc88c1c 09-16, 0068bee7 09-17, 36387da5 09-18, b44f4e8c 09-20) each hung on one test binary (pool heartbeat "1 test binaries still running" forever). For every one the orchestrator logged, at exactly 10800 s: ``` WARN job exceeded wall-clock timeout; aborting request_id=... secs=10800 ``` but the qemu process kept running (up to 137 h) and the job stayed `running` in `jobs`. With MAX_CONCURRENCY=5 the five zombies consumed every slot, so from 2026-09-18 nothing ran: `solstice.jobs.v1` backed up (62 ready), Forgejo runs sat "waiting", and all eight proxies held in-flight tasks. Recovery was `kill -TERM` of the five qemu pids. The moment each VM exited the orchestrator logged `WARN ssh failed error=wall-clock timeout request_id=...`, recorded the job as failed (`ssh error: wall-clock timeout`, exit 1) and started five queued jobs within seconds. So the abort path evidently awaits the SSH session and only unwinds once the guest goes away; it never issues the ACPI shutdown / SIGTERM / SIGKILL sequence that the normal teardown uses ("ACPI shutdown timed out, sending SIGTERM" appears for other jobs, never for these five). Expected: on wall-clock timeout, tear the VM down with the same escalation as normal teardown (ACPI → SIGTERM → SIGKILL), independent of whether the SSH command future has returned, then mark the job failed and release the slot. Related, not this repo's bug: the Anima nightly full suite itself hangs on one test binary every night; the same 2af564a9 commit hung on 09-20 and 8a4ee8aa on 09-15..09-18. Anima's `.solstice/job.sh` pool should probably enforce a per-binary timeout so a single hung test cannot eat the run.

toasterson referenced this issue from a commit

2026-09-20 19:21:47 +00:00

ci(nightly): bound every database-integration test binary

toasterson referenced this issue from toasterson/Anima

2026-09-20 19:21:52 +00:00

ci(nightly): bound every database-integration test binary #732

toasterson referenced this issue from a commit

2026-09-20 20:34:49 +00:00

ci(nightly): bound every database-integration test binary

toasterson referenced this issue

2026-09-20 20:34:59 +00:00

Forgejo proxy watchdog fails a task that is still queued in Solstice (STALE_JOB_TIMEOUT counts queue time) #10