Forgejo proxy watchdog fails a task that is still queued in Solstice (STALE_JOB_TIMEOUT counts queue time) #10

New issue

Open

opened 2026-09-20 20:34:59 +00:00 by toasterson · 0 comments

toasterson commented

2026-09-20 20:34:59 +00:00

Owner

Observed 2026-09-20 right after the executor lane was unwedged (see #9): solstice.jobs.v1 had ~60 ready messages, so a newly pushed Anima commit (25aa89e0, run 8787) waited in the Solstice queue for more than 30 minutes. Its Forgejo test job was then marked failed with

solstice-ci: task timed out after 2099s with no result from orchestrator; pruned by watchdog

while the Solstice job for the same commit was still queued and later ran fine. The proxy's stale-task watchdog starts counting when the JobRequest is published, so any queue backlog longer than STALE_JOB_TIMEOUT_SECS (1800) turns green work into red Forgejo runs, and the only remedy is a new push.

Expected: the watchdog should count from when the orchestrator actually starts the job (or reset when the job is observed queued/running in Solstice), and ideally the proxy should keep the Forgejo task alive with an UpdateTask/log line ("queued behind N jobs") while waiting.

Observed 2026-09-20 right after the executor lane was unwedged (see #9): `solstice.jobs.v1` had ~60 ready messages, so a newly pushed Anima commit (25aa89e0, run 8787) waited in the Solstice queue for more than 30 minutes. Its Forgejo `test` job was then marked failed with ``` solstice-ci: task timed out after 2099s with no result from orchestrator; pruned by watchdog ``` while the Solstice job for the same commit was still `queued` and later ran fine. The proxy's stale-task watchdog starts counting when the JobRequest is published, so any queue backlog longer than STALE_JOB_TIMEOUT_SECS (1800) turns green work into red Forgejo runs, and the only remedy is a new push. Expected: the watchdog should count from when the orchestrator actually starts the job (or reset when the job is observed `queued`/`running` in Solstice), and ideally the proxy should keep the Forgejo task alive with an UpdateTask/log line ("queued behind N jobs") while waiting.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

CloudNebulaProject/solstice-ci#10

No description provided.

Rows
Columns