CloudNebulaProject/solstice-ci

SHA256

Watch

Fork 0

runner-integration: the same Forgejo job is executed by several runners concurrently (duplicate dispatch) #8

New issue

Open

opened 2026-09-19 21:29:33 +00:00 by toasterson · 1 comment

toasterson commented

2026-09-19 21:29:33 +00:00

Owner

Symptom

Forgejo Actions jobs for OpenIndiana/ips (runs 3 and 4 of that repo, 2026-09-19 evening UTC) are executed by several runner instances at once. The job log for run 4 / job Linux shows the runner bootstrap and every step start three times with different timestamps (1789852187, 1789852191, 1789852195/98, 1789852202), e.g.

1789852187 starting step: Toolchain
1789852191 starting step: Toolchain
1789852195 starting step: Format check
1789852198 starting step: Format check
1789852202 starting step: Format check

and the illumos job of run 3 prints system: SunOS job-4e6afc84-... and runner starting repo=... sha=c0f44e1 more than once. In the Forgejo UI this looks like "Set up job" running for 15-35 minutes with a duration counter that keeps resetting, later steps flipping between running and waiting, and finally the job being marked failure with no step failing (run 4 / Linux: steps Toolchain, Install cargo-nextest and Format check still running, Complete job failure, empty log). The illumos job of run 3 did reach a real (still unexplained, silent) ld returned 1 link failure, so VMs do come up, but each job is being provisioned N times.

All eight solstice-forgejo-proxy-N runners advertise the same label set, and the 2026-09-14 commits "runner-integration: force a Forgejo task pick every 20 s" and "scan restored in-flight tasks at startup" landed just before this. Hypothesis: multiple proxies claim the same task (or a restarted proxy re-claims an in-flight task) and all execute it, with their runner-side step reports racing on the same job.

Where to look

crates/runner-integration: task claim/lease handling across the eight proxy processes; whether the in-flight-task scan at startup re-dispatches tasks another proxy still owns.
Orchestrator: is there any dedupe keyed on Forgejo task id before creating a VM?

Repro

Push to OpenIndiana/ips main (workflow .forgejo/workflows/ci.yml, jobs on rust and illumos), then open the job log and count runner starting / starting step: Toolchain lines.

Reported from the Rust IPS CI migration (OpenIndiana/ips#8).

## Symptom Forgejo Actions jobs for `OpenIndiana/ips` (runs 3 and 4 of that repo, 2026-09-19 evening UTC) are executed by **several runner instances at once**. The job log for run 4 / job `Linux` shows the runner bootstrap and every step start **three times** with different timestamps (1789852187, 1789852191, 1789852195/98, 1789852202), e.g. ``` 1789852187 starting step: Toolchain 1789852191 starting step: Toolchain 1789852195 starting step: Format check 1789852198 starting step: Format check 1789852202 starting step: Format check ``` and the illumos job of run 3 prints `system: SunOS job-4e6afc84-... ` and `runner starting repo=... sha=c0f44e1` more than once. In the Forgejo UI this looks like "Set up job" running for 15-35 minutes with a duration counter that keeps resetting, later steps flipping between `running` and `waiting`, and finally the job being marked **failure with no step failing** (run 4 / Linux: steps Toolchain, Install cargo-nextest and Format check still `running`, Complete job `failure`, empty log). The illumos job of run 3 did reach a real (still unexplained, silent) `ld returned 1` link failure, so VMs do come up, but each job is being provisioned N times. All eight `solstice-forgejo-proxy-N` runners advertise the same label set, and the 2026-09-14 commits "runner-integration: force a Forgejo task pick every 20 s" and "scan restored in-flight tasks at startup" landed just before this. Hypothesis: multiple proxies claim the same task (or a restarted proxy re-claims an in-flight task) and all execute it, with their runner-side step reports racing on the same job. ## Where to look - `crates/runner-integration`: task claim/lease handling across the eight proxy processes; whether the in-flight-task scan at startup re-dispatches tasks another proxy still owns. - Orchestrator: is there any dedupe keyed on Forgejo task id before creating a VM? ## Repro Push to `OpenIndiana/ips` `main` (workflow `.forgejo/workflows/ci.yml`, jobs on `rust` and `illumos`), then open the job log and count `runner starting` / `starting step: Toolchain` lines. Reported from the Rust IPS CI migration (OpenIndiana/ips#8).

toasterson referenced this issue from OpenIndiana/ips

2026-09-19 21:29:38 +00:00

Migrate CI to Solstice CI #8

toasterson commented

2026-09-19 21:30:09 +00:00

Author

Owner

Additional data point from OpenIndiana/ips run 4, job illumos: the only log line is

solstice-ci: task timed out after 2098s with no result from orchestrator; pruned by watchdog

so while the Linux job of the same run was being executed three times over, the illumos task never got a VM at all within the 35 min watchdog window. Both symptoms are consistent with the proxies fighting over task ownership / orchestrator capacity.

Additional data point from OpenIndiana/ips run 4, job `illumos`: the only log line is ``` solstice-ci: task timed out after 2098s with no result from orchestrator; pruned by watchdog ``` so while the Linux job of the same run was being executed three times over, the illumos task never got a VM at all within the 35 min watchdog window. Both symptoms are consistent with the proxies fighting over task ownership / orchestrator capacity.

toasterson referenced this issue from OpenIndiana/ips

2026-09-27 13:14:03 +00:00

V1: IPS/SMF deployment with direct TLS, proxy mode and ACME #41

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

CloudNebulaProject/solstice-ci#8

No description provided.

Rows
Columns