sync_backfill returns a gateway timeout even when the import succeeds #11
Loading…
Reference in a new issue
No description provided.
Delete branch "claude/wi-019f907a-sync-backfill-returns-a-gateway-timeout"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Anima work item
019f907a-101a-71c2-a950-36fbd2e94d9e.Symptom
mcp__sia__sync_backfill(and a broadsync_search) reliably returnStreamable HTTP error: upstream request timeoutfrom the MCP gateway, even when the underlying work completes. Observed 2026-07-23 backfilling the twoillumos/ipdPR threads: both calls timed out at the HTTP layer, yet the messages were fetched, classified, and routed successfully (decisions #1375/#1376,MailingList; 2/2 ok). The caller can only confirm success by inspectingdecision_recentafterward.Cause
The backfill path does real work synchronously inside the request: fetch full bodies from the provider → run the tools-less sandbox classifier (LLM) → anima gRPC upserts. For more than a message or two this exceeds the MCP gateway's response window, so the HTTP response times out while the server keeps working. The gateway timeout is unrelated to whether the import succeeds.
Impact
Options
Prefer (1) for anything beyond a handful of messages; (3) is an acceptable stopgap.
Repro
mcp__sia__sync_backfillwith aquery:that resolves to >2 messages, orsync_searchover a broad query, against the personal Gmail mailbox. HTTP times out;decision_recentshows the work landed.b9c2a6536d67aeaf60efWIP: sync_backfill returns a gateway timeout even when the import succeedsto sync_backfill returns a gateway timeout even when the import succeeds`get_project_ci_status` reported `resolvable: false` for Sia — no `default_ci_steps` and no `.solstice/job.sh` — so every native dispatch failed at config resolution before running anything. That failure is not neutral. anima-server's solstice.rs routes it to `handle_config_failure`, which parks the work item with `ParkedReason::CiConfigUnresolvable`. The ABSENCE of this file reads as "this work item's CI is broken" and parks real work. akh-medu learned this the expensive way (8/8 native dispatches failed, the WI 1/2/3 series parked) and its job.sh states the rule: a missing file must not read as "the code is red". Sia had CI only on the Forgejo-Actions lane, so the native lane — the one Anima's exact-head gate reads — had never produced a verdict at all. One definition of the checks, two lanes --------------------------------------- scripts/ci-checks.sh is now the only place the cargo commands are written. Both .solstice/job.sh and .forgejo/workflows/ci.yml call it; neither re-lists them. akh-medu's rule is that a native verdict must never be red for work the Actions lane would call green, because the disagreement parks work items on phantom failures. Two inline copies drift the first time someone edits one; sharing the script makes that structurally impossible rather than a reviewer's responsibility. job.sh only provisions. It deliberately differs from the Anima/akh-medu scripts in two places: - It does not assume passwordless sudo. Those scripts call `sudo apt-get` unconditionally because their image grants it; a runner without sudo would fail with a bare "sudo: command not found" that classifies as a code failure and parks the item. Privilege is resolved once, only missing packages are installed, and the no-privilege case exits with a message naming the packages and the reason. - It does not `rm -rf ../anima`. The workflow did, and on a persistent or shared runner that deletes a checkout the job does not own. An existing sibling is reused; a non-git path there is refused rather than removed. The resolved revision is always logged so an upstream anima-proto change is attributable. ANIMA_SIBLING_REV pins a run without freezing CI to a revision the shipped image does not use. protoc keeps ci.yml's pinned 25.3 release, since Ubuntu's 3.12 rejects anima-proto's proto3 `optional` fields, and installs into the checkout so it needs no privileges. The build is debug: release codegen is what OOM-killed rustc on the Solstice VMs for akh-medu, and an OOM reads as red on healthy code. Sia's release build is still covered by the `docker` job on push to main. Validation ---------- scripts/test-ci-job.sh replaces every external command with a logging stub and asserts on the control flow — the branches that only execute on a runner in an unusual state, i.e. never until the day they matter. 10 scenarios, 36 assertions, all passing; it needs no toolchain and runs in seconds, so it is wired into ci-checks.sh and guards the CI definition on both lanes. It earned its keep immediately, catching three defects before any runner saw the script: `"${profile_args[@]}"` on an empty array is an unbound-variable error under `set -u` on bash < 4.4, so the script would have worked on the runner but not locally, defeating the point of sharing it; the openssl probe hardcoded /usr/include/openssl/ssl.h and reported a false "missing" on any other layout, triggering a pointless privileged install; and the harness's own no-sudo case was inheriting the host's cached sudo credentials, leaving the no-privilege branch untested exactly where it was most likely to be wrong. No application change: no Cargo.toml edit, no dependency version change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>e5ead22All four were real. Reproduced each before fixing. 1. The install died as root. `$SUDO DEBIAN_FRONTEND=noninteractive apt-get` looks like an assignment prefix but is not one: bash decides that from the text BEFORE expansion, and the first word is `$SUDO`. Once that expands to nothing — i.e. running as root, the case that should always work — the assignment becomes the command name: $ SUDO=""; $SUDO DEMO_ONLY=1 true bash: DEMO_ONLY=1: command not found (rc=127) Routed through `env`, which is a real command in both shapes. The harness now exercises root explicitly via an `id` stub rather than only covering sudo/non-root, so the escalation-free path has real coverage. 2. A git worktree's `.git` is a FILE holding a gitdir pointer, so `[ -d .git ]` rejected a perfectly valid shared worktree — and the next branch then refused to run at all. This checkout is itself a worktree, so the script would have rejected the very tree it was written in. Now asks git (`rev-parse --is-inside-work-tree`) and confirms the path is that worktree's ROOT by comparing realpaths, so a subdirectory of an unrelated repo above it is still refused — non-destructively. 3. ANIMA_SIBLING_REV claimed a pin it never enforced: it was only applied on the clone path, so with an existing checkout the requested revision was silently ignored and the verdict was attributed to a revision that was never built. Now validated against the existing checkout and REFUSED on mismatch, without touching a checkout this job does not own. An unknown rev is likewise refused rather than fetched into it. ANIMA_SIBLING_DIR is gone. It was misleading: cargo reads the override path from the workspace [patch] table, so pointing the variable elsewhere provisioned a directory cargo would never look at while the build still failed on the real path. The sibling location is now derived from Cargo.toml, which is the only thing that actually decides it, and a missing [patch] path fails loudly instead of guessing. 4. `clippy --workspace --all-targets -- -D warnings` was stricter than the ci.yml it replaced (libs/bins only). Pulling tests, benches and examples into a -D warnings gate for the first time risks an unrelated baseline failure landing as a red verdict on unrelated work — the phantom-failure park this work item exists to stop. Restored to the previously-verified scope. Widening is a separate change that needs a clean baseline run and then moves both lanes together. Also fixes unbounded recursion introduced when the harness was wired into ci-checks.sh: the harness runs job.sh, which execs ci-checks.sh, which ran the harness. Guarded with CI_SKIP_JOB_HARNESS, which the harness sets for its children, and the guard itself is now covered. Harness: 18 scenarios, 62 assertions, all passing. New fixtures are real, not mocked around — an actual shared worktree created with `git worktree add`, an actual non-root checkout, actual commit SHAs for the pin cases, and a real Cargo.toml with a [patch] table so the derivation runs the same code path the runner takes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>The branch already had the right shape — sync.backfill/sync.search enqueue a background job and return {job_id, status:"pending"} immediately, so the MCP gateway's response window no longer decides whether a successful import looks like a failure. Completing it here rather than rebuilding it. Two gaps remained, both named by the work item. Idempotency. The WI is explicit that dedup-by-external_ref downstream is "luck, not design": re-running a backfill that *looked* like it failed enqueued a second full pass — more provider fetches, more LLM classification — and only the final anima upsert collapsed. Live jobs are now keyed by a fingerprint of the request, and an identical in-flight call joins the existing job instead of starting another, returning the same job_id with deduplicated:true. The boundaries are deliberate: - only Pending/Running jobs are joined; an identical request after a job has finished is a re-run, and silently returning the old result would hide new messages from the caller - a failed job is likewise not joined, or a retry would never retry - `execute` is part of the backfill fingerprint, so a real run cannot be absorbed by an in-flight dry run and silently skipped - explicit id lists are sorted, so the same set in a different order is one job - a None fingerprint always produces a distinct job Completion errors. Two paths could leave a job that no caller could ever resolve: - the registry row is inserted before the work is handed to the worker, so a closed channel returned an error to the tool while leaving a job Pending forever — visible in sync.jobs as about to start, and never starting. fail_if_live now closes it out, and refuses to clobber a terminal state so a late error path cannot rewrite a fast worker's success as a failure. - a worker that panics or returns early leaves the job Running forever, which a poller cannot distinguish from slow progress. JobGuard records "terminated without recording an outcome" on any path that did not report its own. Also bounds the registry. It is in-process and lives for the life of the server, so it accumulated every job ever run; finished jobs are now evicted past a cap while live jobs are never evicted — dropping one would strand a caller polling sync.job with "no such job" while the work was still running. Tests: 11 new, 49 passing in the crate. They cover joining a live duplicate, NOT joining a finished or failed one, key/kind isolation, keyless opt-out, closing out a failed handoff, refusing to clobber a terminal state, the dropped-guard and disarmed-guard paths, and that eviction is bounded without dropping an in-flight job. No production mail actions were taken; all verification is in-process. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.