Gmail runtime OAuth fallback blocks ingestion indefinitely after connection failure #14

New issue

Open

opened 2026-10-03 13:58:13 +00:00 by toasterson · 0 comments

toasterson commented

2026-10-03 13:58:13 +00:00

Owner

Problem and production evidence

On 2026-10-01, Gmail polling repeatedly failed with history_list: client error (Connect). At 23:18 UTC the poller entered the installed-app interactive Google authorization flow and stopped making progress while the SIA pod remained Ready. At recovery on October 3 the cached gmail.modify refresh token still existed, DNS/TLS worked, and a process restart refreshed that same token successfully without user login or cursor reset. Ingestion resumed: 47 backlog messages, 37 successfully routed to Anima/Seshat and 10 intentionally left alone.

Incident: https://github.com/Toasterson/talos/blob/main/docs/incidents/2026-10-03-sia-ingestion-recovery.md
Deployed source: bfc8770ef5 (same tree as 3012b550).

Scope

Implement the durable code fix; operational recovery is already complete.

crates/sia-mail-gmail/src/lib.rs: build_authenticator, build_hub, bootstrap, and runtime token acquisition. Runtime currently uses InstalledFlowAuthenticator and assumes a cache file guarantees no interactive fallback. That assumption failed in production. Confirm the precise yup-oauth2 fallback path with a local deterministic reproduction.
crates/sia-bin/src/pipeline.rs: bound provider polling/authentication so a blocked provider cannot indefinitely stop the poll loop or other mailboxes.
Sync-health persistence and crates/sia-otlp: expose a reliable last-successful-poll heartbeat; failed attempts must not look like successful syncs.

Acceptance criteria

Background/runtime Gmail use never prints a login URL, opens a browser, binds the OAuth callback listener, or waits for user input. Interactive login remains available only through explicit bootstrap.
Transient token-refresh/network failures return bounded, actionable, redacted errors and retry with bounded backoff. Restored connectivity resumes from the existing cursor automatically, without restart or token replacement. Permanent revocation/missing credentials reports reauthorization required and remains responsive.
Preserve refresh-token storage, requested scope, mailbox identity and cursor. Do not log token values or reset production state.
Regression tests exercise the actual runtime authenticator/provider path with controlled OAuth/network responses: failed refresh then recovery; rejected/revoked refresh; missing/malformed cache; and a hanging auth request. Verify no interactive listener and successful next poll, plus isolation of another mailbox. No real Gmail account or credentials in tests.
Export/test last-successful-poll health that does not advance on errors; document the stale-ingestion alert expression and its startup/no-success behavior. Scope any live alert deployment as a separate coordinated step.
Run relevant tests, formatting and lint checks, submit the fix for normal review/CI, and report evidence. Respect existing native CI work (PR #13) rather than bypassing failed gates.

Keep the change focused: registry retention, node03 disk capacity and stor02 CPU capacity are separate operational follow-ups. The operator explicitly requested this issue be sent for implementation.

## Problem and production evidence On 2026-10-01, Gmail polling repeatedly failed with `history_list: client error (Connect)`. At 23:18 UTC the poller entered the installed-app interactive Google authorization flow and stopped making progress while the SIA pod remained Ready. At recovery on October 3 the cached gmail.modify refresh token still existed, DNS/TLS worked, and a process restart refreshed that same token successfully without user login or cursor reset. Ingestion resumed: 47 backlog messages, 37 successfully routed to Anima/Seshat and 10 intentionally left alone. Incident: https://github.com/Toasterson/talos/blob/main/docs/incidents/2026-10-03-sia-ingestion-recovery.md Deployed source: bfc8770ef5a41aff76dd4e23e42de37339601b12 (same tree as 3012b550). ## Scope Implement the durable code fix; operational recovery is already complete. - `crates/sia-mail-gmail/src/lib.rs`: `build_authenticator`, `build_hub`, `bootstrap`, and runtime token acquisition. Runtime currently uses InstalledFlowAuthenticator and assumes a cache file guarantees no interactive fallback. That assumption failed in production. Confirm the precise yup-oauth2 fallback path with a local deterministic reproduction. - `crates/sia-bin/src/pipeline.rs`: bound provider polling/authentication so a blocked provider cannot indefinitely stop the poll loop or other mailboxes. - Sync-health persistence and `crates/sia-otlp`: expose a reliable last-successful-poll heartbeat; failed attempts must not look like successful syncs. ## Acceptance criteria 1. Background/runtime Gmail use never prints a login URL, opens a browser, binds the OAuth callback listener, or waits for user input. Interactive login remains available only through explicit bootstrap. 2. Transient token-refresh/network failures return bounded, actionable, redacted errors and retry with bounded backoff. Restored connectivity resumes from the existing cursor automatically, without restart or token replacement. Permanent revocation/missing credentials reports reauthorization required and remains responsive. 3. Preserve refresh-token storage, requested scope, mailbox identity and cursor. Do not log token values or reset production state. 4. Regression tests exercise the actual runtime authenticator/provider path with controlled OAuth/network responses: failed refresh then recovery; rejected/revoked refresh; missing/malformed cache; and a hanging auth request. Verify no interactive listener and successful next poll, plus isolation of another mailbox. No real Gmail account or credentials in tests. 5. Export/test last-successful-poll health that does not advance on errors; document the stale-ingestion alert expression and its startup/no-success behavior. Scope any live alert deployment as a separate coordinated step. 6. Run relevant tests, formatting and lint checks, submit the fix for normal review/CI, and report evidence. Respect existing native CI work (PR #13) rather than bypassing failed gates. Keep the change focused: registry retention, node03 disk capacity and stor02 CPU capacity are separate operational follow-ups. The operator explicitly requested this issue be sent for implementation.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/sia#14

No description provided.

Rows
Columns