Supervise RedDwarf components and derive readiness from live checks #3

Merged

toasterson merged 2 commits from codex/component-supervision-20261001 into main

2026-10-01 20:50:38 +00:00

toasterson commented

2026-10-01 20:30:07 +00:00

Owner

RedDwarf WI-2: 01a0f6e0-dc28-7072-8fc9-972d7beb2525

Stacked on #2.

Essential component errors, panics and unexpected exits now stop peers and fail the process. Readiness tracks initialized, successful and unexpired storage/controller/scheduler/heartbeat/DNS/node-health checks. Workers expose a separate private health listener. Agent startup validates configuration/TLS and reserves API/DNS listeners before ZFS initialization. Shutdown drains controllers before the API in bounded stages, preserves durable resource ownership, and cancels/kills direct command children. Failed or unknown cleanup retains the API tombstone and persistent IPAM owner instead of finalizing prematurely.

Validation at de378a3264: 200 native workspace tests, binary build, fmt, strict all-targets Clippy, SMF validation/method syntax, API/TLS smoke, two-worker topology and lifecycle smoke passed on oibldofgate. Tests inject controller and drain errors, hung peers, expired/concurrent health updates, command timeout/cancellation and cleanup failure with database reopen; native process checks include occupied API/DNS listeners, malformed TLS/CA/configuration, readiness transitions, restart/outages and rejected registration. Local tests, lint, build and lifecycle smoke passed as well.

Tests use explicit mock zones/storage/NAT on illumos and only task-owned loopback processes. This adds no binary target and activates no SMF service or real zone/network backend. Real image/zone/network/authentication acceptance remains in subsequent milestones. Anima CI is currently unconfigured; native SSH checks do not clear that gate.

RedDwarf WI-2: 01a0f6e0-dc28-7072-8fc9-972d7beb2525 Stacked on https://code.aopc.cloud/CloudNebulaProject/reddwarf/pulls/2. Essential component errors, panics and unexpected exits now stop peers and fail the process. Readiness tracks initialized, successful and unexpired storage/controller/scheduler/heartbeat/DNS/node-health checks. Workers expose a separate private health listener. Agent startup validates configuration/TLS and reserves API/DNS listeners before ZFS initialization. Shutdown drains controllers before the API in bounded stages, preserves durable resource ownership, and cancels/kills direct command children. Failed or unknown cleanup retains the API tombstone and persistent IPAM owner instead of finalizing prematurely. Validation at de378a32641b195d79c920118c49d7b05d2aa829: 200 native workspace tests, binary build, fmt, strict all-targets Clippy, SMF validation/method syntax, API/TLS smoke, two-worker topology and lifecycle smoke passed on oibldofgate. Tests inject controller and drain errors, hung peers, expired/concurrent health updates, command timeout/cancellation and cleanup failure with database reopen; native process checks include occupied API/DNS listeners, malformed TLS/CA/configuration, readiness transitions, restart/outages and rejected registration. Local tests, lint, build and lifecycle smoke passed as well. Tests use explicit mock zones/storage/NAT on illumos and only task-owned loopback processes. This adds no binary target and activates no SMF service or real zone/network backend. Real image/zone/network/authentication acceptance remains in subsequent milestones. Anima CI is currently unconfigured; native SSH checks do not clear that gate.

toasterson added 2 commits

2026-10-01 20:30:07 +00:00