OpenIndiana/ips

Watch

Fork 0

V1: atomically ingest one multi-package p5p per CI run #38

New issue

Open

opened 2026-09-22 12:11:13 +00:00 by forge · 2 comments

forge commented

2026-09-22 12:11:13 +00:00

Part of #24

Read availability and catalog replacement — 2026-09-22

Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (fs.rename), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage.

Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename.

Contract

CI finishes delivering one .p5p containing all packages for a run, then notifies the import script. No multi-archive batch-marker protocol is required by this handoff.
Stage and validate the entire archive before publication. All packages become visible in one atomic repository/catalog transition, or none do; readers must not observe a partly imported batch.
Reimporting the identical committed archive succeeds as a no-op, including retry after a crash between commit and acknowledgement. Use a durable identity/result design; checksum identity was proposed and needs implementation design.
Identical existing packages may be skipped; the remaining new packages become visible together. Different manifest or payload content for an existing exact FMRI rejects the entire batch, naming conflicts. Never silently replace an FMRI.
Establish restart recovery and serialization with concurrent imports/retirement before implementation. Test failures before/during/after commit and real pkg5 reads across commit.
This local ingest path is the v1 deliverable; the REST publish API in #3 is deferred. Coordinate transaction semantics with #26.

Source: maintainer grilling, 2026-09-22. Requirements only; no implementation or validation claimed.

Part of #24 ## Read availability and catalog replacement — 2026-09-22 Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (`fs.rename`), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage. Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename. ## Contract - CI finishes delivering one .p5p containing all packages for a run, then notifies the import script. No multi-archive batch-marker protocol is required by this handoff. - Stage and validate the entire archive before publication. All packages become visible in one atomic repository/catalog transition, or none do; readers must not observe a partly imported batch. - Reimporting the identical committed archive succeeds as a no-op, including retry after a crash between commit and acknowledgement. Use a durable identity/result design; checksum identity was proposed and needs implementation design. - Identical existing packages may be skipped; the remaining new packages become visible together. Different manifest or payload content for an existing exact FMRI rejects the entire batch, naming conflicts. Never silently replace an FMRI. - Establish restart recovery and serialization with concurrent imports/retirement before implementation. Test failures before/during/after commit and real pkg5 reads across commit. - This local ingest path is the v1 deliverable; the REST publish API in #3 is deferred. Coordinate transaction semantics with #26. Source: maintainer grilling, 2026-09-22. Requirements only; no implementation or validation claimed.

forge added this to the Release 1 (server only) milestone

2026-09-22 12:11:13 +00:00

forge added the

Kind/Feature

wayfinder:task

labels

2026-09-22 12:11:13 +00:00

forge referenced this issue

2026-09-22 12:11:35 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge referenced this issue

2026-09-22 12:11:41 +00:00

Decide the publish transaction model: one FMRI per transaction vs many packages per transaction #26

forge referenced this issue

2026-09-22 12:12:31 +00:00

Server Publish #3

forge referenced this issue

2026-09-22 12:12:36 +00:00

OpenID Connect support to authenticate users #18

forge referenced this issue

2026-09-22 12:12:43 +00:00

RBAC Model to authorize people to admin endpojnts #19

forge referenced this issue

2026-09-22 12:12:48 +00:00

Decide the RBAC model: token-claim roles vs stored per-publisher roles, and admin bootstrap #27

forge referenced this issue

2026-09-22 12:12:54 +00:00

Choose the OIDC provider and token flows for CI publishers and humans #28

forge referenced this issue

2026-09-22 12:12:59 +00:00

Task: provision an OIDC test realm/client for pkg6depotd integration tests and CI #36

forge referenced this issue

2026-09-22 12:13:05 +00:00

Wayfinder map: Release 1 (server only) #24

forge added a new dependency

2026-09-22 12:13:12 +00:00

#26 Decide the publish transaction model: one FMRI per transaction vs many packages per transaction

forge referenced this issue

2026-09-23 07:16:40 +00:00

WIP: Build tagged release candidates and an illumos IPS depot package #43

forge referenced this issue

2026-09-23 08:01:15 +00:00

WIP: Import pkg5 archives through atomic repository generations #45

forge commented

2026-09-23 08:01:41 +00:00

Author

Draft PR #45 at ffbb578 implements the atomic multi-publisher pkg5 p5p import through one repository generation pointer rename. It validates the whole batch, makes exact archive retries a no-op, rejects conflicting exact FMRIs, recovers interrupted switches, and keeps stable pkg5 catalog paths. Thirty importer tests pass. A real pkg5-produced p5p imports and lists correctly on the Mac; exact retry succeeds unchanged. Native CI, live pkg5 client, and full-scale load validation remain open.

Draft PR #45 at ffbb578 implements the atomic multi-publisher pkg5 p5p import through one repository generation pointer rename. It validates the whole batch, makes exact archive retries a no-op, rejects conflicting exact FMRIs, recovers interrupted switches, and keeps stable pkg5 catalog paths. Thirty importer tests pass. A real pkg5-produced p5p imports and lists correctly on the Mac; exact retry succeeds unchanged. Native CI, live pkg5 client, and full-scale load validation remain open.

forge commented

2026-09-23 09:55:23 +00:00

Author

The native pkg5 client acceptance test is draft PR #51. It imports a release .p5p and a dependency-free dummy into a scratch repository, then exercises OpenIndiana pkg(1) catalog/manifest/install/verify/uninstall and pkgrecv manifest/payload retrieval against the running depot on the throwaway release VM. Local shell syntax and 272 offline assertions pass; independent Opus review checked command syntax and p5p layout against local OpenIndiana pkg5 source. This is not yet runtime proof: the tagged illumos release CI job must execute it before closing the pkg5 compatibility gate.

The native pkg5 client acceptance test is draft PR #51. It imports a release `.p5p` and a dependency-free dummy into a scratch repository, then exercises OpenIndiana `pkg(1)` catalog/manifest/install/verify/uninstall and `pkgrecv` manifest/payload retrieval against the running depot on the throwaway release VM. Local shell syntax and 272 offline assertions pass; independent Opus review checked command syntax and p5p layout against local OpenIndiana pkg5 source. This is **not yet runtime proof**: the tagged illumos release CI job must execute it before closing the pkg5 compatibility gate.