OpenIndiana/ips

Watch

Fork 0

Decide the publish transaction model: one FMRI per transaction vs many packages per transaction #26

New issue

Closed

opened 2026-09-19 19:58:47 +00:00 by toasterson · 1 comment

toasterson commented

2026-09-19 19:58:47 +00:00

Owner

Part of #24

Read availability and catalog replacement — 2026-09-22

Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (fs.rename), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage.

Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename.

Maintainer decisions — 2026-09-22

V1 transaction boundary: one completed multi-package .p5p from one CI run, imported locally by a notified script. All packages become visible atomically or none do. Identical archive retries succeed as no-ops even after commit-before-ack crashes; exact-FMRI content conflicts reject the whole batch; identical existing packages can be skipped. Implement via #38.

REST publication and its multi-upload protocol are deferred with #3. This ticket remains open for local batch commit/recovery/locking design and ADR capture; do not block v1 on choosing deferred HTTP upload shapes.

These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion.

Earlier brief and context

Question

The plan in docs/ai/plans/2026-04-09-publish-api-oidc.md opens a transaction per FMRI (POST /{publisher}/open/0/{fmri}), while the Server Publish ticket (#3) wants one transaction to carry multiple archives or packages and to be resumable after a client crash. These conflict. Decide the model.

Grill on:

Transaction granularity: per package, per batch, or both (batch = set of per-package sub-transactions)? What does atomic repository update mean for a batch (all-or-nothing catalog commit)?
Resumability: status/0/{txn} returning still-needed hashes is in the plan; is the needed-file list computed from manifests only, and how are shared payloads across packages deduplicated?
Identity of a transaction: server-generated ID vs client-chosen idempotency key (CI retries).
Where transactions live: {repo}/trans/{id}/ on disk (plan) vs SQLite state; recovery on server restart; abandonment policy (no timeout decided, but is there a manual sweep?).
Commit semantics: move payloads into the two-level hash store, write manifest, update catalog incrementally (update log) or trigger full rebuild; what happens to a commit that fails midway.
Which HTTP shapes ship: multipart vs raw body with X-IPkg-File-Hash; archive upload endpoint as a batch transaction or a separate path.

Zoom into libips/src/repository/file_backend.rs (Transaction, begin_transaction, publish_files) and doc/pkg5_docs/guide-publication-protocol.rst for prior art. Record as an ADR and update the plan; the resolution should let /to-tickets slice #3.

Links

Informs #3

Part of #24 ## Read availability and catalog replacement — 2026-09-22 Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (`fs.rename`), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage. Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename. ## Maintainer decisions — 2026-09-22 V1 transaction boundary: one completed multi-package .p5p from one CI run, imported locally by a notified script. All packages become visible atomically or none do. Identical archive retries succeed as no-ops even after commit-before-ack crashes; exact-FMRI content conflicts reject the whole batch; identical existing packages can be skipped. Implement via #38. REST publication and its multi-upload protocol are deferred with #3. This ticket remains open for local batch commit/recovery/locking design and ADR capture; do not block v1 on choosing deferred HTTP upload shapes. These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion. <details> <summary>Earlier brief and context</summary> ## Question The plan in `docs/ai/plans/2026-04-09-publish-api-oidc.md` opens a transaction per FMRI (`POST /{publisher}/open/0/{fmri}`), while the Server Publish ticket (#3) wants one transaction to carry multiple archives or packages and to be resumable after a client crash. These conflict. Decide the model. Grill on: - Transaction granularity: per package, per batch, or both (batch = set of per-package sub-transactions)? What does atomic repository update mean for a batch (all-or-nothing catalog commit)? - Resumability: `status/0/{txn}` returning still-needed hashes is in the plan; is the needed-file list computed from manifests only, and how are shared payloads across packages deduplicated? - Identity of a transaction: server-generated ID vs client-chosen idempotency key (CI retries). - Where transactions live: `{repo}/trans/{id}/` on disk (plan) vs SQLite state; recovery on server restart; abandonment policy (no timeout decided, but is there a manual sweep?). - Commit semantics: move payloads into the two-level hash store, write manifest, update catalog incrementally (update log) or trigger full rebuild; what happens to a commit that fails midway. - Which HTTP shapes ship: multipart vs raw body with `X-IPkg-File-Hash`; archive upload endpoint as a batch transaction or a separate path. Zoom into `libips/src/repository/file_backend.rs` (`Transaction`, `begin_transaction`, `publish_files`) and `doc/pkg5_docs/guide-publication-protocol.rst` for prior art. Record as an ADR and update the plan; the resolution should let `/to-tickets` slice #3. ## Links - Informs #3 </details>

toasterson added this to the Release 1 (server only) milestone

2026-09-19 19:58:47 +00:00

toasterson added the

wayfinder:grilling

label

2026-09-19 19:58:47 +00:00

toasterson added a new dependency

2026-09-19 19:59:11 +00:00

#25 Define what 'v1.0 server release' must include and its acceptance criteria

toasterson added a new dependency

2026-09-19 19:59:22 +00:00

#32 Research: how pkg5 models publication transactions, the trans/ directory and catalog updates after publish

toasterson added a new dependency

2026-09-19 19:59:31 +00:00

#3 Server Publish

toasterson referenced this issue

2026-09-19 19:59:46 +00:00

Wayfinder map: Release 1 (server only) #24

forge referenced this issue

2026-09-22 12:11:13 +00:00

V1: atomically ingest one multi-package p5p per CI run #38

forge added a new dependency

2026-09-22 12:13:12 +00:00

#38 V1: atomically ingest one multi-package p5p per CI run

toasterson commented

2026-09-27 13:13:39 +00:00

Author

Owner

Closed as out of scope for Release 1 (wayfinder). Remote publication is deferred with no target release (maintainer, 2026-09-22), so there is no publish transaction to model. The local import model this question also covered is settled by ADR-007: many packages per import, validated whole, published by one generation switch, identical re-import a no-op, exact-FMRI conflict rejects the batch. Reopen or re-chart when remote publish is scheduled; research #32 still applies.

Closed as **out of scope for Release 1** (wayfinder). Remote publication is deferred with no target release (maintainer, 2026-09-22), so there is no publish transaction to model. The local import model this question also covered is settled by ADR-007: many packages per import, validated whole, published by one generation switch, identical re-import a no-op, exact-FMRI conflict rejects the batch. Reopen or re-chart when remote publish is scheduled; research #32 still applies.

toasterson closed this issue

2026-09-27 13:13:42 +00:00

toasterson referenced this issue

2026-09-27 13:14:15 +00:00

Wayfinder map: Release 1 (server only) #24