Research: how pkg5 models publication transactions, the trans/ directory and catalog updates after publish #32

New issue

Closed

opened 2026-09-19 19:59:00 +00:00 by toasterson · 1 comment

toasterson commented

2026-09-19 19:59:00 +00:00

Owner

Part of #24

Question

Summarise, from doc/pkg5_docs/ (guide-publication-protocol.rst, guide-repository-format.rst, catalog-v1.txt, dev-guide/) and the pkg5 sources in the workspace (repos/OpenIndiana/pkg5 under src/modules/server/transaction.py, src/modules/server/repository.py), how pkg5 handles: transaction lifecycle and on-disk state under trans/, multi-package publication (does pkgsend publish batch?), what close does to the catalog (incremental update log vs rebuild), refresh-index behaviour, and how crashes mid-transaction are recovered. Note which of these behaviours legacy clients observe via the depot API and which are internal.

Capture the findings as docs/research/pkg5-publication-model.md and link it from this ticket. This feeds the publish transaction model decision.

Links

Informs #3

Part of #24 ## Question Summarise, from `doc/pkg5_docs/` (`guide-publication-protocol.rst`, `guide-repository-format.rst`, `catalog-v1.txt`, `dev-guide/`) and the pkg5 sources in the workspace (`repos/OpenIndiana/pkg5` under `src/modules/server/transaction.py`, `src/modules/server/repository.py`), how pkg5 handles: transaction lifecycle and on-disk state under `trans/`, multi-package publication (does `pkgsend publish` batch?), what `close` does to the catalog (incremental update log vs rebuild), refresh-index behaviour, and how crashes mid-transaction are recovered. Note which of these behaviours legacy clients observe via the depot API and which are internal. Capture the findings as `docs/research/pkg5-publication-model.md` and link it from this ticket. This feeds the publish transaction model decision. ## Links - Informs #3

toasterson added this to the Release 1 (server only) milestone

2026-09-19 19:59:00 +00:00

toasterson added the

wayfinder:research

label

2026-09-19 19:59:00 +00:00

toasterson added a new dependency

2026-09-19 19:59:22 +00:00

#26 Decide the publish transaction model: one FMRI per transaction vs many packages per transaction

toasterson referenced this issue

2026-09-19 19:59:46 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson commented

2026-09-19 20:09:43 +00:00

Author

Owner

Resolution

pkg5 models publication as one transaction per package version: open binds an FMRI, creates trans/<unix-open-time>_<urlquoted fmri>/ with a one-line manifest; add appends action lines and stages gzipped payloads under trans/<id>/<hash> (only for content not already in file/); close renames manifest to pkg/, renames payloads into file/, then (unless X-IPkg-Add-To-Catalog: 0) calls add_package which rewrites the catalog once under the tmp/lock file lock and finally rmtree's the dir; abandon is just rmtree. The documented SUBMITTED/INCOMPLETE states were never implemented. Transactions are not journalled or scanned: a crashed one lingers forever in trans/ (invisible to status/0, no pkgrepo cleanup) but can be lazily reopened by id even after a depot restart. Close is not two-phase: manifest moves before payloads and before the catalog write, so crashes can leave uncatalogued or file-less packages (repair = refresh-packages/rebuild/pkgrepo fix).

Catalog: each close does Catalog.add_package with a single op-time, sets catalog.attrs.last-modified, appends to an hourly update.<YYYYMMDDTHH>Z.C log, and saves via copy-to-tmp + rename swap (leaves old.catalog on crash; never prunes logs). pkgsend publish = one transaction per invocation; batching is only --no-catalog + one admin/0?cmd=refresh-packages (incremental __rebuild, which does emit update logs); rebuild destroys the catalog, changes created, and forces every client to a full refetch. Indexing is never on the publish path (refresh-index is an async background task whose failures are invisible).

Legacy pkg(1) only sees the retrieval side: catalog/1/<file> semantics (created stable, monotonic last-modified, per-part last-modified+signature-sha-1, hourly update-log names listed in attrs.updates, op-time filtering, Last-Modified/If-Modified-Since, Cache-Control) plus manifest/file/publisher/p5i/search/status. Everything else (trans layout, open/add/close/append/abandon, manifest/1 & file/1 POST, X-IPkg-* headers, admin/0, index/0, background queue, lock/tmp files) is internal and free to redesign.

Implications for pkg6depotd: preserve the catalog/1 contract exactly; make multi-package transactions native (one op-time, one update-log append, one atomic catalog commit); persist transaction state explicitly with listing/TTL/status exposure; make close validate-then-move (payloads before manifest, idempotent commit marker); keep renames same-filesystem, catalog writes serialised, indexing async but with reported failures. Note "no pkgsend compat" also drops legacy pkgrecv push and remote pkgrepo rebuild/refresh; pulling from pkg6depotd is unaffected.

Full findings: docs/research/pkg5-publication-model.md on branch research/pkg5-publication-model (research branch, not merged; cherry-pick the file when the decision it feeds is recorded).

## Resolution pkg5 models publication as one transaction per package version: `open` binds an FMRI, creates `trans/<unix-open-time>_<urlquoted fmri>/` with a one-line `manifest`; `add` appends action lines and stages gzipped payloads under `trans/<id>/<hash>` (only for content not already in `file/`); `close` renames manifest to `pkg/`, renames payloads into `file/`, then (unless `X-IPkg-Add-To-Catalog: 0`) calls `add_package` which rewrites the catalog once under the `tmp/lock` file lock and finally rmtree's the dir; `abandon` is just rmtree. The documented SUBMITTED/INCOMPLETE states were never implemented. Transactions are not journalled or scanned: a crashed one lingers forever in `trans/` (invisible to `status/0`, no `pkgrepo` cleanup) but can be lazily `reopen`ed by id even after a depot restart. Close is not two-phase: manifest moves before payloads and before the catalog write, so crashes can leave uncatalogued or file-less packages (repair = `refresh-packages`/`rebuild`/`pkgrepo fix`). Catalog: each close does `Catalog.add_package` with a single op-time, sets `catalog.attrs.last-modified`, appends to an hourly `update.<YYYYMMDDTHH>Z.C` log, and saves via copy-to-tmp + rename swap (leaves `old.catalog` on crash; never prunes logs). `pkgsend publish` = one transaction per invocation; batching is only `--no-catalog` + one `admin/0?cmd=refresh-packages` (incremental `__rebuild`, which does emit update logs); `rebuild` destroys the catalog, changes `created`, and forces every client to a full refetch. Indexing is never on the publish path (`refresh-index` is an async background task whose failures are invisible). Legacy pkg(1) only sees the retrieval side: `catalog/1/<file>` semantics (`created` stable, monotonic `last-modified`, per-part `last-modified`+`signature-sha-1`, hourly update-log names listed in `attrs.updates`, op-time filtering, Last-Modified/If-Modified-Since, Cache-Control) plus manifest/file/publisher/p5i/search/status. Everything else (trans layout, open/add/close/append/abandon, manifest/1 & file/1 POST, X-IPkg-* headers, admin/0, index/0, background queue, lock/tmp files) is internal and free to redesign. Implications for pkg6depotd: preserve the catalog/1 contract exactly; make multi-package transactions native (one op-time, one update-log append, one atomic catalog commit); persist transaction state explicitly with listing/TTL/status exposure; make close validate-then-move (payloads before manifest, idempotent commit marker); keep renames same-filesystem, catalog writes serialised, indexing async but with reported failures. Note "no pkgsend compat" also drops legacy `pkgrecv` push and remote `pkgrepo rebuild/refresh`; pulling from pkg6depotd is unaffected. **Full findings:** [docs/research/pkg5-publication-model.md](https://code.aopc.cloud/OpenIndiana/ips/src/branch/research/pkg5-publication-model/docs/research/pkg5-publication-model.md) on branch `research/pkg5-publication-model` (research branch, not merged; cherry-pick the file when the decision it feeds is recorded).