V1: bounded resumable import of historical pkg5 repositories

toasterson commented

2026-09-26 17:05:57 +00:00

Owner

Part of #24; related to #38 (atomic CI p5p ingestion), #39 (import exclusions and snapshot closure), and #42 (historical OpenIndiana client acceptance).

Historical OpenIndiana directory repositories on the read-only NFS export contain hundreds of thousands of manifests each. The separate OpenSolaris legacy V3 source has 2,108,239 manifest files; it is not a Hipster archive. The maintainer confirmed it must be imported into its own pkg6 repository root under its opensolaris.org publisher. The current import-pkg5 reads every source manifest and payload reference into memory before publication. oirepo6 has a measured 4 GB zone swap limit, so a whole-source import is not a safe operational path. Format recognition alone does not solve this.

Implement a bounded, resumable historical directory import path while preserving the existing one-p5p-per-CI-run atomic contract. Bounded batches must stage invisibly with durable progress, resume against the same source and policy identity, and revalidate bytes when consumed. One final generation switch per source and target repository must make all accepted packages in that import visible together; a partial run must never expose a subset. Hipster history shares the Hipster root, while legacy, oi-sparc and unleashed have independent repository roots and SMF depot instances. oi-sparc must not share Hipster's catalog even though both use openindiana.org. Do not import solarm.org or desktop-2023. The repositories have no cross-root atomic switch. Preserve byte-exact manifests and verified pkg5 payloads. Never silently reinterpret a source as Hipster or replace an existing exact FMRI.

Import exclusions require full-source preflight: evaluate KDL exclude/keep rules against the complete selected source and protected snapshot/dependency closure before the first batch becomes visible. If that cannot be done in the first bounded implementation, reject exclusion policies explicitly for that mode. Do not check each batch in isolation, as a later package may depend on an excluded earlier one. No automatic purge is involved.

Acceptance: measure memory on the 4 GB zone using representative NFS sources; interrupt and resume without duplicate or partial batches; verify pkg5 listing, exact manifest/payload retrieval and installation against an isolated repository; record source identity and package counts, including missing payloads, without claiming the 2020 ISO support promise until its separate client tests pass. Do not operate on the live repository for this ticket's initial verification.

Part of #24; related to #38 (atomic CI p5p ingestion), #39 (import exclusions and snapshot closure), and #42 (historical OpenIndiana client acceptance). Historical OpenIndiana directory repositories on the read-only NFS export contain hundreds of thousands of manifests each. The separate OpenSolaris `legacy` V3 source has 2,108,239 manifest files; it is **not** a Hipster archive. The maintainer confirmed it must be imported into its own pkg6 repository root under its `opensolaris.org` publisher. The current `import-pkg5` reads every source manifest and payload reference into memory before publication. oirepo6 has a measured 4 GB zone swap limit, so a whole-source import is not a safe operational path. Format recognition alone does not solve this. Implement a bounded, resumable historical directory import path while preserving the existing one-p5p-per-CI-run atomic contract. Bounded batches must stage invisibly with durable progress, resume against the same source and policy identity, and revalidate bytes when consumed. One final generation switch per source and target repository must make all accepted packages in that import visible together; a partial run must never expose a subset. Hipster history shares the Hipster root, while `legacy`, `oi-sparc` and `unleashed` have independent repository roots and SMF depot instances. `oi-sparc` must not share Hipster's catalog even though both use `openindiana.org`. Do not import `solarm.org` or `desktop-2023`. The repositories have no cross-root atomic switch. Preserve byte-exact manifests and verified pkg5 payloads. Never silently reinterpret a source as Hipster or replace an existing exact FMRI. Import exclusions require full-source preflight: evaluate KDL exclude/keep rules against the complete selected source and protected snapshot/dependency closure before the first batch becomes visible. If that cannot be done in the first bounded implementation, reject exclusion policies explicitly for that mode. Do not check each batch in isolation, as a later package may depend on an excluded earlier one. No automatic purge is involved. Acceptance: measure memory on the 4 GB zone using representative NFS sources; interrupt and resume without duplicate or partial batches; verify pkg5 listing, exact manifest/payload retrieval and installation against an isolated repository; record source identity and package counts, including missing payloads, without claiming the 2020 ISO support promise until its separate client tests pass. Do not operate on the live repository for this ticket's initial verification.

toasterson added the

wayfinder:task

label

2026-09-26 17:05:57 +00:00

toasterson referenced this issue

2026-09-26 17:06:19 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson added this to the Release 1 (server only) milestone

2026-09-26 17:06:38 +00:00

toasterson added a new dependency

2026-09-26 17:06:39 +00:00

#38 V1: atomically ingest one multi-package p5p per CI run

toasterson added a new dependency

2026-09-26 17:06:39 +00:00

#39 V1: explicit import filters and two-stage repository retirement

toasterson referenced this issue

2026-09-26 18:05:26 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson commented

2026-09-26 18:05:29 +00:00

Author

Owner

legacy is mandatory for v1 as a separate opensolaris.org publisher in the final multi-publisher repository. RC12 (690be90) adds strict flat V3 source recognition, path-derived FMRI when pkg.fmri is absent, and a whole-batch import guard; a real sample imports and serves to a native pkg5 client under the correct publisher. The full 2,108,239-manifest source exceeds that guarded atomic importer by design.

A separate import-historical plan/run/publish implementation is under review. It uses disk-backed inventory and a full-source exclusion preflight before staged batches, then one final generation switch. Local full workspace tests currently pass (446 passed, one opt-in test skipped). Independent review found three quadratic SQLite/dependency-walk paths and retry/integrity gaps; fixes and native scale validation are required before this issue can close. No full legacy import, 4 GB memory proof, or live repository adoption is claimed yet. The tagged RC12 source does not contain this work.

`legacy` is mandatory for v1 as a separate `opensolaris.org` publisher in the final multi-publisher repository. RC12 (`690be90`) adds strict flat V3 source recognition, path-derived FMRI when `pkg.fmri` is absent, and a whole-batch import guard; a real sample imports and serves to a native pkg5 client under the correct publisher. The full 2,108,239-manifest source exceeds that guarded atomic importer by design. A separate `import-historical` plan/run/publish implementation is under review. It uses disk-backed inventory and a full-source exclusion preflight before staged batches, then one final generation switch. Local full workspace tests currently pass (446 passed, one opt-in test skipped). Independent review found three quadratic SQLite/dependency-walk paths and retry/integrity gaps; fixes and native scale validation are required before this issue can close. No full `legacy` import, 4 GB memory proof, or live repository adoption is claimed yet. The tagged RC12 source does not contain this work.

toasterson referenced this issue

2026-09-26 18:20:02 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson commented

2026-09-26 18:51:24 +00:00

Author

Owner

RC13 source is implemented and tagged at f6691a0 / v1.0.0-rc.13. pkg6repo import-historical plan|run|publish|status now inventories a V3/V4 directory on disk, checks the complete selected source and KDL exclusion/dependency/snapshot policy before staging, stages bounded resumable batches (default caps: 10,000 manifests / 64 MiB), and exposes the accepted packages in one final generation switch. It preserves the source publisher: OpenSolaris legacy is opensolaris.org, separate from Hipster openindiana.org. A changed source, conflicting exact FMRI, or unsafe exclusion fails closed. No automatic purge is part of this path.

Local full workspace: 457 passed, one opt-in test skipped. Native oibuild release-profile workspace: 457 passed, one skipped; release self-tests 543 passed; IPS baseline package build and unprivileged pkg5 smoke passed. A separate native run used one actual legacy V3 manifest and its three payloads: plan/run/publish reported one opensolaris.org package; its stored manifest and all stored payloads were byte-exact to the NFS-derived sample; native pkg5 pkgrecv listed and retrieved the package and all three payloads from an isolated depot. pkgrecv exported its normal pkg.fmri first action in place of the source V3 comment header; the remaining manifest actions and decompressed payloads matched. This test depot has been stopped.

Still open: the full 2,108,239-manifest NFS source has not been planned, imported, or measured within oirepo6's 4 GB limit. The one-package sample does not prove full-source memory, publication latency, interruption/restart at scale, or installation of an old client with its complete dependency set. No live repository was touched for this test. Keep #53 open until those acceptance gates are recorded; the historical infrastructure import remains deferred.

RC13 source is implemented and tagged at [`f6691a0`](https://code.aopc.cloud/OpenIndiana/ips/commit/f6691a0349662c294a6b8a2d19d6739ec663c988) / [`v1.0.0-rc.13`](https://code.aopc.cloud/OpenIndiana/ips/releases/tag/v1.0.0-rc.13). `pkg6repo import-historical plan|run|publish|status` now inventories a V3/V4 directory on disk, checks the complete selected source and KDL exclusion/dependency/snapshot policy before staging, stages bounded resumable batches (default caps: 10,000 manifests / 64 MiB), and exposes the accepted packages in one final generation switch. It preserves the source publisher: OpenSolaris `legacy` is `opensolaris.org`, separate from Hipster `openindiana.org`. A changed source, conflicting exact FMRI, or unsafe exclusion fails closed. No automatic purge is part of this path. Local full workspace: 457 passed, one opt-in test skipped. Native `oibuild` release-profile workspace: 457 passed, one skipped; release self-tests 543 passed; IPS baseline package build and unprivileged pkg5 smoke passed. A separate native run used one actual `legacy` V3 manifest and its three payloads: plan/run/publish reported one `opensolaris.org` package; its stored manifest and all stored payloads were byte-exact to the NFS-derived sample; native pkg5 `pkgrecv` listed and retrieved the package and all three payloads from an isolated depot. `pkgrecv` exported its normal `pkg.fmri` first action in place of the source V3 comment header; the remaining manifest actions and decompressed payloads matched. This test depot has been stopped. Still open: the full 2,108,239-manifest NFS source has **not** been planned, imported, or measured within oirepo6's 4 GB limit. The one-package sample does not prove full-source memory, publication latency, interruption/restart at scale, or installation of an old client with its complete dependency set. No live repository was touched for this test. Keep #53 open until those acceptance gates are recorded; the historical infrastructure import remains deferred.

toasterson referenced this issue

2026-09-26 19:00:07 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson commented

2026-09-26 22:23:52 +00:00

Author

Owner

RC13 is now the running depot on oirepo6's local port 8080. The tagged package/pkg6/depotd@1.0.0-0.3.13 is installed and pkg verify passes. I disabled the old svc:/network/pkg6depotd:default (Rust 0.5) instance and restarted the packaged svc:/application/pkg/pkg6depotd:default from /usr/lib/pkg6depotd, with /etc/pkg6depotd/pkg6depotd-oirepo6.kdl pointing read-only at the repaired /zdata/oirepo6/repo-rc11-test clone. The packaged service is online on 8080; 8083 has no listener; local health and pkg5 catalog requests return 200. Native pkg5 pkg list -afv -g http://10.1.0.13:8080 entire on oibldofgate exited 0 and returned an openindiana.org/entire FMRI. The public proxy was not changed; oirepo6 is not yet public. The original repository and disabled 0.5 service remain for rollback. Historical NFS imports and pre-generation adoption have not run; those should be staged in a separate clone before a final switch. The current BE is zbe-1, but another BE is marked for next reboot, so reboot persistence remains unverified and no BE was activated.

RC13 is now the running depot on **oirepo6's local port 8080**. The tagged `package/pkg6/depotd@1.0.0-0.3.13` is installed and `pkg verify` passes. I disabled the old `svc:/network/pkg6depotd:default` (Rust 0.5) instance and restarted the packaged `svc:/application/pkg/pkg6depotd:default` from `/usr/lib/pkg6depotd`, with `/etc/pkg6depotd/pkg6depotd-oirepo6.kdl` pointing read-only at the repaired `/zdata/oirepo6/repo-rc11-test` clone. The packaged service is online on 8080; 8083 has no listener; local health and pkg5 catalog requests return 200. Native pkg5 `pkg list -afv -g http://10.1.0.13:8080 entire` on oibldofgate exited 0 and returned an `openindiana.org/entire` FMRI. **The public proxy was not changed**; oirepo6 is not yet public. The original repository and disabled 0.5 service remain for rollback. Historical NFS imports and pre-generation adoption have not run; those should be staged in a separate clone before a final switch. The current BE is `zbe-1`, but another BE is marked for next reboot, so reboot persistence remains unverified and no BE was activated.

toasterson commented

2026-09-26 22:39:21 +00:00

Author

Owner

With the maintainer's explicit approval, beadm activate zbe-1 completed on oirepo6. beadm list -H now marks zbe-1 as NR (running and selected for the next boot); the previous openindiana-2026:09:22 BE has no active flag. The packaged RC13 depot remains online on 8080, the 0.5 service remains disabled, and health is OK. No reboot was performed, so boot-time startup is still untested. Before a reboot, the previous boot choice can be restored with beadm activate openindiana-2026:09:22.

With the maintainer's explicit approval, `beadm activate zbe-1` completed on oirepo6. `beadm list -H` now marks `zbe-1` as `NR` (running and selected for the next boot); the previous `openindiana-2026:09:22` BE has no active flag. The packaged RC13 depot remains online on 8080, the 0.5 service remains disabled, and health is OK. No reboot was performed, so boot-time startup is still untested. Before a reboot, the previous boot choice can be restored with `beadm activate openindiana-2026:09:22`.

toasterson referenced this issue

2026-09-27 13:13:25 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

toasterson commented

2026-09-27 13:14:08 +00:00

Author

Owner

ADR-015 item 2 (2026-09-27): historical import is a capability gate for v1.0, not a completion gate. Gate: one full Hipster archive imported end to end on oirepo6 under the measured 4 GB zone.max-swap limit with peak memory recorded, plus the legacy V3 sample under opensolaris.org. The full 2,108,239-manifest legacy import and the proxy cutover are the first v1.0.x operations tracked here. The command names change under #55 before the tag.

ADR-015 item 2 (2026-09-27): historical import is a **capability** gate for v1.0, not a completion gate. Gate: one full Hipster archive imported end to end on oirepo6 under the measured 4 GB `zone.max-swap` limit with peak memory recorded, plus the `legacy` V3 sample under `opensolaris.org`. The full 2,108,239-manifest `legacy` import and the proxy cutover are the first v1.0.x operations tracked here. The command names change under #55 before the tag.

toasterson referenced this issue

2026-09-27 13:14:15 +00:00

Wayfinder map: Release 1 (server only) #24

toasterson commented

2026-09-27 13:53:33 +00:00

Author

Owner

Source placement is corrected in ADR-015 and the operator book at commit 5cbf36a on feat/separate-depots: Hipster history stays together; legacy (opensolaris.org), oi-sparc (openindiana.org) and unleashed each get a separate repository root and depot instance. solarm.org and desktop-2023 are excluded. The historical importer still publishes one generation atomically per root; no cross-root atomic switch is claimed. Full-source imports have not run. Native RC14 package validation is in progress.

Source placement is corrected in ADR-015 and the operator book at commit 5cbf36a on feat/separate-depots: Hipster history stays together; legacy (opensolaris.org), oi-sparc (openindiana.org) and unleashed each get a separate repository root and depot instance. solarm.org and desktop-2023 are excluded. The historical importer still publishes one generation atomically per root; no cross-root atomic switch is claimed. Full-source imports have not run. Native RC14 package validation is in progress.

toasterson commented

2026-09-27 14:35:59 +00:00

Author

Owner

RC14 target layout is now deployed privately on oirepo6: Hipster retains its existing root; legacy (opensolaris.org), oi-sparc (openindiana.org plus hipster-encumbered), and unleashed have distinct roots and named SMF services. solarm.org and desktop-2023 remain excluded by operator choice. The NFS export 10.1.0.11:/zdata/oirepo is mounted read-only at /zdata/oirepo6/source-nfs.

One actual legacy V3 manifest, its cfg_cache, and three payloads were compared byte-for-byte with the NFS source, imported into the separate legacy root, and listed/fetched through port 8081 by the native pkg5 client on oibldofgate. This is a one-package compatibility proof, not a full import or memory-bound measurement. Full source plan/run/publish and validation remain open; do not run them against the serving root without the staged clone procedure. See PR #56.

RC14 target layout is now deployed privately on oirepo6: Hipster retains its existing root; `legacy` (`opensolaris.org`), `oi-sparc` (`openindiana.org` plus `hipster-encumbered`), and `unleashed` have distinct roots and named SMF services. `solarm.org` and `desktop-2023` remain excluded by operator choice. The NFS export `10.1.0.11:/zdata/oirepo` is mounted read-only at `/zdata/oirepo6/source-nfs`. One actual `legacy` V3 manifest, its `cfg_cache`, and three payloads were compared byte-for-byte with the NFS source, imported into the separate legacy root, and listed/fetched through port 8081 by the native pkg5 client on oibldofgate. This is a one-package compatibility proof, not a full import or memory-bound measurement. Full source plan/run/publish and validation remain open; do not run them against the serving root without the staged clone procedure. See [PR #56](https://code.aopc.cloud/OpenIndiana/ips/pulls/56).

toasterson commented

2026-09-27 16:13:59 +00:00

Author

Owner

An operator import driver is now installed on private oirepo6 at /zdata/oirepo6/import-history.py (source in workspace bin/oirepo6-import-history.py, SHA-256 1b9b09bb09c65a4d831fbe7624fa5461bf2216fd1e9e6543ac1154986e18ffe9). check all verifies all 15 expected read-only NFS source mounts and the target SMF roots. run all plans, stages and publishes oi-sparc, unleashed and legacy into their own roots; it imports all Hipster sources into a ZFS clone of the current pre-generation root and leaves port 8080 unchanged for a later reconciled cutover. The driver resumes interrupted batches, keeps full logs, prints sparse progress, and stops an importer above 2560 MiB RSS or below 100 GiB free. An isolated one-package V3 test completed plan/run/publish through the wrapper as pkg6srv. No full-source import has been started; the Hipster clone creation and 4 GB-zone full-source memory gate remain untested.

An operator import driver is now installed on private oirepo6 at `/zdata/oirepo6/import-history.py` (source in workspace `bin/oirepo6-import-history.py`, SHA-256 `1b9b09bb09c65a4d831fbe7624fa5461bf2216fd1e9e6543ac1154986e18ffe9`). `check all` verifies all 15 expected read-only NFS source mounts and the target SMF roots. `run all` plans, stages and publishes `oi-sparc`, `unleashed` and `legacy` into their own roots; it imports all Hipster sources into a ZFS clone of the current pre-generation root and leaves port 8080 unchanged for a later reconciled cutover. The driver resumes interrupted batches, keeps full logs, prints sparse progress, and stops an importer above 2560 MiB RSS or below 100 GiB free. An isolated one-package V3 test completed plan/run/publish through the wrapper as `pkg6srv`. No full-source import has been started; the Hipster clone creation and 4 GB-zone full-source memory gate remain untested.

Rows
Columns

V1: bounded resumable import of historical pkg5 repositories #53