V1: explicit import filters and two-stage repository retirement #39

New issue

Open

opened 2026-09-22 12:11:17 +00:00 by forge · 1 comment

forge commented

2026-09-22 12:11:17 +00:00

Part of #24

Preserved snapshot support promise — 2026-09-22

Maintainer decision: selected historical ISO snapshots must support maintaining the historical system, not merely booting/installing and upgrading to current Hipster. Users must be able to install additional packages and repair installed packages while retaining that snapshot's original version constraints. Selected snapshots remain protected indefinitely under the agreed retention policy.

Consequences for design and acceptance:

Inventorying only packages already installed in the ISO root filesystem is insufficient. Reconstruct and preserve a usable historical package set, including compatible additional packages, relevant incorporation metadata, dependencies, manifests and payloads. Record provenance/ISO hashes and report unavailable historical content explicitly rather than claiming complete reconstruction.
Packages required for the supported snapshot must remain discoverable to the legacy OpenIndiana pkg5 solver, not just retrievable by an exact-FMRI HTTP request. Design a usable snapshot catalog/view if needed; endpoint/layout mechanics are delegated to engineering and must be validated against local pkg5.
The earlier newest-only incorporation proposal is subordinate to this support promise. Do not remove or hide metadata needed for historical maintenance unless the snapshot view preserves the required semantics and native tests prove it works.
Import exclusions conflicting with the protected historical set still fail preflight and require explicit KDL keep exceptions. Retirement/purge must preserve the full protected snapshot set and shared payload references. Main-catalog hiding must not make a supported snapshot unusable.
Acceptance: from selected historical ISO-installed clients (Hipster approximately 2020 onward), install a package absent from the original image and repair installed content without forcing a move to current package constraints. Also validate the supported update path separately. Use the real local OpenIndiana pkg5 implementation and retain evidence.

This settles the product-level snapshot promise. Reconstruction, snapshot catalog design and validation remain engineering work, not completed functionality.

Configurable retirement and KDL policy — 2026-09-22

Latest maintainer direction:

Retirement must be enable/disable configurable, and its timing must be configurable rather than hard-coded to three months. The maintainer subsequently set the default policy interval to 12 months, superseding the earlier three-month default. Both inactivity and post-hide grace intervals default to 12 months and remain configurable. Earlier three-month references below are historical and superseded.
The import policy file uses KDL, superseding earlier text/JSON alternatives. Regex exclusions and exact-FMRI keep exceptions remain the intended rules: import unmatched packages, explicit keeps override exclusions, invalid patterns or unresolved protected-dependency conflicts fail preflight. Provide explicit proposed exceptions for the operator to add to the KDL file.
Remaining design mechanics are delegated to engineering; do not repeatedly seek maintainer approval for schema spelling, parser choices, CLI spelling or ordinary implementation details. Preserve all agreed semantics.

Engineering defaults/constraints to carry into the design (defaults selected here under that delegation, not independently quoted maintainer decisions): retirement is opt-in/disabled by default; disabling pauses policy-driven retirement without purging content or automatically restoring hidden versions. Continue serving hidden exact-FMRI content. Keep access tracking separately durable. Manual purge remains an explicit operator action with retention, snapshot and dependency checks; enabling retirement must never enable automatic purge. Document configuration effects and validate durations/rules before any mutation.

Still require evidence rather than assumptions: actual local pkg5 behavior, snapshot dependency closure, safe catalog replacement, and historical-client compatibility. No implementation or runtime validation is claimed by this decision update.

Access to hidden versions — 2026-09-22

Maintainer decision: a qualifying access to a hidden version resets its purge eligibility clock but leaves it hidden. It must not automatically reappear in catalogs or search. Restoring catalog/search visibility requires an explicit local restore/unretire command; exact CLI spelling remains part of the command review. Access qualification and snapshot/dependency protection continue to apply. This is a required behavior, not an implementation-complete claim.

Agreed contract

Support a text or JSON policy file matching FMRIs with regex/grep-like rules. Exact syntax and canonical FMRI matching remain to be designed.
Preflight fails if exclusions conflict with protected snapshots/dependencies. Generate explicit exact-FMRI exceptions for the operator to add to the same file; never silently override exclusions. Required incorporated dependency versions must remain available for retained snapshots.
Discover designated historical snapshot contents from ISO images through an agent-assisted, provenance-preserving inventory. Selected snapshots are retained indefinitely.
Superseded versions become retirement candidates after three months without qualifying access. Manifest/payload requests count; catalog/search and our import/audit/retention scans do not. Shared payload requests must not falsely refresh every referencing package version; attribution remains to be designed.
First stage hides versions from catalogs/search while continuing exact-FMRI manifest and payload retrieval.
Second stage is operator-triggered purge, eligible no earlier than three months after hiding or the latest qualifying access, whichever is later. Snapshot pins and dependency checks block purge. Shared blobs cannot be deleted while retained versions reference them.
Durable access accounting is independent of lossy OTel queues. Automated purge and S3 tiering are deferred; hidden content remains local in v1.
Preserve pkg5 pkg.obsolete semantics. Introduce explicit retirement terminology and review/remove the overloaded obsolete-package command (see command-review ticket).

Requires investigation / further grilling

Maintainer proposes retaining only the newest advertised userland-incorporation because old clients hold their installed incorporation locally. Prove this with local pkg5 source and ISO-installed clients before accepting it as safe, including interaction with retained snapshots and pinned dependency versions. Do not treat this as permission to purge snapshot dependencies.
Exact import-grace policy for versions with no access history was not settled (three months from import was proposed, not accepted).
Three calendar months vs fixed duration, payload access attribution, unavailable access history, recovery/unhide, and snapshot visibility remain to be specified.
Hiding may affect solver discovery even when exact fetch still works; acceptance tests must cover supported client operations, not just HTTP fetches.

Source: maintainer grilling, 2026-09-22. Requirements only; no implementation or validation claimed.

Part of #24 ## Preserved snapshot support promise — 2026-09-22 Maintainer decision: selected historical ISO snapshots must support maintaining the historical system, not merely booting/installing and upgrading to current Hipster. Users must be able to install additional packages and repair installed packages while retaining that snapshot's original version constraints. Selected snapshots remain protected indefinitely under the agreed retention policy. Consequences for design and acceptance: - Inventorying only packages already installed in the ISO root filesystem is insufficient. Reconstruct and preserve a usable historical package set, including compatible additional packages, relevant incorporation metadata, dependencies, manifests and payloads. Record provenance/ISO hashes and report unavailable historical content explicitly rather than claiming complete reconstruction. - Packages required for the supported snapshot must remain discoverable to the legacy OpenIndiana pkg5 solver, not just retrievable by an exact-FMRI HTTP request. Design a usable snapshot catalog/view if needed; endpoint/layout mechanics are delegated to engineering and must be validated against local pkg5. - The earlier newest-only incorporation proposal is subordinate to this support promise. Do not remove or hide metadata needed for historical maintenance unless the snapshot view preserves the required semantics and native tests prove it works. - Import exclusions conflicting with the protected historical set still fail preflight and require explicit KDL keep exceptions. Retirement/purge must preserve the full protected snapshot set and shared payload references. Main-catalog hiding must not make a supported snapshot unusable. - Acceptance: from selected historical ISO-installed clients (Hipster approximately 2020 onward), install a package absent from the original image and repair installed content without forcing a move to current package constraints. Also validate the supported update path separately. Use the real local OpenIndiana pkg5 implementation and retain evidence. This settles the product-level snapshot promise. Reconstruction, snapshot catalog design and validation remain engineering work, not completed functionality. ## Configurable retirement and KDL policy — 2026-09-22 Latest maintainer direction: - Retirement must be enable/disable configurable, and its timing must be configurable rather than hard-coded to three months. The maintainer subsequently set the default policy interval to 12 months, superseding the earlier three-month default. Both inactivity and post-hide grace intervals default to 12 months and remain configurable. Earlier three-month references below are historical and superseded. - The import policy file uses KDL, superseding earlier text/JSON alternatives. Regex exclusions and exact-FMRI keep exceptions remain the intended rules: import unmatched packages, explicit keeps override exclusions, invalid patterns or unresolved protected-dependency conflicts fail preflight. Provide explicit proposed exceptions for the operator to add to the KDL file. - Remaining design mechanics are delegated to engineering; do not repeatedly seek maintainer approval for schema spelling, parser choices, CLI spelling or ordinary implementation details. Preserve all agreed semantics. Engineering defaults/constraints to carry into the design (defaults selected here under that delegation, not independently quoted maintainer decisions): retirement is opt-in/disabled by default; disabling pauses policy-driven retirement without purging content or automatically restoring hidden versions. Continue serving hidden exact-FMRI content. Keep access tracking separately durable. Manual purge remains an explicit operator action with retention, snapshot and dependency checks; enabling retirement must never enable automatic purge. Document configuration effects and validate durations/rules before any mutation. Still require evidence rather than assumptions: actual local pkg5 behavior, snapshot dependency closure, safe catalog replacement, and historical-client compatibility. No implementation or runtime validation is claimed by this decision update. ## Access to hidden versions — 2026-09-22 Maintainer decision: a qualifying access to a hidden version resets its purge eligibility clock but leaves it hidden. It must not automatically reappear in catalogs or search. Restoring catalog/search visibility requires an explicit local restore/unretire command; exact CLI spelling remains part of the command review. Access qualification and snapshot/dependency protection continue to apply. This is a required behavior, not an implementation-complete claim. ## Agreed contract - Support a text or JSON policy file matching FMRIs with regex/grep-like rules. Exact syntax and canonical FMRI matching remain to be designed. - Preflight fails if exclusions conflict with protected snapshots/dependencies. Generate explicit exact-FMRI exceptions for the operator to add to the same file; never silently override exclusions. Required incorporated dependency versions must remain available for retained snapshots. - Discover designated historical snapshot contents from ISO images through an agent-assisted, provenance-preserving inventory. Selected snapshots are retained indefinitely. - Superseded versions become retirement candidates after three months without qualifying access. Manifest/payload requests count; catalog/search and our import/audit/retention scans do not. Shared payload requests must not falsely refresh every referencing package version; attribution remains to be designed. - First stage hides versions from catalogs/search while continuing exact-FMRI manifest and payload retrieval. - Second stage is operator-triggered purge, eligible no earlier than three months after hiding or the latest qualifying access, whichever is later. Snapshot pins and dependency checks block purge. Shared blobs cannot be deleted while retained versions reference them. - Durable access accounting is independent of lossy OTel queues. Automated purge and S3 tiering are deferred; hidden content remains local in v1. - Preserve pkg5 pkg.obsolete semantics. Introduce explicit retirement terminology and review/remove the overloaded obsolete-package command (see command-review ticket). ## Requires investigation / further grilling - Maintainer proposes retaining only the newest advertised userland-incorporation because old clients hold their installed incorporation locally. Prove this with local pkg5 source and ISO-installed clients before accepting it as safe, including interaction with retained snapshots and pinned dependency versions. Do not treat this as permission to purge snapshot dependencies. - Exact import-grace policy for versions with no access history was not settled (three months from import was proposed, not accepted). - Three calendar months vs fixed duration, payload access attribution, unavailable access history, recovery/unhide, and snapshot visibility remain to be specified. - Hiding may affect solver discovery even when exact fetch still works; acceptance tests must cover supported client operations, not just HTTP fetches. Source: maintainer grilling, 2026-09-22. Requirements only; no implementation or validation claimed.

forge added this to the Release 1 (server only) milestone

2026-09-22 12:11:17 +00:00

forge added the

Kind/Feature

wayfinder:task

labels

2026-09-22 12:11:17 +00:00

forge referenced this issue

2026-09-22 12:11:35 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge referenced this issue

2026-09-22 12:11:57 +00:00

Document the server #6

forge referenced this issue

2026-09-22 12:13:05 +00:00

Wayfinder map: Release 1 (server only) #24

forge added a new dependency

2026-09-22 12:13:18 +00:00

#25 Define what 'v1.0 server release' must include and its acceptance criteria

forge referenced this issue

2026-09-23 07:16:40 +00:00

WIP: Build tagged release candidates and an illumos IPS depot package #43

forge commented

2026-09-23 09:41:44 +00:00

Author

Implementation is in draft PR #50. pkg6repo retire now hides superseded versions from catalogs/search while exact manifests and payloads remain served; restore reverses that state, and no automatic purge exists. Policy is opt-in with a configurable 12-month inactivity default; protected snapshots preserve dependency closure. The access ledger fails closed on write errors, deleted newest logs (including after all depots stopped), and stale apply plans, with a per-depot durable head and commit-time fence. Explicit KDL/text/JSON FMRI exclusions fail preflight if dependencies would break. Focused tests passed: libips retention 35, retirement 31, HTTP 5, retire CLI 4. Independent Opus review found no release blocker after the final durability fix. Integration with the package/manpages and exact-head CI remains before closing.

Implementation is in draft PR #50. `pkg6repo retire` now hides superseded versions from catalogs/search while exact manifests and payloads remain served; restore reverses that state, and no automatic purge exists. Policy is opt-in with a configurable 12-month inactivity default; protected snapshots preserve dependency closure. The access ledger fails closed on write errors, deleted newest logs (including after all depots stopped), and stale apply plans, with a per-depot durable head and commit-time fence. Explicit KDL/text/JSON FMRI exclusions fail preflight if dependencies would break. Focused tests passed: libips retention 35, retirement 31, HTTP 5, retire CLI 4. Independent Opus review found no release blocker after the final durability fix. Integration with the package/manpages and exact-head CI remains before closing.