v1 deployable pkg6 depot: pkg5 imports, retention, TLS, telemetry and release pipeline #46

Closed

forge wants to merge 35 commits from v1-integration-20260923 into main

pull from: v1-integration-20260923

merge into: OpenIndiana:main

OpenIndiana:main

OpenIndiana:feat/separate-depots

OpenIndiana:v1-main-merge

OpenIndiana:feat/historical-import

OpenIndiana:v1-pkg5-smoke-20260923

OpenIndiana:v1-retention-20260923

OpenIndiana:v1-repo-cli-20260923

OpenIndiana:v1-cli-surface-20260923

OpenIndiana:v1-tls-20260923

OpenIndiana:v1-atomic-import-20260923

OpenIndiana:v1-otel-20260923

OpenIndiana:v1-release-pipeline-20260923

OpenIndiana:research/oidc-providers

OpenIndiana:research/illumos-crypto-stack

OpenIndiana:research/otlp-stack

OpenIndiana:research/pkg5-publication-model

OpenIndiana:claude/ecstatic-allen-0a5f4a

Conversation 4 Commits 35 Files changed 158 +39481 -3989

forge commented 2026-09-23 08:06:23 +00:00

First-time contributor

Copy link

An existing OpenIndiana pkg5 client must be able to consume packages from one authoritative pkg6 repository while multi-package imports switch atomically and hidden old versions retain exact serving. This integration adds generation-switched .p5p import with no-op exact retries and conflict rejection; fail-closed KDL FMRI exclusions and protected keeps; opt-in configurable retirement with a 12-month default and no automatic purge; direct/proxy TLS with ACME HTTP-01 and Cloudflare/Gandi DNS-01; bounded OTLP traces, metrics and logs; an IPS package and SMF service; and an operator book with illumos-style installed man pages. The unfinished pkg6 client is outside the release package.

Alpha, beta, RC and stable tags use one semver release workflow. Each platform builds from the immutable tag and uploads its own assets only after its gates pass; prereleases are published when both platform jobs succeed, while stable releases stay draft for operator promotion. The current tag is v1.0.0-rc.10 (f8c456f). Its Solstice run 8977 passed the Linux job, 384 release-script checks, all 407 native tests, and native .p5p construction. Independent local Opus review found no blocker in the latest diagnostic change.

Evidence through RC9: Linux release checks and all 407 OpenIndiana native release-profile tests pass. The .p5p installs and verifies on a disposable Hipster VM; all packaged manpages render; SMF starts the depot as pkg6srv and puts missing or malformed readable KDL into maintenance; the native local pkg5 client refreshes the depot catalog, receives a package, and installs/verifies/uninstalls it in a scratch image. A live package upgrade uses pkg5's documented --no-backup-be only in the disposable VM, preserves operator-edited KDL, delivers the new default as .kdl.new, and leaves the depot online. RC9 reached live pkg uninstall, which failed when the throwaway VM's master SMF restarter lacked its NONPERSISTENT restarter property group. The package must retain disable_fmri, so RC10 added stage-specific master-restarter checks and bounded diagnostics without weakening uninstall. RC10's fresh VM lacked that property group before package install; svc.startd and svc.configd were running but the master instance state was unavailable. The live package test therefore stopped at preflight, and no tagged IPS asset was published. The disposable VM image needs repair before this release gate can pass.

The earlier RC1–RC8 failures and focused fixes are recorded on #41. RC1 scratch testing also exposed a real pkg5 catalog compatibility bug: pkg6 omitted the required empty updates field from catalog.attrs; RC2 corrected it. Historical Hipster 2020 client and protected snapshot-maintenance acceptance in #42 remains open. Historical sources will be copied from NFS and imported locally into one final repository; mount/access details and infrastructure work are deferred. The code-only work is ready for renewed CI once the deferred disposable-VM image repair is done. This PR is not ready for v1 merge or production cutover on current evidence.

Component changes are tracked in #43, #44, #45, #47, #48, #49, #50 and #51; release acceptance is tracked in #25, #41 and #42.

An existing OpenIndiana pkg5 client must be able to consume packages from one authoritative pkg6 repository while multi-package imports switch atomically and hidden old versions retain exact serving. This integration adds generation-switched `.p5p` import with no-op exact retries and conflict rejection; fail-closed KDL FMRI exclusions and protected keeps; opt-in configurable retirement with a 12-month default and no automatic purge; direct/proxy TLS with ACME HTTP-01 and Cloudflare/Gandi DNS-01; bounded OTLP traces, metrics and logs; an IPS package and SMF service; and an operator book with illumos-style installed man pages. The unfinished pkg6 client is outside the release package. Alpha, beta, RC and stable tags use one semver release workflow. Each platform builds from the immutable tag and uploads its own assets only after its gates pass; prereleases are published when both platform jobs succeed, while stable releases stay draft for operator promotion. The current tag is `v1.0.0-rc.10` (`f8c456f`). Its Solstice run 8977 passed the Linux job, 384 release-script checks, all 407 native tests, and native `.p5p` construction. Independent local Opus review found no blocker in the latest diagnostic change. Evidence through RC9: Linux release checks and all 407 OpenIndiana native release-profile tests pass. The `.p5p` installs and verifies on a disposable Hipster VM; all packaged manpages render; SMF starts the depot as `pkg6srv` and puts missing or malformed readable KDL into maintenance; the native local pkg5 client refreshes the depot catalog, receives a package, and installs/verifies/uninstalls it in a scratch image. A live package upgrade uses pkg5's documented `--no-backup-be` only in the disposable VM, preserves operator-edited KDL, delivers the new default as `.kdl.new`, and leaves the depot online. RC9 reached live `pkg uninstall`, which failed when the throwaway VM's master SMF restarter lacked its NONPERSISTENT `restarter` property group. The package must retain `disable_fmri`, so RC10 added stage-specific master-restarter checks and bounded diagnostics without weakening uninstall. RC10's fresh VM lacked that property group **before package install**; `svc.startd` and `svc.configd` were running but the master instance state was unavailable. The live package test therefore stopped at preflight, and no tagged IPS asset was published. The disposable VM image needs repair before this release gate can pass. The earlier RC1–RC8 failures and focused fixes are recorded on #41. RC1 scratch testing also exposed a real pkg5 catalog compatibility bug: pkg6 omitted the required empty `updates` field from `catalog.attrs`; RC2 corrected it. Historical Hipster 2020 client and protected snapshot-maintenance acceptance in #42 remains open. Historical sources will be copied from NFS and imported locally into one final repository; mount/access details and infrastructure work are deferred. The code-only work is ready for renewed CI once the deferred disposable-VM image repair is done. This PR is not ready for v1 merge or production cutover on current evidence. Component changes are tracked in #43, #44, #45, #47, #48, #49, #50 and #51; release acceptance is tracked in #25, #41 and #42.

forge added 7 commits 2026-09-23 08:06:24 +00:00

Keep pkg5 obsoletion visible and restrict depot HTTP to reads 059db39d44

Build tagged release candidates and package the illumos depot 4cd8aad678

Export depot telemetry over bounded OTLP HTTP 9dd36eb69f

Import pkg5 archives through atomic repository generations ffbb578475

Merge v1-release-pipeline-20260923 into the atomic pkg5 import line ... ff9567fc39

Conflict: pkg6repo/src/e2e_tests.rs. The release branch rewrote the e2e
tests as a self-contained integration target; this line had already
removed `pkg6repo obsolete-package` (ADR-004). Take the rewritten file
and drop its test_e2e_obsoleted_packages, which drove the removed
command; tests.rs still asserts the command stays gone.

Release gate: lower the pkg6repo::e2e_tests minimum to the remaining 5
tests and require the atomic importer suites (libips atomic_pkg5_import,
pkg6repo import_pkg5_cli, pkg6depotd atomic_import_http_tests) and the
read-only HTTP suite, with script self-tests for the importer suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge v1-otel-20260923 into the v1 integration line ... fae83561e2

No textual conflicts; Cargo.lock merged cleanly and passes
`cargo metadata --locked`.

Semantic reconciliation: the telemetry route classifier kept an `index`
operation label, but this line no longer routes /{publisher}/index/0
(ADR-004 read-only surface). Drop HttpOperation::Index so the label set
matches the routes; the route template now classifies as unmatched.

Release gate: require pkg6depotd::telemetry_otlp (10 tests) and raise
config_fail_closed to 11 and packaged_config to 2 for the telemetry
config tests; self-tests cover the telemetry suite as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Renumber the atomic pkg5 import ADR to 007 ... 13ba9d7708

The release pipeline (ADR-005) and OTLP export (ADR-006) decisions were
written in parallel with the atomic importer, which also took 005. Give
the importer ADR-007 and update its references (plan, CLAUDE.md,
obsoleted packages doc). Note the importer and telemetry suites in the
release process description of the test inventory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

forge added 1 commit 2026-09-23 08:38:11 +00:00

ci: Bound the report-only clippy log ... 50fa6ec237

The Linux job streamed every clippy warning with its source snippet into
the job log. Run clippy through .forgejo/scripts/clippy-report.sh
instead: diagnostics are captured as JSON and the step prints a
heartbeat, per-lint counts, one line per unique warning (capped at 300),
every error diagnostic in full and cargo's stderr tail on failure. The
step keeps clippy's exit status. A fake-cargo self-test runs first.

The release workflow does not run clippy and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

forge added 2 commits 2026-09-23 08:51:11 +00:00

Serve depot HTTPS with ACME HTTP and DNS challenges 4031e6ae7a

Merge v1-tls-20260923 into the v1 integration line ... 7729974490

Direct HTTPS with certificate files or ACME (http-01, dns-01 via
Cloudflare/Gandi) now coexists with OTLP telemetry and graceful shutdown.

- lib.rs: load_config validates telemetry and TLS, both fail closed.
  serve() installs the SIGHUP handler first, then serves plain HTTP or
  TLS; both paths use SIGTERM/SIGINT drain before the telemetry flush.
- http/server.rs: one generic serve_until drains any axum Listener;
  run_tls_until wraps it with TlsListener. The auto-merge had dropped
  the drain body from run_until.
- pkg6depotd.8: config-test exits non-zero on invalid TELEMETRY or TLS
  settings and makes no ACME request.
- Config literals in HEAD-side tests gain tls: None.
- packaged_config: TLS examples are parsed with OTLP export enabled.
- TLS ADR renumbered to ADR-008 (release 005, OTel 006, atomic 007).
- Release inventory gates tls_config=7, tls_acme=19, tls_depot_e2e=3,
  packaged_config=3 and the pkg6depotd library suite=35.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

forge added 8 commits 2026-09-23 09:53:18 +00:00

docs: add checked operator mdBook for v1 depot 328b592723

cli: reject unsupported depot and client operations 2795a7c579

cli: make repository and receive commands fail explicitly f0ec530ead

release: ship server and operator binaries in v1 tarballs 4722b69e0c

Merge reviewed v1 depot and client CLI cleanup 2dc4509748

Merge reviewed repository and receive command cleanup 92c66e7e0f

feat: policy retirement snapshots and explicit import exclusions cd22b951cb

Merge reviewed retirement and snapshot policy ... e2d7cc52bf

# Conflicts:
#	pkg6depotd/src/http/server.rs
#	pkg6depotd/src/lib.rs
#	pkg6repo/src/main.rs

forge changed title from WIP: Integrate pkg5 import, telemetry and tagged IPS releases to v1 deployable pkg6 depot: pkg5 imports, retention, TLS, telemetry and release pipeline 2026-09-23 09:53:38 +00:00

forge added 4 commits 2026-09-23 10:21:36 +00:00

test: verify pkg5 clients against installed depot package 0c17c6ce5b

build: prepare unified prerelease native gates fed23a8792

fix: preserve access gaps across interrupted depot shutdown 394c0e3244

test: verify pkg5 clients and manual pages in release package 48f8b64aea

forge referenced this pull request 2026-09-23 10:23:17 +00:00

Decide the documentation format, location and minimum content for the server release #30

forge referenced this pull request 2026-09-23 10:23:21 +00:00

V1: validate OpenIndiana Hipster clients from 2020 onward #42

forge commented 2026-09-23 11:11:30 +00:00

Author

First-time contributor

Copy link

RC1 gate update for exact tag v1.0.0-rc.1 (48f8b64): PR CI and both release-profile suites passed 403/403 tests. The Linux release job succeeded and uploaded its tarball, checksums and provenance. The illumos job built the IPS archive, then failed in Test IPS package on this throwaway VM. The Forgejo log stream stops after the passing test suite, so the failed assertion is not yet visible; native scratch-image reproduction is under way. The release remains a draft with Linux assets only. The RC tag will not be moved; a fix will use a new commit and v1.0.0-rc.2. Historical Hipster 2020 client validation is still separate and pending.

RC1 gate update for exact tag `v1.0.0-rc.1` (`48f8b64`): PR CI and both release-profile suites passed 403/403 tests. The Linux release job succeeded and uploaded its tarball, checksums and provenance. The illumos job built the IPS archive, then failed in **Test IPS package on this throwaway VM**. The Forgejo log stream stops after the passing test suite, so the failed assertion is not yet visible; native scratch-image reproduction is under way. The release remains a draft with Linux assets only. The RC tag will not be moved; a fix will use a new commit and `v1.0.0-rc.2`. Historical Hipster 2020 client validation is still separate and pending.

forge added 1 commit 2026-09-23 12:02:21 +00:00

fix: preserve pkg5 catalog and release FMRI compatibility ca42462105

forge referenced this pull request 2026-09-23 12:43:49 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge added 1 commit 2026-09-23 12:45:19 +00:00

fix: avoid Hipster pkgtree collision in IPS package ebcc192148

forge added 1 commit 2026-09-23 12:52:28 +00:00

test: match pkg5 upgrade publication and listing semantics a5d3cc5e1e

forge added 1 commit 2026-09-23 13:36:57 +00:00

test: inspect protected IPS config with root in live package test 2ff3e08cdc

forge added 1 commit 2026-09-23 14:15:19 +00:00

ci: install manual reader in disposable IPS package test VM ef92cee2fb

forge added 1 commit 2026-09-23 14:50:59 +00:00

test: avoid pipefail false failures on long package output 6df21d2ca0

forge added 1 commit 2026-09-23 15:22:31 +00:00

fix: reject invalid SMF config before marking depot online 55cb6a068b

forge added 1 commit 2026-09-23 15:53:41 +00:00

test: run live pkg5 upgrade without backup BE in Solstice 3371770f38

forge added 1 commit 2026-09-23 16:33:15 +00:00

test: handle absent pkg5 lost+found in live uninstall check ce82d704dc

forge added 1 commit 2026-09-23 17:27:56 +00:00

test: diagnose SMF restarter during live IPS lifecycle f8c456f2cd

forge referenced this pull request 2026-09-23 18:02:39 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge added 1 commit 2026-09-23 19:18:07 +00:00

test: add unprivileged pkg5 release smoke for OpenIndiana a1d715b3db

forge referenced this pull request 2026-09-23 19:19:28 +00:00

V1: IPS/SMF deployment with direct TLS, proxy mode and ACME #41

forge referenced this pull request 2026-09-23 19:19:33 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge commented 2026-09-23 19:19:37 +00:00

Author

First-time contributor

Copy link

oibuild native validation of the immutable RC10 source now passes: release workspace build (-j2, 16m 01s), pkg5 .p5p creation, 155 focused pkg6repo/pkg6depotd tests, and the new unprivileged pkg5 smoke (twice, including the reviewed cleanup/proxy fixes). Commit a1d715b adds the reusable pre-tag check and documents how to build the candidate archive from current HEAD. It does not move the RC10 tag or alter the tag-triggered release pipeline.

The Solstice live-image package test remains the release blocker. Job 22790 failed before pkg install because the fresh VM's SMF master restarter lacks its NONPERSISTENT state group. The shared oibuild zone was not changed at the OS/package/SMF level. Evidence and exact commands are in the workspace note notes/ips-v1-oibuild-rc10.md; issue #41 tracks the live gate.

`oibuild` native validation of the immutable RC10 source now passes: release workspace build (`-j2`, 16m 01s), pkg5 `.p5p` creation, 155 focused `pkg6repo`/`pkg6depotd` tests, and the new unprivileged pkg5 smoke (twice, including the reviewed cleanup/proxy fixes). Commit `a1d715b` adds the reusable pre-tag check and documents how to build the candidate archive from current `HEAD`. It does not move the RC10 tag or alter the tag-triggered release pipeline. The Solstice live-image package test remains the release blocker. Job 22790 failed before `pkg install` because the fresh VM's SMF master restarter lacks its NONPERSISTENT state group. The shared `oibuild` zone was not changed at the OS/package/SMF level. Evidence and exact commands are in the workspace note `notes/ips-v1-oibuild-rc10.md`; issue #41 tracks the live gate.

forge added 1 commit 2026-09-24 16:12:36 +00:00

chore: prepare v1.0.0-rc.11 480dbaef64

forge added 1 commit 2026-09-24 17:48:27 +00:00

ci: publish illumos release assets from oibuild temporarily dfbb618a7a

forge referenced this pull request 2026-09-24 17:53:11 +00:00

Define what 'v1.0 server release' must include and its acceptance criteria #25

forge commented 2026-09-24 17:53:18 +00:00

Author

First-time contributor

Copy link

RC11 release retry and temporary fallback are now concrete. Solstice run 8984: Linux success; illumos built and passed 407/407 release-profile tests, then was marked failed after the last visible test step (the final log does not identify the failing step). The exact annotated RC11 tag was built/tested on oibuild with 407/407 nextest, native pkg5 smoke, IPS package build, checksums and provenance. The five illumos assets were uploaded with the reviewed script from this PR, and Forgejo downloads match their local SHA-256 values. Published prerelease: https://code.aopc.cloud/OpenIndiana/ips/releases/tag/v1.0.0-rc.11 . Its body explicitly says the live-image IPS/SMF gate was not run in the oibuild validation.

Head dfbb618 adds ADR-011, suspends the slow Solstice illumos release job for future tags, and keeps branch CI full. Local validation: release script tests 443/443, native link retry tests 21/21, operator book checks, bash syntax and diff checks; independent Opus review found no blocker. Exact-head branch CI run 8985 passed both Linux and illumos jobs (407/407 tests on each platform). This PR remains open for its normal review/CI gates; the live-image/SMF validation remains a separate release acceptance gap.

RC11 release retry and temporary fallback are now concrete. Solstice run 8984: Linux success; illumos built and passed 407/407 release-profile tests, then was marked failed after the last visible test step (the final log does not identify the failing step). The exact annotated RC11 tag was built/tested on oibuild with 407/407 nextest, native pkg5 smoke, IPS package build, checksums and provenance. The five illumos assets were uploaded with the reviewed script from this PR, and Forgejo downloads match their local SHA-256 values. Published prerelease: https://code.aopc.cloud/OpenIndiana/ips/releases/tag/v1.0.0-rc.11 . Its body explicitly says the live-image IPS/SMF gate was not run in the oibuild validation. Head dfbb618 adds ADR-011, suspends the slow Solstice illumos *release* job for future tags, and keeps branch CI full. Local validation: release script tests 443/443, native link retry tests 21/21, operator book checks, bash syntax and diff checks; independent Opus review found no blocker. Exact-head branch CI run 8985 passed both Linux and illumos jobs (407/407 tests on each platform). This PR remains open for its normal review/CI gates; the live-image/SMF validation remains a separate release acceptance gap.

toasterson referenced this pull request 2026-09-27 13:10:12 +00:00

v1 server release line: RC13 source, acceptance criteria (ADR-015) and glossary #54

toasterson commented 2026-09-27 13:12:49 +00:00

Owner

Copy link

Superseded by #54, which brings the whole RC13 line (feat/historical-import, a strict superset of v1-integration-20260923) to main in one pull request, as decided in ADR-015 item 6. Nothing from this branch is lost; closing to keep one integration path.

Superseded by #54, which brings the whole RC13 line (`feat/historical-import`, a strict superset of `v1-integration-20260923`) to `main` in one pull request, as decided in ADR-015 item 6. Nothing from this branch is lost; closing to keep one integration path.

toasterson closed this pull request 2026-09-27 13:12:51 +00:00

All checks were successful

Hide checks

CI / Linux (fmt / clippy / nextest / e2e / docs) (pull_request) Successful in 6m18s

Details

CI / illumos (OpenIndiana) build + nextest (pull_request) Successful in 10m55s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

Compat/Breaking

Breaking change that won't be backward compatible

  

Crazy Idea

Something that went through my mind which is not part of the core use case but fun

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority/Critical

The priority is critical

  

Priority/High

The priority is high

  

Priority/Low

The priority is low

  

Priority/Medium

The priority is medium

  

Reviewed/Confirmed

Issue has been confirmed

  

Reviewed/Duplicate

This issue or pull request already exists

  

Reviewed/Invalid

Invalid issue

  

Reviewed/Won't Fix

This issue won't be fixed

  

Status/Abandoned

Somebody has started to work on this but abandoned work

  

Status/Blocked

Something is blocking this issue or pull request

  

Status/Need More Info

Feedback is required to reproduce issue or to continue work

  

needs-info

Waiting on reporter for more information

  

needs-triage

Maintainer needs to evaluate this issue

  

ready-for-agent

Fully specified, ready for an AFK agent

  

ready-for-human

Requires human implementation

  

wayfinder:grilling

Wayfinder decision ticket: grilling

  

wayfinder:map

Wayfinder map issue (parent of decision tickets)

  

wayfinder:prototype

Wayfinder decision ticket: prototype

  

wayfinder:research

Wayfinder decision ticket: research

  

wayfinder:task

Wayfinder ticket: task that unblocks a decision

  

wontfix

Will not be actioned

No labels

Compat/Breaking

Crazy Idea

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority/Critical

Priority/High

Priority/Low

Priority/Medium

Reviewed/Confirmed

Reviewed/Duplicate

Reviewed/Invalid

Reviewed/Won't Fix

Status/Abandoned

Status/Blocked

Status/Need More Info

needs-info

needs-triage

ready-for-agent

ready-for-human

wayfinder:grilling

wayfinder:map

wayfinder:prototype

wayfinder:research

wayfinder:task

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

OpenIndiana/ips!46

No description provided.