Define the SQLite catalog load-testing plan: datasets, operations, pass criteria, where it runs #31

New issue

Open

opened 2026-09-19 19:58:58 +00:00 by toasterson · 0 comments

toasterson commented

2026-09-19 19:58:58 +00:00

Owner

Part of #24

Load testing scope correction — 2026-09-22

The maintainer rejected the proposed mandatory 24-hour pre-cutover soak. Sustained-load testing/observation can take place after cutover and is not a v1 release gate. The workload is predominantly file downloads at low concurrency; search is the main query-computation concern highlighted by the maintainer. Do not infer that ingestion/catalog maintenance requires no server computation.

Keep pre-release correctness and targeted functional checks for real OpenIndiana pkg5 clients, search, atomic batch imports, and continuous reads across stable-path catalog rename. OTel failure isolation remains a required behavior. Do not add a mandatory soak duration, synthetic concurrency target, or numeric performance threshold without a new decision. Current oirepo remains the qualitative operational baseline. This supersedes earlier proposed soak/load release gates; it does not authorize production cutover or claim tests have run.

Read availability and catalog replacement — 2026-09-22

Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (fs.rename), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage.

Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename.

Deployment and load clarification — 2026-09-22

Maintainer-provided target: oirepo6, a zone on the same existing host as oirepo, behind the same reverse proxy. Neither zone has explicit resource limits. This describes the intended deployment; no live host/resource inspection has yet been performed.

The intended eventual role is the main repository host. V1 qualification still uses the separate depot; this does not authorize production cutover. It must serve the existing client population comfortably, with performance comparable to the current service. Traffic is predominantly a few requests at a time, accumulating substantial volume over time, rather than large simultaneous bursts.

For #31/#5, use the current service as the comparative baseline and representative low-concurrency sustained traffic as the workload. Actual request-rate/concurrency distribution and resource consumption still need measurement; no numerical latency/RSS/duration budget has been agreed. Include service behavior during imports/catalog maintenance. Absence of zone caps is not an allocation of the shared host's resources; establish a safe test budget before load generation.

Maintainer decisions — 2026-09-22

OpenIndiana historical repositories and Hipster clients from approximately 2020 onward are the v1 acceptance scope (#42). OmniOS validation is deferred to v1.2. Exercise atomic multi-package p5p imports, catalog/search with hidden versions, exact-FMRI retrieval, and serving during OTel collector failure. Numerical p95 latency, concurrency, RSS/rebuild budgets, fixture sizes and execution host are still unresolved; do not invent passing thresholds.

These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion.

Earlier brief and context

Question

Ticket #5 asks to test the SQLite catalog against OpenIndiana and OmniOS repositories. Decide what a passing load test is.

Grill on:

Datasets: full OpenIndiana hipster repo, OmniOS core+extra (2254 packages used for the search work), synthetic scaled-up sets?
Operations: catalog rebuild, incremental update, /catalog/1/ full fetch, manifest fetch, search v0/v1, pkg refresh and pkg install --dry-run from a real pkg(1) client.
Pass criteria: p95 latency per endpoint, peak RSS during rebuild (the recent streaming fixes targeted OOMs), rebuild wall time, concurrent client count.
Tooling: a cargo xtask loadtest, oha/wrk scripts, or a pkg6repo bench verb; where results live (docs/perf/).
Where it runs: Solstice illumos VM (8 GB), the oibuild host, or a dedicated box; whether it is part of CI or a release checklist step.

Record the plan under docs/ai/plans/ and slice #5 from it.

Links

Informs #5

Part of #24 ## Load testing scope correction — 2026-09-22 The maintainer rejected the proposed mandatory 24-hour pre-cutover soak. Sustained-load testing/observation can take place after cutover and is not a v1 release gate. The workload is predominantly file downloads at low concurrency; search is the main query-computation concern highlighted by the maintainer. Do not infer that ingestion/catalog maintenance requires no server computation. Keep pre-release correctness and targeted functional checks for real OpenIndiana pkg5 clients, search, atomic batch imports, and continuous reads across stable-path catalog rename. OTel failure isolation remains a required behavior. Do not add a mandatory soak duration, synthetic concurrency target, or numeric performance threshold without a new decision. Current oirepo remains the qualitative operational baseline. This supersedes earlier proposed soak/load release gates; it does not authorize production cutover or claim tests have run. ## Read availability and catalog replacement — 2026-09-22 Maintainer decision: reads must remain available throughout imports and catalog rebuilds. Continue serving the previous complete catalog until the replacement is ready, then switch atomically by filesystem rename (`fs.rename`), retaining the existing on-disk catalog pathname. No routine-ingestion maintenance outage. Implementation/verification obligations (not claims about current behavior): prepare replacement on the same filesystem; preserve consistency for existing readers and new readers; ensure every newly advertised manifest/payload is available before publication. Inspect actual SQLite/shard/sidecar layout and connection lifetimes: renaming one file must not be assumed to atomically replace a multi-file catalog or safely handle live WAL/SHM state. Design a consistent publication boundary that honors the stable-path rename requirement. Verify crash durability/recovery and real pkg5 requests across the switch, including failure before and after rename. ## Deployment and load clarification — 2026-09-22 Maintainer-provided target: `oirepo6`, a zone on the same existing host as `oirepo`, behind the same reverse proxy. Neither zone has explicit resource limits. This describes the intended deployment; no live host/resource inspection has yet been performed. The intended eventual role is the main repository host. V1 qualification still uses the separate depot; this does not authorize production cutover. It must serve the existing client population comfortably, with performance comparable to the current service. Traffic is predominantly a few requests at a time, accumulating substantial volume over time, rather than large simultaneous bursts. For #31/#5, use the current service as the comparative baseline and representative low-concurrency sustained traffic as the workload. Actual request-rate/concurrency distribution and resource consumption still need measurement; no numerical latency/RSS/duration budget has been agreed. Include service behavior during imports/catalog maintenance. Absence of zone caps is not an allocation of the shared host's resources; establish a safe test budget before load generation. ## Maintainer decisions — 2026-09-22 OpenIndiana historical repositories and Hipster clients from approximately 2020 onward are the v1 acceptance scope (#42). OmniOS validation is deferred to v1.2. Exercise atomic multi-package p5p imports, catalog/search with hidden versions, exact-FMRI retrieval, and serving during OTel collector failure. Numerical p95 latency, concurrency, RSS/rebuild budgets, fixture sizes and execution host are still unresolved; do not invent passing thresholds. These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion. <details> <summary>Earlier brief and context</summary> ## Question Ticket #5 asks to test the SQLite catalog against OpenIndiana and OmniOS repositories. Decide what a passing load test is. Grill on: - Datasets: full OpenIndiana hipster repo, OmniOS core+extra (2254 packages used for the search work), synthetic scaled-up sets? - Operations: catalog rebuild, incremental update, `/catalog/1/` full fetch, manifest fetch, search v0/v1, `pkg refresh` and `pkg install --dry-run` from a real `pkg(1)` client. - Pass criteria: p95 latency per endpoint, peak RSS during rebuild (the recent streaming fixes targeted OOMs), rebuild wall time, concurrent client count. - Tooling: a `cargo xtask loadtest`, `oha`/`wrk` scripts, or a `pkg6repo bench` verb; where results live (docs/perf/). - Where it runs: Solstice illumos VM (8 GB), the `oibuild` host, or a dedicated box; whether it is part of CI or a release checklist step. Record the plan under `docs/ai/plans/` and slice #5 from it. ## Links - Informs #5 </details>

toasterson added this to the Release 1 (server only) milestone

2026-09-19 19:58:58 +00:00

toasterson added the

wayfinder:grilling

label

2026-09-19 19:58:58 +00:00

toasterson added a new dependency

2026-09-19 19:59:18 +00:00

#25 Define what 'v1.0 server release' must include and its acceptance criteria

toasterson added a new dependency

2026-09-19 19:59:44 +00:00

#5 Catalog/search checks and post-cutover load observation