OpenIndiana/ips

Watch

Fork 0

Decide telemetry scope for v1: which OTLP signals, exporter, and whether it gates the release #29

New issue

Closed

opened 2026-09-19 19:58:53 +00:00 by toasterson · 2 comments

toasterson commented

2026-09-19 19:58:53 +00:00

Owner

Part of #24

Maintainer decisions — 2026-09-22

V1 must ship traces, metrics and structured logs over OTLP/HTTP using the researched stack (#34). Cover request latency/errors, import outcomes/duration, catalog rebuilds and retirement/purge operations.

Collector/export failure must not block serving or imports. Bound the export queues; on overflow drop oldest queued records and count drops locally. Retention/access tracking and committed import outcomes need independent durable records and cannot rely on sampled or dropped telemetry.

Still open: collector endpoint, numerical queue limits, sampling/cardinality policy, batching/shutdown behavior and ADR/implementation plan. Signal/scope decisions are agreed; implementation is not complete.

These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion.

Earlier brief and context

Question

pkg6depotd/src/telemetry/mod.rs has a TODO: Add OTLP layer if configured; TelemetryConfig already has otlp_endpoint, service_name, log_format. Decide what v1 actually ships.

Grill on:

Signals: traces for HTTP requests (tower-http + tracing-opentelemetry), logs via OTLP, metrics (request counts, catalog rebuild duration, transaction counts)? Which are must-have for operating the production depot?
Exporter and protocol: OTLP/gRPC vs OTLP/HTTP (affects tonic/protobuf build deps on illumos), sampling defaults.
Where it goes: which collector/backend the OpenIndiana infrastructure runs.
Whether telemetry is release-blocking or a v1.1 item.

Waits on the research ticket about the crate stack. Record the outcome in the plan for #7.

Links

Informs #7

Part of #24 ## Maintainer decisions — 2026-09-22 V1 must ship traces, metrics and structured logs over OTLP/HTTP using the researched stack (#34). Cover request latency/errors, import outcomes/duration, catalog rebuilds and retirement/purge operations. Collector/export failure must not block serving or imports. Bound the export queues; on overflow drop oldest queued records and count drops locally. Retention/access tracking and committed import outcomes need independent durable records and cannot rely on sampled or dropped telemetry. Still open: collector endpoint, numerical queue limits, sampling/cardinality policy, batching/shutdown behavior and ADR/implementation plan. Signal/scope decisions are agreed; implementation is not complete. These decisions supersede conflicting earlier scope below. This update records maintainer decisions, not implementation or test completion. <details> <summary>Earlier brief and context</summary> ## Question `pkg6depotd/src/telemetry/mod.rs` has a `TODO: Add OTLP layer if configured`; `TelemetryConfig` already has `otlp_endpoint`, `service_name`, `log_format`. Decide what v1 actually ships. Grill on: - Signals: traces for HTTP requests (tower-http + tracing-opentelemetry), logs via OTLP, metrics (request counts, catalog rebuild duration, transaction counts)? Which are must-have for operating the production depot? - Exporter and protocol: OTLP/gRPC vs OTLP/HTTP (affects tonic/protobuf build deps on illumos), sampling defaults. - Where it goes: which collector/backend the OpenIndiana infrastructure runs. - Whether telemetry is release-blocking or a v1.1 item. Waits on the research ticket about the crate stack. Record the outcome in the plan for #7. ## Links - Informs #7 </details>

toasterson added this to the Release 1 (server only) milestone

2026-09-19 19:58:53 +00:00

toasterson added the

wayfinder:grilling

label

2026-09-19 19:58:53 +00:00

toasterson added a new dependency

2026-09-19 19:59:14 +00:00

#25 Define what 'v1.0 server release' must include and its acceptance criteria

toasterson added a new dependency

2026-09-19 19:59:27 +00:00

#34 Research: OTLP exporter stack for tracing-subscriber + tokio + axum, and its illumos build footprint

toasterson added a new dependency

2026-09-19 19:59:37 +00:00

#7 V1: OpenTelemetry traces, metrics and logs with bounded drop-oldest export

toasterson referenced this issue

2026-09-19 19:59:46 +00:00

Wayfinder map: Release 1 (server only) #24

forge referenced this issue

2026-09-22 12:12:14 +00:00

V1: OpenTelemetry traces, metrics and logs with bounded drop-oldest export #7

forge referenced this issue

2026-09-23 07:16:40 +00:00

WIP: Build tagged release candidates and an illumos IPS depot package #43

forge referenced this issue

2026-09-23 07:54:53 +00:00

WIP: Export bounded depot telemetry over OTLP/HTTP #44

forge commented

2026-09-23 07:55:40 +00:00

Implementation is in draft PR #44 at 9dd36eb. pkg6depotd now exports OTLP/HTTP protobuf traces, metrics and logs through bounded per-signal queues with oldest-first eviction. Collector failure stays off the request path; configured shutdown was tested against a slow collector. The packaged KDL sample and pkg6depotd(8) document configuration and limits. Local depot suite: 51 passed; independent Opus review found no release blocker. Native illumos, merged integration, and release-candidate CI remain open before this ticket can close.

Implementation is in draft PR #44 at 9dd36eb. pkg6depotd now exports OTLP/HTTP protobuf traces, metrics and logs through bounded per-signal queues with oldest-first eviction. Collector failure stays off the request path; configured shutdown was tested against a slow collector. The packaged KDL sample and pkg6depotd(8) document configuration and limits. Local depot suite: 51 passed; independent Opus review found no release blocker. Native illumos, merged integration, and release-candidate CI remain open before this ticket can close.

toasterson commented

2026-09-27 13:13:30 +00:00

Author

Owner

Resolution

Decided 2026-09-22 with the maintainer and recorded as ADR-006 (now Accepted on PR #54): v1 requires traces, metrics and structured logs over OTLP/HTTP protobuf via reqwest, with bounded per-signal queues that drop oldest on overflow and count drops; a collector outage never blocks serving or imports; durable access records for retirement are independent of telemetry. Implemented and running on oirepo6 as RC13. The remaining native verification is build work and lives in #7, which this ticket no longer blocks.

## Resolution Decided 2026-09-22 with the maintainer and recorded as **ADR-006** (now Accepted on PR #54): v1 requires traces, metrics and structured logs over OTLP/HTTP protobuf via reqwest, with bounded per-signal queues that drop oldest on overflow and count drops; a collector outage never blocks serving or imports; durable access records for retirement are independent of telemetry. Implemented and running on oirepo6 as RC13. The remaining native verification is build work and lives in #7, which this ticket no longer blocks.

toasterson closed this issue

2026-09-27 13:13:32 +00:00

toasterson referenced this issue

2026-09-27 13:14:15 +00:00

Wayfinder map: Release 1 (server only) #24