Research: OTLP exporter stack for tracing-subscriber + tokio + axum, and its illumos build footprint #34

New issue

Closed

opened 2026-09-19 19:59:04 +00:00 by toasterson · 1 comment

toasterson commented

2026-09-19 19:59:04 +00:00

Owner

Part of #24

Question

Determine the current recommended crate set for OpenTelemetry in a tokio/axum service (opentelemetry, opentelemetry_sdk, opentelemetry-otlp, tracing-opentelemetry, tower-http tracing), whether OTLP/HTTP can avoid tonic/protoc, MSRV and version compatibility with the workspace's tracing-subscriber, and the native build requirements on illumos (protobuf, TLS provider). Include a minimal wiring sketch for pkg6depotd/src/telemetry/mod.rs.

Capture as docs/research/otlp-stack.md. This feeds the telemetry scope decision.

Links

Informs #7

Part of #24 ## Question Determine the current recommended crate set for OpenTelemetry in a tokio/axum service (`opentelemetry`, `opentelemetry_sdk`, `opentelemetry-otlp`, `tracing-opentelemetry`, `tower-http` tracing), whether OTLP/HTTP can avoid tonic/protoc, MSRV and version compatibility with the workspace's `tracing-subscriber`, and the native build requirements on illumos (protobuf, TLS provider). Include a minimal wiring sketch for `pkg6depotd/src/telemetry/mod.rs`. Capture as `docs/research/otlp-stack.md`. This feeds the telemetry scope decision. ## Links - Informs #7

toasterson added this to the Release 1 (server only) milestone

2026-09-19 19:59:04 +00:00

toasterson added the

wayfinder:research

label

2026-09-19 19:59:04 +00:00

toasterson added a new dependency

2026-09-19 19:59:27 +00:00

#29 Decide telemetry scope for v1: which OTLP signals, exporter, and whether it gates the release

toasterson referenced this issue

2026-09-19 19:59:46 +00:00

Wayfinder map: Release 1 (server only) #24

forge referenced this issue from a commit

2026-09-19 20:10:26 +00:00

research: OTLP exporter stack for pkg6depotd (wayfinder #34)

toasterson commented

2026-09-19 20:10:27 +00:00

Author

Owner

Resolution

Latest: opentelemetry / _sdk / -otlp / -appender-tracing 0.33.0 (2026-09-18, MSRV 1.75); tracing-opentelemetry 0.33.0 (May 2026) targets OTel 0.32; PR #268 for OTel 0.33 merged 2026-09-19 but is unreleased. axum-tracing-opentelemetry 0.39.1 needs OTel 0.32 + tracing-otel 0.33, MSRV 1.91.
Matrix: tracing-otel 0.30→otel 0.29, 0.31→0.30, 0.32→0.31 (= our Cargo.lock), 0.33→0.32. tracing-otel 0.33 needs tracing-subscriber ≥0.3.22 (we have exactly 0.3.22).
Gotcha: opentelemetry-otlp/-http 0.32 and 0.33 require reqwest ^0.13; the workspace pins reqwest 0.12 in four crates. So today only two consistent sets exist: (A) OTel 0.31 + tracing-otel 0.32 + reqwest 0.12 (current lock), or (B) OTel 0.32 + tracing-otel 0.33 + reqwest 0.13. The hyper-client feature is the reqwest-free escape hatch.
OTLP/HTTP is tonic- and protoc-free: http-proto enables only opentelemetry-proto/gen-tonic-messages (= prost); the generated code is checked into the repo (no build.rs). tonic 0.14/prost currently compile solely because of our grpc-tonic feature. The tls-ring/tls-aws-lc exporter features only apply to tonic. Batch processors use their own thread since SDK 0.28, so rt-tokio is unnecessary.
illumos: nothing new. aws-lc-sys already needs bindgen+libclang+cmake there (handled in Cargo.toml + ci.yml); no protobuf package needed. Side finding: reqwest 0.12's rustls-tls pulls ring while axum-server/jsonwebtoken pull aws-lc-rs, so both rustls providers are compiled in and rustls's ClientConfig/ServerConfig::builder() panics unless install_default() is called; pkg6depotd never calls it (latent, since it serves plain HTTP today). reqwest 0.13 switches to aws-lc-rs and fixes the duplication.
Recommendation: for #34 stay on set (A) and change opentelemetry-otlp to default-features=false, features=["http-proto","reqwest-client","trace","logs","internal-logs"], add opentelemetry-appender-tracing 0.31, drop grpc-tonic/rt-tokio; wire SdkTracerProvider + SdkLoggerProvider with batch HTTP exporters ({endpoint}/v1/traces, /v1/logs), filter hyper/h2/reqwest/opentelemetry off the log bridge to avoid feedback loops, return a guard whose shutdown() runs after graceful axum::serve, and initialise telemetry after daemonize() (batch threads don't survive the fork). Metrics: same pattern via MetricExporter/PeriodicReader, deferred. Next dep sweep: reqwest 0.13, then OTel 0.32/0.33 + axum-tracing-opentelemetry; install the aws-lc-rs provider explicitly in main.

Full findings: docs/research/otlp-stack.md on branch research/otlp-stack (research branch, not merged; cherry-pick the file when the decision it feeds is recorded).

## Resolution - Latest: opentelemetry / _sdk / -otlp / -appender-tracing **0.33.0** (2026-09-18, MSRV 1.75); tracing-opentelemetry **0.33.0** (May 2026) targets OTel **0.32**; PR #268 for OTel 0.33 merged 2026-09-19 but is unreleased. axum-tracing-opentelemetry 0.39.1 needs OTel 0.32 + tracing-otel 0.33, MSRV 1.91. - Matrix: tracing-otel 0.30→otel 0.29, 0.31→0.30, 0.32→0.31 (= our Cargo.lock), 0.33→0.32. tracing-otel 0.33 needs tracing-subscriber ≥0.3.22 (we have exactly 0.3.22). - Gotcha: opentelemetry-otlp/-http 0.32 and 0.33 require **reqwest ^0.13**; the workspace pins reqwest 0.12 in four crates. So today only two consistent sets exist: (A) OTel 0.31 + tracing-otel 0.32 + reqwest 0.12 (current lock), or (B) OTel 0.32 + tracing-otel 0.33 + reqwest 0.13. The `hyper-client` feature is the reqwest-free escape hatch. - OTLP/HTTP is tonic- and protoc-free: `http-proto` enables only `opentelemetry-proto/gen-tonic-messages` (= prost); the generated code is checked into the repo (no build.rs). tonic 0.14/prost currently compile solely because of our `grpc-tonic` feature. The `tls-ring`/`tls-aws-lc` exporter features only apply to tonic. Batch processors use their own thread since SDK 0.28, so `rt-tokio` is unnecessary. - illumos: nothing new. aws-lc-sys already needs bindgen+libclang+cmake there (handled in Cargo.toml + ci.yml); no protobuf package needed. Side finding: reqwest 0.12's `rustls-tls` pulls **ring** while axum-server/jsonwebtoken pull **aws-lc-rs**, so both rustls providers are compiled in and rustls's `ClientConfig/ServerConfig::builder()` panics unless `install_default()` is called; pkg6depotd never calls it (latent, since it serves plain HTTP today). reqwest 0.13 switches to aws-lc-rs and fixes the duplication. - Recommendation: for #34 stay on set (A) and change `opentelemetry-otlp` to `default-features=false, features=["http-proto","reqwest-client","trace","logs","internal-logs"]`, add `opentelemetry-appender-tracing 0.31`, drop `grpc-tonic`/`rt-tokio`; wire SdkTracerProvider + SdkLoggerProvider with batch HTTP exporters (`{endpoint}/v1/traces`, `/v1/logs`), filter `hyper/h2/reqwest/opentelemetry` off the log bridge to avoid feedback loops, return a guard whose `shutdown()` runs after graceful `axum::serve`, and initialise telemetry after `daemonize()` (batch threads don't survive the fork). Metrics: same pattern via `MetricExporter`/`PeriodicReader`, deferred. Next dep sweep: reqwest 0.13, then OTel 0.32/0.33 + axum-tracing-opentelemetry; install the aws-lc-rs provider explicitly in `main`. **Full findings:** [docs/research/otlp-stack.md](https://code.aopc.cloud/OpenIndiana/ips/src/branch/research/otlp-stack/docs/research/otlp-stack.md) on branch `research/otlp-stack` (research branch, not merged; cherry-pick the file when the decision it feeds is recorded).

toasterson closed this issue

2026-09-19 20:10:29 +00:00

forge referenced this issue

2026-09-22 12:11:46 +00:00

Decide telemetry scope for v1: which OTLP signals, exporter, and whether it gates the release #29

toasterson referenced this issue

2026-09-27 13:14:15 +00:00

Wayfinder map: Release 1 (server only) #24