Skip to content

feat: collect the OTLP data exported by the benchmarks - #589

Draft
lvaroqui wants to merge 2 commits into
mainfrom
cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet
Draft

lvaroqui wants to merge 2 commits into
mainfrom
cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet

Conversation

@lvaroqui

Copy link
Copy Markdown
Contributor

Add an experimental --experimental-otel flag (CODSPEED_EXPERIMENTAL_OTEL) that collects the OpenTelemetry data the benchmarks export, and uploads it with the results.

When the flag is set, each run part starts an OTLP/HTTP receiver on a free localhost port and passes its URL to the benchmark in CODSPEED_OTEL_ENDPOINT. The receiver accepts protobuf and JSON exports of traces, logs and metrics, plain or gzipped, on /v1/{traces,logs,metrics}.

While the benchmark runs, the receiver only writes each request body to disk as received, so it adds no compression work next to the measured process. A request is stored before it is acknowledged, so everything the process sent before exiting is kept. Once the process exits, the requests are decompressed and packed into a single otel.tar.zst in the profile folder. Compressing them as one stream with long-distance matching takes advantage of the content that repeats across export batches. Nothing is written when the benchmark exports nothing.

To put the spans and logs on the same clock as the benchmark markers, instrument-hooks is bumped for instrument_hooks_monotonic_offset: integrations timestamp with the monotonic clock shifted to read as wall clock, and record the offset in the codspeed.clock.monotonic_offset_ns resource attribute. The bindings expose it as InstrumentHooks::monotonic_offset and MONOTONIC_OFFSET_ATTRIBUTE.

tests/otel.rs runs the divan otel_example end to end through codspeed run --experimental-otel in walltime mode, and checks that every span and log record, shifted back by its resource's offset, falls within the benchmark markers. It needs a codspeed-rust checkout next to this repository, so it is ignored by default: cargo test --test otel -- --ignored. Set CODSPEED_OTEL_TEST_PROFILE_FOLDER to keep the profile folder for inspection.

Stacked on #587.

Closes COD-3863

@lvaroqui
lvaroqui added this pull request to stack #590 October 16, 2026 12:56
@codspeed

codspeed Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 8 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

✅ 37 untouched benchmarks
⏩ 10 skipped benchmarks1


Comparing cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet (33db967) with main (53c7660)

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Base automatically changed from cod-3863-chore-avoid-blocking-in-runner to main October 16, 2026 13:07
@lvaroqui
lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch from 518b466 to 1ff3345 Compare October 16, 2026 13:07
lvaroqui and others added 2 commits October 16, 2026 18:13
Add an experimental `--experimental-otel` flag (CODSPEED_EXPERIMENTAL_OTEL).
When set, each run part starts an OTLP/HTTP receiver on a free localhost
port and passes its URL to the benchmark in CODSPEED_OTEL_ENDPOINT.

The receiver accepts protobuf and JSON exports of traces, logs and
metrics, plain or gzipped. While the benchmark runs, it only writes each
request body to disk as received, so it adds no compression work next to
the measured process. Once the process exits, the requests are
decompressed and packed into a single `otel.tar.zst` in the profile
folder, uploaded with the other results. Compressing all requests as one
stream with long-distance matching takes advantage of the content that
repeats across export batches.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build the divan `otel_example` from a codspeed-rust checkout next to this
repository, run it through `codspeed run --experimental-otel` in walltime
mode, and check the profile folder holds an `otel.tar.zst` whose trace
requests contain the benchmark's spans.

The test is ignored by default since CI has no codspeed-rust checkout.
Run it with `cargo test --test otel -- --ignored`.

Refs COD-3863
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lvaroqui
lvaroqui force-pushed the cod-3863-featrunner-collect-otlp-with-rotel-and-upload-it-as-parquet branch from 1ff3345 to 33db967 Compare October 16, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant