Skip to content

feat: support running TypeScript benchmarks with tsx - #100

Draft
adriencaccia wants to merge 9 commits into
mainfrom
cod-3900-use-tsx-instead-of-esbuild-register-for-nodejs-typescript
Draft

adriencaccia wants to merge 9 commits into
mainfrom
cod-3900-use-tsx-instead-of-esbuild-register-for-nodejs-typescript

Conversation

@adriencaccia

Copy link
Copy Markdown
Member

Make tsx safe to benchmark with CodSpeed, so the docs can recommend it instead of the unmaintained esbuild-register, which cannot run a top-level await. Under CodSpeed, a preloaded loader turns off the esbuild options tsx forces (keepNames ran __name() calls inside the measured code, up to +80% on closure-heavy benchmarks, and hid arrow function names in flamegraphs) and the --ping of esbuild's service, whose handling every second was counted in random simulation measurements; scripts started with the tsx CLI rerun themselves with the runner's node flags, and tinybench now lets promise jobs left by module loading run before its first measurement. On Node 22.13 to 26.8, simulation instruction counts, walltime and memory results match esbuild-register.

Refs COD-3900

adriencaccia and others added 9 commits October 17, 2026 07:16
When a benchmark runs with tsx under CodSpeed, the introspected flags now
preload a loader ahead of tsx that patches the esbuild instance tsx uses:

- `keepNames` and `minifyWhitespace` are turned off for tsx's own
  transforms. `keepNames` wraps functions in `__name()` calls that run
  inside the measured code (+30% on closure-heavy benchmarks) and hide
  arrow function names from V8, so flamegraphs showed
  `anonymous function at line 2` instead of the name.
- esbuild's service processes start without `--ping`. With it, they ping
  their host thread every second, and that thread's work was counted in
  random simulation measurements.

The loader is a `--require` preload, so it runs before tsx's own preloads.
When tsx cannot be patched, it warns and leaves tsx's defaults.

Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
…mark

When the benchmark file calls `run()` while an ES module loader still has
promise jobs queued, those jobs run between the awaits of the first
measurements and add up to 250k instructions to a random early benchmark.
Draining one event loop turn first makes these runs deterministic.

Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
esbuild-register is no longer maintained. Use tsx, which the ES module
examples already use, and drop esbuild-register.

Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
Run the simple ES module example with a top-level `await bench.run()` and
add an async benchmark, the shape users write when running TypeScript
benchmarks with tsx.

Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
`tsx bench.ts` runs the script in a child node process, which gets neither
the node flags CodSpeed adds to the CLI's process nor the tsx loader. When
the plugin loads in such a process without the flags, rerun it with them,
in place with `process.execve` when available, so no parent process stays
alive during the measurements. A process reruns at most once, and never
from a worker thread or with an IPC channel to its parent.

The runner splits the introspected flags on whitespace, so flags holding a
path with spaces are left out of them; under tsx, the rerun adds them back.

Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
Refs COD-3900
Co-Authored-By: Claude <noreply@anthropic.com>
Check that tsx transforms without keepNames and minifyWhitespace, that
esbuild's service starts without `--ping`, and that a script run with the
`tsx` CLI under CodSpeed reruns once with the node flags it needs.

Refs COD-3900
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsx registers its module hooks off-thread before Node 22.22.3 and 24.11.1,
and Node only has `module.registerHooks` from 22.15, so the latest 22, 24
and 26 releases never exercise those loader paths.

Refs COD-3900
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Oct 17, 2026

Copy link
Copy Markdown

Merging this PR will regress 4 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 14 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

ℹ️ 22 benchmarks have no flame graph

The flamegraph could not be generated because too much time was spent in runtime internals, making the visualization not useful for performance analysis.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 36 improved benchmarks
❌ 4 regressed benchmarks
✅ 223 untouched benchmarks
🆕 3 new benchmarks
⏩ 57 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Memory test_recursive_cached_fibo_10 784 B 1,440 B -45.56%
❌ Simulation recursive fibo 15 350.7 µs 635.6 µs -44.83%
❌ Memory test sync baz 10 784 B 1,312 B -40.24%
❌ WallTime switch 2 84 ns 108 ns -22.22%
⚡ Memory switch 2 13,384 B 656 B ×20
⚡ WallTime test_recursive_cached_fibo_10 2,544 ns 612 ns ×4.2
⚡ WallTime test_recursive_cached_fibo_20 4 µs 1.7 µs ×2.4
⚡ WallTime test_recursive_cached_fibo_30 4.7 µs 2.2 µs ×2.1
⚡ Simulation test_recursive_cached_fibo_10 85.3 µs 45.7 µs +86.71%
⚡ Simulation recursive fibo 15 with hooks 644.1 µs 362 µs +77.94%
⚡ Simulation recursive fibo 15 with hooks 644.2 µs 362.2 µs +77.89%
⚡ ⚠️ Simulation wait 500ms 47.9 ms 27.8 ms +72.52%
⚡ ⚠️ Simulation wait 500ms 47.1 ms 28.1 ms +67.77%
⚡ ⚠️ Simulation wait 1sec 85.1 ms 56.2 ms +51.46%
⚡ ⚠️ Simulation wait 1sec 83.7 ms 56.9 ms +47.07%
⚡ Simulation test_recursive_cached_fibo_20 81.9 µs 56.7 µs +44.52%
⚡ Simulation test_recursive_cached_fibo_30 94.8 µs 69.7 µs +36.06%
⚡ Simulation test_recursive_cached_fibo_20 109.8 µs 81.8 µs +34.23%
⚡ ℹ️ Simulation test_recursive_cached_fibo_10 112 µs 83.5 µs +34.22%
⚡ Simulation test_recursive_cached_fibo_30 117.6 µs 89.6 µs +31.25%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3900-use-tsx-instead-of-esbuild-register-for-nodejs-typescript (9fab634) with main (d15b97d)

Open in CodSpeed

Footnotes

  1. 57 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant