Repository navigation
Conversation
Runs the Mutants gate and Test JS's run-tests job on the current Blacksmith runner, on GitHub's runner and on several RunsOn sizes, in one workflow run, so the runners can be compared on the same commit before choosing one. Collapse each matrix back to one runner before merging. The RunsOn labels carry the two fixes cipherstash/vitaminc#487 found: an 80 GB volume, because the default root volume fills during a Rust build and rust-lld dies with SIGBUS, and the s3-cache extra, which serves actions/cache from the RunsOn stack's own bucket instead of fetching GitHub's cache over the internet. The Mutants entries also ask for the Ubuntu 22.04 image to match the Blacksmith baseline. The Mutants entries pin the c8a family. cargo-mutants builds and tests one mutant at a time, so per-core speed decides its run time, and c8a has the best x64 single-thread score in RunsOn's EC2 benchmark at about the price of m8i. The Mutants trial mutates a fixed stack-auth/stack-encrypt commit range (15 mutants) instead of the PR diff, which touches no crate source and would yield no mutants. Its mise cache is off, because a matrix runs-on names no image to key the cache by, which workflow-mise-cache-image.test.mjs requires. Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
|
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 68 | 0 | 57 | 0 |
Every mutant in the changed lines was caught by a test.
Contributor
Author
Trial results: every runner passesAll 15 trial jobs passed on the first attempt: Mutants 5 of 5, Test JS 10 of 10. They passed again on a re-run (attempt 2). The re-run gives warm-cache timings for every runner. On attempt 1 only Blacksmith had a warm cache, restored from Times are in minutes. Attempt 2 is the warm run. Mutants (mutation testing, 15 mutants)
Test JS (
|
| Runner | Instance | Node 22 cold / warm | Node 24 cold / warm |
|---|---|---|---|
| Blacksmith 4vcpu (current) | 3.8 / 3.8 | 3.7 / 3.4 | |
| RunsOn 8cpu | m8id.2xlarge, c8i-flex.2xlarge (spot) | 5.0 / 3.7 | 4.8 / 3.7 |
GitHub ubuntu-latest |
7.7 / 4.3 | 9.2 / 4.8 | |
| RunsOn 4cpu | m8id.xlarge, m8i-flex.xlarge (spot) | 6.7 / 4.8 | 6.5 / 4.6 |
| RunsOn 2cpu | m8id.large, m8i-flex.large, m8i.large (spot) | 11.7 / 7.1 | 11.2 / 6.8 |
- Warm, RunsOn 8cpu matches Blacksmith 4vcpu, and RunsOn 4cpu roughly matches GitHub.
- The cold run is slow because of the Rust binding builds. The protect-ffi binding took 1.2–4.0 minutes to build cold, and 0.0–0.1 minutes once cached.
- RunsOn's built-in
Ncpu-linux-x64runners chose a different instance family on each run (m8id, m8i-flex, c8i-flex, m8i). That makes Test JS timings noisier than the Mutants timings, which are pinned to c8a.
Not yet measured
- Cost per job. RunsOn's benchmark gives spot prices for the 2 vCPU size in us-east-1. Price per job needs this stack's actual region, spot price and RunsOn licence.
- Test JS on a pinned family. For example
family=c8a/cpu=4orcpu=8. It would give cleaner Test JS numbers. - Mutants with mise's cache on. It would show the job times a merged setup would actually get.
The first trial left two questions open. RunsOn's built-in Ncpu-linux-x64 runners chose a different instance family on every Test JS run (m8id, m8i-flex, c8i-flex, m8i), so the size comparison mixed CPU generations. Pinning family=c8a, as the Mutants entries already do, holds the CPU constant across sizes. The Mutants workload was 15 mutants, about two minutes of mutation testing, while the real gate averaged 11.9 minutes a run last week. The fb75f40..9b7e122 range of stack-auth/stack-encrypt changes yields 125 mutants, so differences between sizes show up at the scale the gate actually runs. Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
Runs each job of Test EQL, Tests (Go), Tests (crates), Test JS, the four integration suites, Fuzz (crates), Unused dependencies (crates) and CRAP gate (crates) on its current Blacksmith runner, on GitHub's runner, and on three RunsOn c8a sizes, so every job gets the same side-by-side comparison the Mutants and run-tests trials gave. Each job's sizes follow its current tier: a 16 vCPU Ubuntu 22.04 job runs on c8a 4, 8 and 16 cpu with the ubuntu22 image, and a 4 vCPU Ubuntu 24.04 job on c8a 2, 4 and 8 cpu with the ubuntu24 image. GitHub's entry names the same Ubuntu release, so every entry of a job runs one image. That is what lets the mise caches stay on. workflow-mise-cache-image now resolves a matrix runs-on to the labels of its axis and accepts the prefix when every label names the same image, reading a RunsOn label's image= segment. A RunsOn label without one names no image, because its default is stack configuration the label does not show. Fuzz's include-only matrix becomes an explicit target-by-runner cross product: a runner axis beside include rows would merge every row into every runner, and each later row would overwrite the one before. Two jobs upload an artifact other jobs download (Test EQL's nextest-archive, Go's wasm-guests). An artifact name is unique per run, so only the Blacksmith entry uploads, and every runner's consumers test that build. Left on their current runners: Test EQL's changes, setup and ci-required, which publish outputs or aggregate results; Go's macOS and Windows jobs, which RunsOn does not cover; and the schedule-only fuzz campaign, which a pull request never runs. Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
Contributor
Author
Experiments 1 and 2: pinned c8a for Test JS, 125 mutants for MutantsBoth experiments passed on every runner (commit 0f2123b). Times are in minutes. Every runner had a warm cache, except where a note says otherwise. Experiment 1: Test JS
|
| Runner | Instance | Node 22 | Node 24 |
|---|---|---|---|
| RunsOn c8a 8cpu | c8a.2xlarge (spot) | 3.1 | 2.9 |
| RunsOn c8a 4cpu | c8a.xlarge (spot) | 3.4 | 3.3 |
| Blacksmith 4vcpu (current) | 3.7 | 4.0 | |
| RunsOn c8a 2cpu | c8a.large (spot) | 5.1 | 4.9 |
GitHub ubuntu-latest |
6.5 | 6.3 |
- c8a 4cpu is about 12% faster than Blacksmith 4vcpu, with the same number of vCPUs. c8a 8cpu is about 22% faster.
- Pinning the family helped. On the warm run of the built-in runners, 4cpu (m8i-flex) took 4.6–4.8 minutes, and 2cpu (m8i) took 6.8–7.1. On c8a they took 3.3–3.4 and 4.9–5.1.
- The RunsOn S3 cache carried over from the earlier runs, so these RunsOn entries were warm on their first attempt.
Experiment 2: Mutants with 125 mutants
The trial now mutates fb75f40..9b7e122. That is 125 mutants, so the Blacksmith job now takes 12.9 minutes. That is close to the real gate's average of 11.9 minutes per run last week.
| Runner | Instance | Job | cargo-mutants step |
|---|---|---|---|
| RunsOn c8a 16cpu | c8a.4xlarge (spot) | 10.6 | 9.2 |
| Blacksmith 16vcpu (current) | 12.9 | 11.4 | |
| RunsOn c8a 8cpu | c8a.2xlarge (spot) | 14.0 | 12.2 |
| RunsOn c8a 4cpu | c8a.xlarge (spot) | 20.5 | 17.9 |
GitHub ubuntu-22.04 |
28.8 | 25.0 |
- c8a 16cpu is about 18% faster than Blacksmith 16vcpu over the whole job, and 19% faster in the mutation step.
- c8a 8cpu is about 7% slower than Blacksmith 16vcpu in the mutation step, with half the vCPUs.
- Going from 8 to 16 cpu saves about 25% of the time while doubling the vCPUs. cargo-mutants tests one mutant at a time, so more cores help less than linearly.
- mise's tool install is still uncached in this job (1.1–3.6 minutes), as noted in the PR description.
Next
The next commit extends the same comparison to every job in Test EQL, Tests (Go), Tests (crates), Test JS, the four integration suites, Fuzz (crates), Unused dependencies (crates) and CRAP gate (crates).
The GitHub entry of Shard PG17 2/4 failed nine payload_schema_tests
with "failed to read /home/runner/_work/stack/stack/packages/eql/...".
Those tests locate their schema files through
env!("CARGO_MANIFEST_DIR"), which the compiler bakes into the binary,
and the shards ran the archive Blacksmith built. Blacksmith and
RunsOn check out under _work/, GitHub-hosted runners under work/, so
only GitHub's shards looked in a directory that was not there.
Each build-archive entry now uploads nextest-archive-<runner id> and
each shard downloads its own runner's, which also times the whole
build-then-test pipeline on every runner.
Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
Contributor
Author
Phase 2 results: every job in 12 workflows, on 5 runners eachCommit 8f19afd ran 44 jobs on 5 runners each, 220 trial jobs in total. The jobs come from Test EQL, Tests (Go), Tests (crates), Test JS, the four integration suites, Fuzz (crates), Unused dependencies (crates), CRAP gate (crates) and Mutants. The runners for each job are its current Blacksmith runner, GitHub's own runner, and RunsOn c8a at three sizes. 216 of 220 jobs passed. The 4 failures are Test EQL's shards on GitHub's runner, and they come from the trial setup, not from the runner. The shards replayed a test archive that Blacksmith built. That archive has Blacksmith's checkout path compiled into it ( Summary
Options, projected over last week's volumeThe projection takes each job's cost in this run and multiplies it by the number of times that job ran last week. These 44 jobs ran 16,017 times last week.
Price per minute
By tier16 vCPU tier (Ubuntu 22.04): 22 jobs
Last week these 22 jobs ran 7,940 times for 26,292 minutes on blacksmith 16vcpu. 4 vCPU tier (Ubuntu 24.04): 22 jobs
Last week these 22 jobs ran 8,077 times for 34,250 minutes on blacksmith 4vcpu. Queue to start, and spot against on-demand
Where RunsOn was slower, and why
Method
NextPushing 4058c00 runs all twelve workflows again. Their caches are now warm on GitHub and RunsOn too, so that run gives the like-for-like comparison. I'll post its numbers here. Per job (minutes; RunsOn cost in brackets)16 vCPU tier (Ubuntu 22.04)
4 vCPU tier (Ubuntu 24.04)
|
Contributor
Author
Warm run results: all 220 trial jobs passCommit 4058c00 gives each Test EQL runner its own test archive. With it, every job passed on every runner, including the four Test EQL shards on GitHub's runner that failed on the previous run. All 44 jobs had run once before on every runner, so this run is the like-for-like comparison with warm caches. A correction to the previous comment. It said Blacksmith's caches were warm from Summary
Options, projected over last week's volume (warm run)
Recommendationc8a 8cpu for both tiers. It is the cheapest option that is faster than today on both tiers:
One job is worth keeping at 16 cpu: Mutants. It took 13.9 minutes on c8a 8cpu, against 10.7 on c8a 16cpu and 11.6 on Blacksmith. On c8a 16cpu it would cost about $0.07 a run. Before moving, two things are worth checking:
By tier (warm run)16 vCPU tier (Ubuntu 22.04): 22 jobs
Last week these 22 jobs ran 7,940 times for 26,292 minutes on blacksmith 16vcpu. 4 vCPU tier (Ubuntu 24.04): 22 jobs
Last week these 22 jobs ran 8,077 times for 34,250 minutes on blacksmith 4vcpu. Queue to start, and spot against on-demand
Per job, warm run (minutes; RunsOn cost in brackets)16 vCPU tier (Ubuntu 22.04)
4 vCPU tier (Ubuntu 24.04)
|
Contributor
Author
Indicative runner costs from this trialEvery successful Mutants gate and Test JS job on this PR across all four commits (including re-runs), grouped by runner option. Time is the median job duration; cost is the average per job. Mutants gate
Node tests (Node 22 + Node 24)
Other Test JS jobs (median time / cost per job)
* One of the two jobs fell back to an on-demand instance, which costs about 2.4× the spot price. Takeaways
How these were calculated
|
Adds c8g at 16 and 8 cpu to the Mutants gate and at 4 and 8 cpu to each Test JS job (Node 22, Node 24, E2E, Bun, WASM E2E, Lint), beside the c8a x64 entries of the same sizes, so arm64 can be compared on speed and cost one CPU against another. The family is pinned to c8g alone for the same reason c8a is. The images keep each job's Ubuntu release (ubuntu22-full-arm64 for Mutants, ubuntu24-full-arm64 for Test JS). Nothing else needed to change for the caches to stay apart by architecture: rust-cache, the mise cache and the protect-ffi index.node cache all put the runner's architecture in their keys, and the protect-ffi WASM cache holds wasm32 output, which is the same on either host. workflow-mise-cache-image now pins that an arm64 RunsOn label names its Ubuntu release like an x64 one, and that one prefix serves both. Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
family=c8g is a prefix match, so RunsOn also launched c8gd (local NVMe) and c8gn (network-optimised) instances for those entries: of the arm64 jobs on b52c3d6, the 16 cpu Mutants entry ran on c8g.4xlarge and c8gd.4xlarge, and the 4 cpu Test JS entries on c8g, c8gd and c8gn. That mixes families into a comparison meant to be c8g against c8a. RunsOn's wildcard form, family=c8g.*, matches the c8g family alone. Claude-Session: https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U
Contributor
Author
arm64 (Graviton) results: c8g against c8aCommits b52c3d6 and dd4df4d added AWS Graviton4 runners, RunsOn family
Every option, x64 and arm64, has 3 successful runs on dd4df4d. Every arm64 job there ran on a plain Summary
Median of 3 runs per option
What arm64 neededNo fixes for architecture. No arm64 job failed because of the CPU architecture. The caches already keep arm64 and x64 apart, because their keys include the architecture:
Two changes that were not architecture fixes:
Other incidents during these runs (not arm64)
|
Contributor
Author
Follow-up: arm64 (Graviton) resultsThis compares the arm64 entries against everything else, using only the runs where both architectures ran side by side: commits Short version: Graviton ( Mutants gate
On every one of the six attempts, arm64 took about twice as long as Test JS (median time / cost per job)
At the same CPU count, arm64 is 20–50% slower on everything except Lint, for about the same or higher cost. Recommendation
Notes
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. This is a trial.
Summary
Our CI runs on Blacksmith runners, which are hosted machines that GitHub Actions jobs run on. We are evaluating RunsOn as a replacement. RunsOn starts a fresh EC2 machine for each job, in our own AWS account. Before we choose a runner, we need two things: proof that our builds pass on RunsOn, and a speed comparison across machine sizes.
This PR runs two jobs on several runners in the same workflow run: the current Blacksmith runner, GitHub's own runner, and three RunsOn sizes. All the runners build the same commit, so their times are directly comparable.
Changes
mutants.yml, the mutation-testing gate for thestack-authandstack-encryptRust crates): a matrix overblacksmith-16vcpu-ubuntu-2204(current),ubuntu-22.04(GitHub) and RunsOnc8aat 4, 8 and 16 vCPU.tests.yml, jobrun-tests, the main TypeScript test suite with live encryption): a newrunneraxis overblacksmith-4vcpu-ubuntu-2404(current),ubuntu-latest(GitHub) and RunsOn2cpu,4cpuand8cpu. It runs for both Node 22 and Node 24, so there are 10 jobs.volume=80gb: the default disk fills during a Rust build, and the linker then crashes (found in ci(test): trial RunsOn runner sizes (do not merge) vitaminc#487).extras=s3-cache: the job cache is served from the RunsOn stack's own S3 bucket. Fetching GitHub's cache from AWS cost 45–100 s per job in that trial.image=ubuntu22-full-x64, to match the Ubuntu 22.04 that its Blacksmith baseline uses. RunsOn defaults to 24.04.cef0a1c..9b7e122, the 2026-10-07/08 error-message work, 15 mutants).workflow-mise-cache-image.test.mjs) requires this whenruns-oncomes from a matrix. So every entry installscargo-nextestandcargo-mutantsfrom scratch. To compare the runners themselves, use the time of the "Run cargo-mutants" step, not the job total.fail-fast: false, so a failure on one runner does not cancel the others.Verification
pnpm run test:scripts: 79 files passed, 1,387 tests passed (29 skipped). This includes the guard tests that read these two workflows: the mise cache key, the pinned Node and Postgres matrix, the binding step order and node-gyp.e2e/tests/supply-chain.e2e.test.ts: 29 tests passed.Related
Review notes
main. A re-run gives warm timings for all runners.c8a, so that each size runs on the same CPU and the timings compare cleanly. If spot capacity for c8a is short, a later change can add a fallback family (for examplefamily=c8a+c8i), at the cost of noisier timings..cargo/mutants.tomlsets nojobs). So the larger sizes may help less than their price grows. Setting--jobsis a separate experiment.Ncpu-linux-x64runners for now. The c8a choice was made for the CPU-heavy Mutants job, and the same change can be tried on Test JS later.workflow-mise-cache-image.test.mjsto read the image from a RunsOn label, so the mise cache can come back on.🤖 Generated with Claude Code
https://claude.ai/code/session_017GMWYNGir7bpQacaUzN84U