Add visual regression testing for RNTester

[robhogan]

RNTester's rendering coverage, particularly text, currently lives in Meta's internal jest-e2e suite. To move it into public CI we need screenshot comparison that works for fork PRs and doesn't add PNGs to the repo. This adds that, as a separate set of visual flows alongside the existing Maestro E2E.

How it works

Visual tests are Maestro flows in packages/rn-tester/visual-tests/flows/. Each one deep-links to an RNTester example, waits for it to be ready and captures a screenshot cropped to a testID. Maestro only captures - it never decides pass or fail.

The work is split across a trust boundary, so a fork PR never runs with credentials:

• Visual capture runs as two jobs in Test All, reusing its RNTester Release builds: test_visual_android_rntester after build_android and test_visual_ios_rntester after test_ios_rntester. Each runs every flow in one Maestro invocation, on a Pixel 7 API 35 emulator (1080x2400, 420dpi) or an iPhone 17 Pro on iOS 26.2, and uploads the PNGs and a manifest recording the commit and device environment as an artifact. They have no secrets.
• Visual Compare RNTester (workflow_run on Test All, from main) treats that artifact as data - it's validated before any image is decoded and nothing in it is executed. It finds the baseline, compares with pixelmatch, posts a check run per platform and a single PR comment with expected, captured and diff images, and on pushes to main publishes the captures as a new baseline. It runs once Test All completes, including when unrelated jobs fail, so the report arrives with the rest of CI rather than as soon as the captures are ready.
• Visual Approve RNTester (pull_request_target, on the visual-change-approved label) records a maintainer's approval of an intentional change. The approval is tied to the exact set of differences, so a later push that changes them removes the label and needs a new review.

Comparison is strict by default: any changed pixel is a difference. A test can be loosened in visual-tests/config.json, with evidence. Captures have been byte-identical across separate builds and runners so far, so I'd expect that to be rare.

What defines the baseline

A baseline is the full set of captures from a commit on main, keyed by commit and by device environment. Every push to main publishes one, and nothing is ever overwritten.

A PR is compared with the baseline for the commit it's based on (the first parent of the merge commit CI tests), or the nearest earlier commit on main that has one, and the report says which. So a PR based on last week's main is compared with last week's rendering, not with whatever has landed since. Approving a PR doesn't promote its images - main is captured again after merge and that becomes the baseline.

The environment key covers everything that affects rendering (OS runtime, device, screen size, density, font scale, locale, appearance, animations, system image and, on Android, the emulator version, which is pinned). If the CI image drifts, the key changes and a fresh baseline is published rather than every PR showing a spurious diff.

Settings

Storage is any GCS bucket, configured through repository variables. Writes use GitHub's OIDC token, so there's no stored secret, and only the two trusted workflows on main can authenticate. They need create-only access (roles/storage.objectCreator). Reads are public.

Variable
VISUAL_BUCKET	Bucket to upload to
VISUAL_PUBLIC_BASE_URL	Public URL for reads and report images - the bucket, or a CDN in front of it
VISUAL_GCP_PROJECT	GCP project for authentication
VISUAL_WORKLOAD_IDENTITY_PROVIDER	Workload identity provider trusting this repo's visual-compare-rntester.yml and visual-approve-rntester.yml on main

Expo can host the bucket and a CDN for this. Reports and approvals expire after 30 days by lifecycle rule. Baselines are small (8-175 KB per image), and at the current rate of pushes to main the cost is cents a month.

packages/rn-tester/visual-tests/README.md covers adding a test and handling a failure. This starts with two text tests and runs non-blocking. The plan is to port high-value jest-e2e text and rendering tests, then make the checks required once they've proven reliable.

Changelog:

[INTERNAL] [ADDED] - Visual regression testing for RNTester in CI

Test Plan:

Ran the full flow in a fork (https://github.com/robhogan/react-native), with Test All trimmed to the jobs this depends on, against a personal GCS bucket:

• A PR with no visual change passes on both platforms (Visual spike: chain-no-change robhogan/react-native#3).
• A deliberate text layout change fails both platforms, with expected, captured and diff images on the checks page and in the PR comment (Visual spike: chain-text-regression robhogan/react-native#4).
• Applying visual-change-approved passes the approval check. Pushing a further, different change removed the label with a comment and failed the check again.
• Captures were byte-identical across 5 Android and 4 iOS runs, from 3 separate builds on fresh runners.
• When an unrelated Test All job fails, the comparison still runs. Run against a Test All run whose Android capture had failed, it reports Android as having no capture and compares iOS as usual.
• Pixel Launcher raised an ANR dialog during emulator boot in 3 of 8 probe runs. After the capture's setup step, none of the 8 had a dialog showing.
• Running all flows in one Maestro invocation brought the iOS capture step to 2m20s for two tests, from 3m49s for one test when each flow started Maestro separately. Failed flows are retried individually.
• New unit tests for artifact validation (path traversal, symlinks, hash and size checks), comparison, storage keys, approval and report rendering.

Not yet tested: a PR from a third-party fork, to confirm it can't reach storage credentials.

[cortinico]

Thats' great - can we add some markdown in the repos to explain how to use this?

[cortinico]

We already build rn_tester in previous workflows, no? Can't we reuse the APK from that jobs?

[cortinico]

I'm a little concerned that we don't have an explicit resolution configured here and this could make the screenshots drift quite a lot once we bump this rule. Potentially we should add like profile: 'pixel' or so?

[cortinico]

@robhogan I'm having a harder time in reviewing this. Can you ask your agent to split it in smaller meaningful commits (doens't need to ne separate PRs), so it would be easier for me to do a pass?

Also my browser is unhappy with the diff size 😅