Skip to content

test(validator): validate real GitHub pull requests in CI e2e and integration tests - #879

Open
roychying wants to merge 1 commit into
chenghan.ying/github-validatorfrom
chenghan.ying/github-validator-e2e
Open

roychying wants to merge 1 commit into
chenghan.ying/github-validatorfrom
chenghan.ying/github-validator-e2e

Conversation

@roychying

Copy link
Copy Markdown
Contributor

Why?

The GitHub validator's unit tests run against a fake. Only the real API shows whether its GraphQL query and its reading of the response are right: field names, isRequired for a given pull request, and errors for a missing pull request. This follows the pattern of the GitHub merger's live tests.

What?

  • TestGitHubLandE2E: e2e-github-queue now sets validator: {type: github}. Every full-stack land (a single pull request and a stack) passes through the validator against the test repository before it is batched.
  • New test/integration/submitqueue/extension/validator/github, which opens throwaway pull requests via githubtestrepo and checks that:
    • an open pull request passes
    • a stack passes
    • a pull request whose head moved is rejected
    • a pull request that doesn't exist fails
  • CI: a new validator-integration-test job, added to required-checks.
    • Same gating as the merger job: SQ_TEST_REPO_TOKEN and SQ_GITHUB_TEST_REPO; it fails rather than skips when the repository opted in but the secret is missing.
    • The workflow-security note now lists it among the live tests that use that token.
  • Make target: make integration-test-submitqueue-validator.
  • Docs:
    • doc/howto/TESTING.md lists the third live suite.
    • Its token instructions add read access to checks and commit statuses, which the validator reads.
    • The validator README gains a "Live tests" section.

The test repository requires no reviews or checks, because the tests merge directly. So the live suites cover the passing path and real-API behaviour. The review and check rules are covered by the parent PR's unit tests.

If SQ_TEST_REPO_TOKEN is a fine-grained token and the test repository is private, it may need read access to checks and commit statuses. Without it, GitHub may refuse the check-related fields, the validator would fail, and the e2e land tests would end in error.

Test Plan

  • Without credentials, both suites skip (verified locally with bazel test //test/integration/submitqueue/extension/validator/...).
  • make lint check-gazelle check-mocks check-tidy pass.

Run against a real test repository (see doc/howto/TESTING.md):

SQ_GITHUB_TOKEN=$(gh auth token) SQ_GITHUB_TEST_REPO=<owner>/<repo> make integration-test-submitqueue-validator
SQ_GITHUB_TOKEN=$(gh auth token) SQ_GITHUB_TEST_REPO=<owner>/<repo> make e2e-test

Issue

@roychying
roychying added this pull request to stack #880 October 16, 2026 18:43
@roychying
roychying requested review from a team, behinddwalls and sbalabanov as code owners October 16, 2026 18:43
@roychying
roychying force-pushed the chenghan.ying/github-validator-e2e branch from 3c6d8f3 to e7868a2 Compare October 16, 2026 18:57
@roychying
roychying force-pushed the chenghan.ying/github-validator-e2e branch from e7868a2 to ecf3329 Compare October 16, 2026 18:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant