GitHub Action: run TestWatch from CI

Set it up

1. In TestWatch, open Settings → API keys and create a key. Only owners and admins can. Keep both scopes ticked: runs:write starts runs, runs:read is needed to wait for the verdict. You can pin the key to one app. The key starts with tw_ and is shown once.
2. Add it to your repository as a secret named TESTWATCH_API_KEY.
3. Add a workflow. The app’s CI & deploys page in TestWatch shows one with your app id filled in.

# .github/workflows/testwatch.yml
name: TestWatch
on:
  deployment_status:      # after each deploy, against the deployed URL
  workflow_dispatch:
jobs:
  e2e:
    if: github.event_name != 'deployment_status' || github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: testwatch-dev/action@v1
        with:
          api-key: ${{ secrets.TESTWATCH_API_KEY }}
          app: <your app id>
          # environment: staging
          # tags: smoke
          # fail-on: real_bug      # only confirmed bugs fail the job
          # junit: testwatch.xml

The action waits for the final verdict: when a test fails, TestWatch first checks whether it is a real bug or a page change, and the verdict is final once that is done. Without a final verdict before timeout, the job fails with the output verdict=timeout.

Verdicts

When several tests fail, the worst verdict wins: real_bug, then needs_review, then infra.

Inputs

Use url or environment-url, not both. url is the strict one: the run is refused when the URL doesn’t match the environment’s preview patterns. environment and environment-url can’t be combined either. On a deployment_status event without environment, url or environment-url, the action uses the deployed URL, and skips the run with a notice when the deployment didn’t succeed.

Outputs

The job summary lists the failing tests with links, and each one is also an annotation.

Recipe: Vercel previews

Vercel sends a deployment_status event for every deploy, previews included. Add your preview hosts as preview patterns on the environment first, for example *-acme.vercel.app (see Preview URLs). The deployed URL is then matched to the environment by those patterns.

on:
  deployment_status:
jobs:
  e2e:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: testwatch-dev/action@v1
        with:
          api-key: ${{ secrets.TESTWATCH_API_KEY }}
          app: <your app id>

Recipe: Netlify deploy previews

Netlify doesn’t send deployment_status by default. Either add a TestWatch deploy hook to Netlify’s deploy notifications, or pass the preview URL yourself:

on:
  pull_request:
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      # ...wait for the Netlify deploy preview here...
      - uses: testwatch-dev/action@v1
        with:
          api-key: ${{ secrets.TESTWATCH_API_KEY }}
          app: <your app id>
          environment: staging
          url: https://deploy-preview-${{ github.event.number }}--acme.netlify.app

When a preview run fails

A run against a preview URL stays out of the app’s main health numbers. When a test needs a fix on that branch, TestWatch proposes a PR fix and the job summary and annotations link to it. Accept it in TestWatch, then re-run the job.

Source

The action is the TestWatch CLI wrapped for GitHub, so both behave the same way. Code: github.com/testwatch-dev/action; listing: TestWatch Playwright E2E on GitHub Marketplace. On other CI systems, use the CLI or the REST API.