TestWatch CLI: run tests from any CI

Quickstart

Create an API key in TestWatch under Settings → API keys (owners and admins). Keep both scopes: runs:write starts runs and runs:read lets the CLI wait for the verdict. Then:

export TESTWATCH_API_KEY=tw_...      # TestWatch: Settings, API keys
npx @testwatch/cli run --app <your app id> --wait --junit testwatch.xml

Without --wait the CLI starts the run, prints its link and exits 0.

Environment variables

Variable	Use
`TESTWATCH_API_KEY`	Your API key. Required.
`TESTWATCH_API_URL`	The API address. Default `https://api.testwatch.dev`; leave it unset.

Commands

testwatch run [options] starts a run of an app’s tests.
testwatch whoami checks the key: which workspace it belongs to, its scopes and the app it is pinned to. It needs the runs:read scope.
--help (-h) and --version (-v).

$ npx @testwatch/cli whoami
Workspace: Acme
Key: CI (runs:write, runs:read)

Run options

Flag	What it does
`--app <id>`	App id. Not needed when the key is pinned to one app.
`--env <name\|id>`	Environment to run against. Default: the app’s default.
`--env-url <url>`	A deployed URL. Matches an environment’s own URL or one of its preview patterns. Not with --env.
`--url <url>`	Run against this preview URL, with --env or the app’s default environment. It must match that environment’s preview patterns. Not with --env-url.
`--tag <tag>`	Only tests with this tag. Repeat it for more tags.
`--tags-mode <or\|and>`	Match any tag (or, the default) or all of them.
`--tests <id,id>`	Only these test ids.
`--account <id>`	Test account to sign in as. Default: the environment’s.
`--wait`	Wait for the final verdict, after failures have been checked.
`--timeout <dur>`	Stop waiting after this long. Default 30m; also 90s, 45m, 1h.
`--poll <dur>`	How often to check while waiting. Default 5s, at least 1s.
`--fail-on <list>`	Verdicts that fail the job. Default real_bug,needs_review,infra; "none" never fails.
`--junit <file>`	Write a JUnit XML report. Needs --wait.
`--json`	Print the final run as JSON.
`--idempotency-key <k>`	The same key returns the same run. Default: derived from the CI job.
`--commit <sha>, --branch <name>, --pr <n>`	CI details for the run. Detected automatically on GitHub Actions and GitLab CI.
`--cancel-on-timeout`	Cancel the run when --timeout is reached.
`--no-cancel-on-abort`	Keep the run going when the job is cancelled (SIGINT or SIGTERM).

Verdicts

Verdict	Meaning
`pass`	Every test passed, flaky passes included.
`real_bug`	The app is broken.
`needs_review`	A test failed and needs a person, often because the app changed.
`infra`	The app or its login wasn’t usable, or the run was cancelled.

Exit codes

Code	Meaning
`0`	Passed, or the verdict isn’t in `--fail-on`, or you didn’t wait.
`1`	The final verdict is in `--fail-on`.
`2`	A usage, auth, plan, network or timeout error.
`130`	Interrupted. The run is cancelled unless `--no-cancel-on-abort` is set.

Network errors, HTTP 429 and 5xx answers are retried with backoff, honouring Retry-After. Other 4xx answers stop the CLI with exit code 2 and the API’s message.

CI detection

On GitHub Actions the CLI reads the commit (the pull request’s head commit, not the merge commit), branch, pull request number, repository and the workflow run link. It writes a job summary, an annotation per failing test and the step outputs run-id, url and verdict. The ready-made GitHub Action wraps the same code.

On GitLab CI it reads the commit, branch, merge request number and the job link. A minimal job:

# .gitlab-ci.yml (TESTWATCH_API_KEY set as a masked CI/CD variable)
e2e:
  image: node:22
  script:
    - npx @testwatch/cli run --app <your app id> --wait --junit testwatch.xml
  artifacts:
    when: always
    reports:
      junit: testwatch.xml

In both, the default idempotency key comes from the CI job and what it runs. A retried request maps to the same run; a re-run of the job, or a second step that targets something else, gets its own run. Elsewhere, pass --commit, --branch and --pr yourself.

Preview URLs and PR fixes

Pass --url to test a preview deploy (see Preview URLs). When TestWatch proposes a fix for a test on that branch, the report prints a link to accept it. Accept it in TestWatch, then re-run the job; tests that ran with an accepted fix are marked in the report. The exit code doesn’t change.

Package: @testwatch/cli on npm.