# TestWatch CLI: run tests from any CI

Source: https://testwatch.dev/docs/cli
Updated: 2026-10-17

@testwatch/cli starts a run, waits for the verdict and exits with a code your CI understands. It has no dependencies and needs Node 20 or newer. API access is on Starter and up.

## Quickstart

Create an API key in TestWatch under **Settings → API keys** (owners and admins). Keep both scopes: `runs:write` starts runs and `runs:read` lets the CLI wait for the verdict. Then:

```
export TESTWATCH_API_KEY=tw_...      # TestWatch: Settings, API keys
npx @testwatch/cli run --app <your app id> --wait --junit testwatch.xml
```

Without `--wait` the CLI starts the run, prints its link and exits `0`.

## Environment variables

| Variable | Use |
| --- | --- |
| `TESTWATCH_API_KEY` | Your API key. Required. |
| `TESTWATCH_API_URL` | The API address. Default `https://api.testwatch.dev`; leave it unset. |

## Commands

- `testwatch run [options]` starts a run of an app’s tests.
- `testwatch whoami` checks the key: which workspace it belongs to, its scopes and the app it is pinned to. It needs the `runs:read` scope.
- `--help` (`-h`) and `--version` (`-v`).

```
$ npx @testwatch/cli whoami
Workspace: Acme
Key: CI (runs:write, runs:read)
```

## Run options

| Flag | What it does |
| --- | --- |
| `--app <id>` | App id. Not needed when the key is pinned to one app. |
| `--env <name\|id>` | Environment to run against. Default: the app’s default. |
| `--env-url <url>` | A deployed URL. Matches an environment’s own URL or one of its preview patterns. Not with --env. |
| `--url <url>` | Run against this preview URL, with --env or the app’s default environment. It must match that environment’s preview patterns. Not with --env-url. |
| `--tag <tag>` | Only tests with this tag. Repeat it for more tags. |
| `--tags-mode <or\|and>` | Match any tag (or, the default) or all of them. |
| `--tests <id,id>` | Only these test ids. |
| `--account <id>` | Test account to sign in as. Default: the environment’s. |
| `--wait` | Wait for the final verdict, after failures have been checked. |
| `--timeout <dur>` | Stop waiting after this long. Default 30m; also 90s, 45m, 1h. |
| `--poll <dur>` | How often to check while waiting. Default 5s, at least 1s. |
| `--fail-on <list>` | Verdicts that fail the job. Default real_bug,needs_review,infra; "none" never fails. |
| `--junit <file>` | Write a JUnit XML report. Needs --wait. |
| `--json` | Print the final run as JSON. |
| `--idempotency-key <k>` | The same key returns the same run. Default: derived from the CI job. |
| `--commit <sha>, --branch <name>, --pr <n>` | CI details for the run. Detected automatically on GitHub Actions and GitLab CI. |
| `--cancel-on-timeout` | Cancel the run when --timeout is reached. |
| `--no-cancel-on-abort` | Keep the run going when the job is cancelled (SIGINT or SIGTERM). |

## Verdicts

| Verdict | Meaning |
| --- | --- |
| `pass` | Every test passed, flaky passes included. |
| `real_bug` | The app is broken. |
| `needs_review` | A test failed and needs a person, often because the app changed. |
| `infra` | The app or its login wasn’t usable, or the run was cancelled. |

## Exit codes

| Code | Meaning |
| --- | --- |
| `0` | Passed, or the verdict isn’t in `--fail-on`, or you didn’t wait. |
| `1` | The final verdict is in `--fail-on`. |
| `2` | A usage, auth, plan, network or timeout error. |
| `130` | Interrupted. The run is cancelled unless `--no-cancel-on-abort` is set. |

Network errors, HTTP 429 and 5xx answers are retried with backoff, honouring `Retry-After`. Other 4xx answers stop the CLI with exit code `2` and the API’s message.

## CI detection

On **GitHub Actions** the CLI reads the commit (the pull request’s head commit, not the merge commit), branch, pull request number, repository and the workflow run link. It writes a job summary, an annotation per failing test and the step outputs `run-id`, `url` and `verdict`. The ready-made [GitHub Action](https://testwatch.dev/docs/github-action) wraps the same code.

On **GitLab CI** it reads the commit, branch, merge request number and the job link. A minimal job:

```
# .gitlab-ci.yml (TESTWATCH_API_KEY set as a masked CI/CD variable)
e2e:
  image: node:22
  script:
    - npx @testwatch/cli run --app <your app id> --wait --junit testwatch.xml
  artifacts:
    when: always
    reports:
      junit: testwatch.xml
```

In both, the default idempotency key comes from the CI job and what it runs. A retried request maps to the same run; a re-run of the job, or a second step that targets something else, gets its own run. Elsewhere, pass `--commit`, `--branch` and `--pr` yourself.

## Preview URLs and PR fixes

Pass `--url` to test a preview deploy (see [Preview URLs](https://testwatch.dev/docs/preview-urls-and-protection)). When TestWatch proposes a fix for a test on that branch, the report prints a link to accept it. Accept it in TestWatch, then re-run the job; tests that ran with an accepted fix are marked in the report. The exit code doesn’t change.

Package: [@testwatch/cli on npm](https://www.npmjs.com/package/@testwatch/cli).