Preview URLs and protected staging

Protected staging

Owners and admins set protection per environment, in the environment’s edit dialog. Members can see what is set but not the secrets. Pick any of these:

These go only to the run’s exact origin: the same scheme, host and port. Third-party scripts, images and APIs on the page never see them. Redirects never carry injected headers either, even back to your own site; the next request gets through on the cookie the gate sets (Vercel and Cloudflare Access) or on Basic auth.

The same protection is used while TestWatch generates a test, when it checks a failure, and on every run. Secrets are encrypted at rest; lists in the dashboard show only the kind, the Basic username, the Cloudflare client id and header names. You can test the settings from the dialog before a run depends on them.

Known limits: a gate that accepts only a custom header (and sets no cookie) fails after a redirect, because the redirected request has no header. WebSockets and Playwright’s request fixture get no injected headers.

Preview URLs

A preview URL run uses an environment’s tests and test accounts against another host, such as a per-branch deploy. Owners and admins add up to 10 preview patterns per environment, in the same dialog. A run may target a preview URL only when its host matches one of them.

Pattern rules

• A pattern is a host. An optional https:// is removed; a port, path, query, user, IP address or non-ASCII name is refused (give international names as xn--).
• Exactly one *, in the leftmost label. It matches one or more of a-z, 0-9 and -, never a dot.
• The part after the wildcard label must be at least two labels. If it is your own domain, the pattern is fine as is.
• On a shared host such as vercel.app, netlify.app or pages.dev, the wildcard label needs at least 4 fixed characters including a letter.
• Public suffixes such as .com or .co.uk, and wildcard-DNS services that resolve any name to any IP, are refused.
• A preview URL matches when it is https on the default port, has the same number of labels, and the wildcard label fits.

*.preview.acme.com        ok: your own domain
pr-*.acme.com             ok: your own domain
*-acme.vercel.app         ok: shared host, 4+ fixed characters
*.vercel.app              refused: matches every Vercel customer
*-acme.com                refused: .com is a public suffix
*.acme.com/path           refused: a pattern is a host only

Shared hosts and secrets

Anyone can create evil-acme.vercel.app, which matches *-acme.vercel.app. So the test-account login and the protection secrets go to a preview host only when its pattern is on your own domain, or when an admin ticks “Trust shared-host patterns with secrets” on the environment. On an untrusted preview, tests that need a login finish as “login withheld” without running, and tests without a login run normally.

Starting a preview run

• Dashboard: the Run dialog’s Preview URL field.
• GitHub Action: url, or environment-url / a deployment_status event, which picks the environment by its URL or preview patterns. See GitHub Action.
• CLI: --url or --env-url. See CLI.
• REST API: baseUrl or environmentUrl on POST /v1/runs. See REST API.
• Deploy hooks: the deployed URL in the hook call. See Deploy hooks.

# GitHub Action
url: https://acme-git-signup-acme.vercel.app

# CLI
npx @testwatch/cli run --app <your app id> --env staging --url https://pr-42.acme.com --wait

# REST API
{ "appId": "<your app id>", "environment": "staging", "baseUrl": "https://pr-42.acme.com" }

Every worker checks the preview URL against the environment’s current patterns before it reads any secret, so removing a pattern takes effect at once.

Preview runs stay separate

Preview runs are left out of the app’s health numbers, the weekly digest and the “last pass” a failed result is compared with. Slack and email rules include them only when the rule has “include previews” on. Webhook channels always get their run.verdict.

PR fixes

When a test fails on a preview because the page changed on that branch, TestWatch proposes a PR fix for that branch only. Besides locators and waits, a PR fix may update the expected text in an assertion when the change is deliberate, such as a renamed button or route. It never removes, loosens or negates an assertion, and you confirm any assertion change when you accept the fix.

1. Accept the fix in TestWatch (“Accept for this PR”). Later preview runs on that branch run the fixed test, so CI goes green.
2. After the branch is merged, when the main environment fails the same way, TestWatch first replays the accepted fix against it. If it passes twice, it becomes the new version of the test, with an Undo.
3. Fixes nobody accepted are never run against your main environment.