Why Playwright tests pass locally but fail in CI

The test file is the same in both places. The machine running it isn’t. When a Playwright test passes on your laptop and fails in CI, the cause is almost always one of a short list of differences between the two machines, and most of them can be pinned in playwright.config.ts.

Below is each difference, how it breaks a test, and the setting that removes it, with Playwright’s documented defaults next to the values we chose.

Start from a config that says what it assumes

Every test suite you export from TestWatch comes with this playwright.config.ts. It is the environment the tests were generated and run in, written down, so a runner reproduces it. The rest of this post refers to its line numbers.

1import { existsSync } from 'node:fs';
2import { defineConfig, devices } from '@playwright/test';
3
4// Exported from TestWatch. Settings live in .env (copy .env.example).
5if (existsSync('.env')) process.loadEnvFile('.env');
6if (!process.env.TARGET_BASE_URL)
7  throw new Error('Set TARGET_BASE_URL (the app under test) in .env — see .env.example.');
8// The tw-<tag> prefix of unique() / testEmail() values — one per test run, so you can purge
9// everything a run created. Set here so every Playwright worker shares it.
10process.env.TW_RUN_TAG ??= 'l' + Date.now().toString(36).slice(-6) + Math.random().toString(36).slice(2, 5);
11
12// Quarantined in TestWatch (flaky or known-broken): skipped unless TW_INCLUDE_QUARANTINED=1.
13const quarantined = [
14
15];
16
17export default defineConfig({
18  testDir: './tests',
19  testIgnore: process.env.TW_INCLUDE_QUARANTINED === '1' ? [] : quarantined,
20  // Tests run one after another: they may share the same login (TARGET_EMAIL), and parallel
21  // sessions on one account can log each other out.
22  fullyParallel: false,
23  workers: 1,
24  forbidOnly: !!process.env.CI,
25  retries: 1,
26  timeout: 90_000,
27  // Web-first assertions wait up to 15 s for slow success states.
28  expect: { timeout: 15_000 },
29  reporter: [['list'], ['html', { open: 'never' }]],
30  use: {
31    baseURL: process.env.TARGET_BASE_URL,
32    // A missing element fails its own step instead of waiting out the whole test timeout.
33    actionTimeout: 30000,
34    navigationTimeout: 60000,
35    // The environment TestWatch generated and ran these tests in.
36    locale: 'en-US',
37    timezoneId: 'UTC',
38    colorScheme: 'light',
39    reducedMotion: 'reduce',
40    trace: 'retain-on-failure',
41    video: 'retain-on-failure',
42    screenshot: 'only-on-failure',
43    headless: true,
44  },
45  projects: [
46    {
47      name: 'desktop-chrome',
48      testMatch: /\.desktop-chrome\.spec\.ts$/,
49      use: { ...devices['Desktop Chrome'], viewport: { width: 1280, height: 720 } },
50    },
51  ],
52});

The clock: timezone and locale

Playwright’s timezoneId defaults to the system timezone (options reference). Your laptop runs in your timezone; most CI runners run in UTC. A test that checks “today”, a delivery date, an expiry or a date picker can pass at 3 pm on your machine and fail on the runner, or pass all day and fail after midnight UTC.

Pin it (line 37): timezoneId: 'UTC'. The locale already defaults to en-US, so dates and numbers format the same way unless someone changes it; we write it down anyway (line 36), so the assumption is in the file and not in a default.

Slower machines: timeouts

The defaults are a 30 s test timeout and a 5 s wait for each web-first assertion; actions and navigations have no timeout of their own (timeouts). A shared runner with fewer cores is slower than your laptop. A confirmation that appears after six seconds passes locally and fails there. And a click on a button that never appears waits until the whole test times out, so the error names the timeout instead of the step.

Our values:

• assertions wait up to 15 s (line 28), enough for a slow success message;
• a single action gives up after 30 s (line 33), so a missing element fails its own step;
• a navigation gets 1 min (line 34);
• the whole test gets 1 min 30 s (line 26).

Raise a budget when the app is slow, never by adding fixed sleeps.

Animations

reducedMotion defaults to no-preference, so the page plays its transitions. Playwright waits for an element to be stable before it clicks, but an assertion can still read a value while a panel is sliding in, and a busy runner changes when that happens. With reducedMotion: 'reduce' (line 39), apps that respect the user’s reduced-motion setting skip the motion, and the test sees the end state.

Parallel workers and shared state

By default Playwright uses half of the number of logical CPU cores as workers (config reference), and test files run in parallel (parallelism). Your laptop and the runner have different core counts, so they split the suite differently. Two tests that log in as the same user, fill the same cart or edit the same record pass in one arrangement and fail in the other.

There are two fixes. Make each test create its own data (clean test data shows how we do it), or run the tests one after another. Our exported config does the second (lines 22–23), because its tests share one login, and two sessions on one account can log each other out.

Retries that hide the problem

Retries default to 0. A common setup turns them on only in CI, which means the flaky test fails locally, passes on the runner’s second try, and nobody looks. Playwright marks such a test as flaky (retries); read that label as a bug report.

We use retries: 1 everywhere (line 25), the same locally and in CI, and TestWatch shows a test that only passed on its retry as “Passed · flaky”.

What only exists on your laptop

Settings in a local file. Locally the base URL and the test account come from a .env file that isn’t committed. On the runner the test starts without them and fails on the first navigation. Our config refuses to start without the address and says where to set it (lines 6–7). The exported workflow reads it from the repository’s variables and the passwords from its secrets.

Browsers without their system libraries. A fresh runner has no browser, and the browser needs operating system libraries your laptop installed long ago. Install both in one step:

npx playwright install --with-deps chromium

A forgotten test.only. It passes locally because it is the only test that ran. forbidOnly (line 24) makes the CI run fail on it instead of quietly skipping the rest of the suite.

Debug the failure you can’t reproduce

Keep the evidence of the run that failed: a trace, a video and a screenshot, only for failed tests (lines 40–42). Upload the report as a build artifact, download it, and open the trace:

npx playwright show-trace test-results/<test>/trace.zip

Then reproduce the runner’s conditions on your machine:

TZ=UTC CI=1 npx playwright test --workers=1

When the test itself is the problem

Some tests fail in CI because the code waits on time instead of on the app: a fixed waitForTimeout, waiting for networkidle, reading a value once with textContent() and then asserting on it, or branching on isVisible(). No config setting fixes those. The test passes on a fast machine and fails on a slow one. TestWatch checks the tests it generates for these shapes and sends them back for a rewrite before one is accepted; the AI Playwright test generator page explains the other checks a generated test has to pass.

What this doesn’t cover

Screenshot comparisons have their own causes, because fonts and rendering differ between operating systems; compare screenshots only on the machine that made the baseline. Third-party services your app calls can be slow or down on the runner and not on your laptop; that is a real failure, not a flaky test. And if your laptop and the runner use different Playwright versions, pin the version in package.json before anything else.

Questions