How to evaluate AI testing tools: count the false greens

Every AI testing tool can show you a green run. A green run proves the test ran. It doesn’t prove the test would fail if your app broke, and that is the only reason to have the test.

A test that passes on a broken app is a false green. It is worse than no test at all: no test leaves you unsure, a false green tells you to ship. So when you trial a tool, count the false greens before anything else.

Why a passing test proves little

When a tool writes a test and then reports that it passes, the tool is grading its own work. The request was “check that checkout works”; the test it wrote may check only that a page loaded. Both pass. Only one of them would catch the bug you care about.

We learned this on our own generator. In our earliest runs (September 2026), counted with today’s rules, 4% of 58 trials were false greens: tests that went out passing and would have stayed green on a broken app. Our own pass check didn’t catch them. The checks below did.

Our definition, the same one our eval page uses: a false green is a test that passed when it should have failed. The only defence is a check that knows the right answer without asking the tool.

Score every trial with five outcomes

A single pass rate hides the difference that matters. A generation that fails is a cost you notice: you lose a few minutes and try again. A test that is wrong is a cost you don’t notice until production. Score each trial as one of five outcomes:

• Good outcomes

  A correct test, or a correct diagnosis: a bug flagged, or a clear reason the flow can’t be tested as asked.

• False green

  The test passed when it should have failed. This is why you review each test and every fix.

• Brittle

  The test caught the bugs but broke on a harmless change, or passed only sometimes.

• Generation failed

  No usable test came out of the generation.

• False blocker

  Generation stopped on a blocker that wasn’t real, or skipped the flow.

A good outcome isn’t always a test. When a flow can’t be tested as asked, a clear reason is the right answer, and a tool that writes a test anyway has done something wrong.

Use checks a demo can’t fake

A demo shows a test passing on a working app. To learn whether the test is any good, you need checks that know the right answer independently of the tool.

Read the outcome where it happens. If the flow places an order, look for the order in the app’s backend, with the right items. A test that passes while no order was created is a false green, whatever the screenshot shows.

Plant bugs. Run the finished test against copies of the app with one bug in each. It must fail on every one. Most useful are bugs that still look like success. Our eval plants 21 different bugs across 29 cases, such as these:

• adding to the cart says it worked, but the cart stays empty
• the order total is 1.00 more than the sum of the items
• saving a ticket shows "Saved", but nothing is stored

Make harmless changes. Run the same test against changes that break nothing: a promo banner added above the page; every class name and id changed; navigation links and products shown in a different order. It must still pass. It must also pass 3 clean runs in all, to catch tests that pass only sometimes.

The check that catches a planted bug is usually one line. Here is the last step of a real test TestWatch wrote for the open-source demo shop:

37await test.step('Verify order confirmation and total', async () => {
38  await expect(page.getByText('Thanks! Your order has been placed.')).toBeVisible();
39  await expect(page.getByTestId('order-total')).toHaveText('$21.00');
40});

When the shop’s total was wrong, line 39 failed the run:

Expected: "$21.00"
Received: "$16.00"

Without that line, the same test passes on the broken shop. When you read a generated test, look for the line that would fail if the outcome were wrong. If you can’t find one, you have found a false green before it ships.

Include cases that should end without a test

Two kinds of case test the judgement of a tool, not its code.

Flows that can’t run as asked. A bot check on the sign-in page, an app that is down, a wrong password, a 2FA code the tool doesn’t have. There is no correct test here; the right outcome is a clear reason, fast. In our run, 12 of 12 such trials ended with the right reason, across 4 cases.

An app that is already broken. Give the tool a request while the bug is live. A tool that writes a passing test has just declared the bug correct behaviour. It should flag the bug instead. In our run, 3 of 6 trials ended well. The 3 that didn’t are listed below, including the run’s only false green.

A worked example: our own eval run

We ran our generator against these checks on 18 October 2026: 29 cases on two fixture apps, each tried 3 times, prompt version gen-v26. Every square below is one trial.

• Good outcomes 82
• False green 1
• Brittle 0
• Generation failed 4
• False blocker 0

1. Everyday flows30 of 30 ended well
2. Error paths5 of 6 ended wellGeneration failed: 1
3. One-line requests5 of 6 ended wellGeneration failed: 1
4. Poor markup + slow app3 of 3 ended well
5. File upload/download12 of 12 ended well
6. Email flows9 of 9 ended well
7. Unique test data3 of 3 ended well
8. Can't-run environments12 of 12 ended well
9. App broken during generation3 of 6 ended wellFalse green: 1Generation failed: 2

94% of the 87 trials ended well (95% interval 87–98%). 1 was a false green (1%), 0 were brittle, 4 generations failed (5%) and 0 stopped on a false blocker. Every trial that didn’t end well:

• Generation failed

  Empty shipping form is rejected, trial 1. The AI provider was at capacity and the generation stopped after its retries, with no test.

• Generation failed

  Buy something (one-line request), trial 1. Hit the step limit before finishing.

• Generation failed

  Place an order while checkout is broken, trial 3. Hit the step limit before flagging the broken order step.

• False green

  Check the order total while it is wrong, trial 2. Shipped a test that passed on the broken app (let the wrong total through).

• Generation failed

  Check the order total while it is wrong, trial 3. Hit the step limit before flagging the wrong total.

The false green came from “Check the order total while it is wrong” (the order total is 1.00 too high during generation), in 1 of its 3 trials. When the app is already wrong while a test is written, whatever the page shows can end up as the expected value. That is the failure to look for in any tool’s output: an expected value taken from the screen rather than from the request.

The full table, with every case and every planted bug, is on How we measure TestWatch. How the generator gets there is on the AI Playwright test generator page.

Read the numbers honestly

Run each case at least three times. The same request can end differently on another try. In our run, 4 of 29 cases had trials that ended differently from each other. One trial per case would have hidden that.

Ask for an interval. 87 trials put our good-outcome rate between 87% and 98% with 95% confidence. That spread is 11 points wide, and it is the honest version of “94%”. A percentage with no trial count is a sales number.

Watch for tuning on the test app. We tuned our prompt mostly on one app, Shop: it had 63 of the 72 trials in earlier runs. Shop scored 58 of 63. Desk, which we used far less while tuning, scored 24 of 24. When a vendor shows you a number, ask which app it was tuned on.

Keep the scoring fixed. We recount older runs with today’s rules, so the rows compare. Our earliest row (September 2026) came to 81% good outcomes over 58 trials; this run came to 94%. Prompts and settings changed in between, so that shows a direction, not what caused it.

Run your own 10-flow trial

You can do this in an afternoon with any tool, on your own app.

1. Pick 10 flows that matter: six everyday ones, two error paths, and two that can’t run as asked (behind a bot check, or with a wrong password).
2. Write each as one or two plain sentences, the way someone on your team would ask.
3. Generate each one three times, and score every trial with the five outcomes above.
4. Plant one bug per flow that still looks like success: a cart that says “added” but stays empty, a total that is off by one. Every test must fail.
5. Make harmless changes: rename the CSS classes, add a banner, reorder a list. Every test must still pass, three runs in a row.
6. Read the code. Find the line that checks the outcome. Check that you can export the suite and run it without the tool.
7. Count the AI calls on a passing run. If every run calls a model, the cost and the variation come with every run. With TestWatch, passing runs make no AI calls.
8. Rank by false greens first. A tool with fewer good outcomes and no false greens is safer than one with more of both.

What this doesn’t prove

Our numbers come from two fixture apps we built, each tested in one version. Real apps are bigger and less tidy. An app already broken while the test is written was measured on only 6 trials, and there is no third-party sign-in or real bot-check service in the set. A trial on your own app, scored like this, tells you more than any vendor’s number, ours included.

Questions