How we measure TestWatch
94% of 87 generations ended well (95% interval 87–98%), measured on 18 October 2026 with prompt gen-v26.
Ended well means TestWatch wrote a correct test, or correctly said why it couldn’t: it flagged a bug in the app, or named the reason the flow can’t be tested as asked. Every case and every trial is below, including the 5 that didn’t end well.
Results
87 trials across 29 cases on two apps. No trial was left out. The shares are rounded so they add up to 100.
- Good outcomes94%82 of 87
A correct test, or a correct diagnosis: a bug flagged, or a clear reason the flow can’t be tested as asked.
- False green1%1 of 87
The test passed when it should have failed. This is why you review each test and every fix.
- Brittle0%0 of 87
The test caught the bugs but broke on a harmless change, or passed only sometimes.
- Generation failed5%4 of 87
No usable test came out of the generation.
- False blocker0%0 of 87
Generation stopped on a blocker that wasn’t real, or skipped the flow.
Split by app and by kind of case
Desk had 9 of 72 trials in all earlier runs, so it was barely used while the prompt was tuned. That makes it the nearest thing to an unseen app here: 24 of 24 ended well.
| Split | Ended well |
|---|---|
| By app | |
| Shop (tuning app) | 58 of 63 |
| Desk (barely used while tuning) | 24 of 24 |
| By kind of case | |
| Everyday flows | 30 of 30 |
| Error paths | 5 of 6 |
| One-line requests | 5 of 6 |
| Poor markup + slow app | 3 of 3 |
| File upload/download | 12 of 12 |
| Email flows | 9 of 9 |
| Unique test data | 3 of 3 |
| Can't-run environments | 12 of 12 |
| App broken during generation | 3 of 6 |
Time per generation
A typical generation took 20 steps and 1 min 43 s (medians). These are small fixture apps; on a larger real app a generation takes more steps. Plans count the tests you keep; a failed generation takes no slot.
How a trial is judged
One trial is one generation from scratch. A trial ends well only if it passes every check below that applies to its case.
- The request
TestWatch gets the plain-English scenario and the app’s address, the same as a customer gives it.
- The working app
The test it writes must pass, and the app’s own backend must record the expected outcome, for example an order created with 2 items. A test that passes without the order being placed doesn’t count.
- Injected bugs
The same test runs against copies of the app with one bug planted in each, such as a cart that silently drops an item. It must fail on every one.
- Harmless changes
It runs against harmless changes (products in a different order, a promo banner, every class name changed) and must pass. Then it gets 3 extra clean runs to catch flakes.
- Can’t-run environments
In 4 cases the flow can’t be tested as asked: a CAPTCHA, the app down, a wrong password, a missing 2FA secret. There is no correct test, so the generation must end with the right diagnosis.
- Broken while generating
In 2 cases the app already has a bug while TestWatch writes the test. It must flag the bug. A test that passes there counts as a false green.
The apps
Shop and Desk are two apps served by one fixture server we built to test TestWatch.
- Shop
A small online store: sign-in, sign-up, products, cart, checkout with shipping and review, orders with a CSV export, a profile photo, password reset and emailed sign-in codes.
- Desk
A support desk: sign-in, a ticket list with filters, ticket details, a new-ticket dialog, CSV import, team invites and settings.
Why the cases hold up
- Randomised
Labels, prices and class names change with every trial, so a test can’t pass by repeating an earlier one.
- Hard variants
Some cases use poor accessibility markup and slow responses.
- Isolated
Every trial runs on its own copy of the app with fresh data.
- Checked in the backend
Success is read from the app’s own backend events, not from screenshots or from the test reporting itself green.
The apps’ code isn’t public. A public fixture ends up in model training data, and then it stops measuring anything. The scenarios and the planted bugs are listed in full below.
All 29 cases
Each row is one case: the request TestWatch got, the bugs its test had to catch, the changes it had to survive, and the result of every trial.
| Case | Trials | Kind | Must catch or report | Must survive | Median steps |
|---|---|---|---|---|---|
| Shop | |||||
Remove an item from the cartshop-cart-removeLog in with the test account. Add any two different products to the cart. Open the cart and remove one of the two products. Verify the removed product is gone and the other one is still in the cart. |
| Everyday flows |
|
| 40 |
Checkout with two itemsshop-checkoutLog in with the test account. Add any two different products from the products page to the cart. Open the cart and verify both products are listed. Go to checkout and fill in the shipping details (any name, address, city, postal code). On the review step, verify the total equals the sum of the two added product prices. Place the order and verify the order confirmation with an order number is shown. |
| Everyday flows |
|
| 64 |
Sign inshop-loginLog in with the test account. Verify the products page is shown after login. |
| Everyday flows |
|
| 10 |
Log outshop-logoutLog in with the test account. Log out. Verify that opening the products page now shows the login page instead. |
| Everyday flows |
|
| 11 |
Empty shipping form is rejectedshop-checkout-validationLog in with the test account. Add any product to the cart and go to checkout. Submit the shipping form without filling in any fields. Verify a validation error is shown and you stay on the shipping step. |
| Error paths |
|
| 18 |
Wrong password is refusedshop-login-invalidTry to log in with the test account's email but a wrong password. Verify an error message is shown and you are not logged in. |
| Error paths |
|
| 9 |
Buy something (one-line request)shop-checkout-vagueMake sure a customer can buy something. |
| One-line requests |
|
| 34 |
Checkout with two items on a hard appshop-checkout-hardLog in with the test account. Add any two different products from the products page to the cart. Open the cart and verify both products are listed. Go to checkout and fill in the shipping details (any name, address, city, postal code). On the review step, verify the total equals the sum of the two added product prices. Place the order and verify the order confirmation with an order number is shown.Variant: poor accessibility markup and slow responses. |
| Poor markup + slow app |
|
| 68 |
Upload any image as the profile photoshop-avatar-sampleLog in with the test account. Open the Profile page, upload any image as the profile photo and save. Verify the photo is shown. |
| File upload/download |
|
| 19 |
Upload a given profile photoshop-avatar-uploadLog in with the test account. Open the Profile page. Upload avatar.png as the profile photo and save. Verify the new photo is shown and its file name appears on the page. |
| File upload/download |
|
| 18 |
Export orders as CSVshop-export-ordersLog in with the test account. Open the Orders page. Export the orders as CSV. Verify the downloaded file contains the newest order number shown in the list. |
| File upload/download |
|
| 17 |
Sign in with an emailed codeshop-email-codeOn the sign-in page choose "Email me a sign-in code". Enter a fresh email address that can receive mail. Enter the code from the email you receive. Verify you are signed in and the products page is shown. |
| Email flows |
|
| 17 |
Reset a forgotten password by emailshop-email-resetCreate an account ("Create account" on the sign-in page) with a fresh email address that can receive mail. Use "Forgot password?" on the sign-in page to request a reset email for that address. Open the reset link from the email and choose a new password. Sign in with the new password and verify the products page is shown. |
| Email flows |
|
| 32 |
Sign up and confirm the emailshop-email-signup-verifyFrom the sign-in page, open "Create account". Sign up with a fresh email address that can receive mail and a password. Open the confirmation link from the email you receive. Sign in with the new account and verify the products page is shown. |
| Email flows |
|
| 21 |
Sign up with fresh test datashop-signup-test-dataFrom the sign-in page, open "Create account". Sign up with a new email address and a password (the email does not need to be read). Verify the "Check your email" confirmation is shown. |
| Unique test data | None |
| 13 |
Sign-in behind a bot checkshop-env-captchaLog in with the test account. Verify the product list is shown.Variant: the sign-in page is behind a CAPTCHA. |
| Can't-run environments | Stop and report that a bot check blocks the sign-in | None | 6 |
App is downshop-env-downLog in with the test account. Verify the product list is shown.Variant: every page returns 503. |
| Can't-run environments | Stop and report that the app is down | None | 5 |
Sign-in needs a 2FA codeshop-env-totpLog in with the test account. Verify the product list is shown.Variant: sign-in asks for a 2FA code and no code source is set up. |
| Can't-run environments | Stop and report that a 2FA code is needed | None | 6 |
Test account has the wrong passwordshop-env-wrong-pwLog in with the test account. Verify the product list is shown.Variant: the test account's saved password is wrong. |
| Can't-run environments | Stop and report that the test account's password is wrong | None | 6 |
Place an order while checkout is brokenshop-bug-checkoutLog in with the test account. Add any product to the cart and go to checkout. Fill in the shipping details (any name, address, city, postal code) and continue. Place the order and verify the order confirmation with an order number is shown.Variant: placing the order shows an error page during generation. |
| App broken during generation | Flag the broken checkout as an app bug instead of shipping a test | None | 51 |
Check the order total while it is wrongshop-bug-totalLog in with the test account. Add any two different products from the products page to the cart. Go to checkout and fill in the shipping details (any name, address, city, postal code). On the review step, verify the total equals the sum of the two added product prices.Variant: the order total is 1.00 too high during generation. |
| App broken during generation | Flag the wrong total as an app bug instead of shipping a test | None | 60 |
| Desk | |||||
Filter tickets by statusdesk-filterLog in with the test account. Filter the ticket list by status Closed and verify only closed tickets are listed. Filter by status Open and verify the "open tickets" count equals the number of listed tickets. |
| Everyday flows |
|
| 20 |
Invite a teammate as Admindesk-inviteLog in with the test account. Invite 'new.agent@example.com' to the team with the role Admin. Verify they appear in the team list with role Admin. |
| Everyday flows |
|
| 29 |
Turn off email notificationsdesk-settingsLog in with the test account. Open settings and turn off email notifications. Save, reload the page and verify email notifications are still off. |
| Everyday flows |
|
| 25 |
Close a ticketdesk-ticket-closeLog in with the test account. Close any open ticket using its action in the ticket list. Open that ticket and verify its status is Closed. |
| Everyday flows |
|
| 61 |
Create a ticketdesk-ticket-createLog in with the test account. Open the new-ticket dialog from the ticket list. Create a ticket titled 'Printer on fire' with priority High. Verify the ticket appears in the list with priority High and status Open. |
| Everyday flows |
|
| 22 |
Change a ticket's prioritydesk-ticket-editLog in with the test account. Open any open ticket from the ticket list. Change its priority to a different value and save. Reload the page and verify the new priority is still selected.Variant: slow responses. |
| Everyday flows |
|
| 20 |
Log a new ticket (one-line request)desk-ticket-create-vagueMake sure a support agent can log a new ticket and see it in the list. |
| One-line requests |
|
| 18 |
Import tickets from a CSV filedesk-import-csvLog in with the test account. Open the Import page and upload tickets.csv. Verify the import confirmation. Open the tickets list and verify every ticket from the file is listed. |
| File upload/download |
|
| 21 |
The 5 that didn’t end well
- Empty shipping form is rejectedGeneration failed
Trial 1. The AI provider was at capacity and the generation stopped after its retries, with no test.
- Buy something (one-line request)Generation failed
Trial 1. Hit the step limit before finishing.
- Place an order while checkout is brokenGeneration failed
Trial 3. Hit the step limit before flagging the broken order step.
- Check the order total while it is wrongFalse green
Trial 2. Shipped a test that passed on the broken app (let the wrong total through).
- Check the order total while it is wrongGeneration failed
Trial 3. Hit the step limit before flagging the wrong total.
History
3 rows, oldest first. The shares are percentages of each row’s trials. Every row is counted with today’s buckets, this run included.
| Run | Trials | Good outcomes | False green | Brittle | Generation failed | False blocker |
|---|---|---|---|---|---|---|
September 2026 (recounted)Prompts gen-v16 to gen-v21 mixed, graph off; recounted with today's buckets.Runs: 2026-09-23_15-57-14 2026-09-23_15-58-37 2026-09-23_18-25-38 2026-09-23_19-55-18 2026-09-26_11-49-45 2026-09-26_16-38-41 2026-09-26_19-37-05 2026-09-27_09-06-11 2026-09-27_09-17-49 2026-09-27_11-10-45 2026-09-27_11-12-43 2026-09-27_11-12-51 2026-09-27_11-13-11 2026-09-27_11-16-31 2026-09-27_11-19-26 | 58 | 81% | 4% | 7% | 3% | 5% |
October 2026, earlier runsPrompts gen-v22 to gen-v26 mixed, graph on and off; recounted with today's buckets.Runs: 2026-10-10_09-43-35 2026-10-12_16-13-57 2026-10-12_16-19-46 2026-10-18_11-14-53 | 14 | 100% | 0% | 0% | 0% | 0% |
October 2026 (this run)Prompt gen-v26, graph on.Run: 2026-10-18_11-32-16 | 87 | 94% | 1% | 0% | 5% | 0% |
What this doesn’t prove
- These are fixture apps we built, not your app. Real apps are larger and less tidy.
- Each app was tested in one version.
- Only 2 cases have the app broken while the test is written, so a test passing on an app that is already broken was measured on only 6 trials.
- No third-party sign-in (Google, GitHub or SSO) and no real CAPTCHA service.
Next
- An app we didn’t build: RealWorld “Conduit”, open source, tested without tuning on it.
- 10 cases where the app is broken while generating, up from 2.
- 5 trials per case instead of 3, for a narrower interval.
This page is regenerated from the next run’s data, with its date and prompt version.