Catching the Anti-Patterns Claude Code Writes Into Your Tests

The test was green. That was the problem.

I'd asked Claude Code to add coverage for a checkout flow, it produced a tidy Playwright spec, the suite passed, and I almost merged it. Then I actually read the diff. The "test" mocked the entire payment module, asserted that a spinner appeared, and never once checked that an order was created. It was green because it tested almost nothing.

If you use Claude Code to write tests — and you should, it's genuinely good at it — this is the tax you pay. The model writes fluent, plausible, well-formatted tests that pass on the first run. Fluent and plausible is exactly what makes the bad ones dangerous. A junior's sloppy test looks sloppy. An AI's sloppy test looks like something a senior wrote, right up until you notice it can never fail.

This post is the review checklist I now run over every AI-generated test before it merges. It's the fourth in a series on using Claude Code as a working QA engineer, and it's the one I wish I'd written first.

The problem: green is not a signal

Here's the uncomfortable framing from Anastassia Voronina's "the quality gate — trusting code you don't read": when the cost of producing a test drops to near zero, the test passing tells you almost nothing. Anyone — or any model — can generate a spec that goes green. The signal was never "it passes." The signal is "it fails when the thing it's testing breaks." AI is very good at the first and indifferent to the second.

So the checklist below isn't about style. Every item is a way a test can be green and worthless at the same time. I've grouped them by how often Claude Code produces them in my own work, most common first.

1. Happy path only

The single most reliable AI test smell. Ask for "a test for the login form" and you get one test: valid credentials, redirect, done. No wrong password, no locked account, no empty field, no expired session. The bugs that reach production live in exactly those branches — the happy path is the one manual QA already covered on day one.

Catch it: count the assertions on failure states versus success states. If the ratio is zero, the test is describing the demo, not the feature.

2. Mocking the thing under test

This is the one that fooled me on checkout. The model reaches for a mock whenever a dependency is slightly inconvenient, and it doesn't distinguish between "mock the third-party email provider" (fine) and "mock the payment service whose behavior is the entire point of this test" (worthless). Over-mocking is how you get a test that verifies your mocks are configured correctly and nothing else.

Catch it: for every mock in the diff, ask "if the real thing I mocked were completely broken, would this test still pass?" If yes for the primary subject, delete the mock.

3. Ignoring existing fixtures and helpers

Claude Code operates on the file it's editing. Left alone, it will happily reinvent a loginAsAdmin helper you already have, hand-roll test data that your factory already produces, or spin up its own beforeEach that duplicates your global setup. Nothing is wrong — it's just a slow drift toward a test suite where every file does auth differently and nobody can refactor anything.

Catch it: this is a prompt fix as much as a review fix. Point the model at your fixtures explicitly (more on that below). In review, grep the diff for setup code that already exists elsewhere.

4. Hard waits dressed up as fixes

When a test is flaky, the model's instinct is to make the symptom go away, not find the cause. The Currents team documented this precisely: ask AI to fix a flaky Playwright test and it will suggest waitForTimeout(5000) before the assertion, or bump a timeout, when the real problem was a hydration delay or CI network latency. As they put it, "it adds hard waits for network issues. It changes selectors for timing bugs." The wrong-layer fix.

waitForTimeout is an anti-pattern Playwright's own docs warn against — it's a hardcoded guess that's simultaneously too slow on fast machines and too fast on slow ones. Auto-waiting assertions (await expect(locator).toBeVisible()) are the correct tool and the model knows about them; it just reaches for the sledgehammer first.

Catch it: git grep waitForTimeout in the diff, every single time. Near-zero false positives.

5. Weakened assertions and brittle locators

Two ends of the same rope. On one end, the model relaxes an assertion to make a test pass — toBeVisible() softened into a try/catch, an exact-text check turned into a substring. On the other, it grabs a locator by a CSS class or DOM position that'll shatter the next time someone touches the markup, instead of a role or a data-testid.

Catch it: any assertion wrapped in a conditional is guilty until proven innocent. For locators, if it's not user-facing (role, label, text, test id), ask why.

The solution: two gates, one for the prompt and one for the diff

You can't review your way out of this at scale — a human reading every AI diff line by line is the bottleneck AI was supposed to remove. So I run two gates. The first shapes what the model writes. The second catches what slips through, and it's automated.

Gate one: give Claude Code the standard up front

Half of these smells are prompt problems. The model writes happy-path-only tests because you asked for "a test," singular. Point it at your conventions and the output changes:

Write Playwright tests for the checkout flow in src/checkout/.

Rules:
- Use the existing fixtures in tests/fixtures/ — do NOT create new auth or
  test-data helpers. Read that folder first.
- Cover failure states, not just the happy path: declined card, expired
  session, empty cart, network error from the payment service.
- Do NOT mock the payment service — that's the subject under test. Mock only
  third-party notifications.
- No waitForTimeout. Use auto-waiting assertions.
- Locators: role, label, or data-testid only. No CSS-class or nth-child selectors.

Before writing, list which fixtures you'll reuse and which failure cases you'll cover.

That last line matters. Making the model state its plan before writing surfaces the happy-path bias while it's still cheap to correct — if the plan lists one test case, you stop it there.

Gate two: a skill that reviews the diff

The prompt reduces the smells; it doesn't eliminate them. For the rest, I keep a /review-ai-tests skill in .claude/skills/ that runs over the staged diff. The point is that a second pass — even by the same model, in a fresh context with an adversarial instruction — catches things the writing pass was blind to. This is the same idea behind Evan Marshall's warning in "Your AI Code Review Is Lying to You": AI review that just re-reads the diff and nods is theater. The review has to be looking for something specific.

---
name: review-ai-tests
description: Flag anti-patterns in AI-generated tests before merge.
---

Review the staged test diff (`git diff --cached`). You are a skeptical senior
QA engineer. For each new or changed test, check and report:

1. Assertion balance — how many assertions cover failure vs. success states?
   Flag any test with zero failure-state assertions.
2. Over-mocking — for each mock, would the test still pass if the mocked
   subject were fully broken? Flag mocks of the primary subject under test.
3. Fixture reuse — does this duplicate setup/helpers that already exist in
   tests/fixtures/ or tests/helpers/? Point to the existing one.
4. Hard waits — flag every waitForTimeout and every raised timeout.
5. Weak assertions — flag assertions inside try/catch or conditionals, and
   exact checks softened to substrings.
6. Brittle locators — flag CSS-class, nth-child, or XPath locators; suggest
   role/label/testid.

Output a table: file:line, smell, why it's a problem, suggested fix.
Do not rewrite the tests. Do not comment on style. Only these six.

Constraining it to six named smells is deliberate. An open-ended "review these tests" gets you generic praise; a closed checklist gets you a grep-with-judgment. Here's the kind of thing it catches on a real diff:

checkout.spec.ts:14  over-mocking   Mocks paymentService.charge() — the
                                    subject under test. Test passes even if
                                    charging is completely broken. Remove mock,
                                    use the sandbox payment fixture.
checkout.spec.ts:9   fixture reuse  New loginAsUser() duplicates
                                    tests/fixtures/auth.ts:loginAsUser. Reuse it.
checkout.spec.ts:22  hard wait      waitForTimeout(3000) before order assertion.
                                    Replace with await expect(orderRow).toBeVisible().

Three real problems, zero style noise, in the time it takes to stage the file.

The flake question: run it five times

One smell needs its own tool, because you can't see it in a diff at all: flakiness. A test can be perfectly written and still pass-fail-pass on identical code, and AI-fixed tests are especially prone to it because the model "fixes" flakiness by hiding it. Nael Marwan's advice — "Stop Guessing Whether Your Test Is Flaky. Run It Five Times." — is the cheapest tool in the box. Playwright ships it built in:

npx playwright test checkout.spec.ts --repeat-each=5

Five green runs is a real green. One flake in five and the test doesn't merge — the review skill can flag design smells, but only repetition exposes timing ones. I wire this into the same pre-merge step as the skill.

Honest limitations

The review skill is not a proof of correctness. It catches the six smells it's told to catch and is blind to everything else — a test can pass all six checks and still assert the wrong business rule, because judging whether the assertion is meaningful requires knowing what the feature is supposed to do, and that lives in your head, not the diff. Don't let a clean skill report talk you out of reading the assertions yourself on anything that matters.

Two more caveats. The prompt gate depends on your fixtures actually being discoverable — if your setup helpers are scattered and unnamed, the model can't reuse what it can't find, and neither can the next human. And --repeat-each catches flake that reproduces in five runs; a one-in-fifty flake will still get through, so it's a filter, not a guarantee.

None of this makes AI-written tests untrustworthy. It makes them reviewable, which is the whole game. The model turns "write the tests" from an hour into a minute; the checklist turns "trust the tests" from a leap of faith into a five-line grep and one skill.

Takeaways

• Green is not a signal. When producing a test is free, "it passes" tells you nothing. "It fails when the feature breaks" is the only thing worth trusting.
• Six smells cover most of it: happy-path-only, mocking the subject under test, ignoring existing fixtures, hard waits, weakened assertions, brittle locators.
• Two gates beat one review. Put your standard in the prompt to shape what gets written; run a closed-checklist skill over the diff to catch the rest. Make the model state its plan before writing.
• git grep waitForTimeout and --repeat-each=5 are the two cheapest, highest-value checks you own. Automate both into pre-merge.
• The skill flags design smells; only repetition finds timing ones, and only you can judge whether an assertion is meaningful. Keep the human on that last one.

Next in the series: a second model as your quality gate — wiring an independent, read-only cross-model reviewer over Claude Code's output, and where a different model catches what the same one, reviewing itself, always misses.