Your Agent Doesn't Know What "Skipped" Means Either

A triage agent I set up to read nightly runs gave me a summary I was very happy with for about four minutes:

41 passed, 0 failed, 6 skipped. Nothing to investigate — the skips are intentional.

Zero failed was true. "The skips are intentional" was not. Four of those six were tests that Playwright never got to run, because an earlier test in the same test.describe.serial group had blown up. The agent read the word skipped, matched it against everything the word means in English, and filed the whole bucket under "somebody decided this doesn't need to run."

Here's the thing that bothered me: the agent wasn't hallucinating. It read the report correctly. The report said skipped. The agent's mistake was assuming that word means one thing — which is exactly the assumption every one of us makes until the day it costs us a release.

This post is about the layer I now put between the test report and the agent, and why "just prompt it better" doesn't fix this.

Four runners, four vocabularies, zero agreement

I went and read the primary sources instead of trusting my memory. It's worse than I thought.

Playwright. A single test result carries one of five statuses (TestResult.status):

"passed" | "failed" | "timedOut" | "skipped" | "interrupted"

skipped covers a test you excluded with test.skip() or test.fixme(). It also covers the serial-mode cascade: the retries docs state that in serial mode, when a test fails and retries are off, all tests after the failure are skipped entirely. One word, two situations with opposite meanings — one is a decision you made, the other is damage from a failure.

And interrupted? That's the run being cut short — by --max-failures, or by you hitting Ctrl-C. It is a very calm-sounding word for "this suite never finished."

Cypress. Cypress splits the exact case Playwright merges, using the exact word Playwright doesn't. From Writing and Organizing Tests:

• A pending test is one "Cypress intentionally doesn't run because you told it not to" — no body, it.skip()/xit(), or excluded via the browser option.
• A skipped test is one Cypress "*meant* to run but couldn't because a shared hook failed."

Read that twice. In Cypress, skipped is a failure symptom. It's arguably the most urgent status in the whole report — a broken beforeEach can silently vaporize an entire describe block. In Playwright, skipped mostly means "we're fine, this was on purpose."

The Module API exposes these as separate counters, totalPending and totalSkipped. So if you're summing "not-run" tests across runners, you're adding two numbers that mean different things to one number that means neither.

Jest. Jest ships seven statuses. Not five, not four — seven. Straight from AssertionResult in @jest/types:

"passed" | "failed" | "skipped" | "pending" | "todo" | "disabled" | "focused"

focused is a status, which is to say Jest will tell you a test passed and that somebody left a .only in the file. That's useful. What Jest will not tell you, anywhere in those seven, is that a test was flaky — even though jest.retryTimes() exists and will happily retry a failing test until it passes. A test that failed twice and passed on the third attempt reports as passed. Same string as a test that has never failed in its life.

The trap I actually walked into: two status fields, one file

The runner-vs-runner mismatch is the famous problem. The one that bit me lives inside a single Playwright report.

Playwright has a second status vocabulary, on TestCase.outcome():

"skipped" | "expected" | "unexpected" | "flaky"

The type definitions are explicit that this is not the same thing as a result status:

"Note that outcome is not the same as testResult.status: — Test that is expected to fail and actually fails is 'expected'. — Test that passes on a second retry is 'flaky'."

Now open the JSON reporter source. _serializeTest() writes status: test.outcome(). _serializeTestResult() writes status: result.status. Both fields are called status. They are one nesting level apart:

{
  "tests": [{
    "status": "unexpected",        // <- outcome() vocabulary
    "results": [
      { "status": "failed",  "retry": 0 },   // <- TestResult vocabulary
      { "status": "passed",  "retry": 1 }
    ]
  }]
}

Three consequences, all of which I've now seen an agent walk into:

1. Grep for "status": "failed" and you'll never match at the test level. A failing test says unexpected there. If your agent (or your jq one-liner) is scanning at the wrong depth, your failure count is zero.
2. "status": "expected" can mean a test that failed. That's what test.fail() does — the failure is the assertion. An agent told to "look at anything not expected" will skip right past it, and will also skip past the far more interesting case: a test.fail() test that started passing, which is a real regression signal reported as unexpected.
3. flaky exists only at the test level, and only if retries are on. With retries: 0, a flaky test is just a failure. Turn retries on and the same test moves buckets without a single line of code changing.

So: same word, different meanings across runners. Same word, different meanings within one file. This is not a vocabulary problem you can prompt your way out of.

Why this hits agents harder than it hits you

You have the same bug. You read "skipped" and relax. The difference is that you've been burned by a cascading beforeEach failure at least once, and now, in your repo, on your suite, you double-check.

An agent's prior comes from every repo on the internet at once. "Skipped" means "deprioritize" in the aggregate, and the aggregate is what it brings to your report. Worse, the failure is silent and confident: you don't get an error, you get a clean summary with a number in it. If you've read my earlier posts in this series, you'll recognize the pattern — the expensive bug in agent tooling is almost never a crash. It's a successful-looking answer that's pointed at the wrong thing.

The instinct is to write a longer prompt. "Remember that in Cypress, skipped means a hook failed..." I tried it. It works most of the time, which is the worst possible outcome — it's just reliable enough that you stop checking, and it degrades the moment the report gets long, or somebody switches runners in one package of the monorepo.

Semantics that a script can determine should not be left to a model to infer. That's the whole lesson.

The fix: normalize before the agent ever sees it

One vocabulary, deterministic, ~60 lines. Here's the target:

blocked is the one that matters, and no runner has it. It's Cypress's skipped, and it's Playwright's serial cascade, and in both cases it's currently wearing the same label as "we don't care about this test." Splitting it out is 90% of the value here.

The Playwright adapter, working from the JSON report:

// scripts/normalize-results.mjs
function fromPlaywright(report) {
  const out = [];
  walk(report.suites, spec => {
    for (const test of spec.tests) {
      const attempts = test.results ?? [];
      const ran = attempts.filter(r => r.status !== 'skipped');

      // Nothing ever executed: decide *why* by looking at annotations,
      // not at the word "skipped".
      if (ran.length === 0) {
        const kinds = (test.annotations ?? []).map(a => a.type);
        out.push({
          id: `${spec.file}:${spec.line} ${spec.title}`,
          status: kinds.includes('skip') || kinds.includes('fixme')
            ? 'skipped-by-code'
            : 'blocked',   // no annotation + never ran => serial cascade / interrupted
        });
        continue;
      }

      const last = ran.at(-1);
      const failedEarlier = ran.slice(0, -1).some(r => r.status !== 'passed');
      const bad = s => s === 'failed' || s === 'timedOut';

      out.push({
        id: `${spec.file}:${spec.line} ${spec.title}`,
        status:
          last.status === 'interrupted' ? 'blocked'
          : test.expectedStatus === 'failed' && bad(last.status) ? 'passed'  // test.fail() did its job
          : bad(last.status) ? 'failed'
          : failedEarlier ? 'flaky-retried'
          : 'passed',
        attempts: ran.length,
      });
    }
  });
  return out;
}

Two details worth calling out, because they're where I got it wrong the first time:

• I derive flaky-retried from the attempt array, not from test.outcome(). Same answer today, but it survives retries: 0, and the identical logic drops onto Jest — which, as established, will never hand you the word "flaky."
• I never read tests[].status. Once you know there are two status fields in that file, the safe move is to stop touching the ambiguous one.

Cypress is shorter, and reads almost like a bug report:

const CYPRESS = {
  passed: 'passed',
  failed: 'failed',
  pending: 'skipped-by-code',  // you told it not to run
  skipped: 'blocked',          // it wanted to run; a shared hook died
};

Jest, mapping all seven:

const JEST = {
  passed: 'passed',
  failed: 'failed',
  skipped: 'skipped-by-code',
  pending: 'skipped-by-code',
  todo: 'skipped-by-code',
  disabled: 'skipped-by-code',
  focused: 'passed',           // ...and raise a separate alarm about the stray .only
};

skipped-by-filter is the one that costs you an extra step, and it's the one place I'll tell you to check your own setup rather than take my word: how a filtered-out test shows up in the report — as skipped, or as nothing at all — is not something the CLI docs spell out, and it can differ by reporter. Either way the robust detection is the same: get the full inventory with npx playwright test --list --reporter=json, and set-difference it against what ran. Wire it up the first time somebody's shard config quietly stops running a third of the suite.

Handing it to Claude Code

The mapper is useless if the agent can still reach around it. So the report never enters the context window:

---
name: test-triage
description: Triage a CI test run. Use whenever asked to look at test results,
  a failing run, or a nightly report.
---

# Test triage

## Non-negotiable
Never read the raw runner report. Run:

    node scripts/normalize-results.mjs <path-to-report> > /tmp/normalized.json

and work only from that. Runner-native statuses ("skipped", "pending",
"expected", "unexpected") are ambiguous across runners and across nesting
levels within one report. The normalizer resolves them; you do not.

## Statuses and what each one means for you
- `failed`         — triage: read the error, find the last passing commit.
- `blocked`        — an upstream failure prevented this from running.
                     Do NOT report as intentionally skipped. Find the cause first;
                     one `blocked` cluster is usually one root cause.
- `flaky-retried`  — passed on retry. Report separately from `failed`, never
                     merge into the pass count.
- `skipped-by-code`— intentional. Report the count and the trend, nothing else.
- `skipped-by-filter` — intentional *if* the filter was intentional. Say which
                     filter excluded it.

## Output
Group by root cause, not by file. Always state how many tests each root cause
blocked.

If you want the hard version, add a PreToolUse hook that rejects a Read on **/test-results*.json and points at the normalizer. Guardrails that live in the repo beat guardrails that live in a prompt, because the repo's version is versioned, reviewable, and applies to every session including the ones you didn't start.

Run the A/B yourself — it takes ten minutes

I'm not going to publish numbers from my suite and ask you to trust them. Run this instead; it's the version of this experiment I'd actually believe. The prompt, used verbatim in both halves:

Triage this test run. Group the problems by root cause and tell me what to fix first. Then give me a one-line summary I can paste into Slack.

1. Take a real failing run. Ideally one with a test.describe.serial block where the first test fails, or a Cypress spec with a broken beforeEach — that's where the whole effect lives.
2. Give Claude Code the raw report plus that prompt. Save the Slack line.
3. Clear the context. Run the normalizer, then give it /tmp/normalized.json plus the same prompt.
4. Diff the two Slack lines.

What I'd expect you to see — and what I did see — is that the raw-report run produces a summary whose shape is right and whose counts are wrong: cascade-skips absorbed into "intentionally skipped," a retried-and-passed test counted as a clean pass, and a root-cause ordering that follows file order rather than blast radius. The normalized run gets the counts right for a boring reason: it isn't inferring them.

Do it on your own suite. If your suite has no serial blocks and no shared hooks, you genuinely have less of this problem, and you should know that about yourself.

Where this doesn't help

Honest limitations:

• The mapper is now a thing that can be wrong. It's parsing logic in your triage path, and it deserves unit tests with fixture reports — including a serial cascade and a test.fail(). I'd rather debug a mapper with tests than a prompt without them, but it's not free.
• skipped-by-filter needs a second command. Full inventory plus a set difference. Skip it until the day it bites you, but know it's the gap.
• These vocabularies are versioned. Everything above I verified against current primary sources; runners add statuses. Pin the version you tested against, and re-read the type definitions when you upgrade.
• The idea came from a vendor blog, and the details didn't survive intact. Currents published a status-vocabulary review at the end of July that made me go look. Its framing is that Playwright and Cypress use "skipped" for opposite cases. Reading both sets of primary docs, I'd put it differently: Cypress splits the two cases into pending and skipped, while Playwright uses skipped for both. That's a meaningfully different problem — Playwright's ambiguity is unresolvable from the status alone, which is exactly why my adapter falls back to annotations. Take the pointer from vendor blogs. Take the facts from the docs.

Takeaways

• Runner statuses are not an interchange format. They're four private vocabularies that happen to share English words.
• Playwright's JSON report has two different status fields one level apart. Read the wrong one and your failure count is zero.
• blocked — meant to run, prevented by an upstream failure — is the status no runner has and every triage needs.
• Anything a script can decide deterministically, decide it in the script. Agents are for judgment, not for disambiguating vocabulary.
• Ship the guardrail into the repo (a skill, a hook, a normalizer with tests), not into a prompt you retype every session.

Next in this series: the same class of bug one layer up — tools that return success for a code path that never executed. Your normalizer can be perfectly correct about a report that was never regenerated.