Playwright Codegen: Should You Trust the Tests It Writes for You?

A new hire is told to add coverage for the checkout flow. Instead of opening the docs and writing locators by hand, they run one command, click through the flow in a browser window, and watch working Playwright code appear line by line as they go. Five minutes later there's a passing test, written without typing a single page.getBy... call. It feels like the setup step of test automation just got skipped entirely.

Two weeks later that same test starts failing in CI, not because the checkout flow broke, but because a product listing changed from "Wireless Mouse" to "Wireless Mouse (2024)" and the generated locator was matching on that exact text. The test wasn't wrong when it was recorded. It just never understood what it was actually supposed to be checking.

What Codegen Actually Does

npx playwright codegen <url> opens two windows: a real browser you interact with normally, and an inspector window that mirrors every action back as Playwright code in real time. Click a button, and a page.getByRole('button', { name: '...' }).click() line appears. Fill a field, and a .fill() call shows up with the exact value typed. It also exposes a "pick locator" mode for inspecting how it would target any element on the page, and toolbar buttons for recording assertions, visibility, text content, or a specific value, directly into the generated script.

It's not unique to Playwright as an idea, Selenium IDE and Cypress both have comparable recorders, but Playwright's version benefits from the same locator engine the rest of the framework uses, so what it generates is normal, runnable Playwright code rather than a separate recorded-macro format.

What It Actually Gets Right

For a first pass, codegen removes a real amount of friction. Getting the exact API syntax right, which locator method applies to which kind of element, how to fill a date picker versus a plain text input, is the kind of boilerplate that's tedious to look up repeatedly, and codegen produces working code for it immediately.

When the underlying page markup is already accessible, with real button roles, associated labels, and meaningful link text, codegen tends to generate the same kind of locator a careful developer would write by hand: getByRole, getByLabel, getByText. In that scenario, recording a flow once and skimming the output is often faster than writing it from scratch, and the generated assertions (from the "assert visibility" and "assert text" toolbar buttons) save the step of remembering the exact expect() syntax.

It's also a legitimately useful exploration tool even when you never keep the generated code: clicking through a flow while watching the locator for each element appear is a fast way to learn how a particular page exposes its elements to the accessibility tree, before writing a locator by hand.

Where the Generated Locators Go Wrong

The problems start exactly where the markup stops being accessible, which is most real applications, most of the time.

Text-based locators baked in from one specific state. If a recorded click targets a locator built from visible text, "Wireless Mouse", a row in an order table, a customer's name on a profile page, that locator is now tied to that exact string. Any copy change, localization, or A/B test variant breaks the test, even though nothing about the actual feature regressed.

Index-based locators in repeated elements. Click the third item in a product grid during recording, and without a unique accessible name to latch onto, codegen can fall back to a structural or positional locator, something resolving to "the third matching element." That's fragile in a way that has nothing to do with the test's intent: reorder the grid, add a promoted item at the top, or change a filter default, and the test now clicks the wrong thing without ever raising an error that makes the real cause obvious.

Generic CSS selectors on custom components. A custom dropdown, modal, or date picker built without proper ARIA roles gives codegen nothing meaningful to anchor to, so it sometimes resolves to a class name or tag-based selector instead. Those tend to be the first thing to break on a styling refactor, since CSS classes change for reasons that have nothing to do with functionality.

Recording against seeded or environment-specific data. A flow recorded against a test account named "Jane Doe" or an order numbered #4821 captures that literal value in the generated locator or assertion. It passes locally against that data and fails the moment it runs against a different environment or a reseeded database.

None of this is really codegen "getting it wrong." It's generating the most specific locator it can find for whatever happened to be on the page during that one recording session, which is a different goal than writing a locator that will still make sense in six months.

The Deeper Limitation: It Records Actions, Not Intent

Even when every generated locator is solid, codegen only knows what you clicked. It doesn't know what the test is actually supposed to prove.

It won't add an assertion unless you explicitly trigger one from the toolbar while recording, so a naive recorded flow often ends with a successful-looking series of actions and no real check that anything worked. It has no way to know that the interesting case isn't "the happy path with valid data" but "what happens when the coupon code is expired," because that case never got clicked during the recording session. And it can't tell the difference between an action that's incidental to the flow, dismissing a cookie banner, say, and one that's the actual point of the test.

A recorded test is a transcript of a session, not a specification of correct behavior. The gap between those two things is exactly the gap a human still has to fill in.

How to Use It Without Inheriting a Brittle Suite

Treat codegen as a fast first draft, not a finished test. Record the flow, then open the generated file and go through it line by line before it goes anywhere near a commit.

Replace locators that were only specific by accident. If a generated locator targets "Wireless Mouse" and the test is actually about adding any item to a cart, swap it for something structural and stable, a data-testid on the add-to-cart button, or a role-based locator scoped to the product card itself, that doesn't depend on which product happened to be on screen during recording.

Collapse index-based and overly specific selectors into something that expresses what the element actually is, not where it happened to sit in a list during one run. If the markup doesn't expose anything stable to select on, that's often a signal worth raising with whoever owns the component, since the same lack of an accessible name or role that makes the locator fragile for testing is often the same gap that makes the component harder to use with a screen reader.

Add the assertions that actually matter, not just the ones the recording session happened to capture. A generated test might stop right after a button click; the real test usually needs an assertion about what changed as a result, a new row appearing, a total updating, a confirmation message rendering.

Fold the result into existing structure rather than leaving it as a standalone recorded script. A repeated login flow or a shared setup step that codegen just recorded inline belongs in a fixture or a page object, same as a hand-written test would.

A Realistic Before and After

A raw recorded line might look like this:

await page.locator('.product-card:nth-child(3) >> text=Add to cart').click();

That passes today, against today's product listing, in today's sort order. A version worth keeping looks more like:

await page.getByRole('article', { name: productName })
  .getByRole('button', { name: 'Add to cart' })
  .click();

The second version expresses what's actually being tested, adding a named product to the cart, instead of what happened to be third in a list during one recording session.

Final Thoughts

Codegen solves the part of writing a test that was never really the hard part, getting the right method call for the right kind of interaction. The hard part was always deciding what a test should actually verify and making sure it keeps verifying that after the page around it changes. That part still has to come from a person who understands the feature, not from a recording of whatever happened to be clicked during one five-minute session.

Used that way, as a fast draft that gets reviewed and tightened before it's trusted, codegen is a genuine time saver. Used as a substitute for actually reading the test it just wrote, it's a reliable way to end up with a suite that passes for the wrong reasons and breaks for reasons that have nothing to do with the feature it was supposed to protect.