Smoke Tests vs. Full Regression: Knowing What to Run When

A team ships a small CSS fix and waits twenty-five minutes for the full regression suite, two thousand tests covering every corner of the product, to confirm that changing a button's padding didn't somehow break payment processing. A few weeks later, a different team, tired of slow deploy gates, trims their pipeline down to a handful of fast checks and ships a change that breaks checkout in production, because nothing in that fast gate happened to touch checkout.

Both teams have the same underlying problem: treating "which tests should run" as a single, fixed answer instead of a question with a different right answer depending on what's actually at stake in that moment.

Two Different Jobs, Not Two Tiers of the Same Thing

Smoke tests answer one narrow question, fast: is the build fundamentally broken? Does the application start, can a user log in, does the homepage render, does the core critical path, usually checkout, signup, or whatever the product's single most important flow is, work end to end. A smoke suite is intentionally small, often ten to thirty tests, built to run in a minute or two and catch the kind of catastrophic failure that makes everything else irrelevant, a deploy that doesn't boot, a broken build, a database migration that locked the app out of its own data.

Full regression answers a much broader question, slowly: does every previously working feature still work, including the edge cases, the less-used settings pages, the permission boundaries, the less common user journeys. A full regression suite can run into the thousands of tests and take anywhere from twenty minutes to several hours, depending on how much of it is UI-driven versus API-level, and it's built to catch the kind of subtle regression that a narrow smoke test would never touch, a permissions check that silently stopped working on a page three clicks deep, a currency formatting bug that only shows up for one locale.

Neither is a smaller or larger version of the other. A smoke suite isn't "regression testing but fewer tests," it's a different question with a different acceptable risk tolerance, and conflating the two is how teams end up either waiting 25 minutes to ship a CSS tweak or shipping a checkout-breaking change through a gate that was never built to catch it.

Matching the Test to the Moment

On every commit, before merge: something fast enough not to break developer flow, usually unit tests and a thin smoke layer covering the most critical paths. The goal here is immediate feedback, a developer should know within a few minutes if something fundamental broke, not find out twenty minutes later after context-switching to something else.

On every deploy to staging or production: the smoke suite, run as a deploy gate, confirming the build is fundamentally healthy before anyone, human or automated, spends more time on it. This is the fast go/no-go check: did the deploy itself work, is the app reachable, does the critical path function.

On a schedule, or before a larger release: the full regression suite, run nightly or as part of a release candidate process, where a 30-60 minute runtime is an acceptable cost for comprehensive coverage. This is where the slower, broader net catches the subtler regressions that a fast gate was never meant to catch.

Targeted regression, on a specific change: when a change touches a specific area, the permissions system, the billing flow, running the regression tests scoped to that area specifically, rather than either the full suite or nothing, is often the most efficient middle ground, catching the regressions most likely to actually be affected without paying for the full suite's runtime.

Why Getting This Wrong Is So Common

Teams tend to drift toward one of two failure modes, both for understandable reasons. Running the full suite on every single change feels safe, more tests obviously means more coverage, but it quietly trains developers to see test runs as a tax to be waited out rather than useful feedback, and a 25-minute gate on every PR is a real, compounding cost to how fast a team can actually ship. Running only a minimal smoke suite feels fast and responsive, but it leaves a real gap: smoke tests are deliberately narrow, and a change that breaks something outside that narrow critical path sails through a gate that was never designed to catch it.

The fix isn't picking one extreme, it's being deliberate about which layer is actually answering which question, and making sure both layers exist and run at the right moment rather than substituting one for the other.

Building a Smoke Suite That's Actually Useful

A smoke suite earns its speed by being genuinely narrow and well-chosen, not just short. The right contents are usually: can a user authenticate, does the single most business-critical flow complete end to end (checkout, the core action the product exists to perform), do the handful of pages or endpoints that, if broken, would be an obvious, embarrassing, immediate problem, actually load. A smoke suite with the wrong thirty tests, ones that happen to be fast to write rather than ones that actually represent what "fundamentally broken" would look like, gives false confidence without actually reducing risk.

It's also worth keeping deliberately stable. A smoke suite that's flaky defeats its own purpose, since a flaky fast gate trains people to rerun and ignore it, which is exactly the failure mode a slow, comprehensive, untrustworthy regression suite falls into as well.

A Realistic Pipeline Shape

# conceptual pipeline stages
on_pull_request:
  - unit_tests          # seconds
  - smoke_tests          # 1-2 minutes

on_merge_to_main:
  - smoke_tests           # confirm the merged build is healthy
  - targeted_regression   # scoped to changed areas

nightly:
  - full_regression       # 30-60 minutes, comprehensive

before_release:
  - full_regression
  - manual_exploratory_pass

The structure isn't rigid, the right breakdown depends on the product, the team's risk tolerance, and how expensive a production incident actually is for that specific business, but the underlying principle holds across most teams: fast, narrow checks gate every change, broad, slow checks run on a cadence that doesn't block day-to-day shipping.

Final Thoughts

A test suite that takes 25 minutes to approve a CSS change and one that approves a checkout-breaking bug in under a minute have the same root cause: treating "how much to test" as one fixed answer instead of a question that depends on what moment it's being asked at. Smoke tests and full regression aren't competing strategies, they're answering different questions at different speeds, and a pipeline that uses both, at the right moments, gets fast feedback where it matters and real coverage where it matters, without forcing a choice between the two.