Your QA Suite Passed. Why Your Website Might Still Be Exposed

Your QA Suite Passed. Why Your Website Might Still Be Exposed

Every test is green. Login works, checkout completes, the signup flow sends its confirmation email, and the regression suite ran clean on the last three deploys. By every measure QA cares about, the site is working.

None of that tells you whether your .env file is publicly downloadable, whether your S3 bucket lets anyone list its contents, or whether someone can send an email that looks like it came from your CEO. QA testing and security exposure are answering two completely different questions, and it's easy to assume a passing test suite covers more ground than it actually does.

What QA Testing Actually Checks

Functional, regression, and end-to-end testing (the kind covered by tools like Playwright) all share the same underlying question: does the application behave the way it's supposed to when a user interacts with it? That's exactly the right question for catching broken forms, failed redirects, or a checkout flow that silently stops working.

But that question only covers intended behavior. It never asks what an attacker who isn't following the intended flow could reach, download, or exploit. A login form can pass every test case you write for it and still be missing the headers that stop clickjacking, or sit behind a server leaking its exact software version to anyone who asks.

The Gap: What a Test Suite Never Touches

None of the following show up in a typical QA pass, because nothing in a functional test is designed to look for them.

Missing security headers. Headers such as Content-Security-Policy, Strict-Transport-Security, and X-Frame-Options provide important browser-level protections. For example, missing framing protections can allow a page to be embedded by another site and potentially enable clickjacking attacks. A functional test has no reason to check for a security header unless that check was explicitly included.

Exposed files and secrets. A .env file left publicly accessible can expose database credentials, API keys, and signing secrets. An exposed .git/config can reveal repository configuration and, depending on how credentials were stored, potentially sensitive authentication information. Your test suite never requests URLs that aren't part of the application's actual flow, so it never stumbles onto them.

Open databases and exposed ports. Services like Redis, MongoDB, or Elasticsearch can sometimes be reachable directly from the internet without appropriate authentication or access controls. A functional test talks to your application through its intended interface, not by scanning the network services exposed by the server.

Public cloud storage. An S3 bucket created for uploads or backups can end up world-readable, or worse, world-writable, without anyone noticing. This lives entirely outside the application layer your tests exercise.

Email authentication gaps. Weak or missing SPF, DKIM, and DMARC configuration can make it easier for attackers to spoof your domain in email-based attacks. These controls help receiving mail systems verify whether messages claiming to come from your domain are authorized. There's no button in your UI that tests this.

Weak or outdated TLS. A server that still accepts deprecated TLS versions or uses an invalid, misconfigured, or soon-to-expire certificate can create security and availability risks. A basic browser test may only confirm that the page loads over HTTPS; it doesn't necessarily inspect the server's full TLS configuration or certificate posture.

Subdomain takeover. A DNS record pointing at a deleted cloud resource, such as an abandoned staging environment, can create a dangling DNS record that may be claimable by an attacker, depending on the provider and configuration. Nothing in a functional suite touches DNS configuration at all.

Why the Gap Exists

This isn't a QA failure. It's a scope difference. Functional testing is built to verify behavior inside the application's intended surface: the pages, forms, and flows a real user would actually touch. Security exposure lives largely outside that surface, in server configuration, DNS records, cloud storage permissions, and HTTP-level details that never appear in a user story.

Closing that gap with more functional tests doesn't really work, because you'd be writing test cases for problems you don't know exist yet. What helps close that gap is a different kind of check: one that examines the site's publicly accessible surface the way an outside observer would, rather than following a predefined user journey.

Closing the Gap Without Hiring a Pentester First

This is the specific gap FlawPilot's scanner is built to check. Point it at a URL and, in under 90 seconds, it runs the site through automated checks covering the exact categories above: HTTP and TLS configuration, security headers, exposed files and secrets, open ports, S3 bucket permissions, DNS and email authentication (SPF/DKIM/DMARC), and subdomain takeover risk, then returns separate security, performance, and infrastructure scores.

The output is written in plain language instead of raw tool output. Instead of "HSTS header not present," the finding reads something closer to: "Your site can be quietly switched from HTTPS to HTTP by an attacker on the same WiFi network, exposing your users' passwords." Instead of "SPF record missing," it's: "Anyone can send emails pretending to be from your domain." Each finding also includes what to actually do about it.

This doesn't replace a full penetration test. It won't catch business-logic flaws or issues that only exist behind authentication. What it does is surface the category of problem that a passing test suite gives zero visibility into, the kind that's genuinely embarrassing to discover from a security researcher's email instead of from your own tooling.

Where This Fits Alongside Your Existing Testing

The practical version of this isn't "replace your QA process." It's adding one more check to a pipeline that already has several:

• Functional and regression tests (Playwright, Cypress, Selenium) confirm the application behaves correctly for real users.
• A security and infrastructure scan confirms the application isn't exposing something to everyone else.

Run the second one before a launch, after any significant infrastructure change (a new S3 bucket, a new subdomain, a DNS update), and periodically afterward, since none of these issues are one-time risks. A bucket permission can drift, a certificate can expire, a staging subdomain can get abandoned six months from now and become exactly the dangling DNS record an attacker is looking for.

Final Thoughts

A green test suite shows that the scenarios covered by your tests are behaving as expected. It was never designed to answer a different question: what can someone reach who isn't following your intended flow at all?

That's not a reason to distrust your QA process, it's a reason to add a layer that actually checks for it. The two aren't competing tools, they're covering different territory, and a site that's fully tested but never scanned for exposure is still one open S3 bucket or one missing header away from a problem nobody was looking for.