Webhook Testing: Signatures, Retries, Duplicates, and Replay

A payment provider's webhook fires the instant a charge succeeds. The integration works perfectly in every manual test, click pay, watch the order confirm a second later, done. Weeks later, during a brief network blip, the provider's retry logic resends the same event twice because the first delivery timed out waiting on a slow database write. The endpoint processes it twice. A customer gets charged once but receives two confirmation emails and two inventory deductions, and nobody notices until support tickets start describing an order that doesn't match what was actually bought.

Nothing about that endpoint was broken in the way a normal test would catch. It worked, every time it was tested by hand. It just was never tested against the specific, structured ways webhook delivery actually behaves in production: retries, duplicates, out-of-order arrival, and requests that aren't what they claim to be.

Why Webhooks Are a Different Testing Problem

A typical API test calls an endpoint and checks the response. A webhook test has to account for the fact that the caller isn't your test suite, it's a third party's delivery system, and that system makes its own decisions about when to send, how many times to send, and what to do if your endpoint doesn't answer fast enough. Testing a webhook endpoint well means testing against that delivery system's actual behavior, not just the shape of a single request.

Most webhook providers, payment processors, source control platforms, messaging services, document the same handful of guarantees and failure modes: signed payloads to prove authenticity, retries with backoff when the endpoint doesn't respond successfully, no guarantee of exactly-once delivery, and no strict guarantee of ordering. A test suite that only ever sends one clean, correctly signed event and checks for a 200 response is testing none of that.

Signature Verification

Most webhook providers sign the payload with a shared secret, commonly HMAC-SHA256, and send the signature in a header alongside the request. The receiving endpoint is supposed to recompute the signature from the raw payload and reject anything that doesn't match, since without that check, anyone who discovers the endpoint URL can send fabricated events.

What to actually test: a request with a valid signature should be accepted. A request with a missing signature header should be rejected. A request with a signature computed from a different secret should be rejected. A request where the payload was modified after signing, even by one byte, should be rejected, since this is the exact scenario signature verification exists to catch. It's a common and costly mistake to implement the happy path (valid signature works) without ever testing that an invalid one is actually refused, since both cases can look identical from the outside until someone deliberately sends a bad one.

Replay Protection

A signed payload that was valid once is still a valid signature forever, unless the endpoint also checks that the request is recent. Most providers include a timestamp in the signed payload specifically so the receiving endpoint can reject anything outside a reasonable window, a few minutes, typically, which prevents someone who's intercepted a legitimate request from resending it later to trigger the same action again.

What to actually test: capture a validly signed request, then resend it after the provider's documented tolerance window has passed, and confirm it's rejected for being stale, not just for a bad signature. A surprising number of real implementations verify the signature correctly but never check the timestamp at all, which means a captured request stays replayable indefinitely.

Duplicate Delivery

Providers almost universally document that webhook delivery is "at least once," not "exactly once." If the endpoint doesn't respond with a success status within a timeout, or the provider's own infrastructure has a transient issue, the same event gets sent again, sometimes more than once. The endpoint is expected to handle that gracefully, which usually means tracking which event IDs have already been processed and treating a repeat as a no-op rather than reprocessing it.

What to actually test: send the same event payload with the same event ID twice in a row, and confirm the side effect, the charge, the inventory update, the email, only happens once. Also test what happens when the two duplicate deliveries arrive close enough together that they could race each other, since a naive "check if processed, then process" implementation without a proper lock or unique constraint can still process both if they overlap in time.

Out-of-Order and Interleaved Events

A sequence of related events, order created, payment confirmed, order shipped, isn't guaranteed to arrive in that order. Network conditions, retries, and provider-side queuing can all cause a later event to arrive before an earlier one. An endpoint that assumes strict ordering, processing a "shipped" event by looking up an order that the "created" event hasn't registered yet, breaks in a way that's easy to miss if every test sends events in the expected sequence.

What to actually test: deliberately send events out of their natural order and confirm the system either handles it correctly or fails in an obvious, loggable way, rather than silently corrupting state. If strict ordering genuinely matters for correctness, the system needs an explicit mechanism, a sequence number, a state machine that rejects out-of-order transitions, not an assumption baked into the code that nothing will ever arrive late.

Timeout and Retry Behavior

Providers typically retry on a schedule with exponential backoff, and most give up after a documented number of attempts or a documented time window. If an endpoint is slow, perhaps it's doing a heavy database write before responding, the provider may consider the delivery failed and retry even though the endpoint eventually would have succeeded, which is itself a source of duplicate processing.

What to actually test: confirm the endpoint responds quickly, typically by doing the minimal work needed to acknowledge receipt and queuing the heavier processing for after the response, rather than making the provider wait on a slow synchronous operation. An endpoint that takes ten seconds to process before responding is one slow dependency away from being retried unnecessarily.

Setting Up a Realistic Test Environment

Testing this well usually means being able to construct and sign payloads independently of the real provider, rather than only testing against live sandbox events, since that's the only way to deliberately produce invalid signatures, stale timestamps, and duplicate event IDs on demand. A test helper that signs an arbitrary JSON payload with the shared secret, the same way the real provider would, makes most of the above scenarios straightforward to script: generate a valid payload, tamper with it, resend it, duplicate it, and assert on each case independently.

function signPayload(payload, secret, timestamp) {
  const body = `${timestamp}.${JSON.stringify(payload)}`;
  return crypto.createHmac('sha256', secret).update(body).digest('hex');
}

test('rejects a request with a stale timestamp', async ({ request }) => {
  const payload = { event: 'payment.succeeded', id: 'evt_123' };
  const staleTimestamp = Math.floor(Date.now() / 1000) - 600; // 10 minutes old
  const signature = signPayload(payload, SECRET, staleTimestamp);

  const res = await request.post('/webhooks/payments', {
    data: payload,
    headers: { 'X-Signature': signature, 'X-Timestamp': String(staleTimestamp) },
  });

  expect(res.status()).toBe(400);
});

Receiving FlawPilot's Own Webhooks

Everything above applies directly if you're building a receiver for FlawPilot's scan webhooks, not just a hypothetical payment provider. FlawPilot sends signed POST requests to an endpoint you register whenever a scan finishes, fails, or changes state, and the delivery mechanics follow the same pattern described throughout this post: the endpoint has to be a public HTTPS address (FlawPilot won't deliver to plain HTTP or to a port outside 443 or the 1024-65535 range), requests are signed so your endpoint can verify they actually came from FlawPilot and not a spoofed sender, and deliveries that fail repeatedly get flagged as failing rather than retried forever, visible directly in the Webhooks settings page alongside when the failures started.

Each endpoint can be scoped to specific event types, a quick scan completing versus failing, a code scan completing versus failing, and so on, rather than receiving every event by default. If a receiver is only built and tested against one clean, successful delivery, the same gaps described above apply here too: an endpoint that's never been tested against a retried delivery, a delivery that arrives after a previous one failed, or a signature it should reject, is exactly the kind of receiver that looks correct in a demo and breaks the first time a real delivery doesn't go perfectly.

Final Thoughts

Webhook endpoints fail in a specific, well-documented set of ways, unsigned or tampered requests, replayed old requests, duplicate deliveries, out-of-order arrival, slow responses triggering unnecessary retries, and almost none of them show up in a test suite that only ever sends one clean, valid event and checks the response code. The fix isn't more tests in general, it's tests built around the actual behavior of webhook delivery systems, not the simplified version of that behavior most integrations are written to assume.