Webhooks: Why a 200 Does Not Make an Integration Reliable

A webhook works in testing. An event arrives, the handler writes a record, returns 200, and everybody is happy.

That does not make it a resilient webhook.

The real problems often appear only when a provider retries a delivery, two events arrive in a different order, a request is sent again after a timeout, or someone attempts to replay an intercepted request. That is when you discover whether the endpoint merely accepts data or actually behaves like a reliable integration boundary.

Stripe explicitly documents retries, duplicate deliveries, multi-day retry behavior in live mode, and delivery without guaranteed ordering. GitHub likewise recommends signature validation, unique delivery identifiers, and a fast 2xx response. These are not exotic edge cases. They are part of normal webhook operations.

The dangerous assumption: The provider is just calling our URL

A publicly reachable webhook endpoint is, first of all, a public URL. Without an additional trust mechanism, anyone can attempt to send requests to it. Business logic therefore cannot rely on a request being probably from the expected service.

A resilient receiver separates at least four questions: Is the request authentic and unmodified? Is this particular delivery recent enough, or could it be a replay? Have we already processed this event? Can processing be repeated without causing the business effect twice?

Those questions sound similar, but they protect against different failure modes.

Signature verification only works when you verify the bytes that were actually signed

Many webhook providers authenticate a request body with a shared secret. Stripe uses HMAC with SHA-256 and includes a timestamp in the signed material. GitHub also uses HMAC-SHA256 for its X-Hub-Signature-256 header. HMAC itself is described as a message-authentication mechanism in RFC 2104.

The practical failure is often not cryptography. It is framework behavior. If middleware has already parsed, normalized, or re-serialized JSON, the byte sequence can differ from the body the provider originally signed. Stripe therefore explicitly requires the unmodified raw request body for signature verification.

A review should not stop at finding a function called verifySignature(). It should establish which exact bytes are verified, whether a JSON parser runs first, whether a proxy or middleware can modify the body or relevant headers, and whether the secret is loaded from protected configuration rather than committed to the repository. If verification is implemented manually, an appropriate constant-time signature comparison matters too.

A signature check that succeeds locally but sees different input behind the production proxy is not a reliable control.

Replay protection is not duplicate protection

A correctly signed request can still be undesirable if it is old and deliberately sent again. Stripe includes a timestamp in the signed payload and documents a default five-minute tolerance in its libraries. GitHub recommends using the unique X-GitHub-Delivery identifier to recognize repeated deliveries.

Replay protection asks whether a delivery is temporally acceptable and belongs to a legitimate transmission. Duplicate protection asks whether the underlying event has already produced its intended business result.

You often need both. A legitimate provider retry can carry a fresh timestamp and a valid signature while still representing the same event. Signature validation alone therefore does not stop your application from executing the same business action twice.

Exactly once is usually a property of your processing, not your transport

Webhook providers retry because networks and applications fail. Consider a payment event. The provider sends a successful-payment event. Your server updates the order and starts an email job, then dies before the 200 response reaches the provider. From the provider's perspective, successful processing is uncertain. Retrying is sensible.

If the second request creates another invoice, consumes the same voucher twice, or launches fulfillment again, the retry was not the design flaw. The missing property was idempotency.

A common resilient pattern is to persist a stable provider event ID, or another appropriate idempotency key, and atomically tie it to the state transition you want to make. A later retry can then determine that the event has already completed and return success without repeating the business effect.

An in-memory set is not enough because it disappears after a restart. A simple check first, write later sequence can also race when two deliveries are processed concurrently. Your database or queue boundary needs to make it impossible for the same identifier to be successfully committed twice at the same time.

Ordering must not become a hidden precondition

A common assumption is that once subscription.created has been processed, invoice.paid can safely depend on it. Stripe does not guarantee that delivery order. GitHub also documents that webhook deliveries can arrive out of the order in which the underlying events occurred.

A more resilient design is state-oriented: the event identifies the affected object, the application evaluates its current known state, retrieves current provider data if necessary, and applies transitions only when they make sense from that state. Late or superseded events should not roll an object back to an older state.

That makes the integration less dependent on timing and network luck.

Acknowledge quickly, work slowly

Webhook endpoints are poor places for long-running business processes. GitHub recommends returning a quick 2xx response and using a queue for longer work. Stripe likewise recommends responding successfully before complex processing can cause a timeout.

A resilient flow therefore often looks like this: Receive → verify signature → identify event → durably enqueue → return 2xx quickly → process asynchronously.

The word durably matters. Returning 200 before the event has been safely stored or handed to a reliable queue can lose data. Doing lengthy synchronous work before responding can create unnecessary retries. The boundary should be explicit: at what point does your application consider the event safely accepted?

Five webhook red flags worth investigating immediately

1. Business logic runs before signature verification

The application may be acting on unauthenticated input.

2. The receiver stores no event or delivery identifier

There is usually no robust basis for deduplication or incident reconstruction.

3. Every failure still returns 200

That may suppress retries while silently losing legitimate events. A 2xx response is appropriate only when the application has actually reached its chosen safe-acceptance point.

4. A timeout inevitably causes duplicate side effects

The business operation is not idempotent enough for a retrying transport.

5. Nobody knows how the signing secret is rotated

Secrets are not a one-time setup. Stripe, for example, supports an overlap period in which multiple signatures can be valid while a secret is rotated. Teams should know how rotation works before an emergency forces them to learn it live.

Reliability also needs operational evidence

Correct code is not enough. Useful signals include received events by type, signature failures separated from application failures, queue lag, processing duration, retry and duplicate rates, the age of the oldest unprocessed event, and permanently failed jobs. Correlation between provider event IDs, internal job IDs, and affected business objects also makes incident analysis much easier.

Sometimes zero events for several hours matters more than a high error percentage. A broken DNS record, certificate problem, or accidentally changed route may produce no application error at all because the request never reaches the application.

What Website-Pflichtencheck would inspect around this boundary

When webhooks are part of a critical website or SaaS flow, a technical review can go beyond the visible frontend. Website-Pflichtencheck can examine reachability, TLS, signature verification, raw-body handling, secret management, replay and duplicate controls, retry behavior, idempotency, queue boundaries, logging, and monitoring.

The goal is not to force every endpoint into one architecture. A newsletter webhook has a different failure impact from a payment, account-provisioning, or fulfillment webhook. Controls should match the business consequence of a missed, forged, or duplicated event.

The better test is not: Does an event arrive?

Instead, test the same event ID twice, related events in reverse order, a valid payload with an invalid signature, an expired signed request where the provider supports timestamp validation, a worker crash after a state change, provider redelivery after a timeout, signing-secret rotation, and queue outage or backlog.

If those scenarios make it unclear which actions may safely happen twice and which business effect must happen only once, that uncertainty is the finding.

A webhook is not reliable because it returned 200 yesterday. It is reliable when repetition, delay, outages, and tampering attempts do not create surprising business outcomes.

If your website or SaaS depends on webhook-driven integrations, Website-Pflichtencheck can include those boundaries in a technical review—before a harmless-looking retry becomes a customer-facing incident.