200 OK Is Not Business Success: How to Make Webhooks Reliable

A customer pays. Stripe reports success through a webhook. Your endpoint returns 200 OK. The provider dashboard marks the delivery as successful.

The customer account is still not activated.

Or worse: it is activated twice, two invoices are created, and two internal notifications are sent.

Webhooks look like ordinary API requests, but in production they behave like a distributed messaging boundary: networks fail, deliveries may be retried, duplicates can occur, messages can arrive later than expected, and retry behavior differs between providers. That is why “the endpoint returned 200” does not prove that the business workflow completed reliably.

Delivery and processing are different outcomes

A webhook provider first needs to know whether your application accepted the message. Your application still needs to decide how the business event should be processed.

Treat these as separate steps:

1. Receive the request.
2. Verify origin and signature.
3. Identify the event.
4. Persist it durably or place it safely on a queue.
5. Respond to the provider quickly.
6. Execute business processing.
7. Record the outcome and failure state.
8. Retry or reconcile with the source when necessary.

If step 5 succeeds but step 6 fails later, the event must not disappear. If step 4 was only in memory and the process crashes immediately after the 200, you have told the provider that delivery succeeded even though nothing durable remains to process.

That is the difference between transport success and business success.

Providers do not all behave the same way

A dangerous assumption is: “Failed webhooks will be retried anyway.” That is not universally true.

Stripe documents automatic retries for live-mode webhook deliveries for up to three days with exponential backoff. Stripe also explicitly warns that the same event can be delivered more than once and that event delivery order is not guaranteed.

GitHub behaves differently. GitHub documents that failed webhook deliveries are not automatically redelivered. They can be redelivered manually or through automation you build yourself. GitHub also recommends returning a 2xx within ten seconds and moving longer work to asynchronous processing.

The operational rule is therefore simple: Retry, ordering, and redelivery behavior are part of each provider integration and must be documented per provider.

Red flag 1: The handler is not idempotent

Imagine that a payment.succeeded event is delivered twice. The first delivery marks the order as paid, creates an invoice, books commission, provisions access, and sends email. The second delivery does exactly the same thing again.

The transport may be behaving correctly; the application is not robust against repetition. RFC 9110 defines idempotence as the property where multiple identical requests have the same intended effect as one request. A webhook POST is not automatically idempotent. The application needs to create that property.

Common controls include:

• persist the provider event ID,
• enforce a unique constraint on that event ID,
• consider business-object ID plus event type where separate events can represent one logical change,
• implement state transitions so that “already paid” does not repeat the same side effects,
• protect downstream operations with idempotency keys or stable business references.

The goal is not to prevent every duplicate delivery. The goal is to make sure a duplicate delivery does not create duplicate business effects.

Red flag 2: The application assumes event order

A subscription implementation expects: subscription created, invoice created, payment succeeds, access is activated. Then invoice.paid arrives before customer.subscription.created.

Stripe explicitly documents that event order is not guaranteed. A handler that assumes earlier events have already been processed can work in testing and fail under retries, load, or network delay.

A more resilient design treats the event as a signal about state, fetches missing current objects from the provider API when needed, evaluates transitions against the current business state, orders dependent work through its own state machine or queue, and avoids treating inbound webhook order as the only source of truth.

A webhook often says, “something happened.” For “what is true now?”, the source system’s API can be a better authority.

Red flag 3: 200 OK is returned before durable acceptance

Responding quickly is correct. Responding before the event is safely accepted is dangerous.

A fragile flow looks like this: verify signature, send 200 OK, pass the event to an in-memory worker, then the process crashes. The provider believes delivery is complete. Your application no longer has the event.

A stronger flow is:

1. Verify signature and basic structure.
2. Persist the event ID and relevant payload or successfully write it to a durable queue.
3. Only then return 2xx.
4. Execute business processing asynchronously.

That turns the endpoint into a reliable ingestion boundary instead of a narrow gap between an HTTP request and background code.

Red flag 4: Too much work happens inside the request

The opposite failure is also common. The webhook request waits for PDF generation, accounting updates, CRM sync, email delivery, and three external APIs.

GitHub recommends responding within ten seconds. Stripe likewise recommends returning success before complex processing and handling events asynchronously. Long synchronous handlers create timeouts, unnecessary redelivery, parallel processing of the same event, ambiguous failures between delivery and business logic, and escalating latency under load.

The robust boundary is: perform only the work required to accept the message safely inside the HTTP request.

Red flag 5: Signature verification works only on the happy path

Webhook secrets are not decorative. Without verification, a public endpoint can be called by anyone with a fabricated payload.

Stripe documents that signature verification requires the original raw request body. If a framework parses, normalizes, or otherwise modifies the JSON first, a legitimate signature can fail verification. GitHub also recommends webhook secrets and HTTPS.

A security review should verify that every relevant delivery is signature-checked, the raw body is preserved where required, secrets are separated by environment and endpoint, old secrets can be rotated safely, replay protections are respected, signatures and secrets do not leak into logs, and unexpected event types are rejected or deliberately ignored.

A valid signature proves origin. It does not prove that business processing is idempotent, ordered correctly, or successful.

Red flag 6: Failure states are only log lines

The worker calls a CRM API and receives 500. A stack trace appears in the logs. Nothing else happens. That is not a retry strategy.

Business-critical webhook processing needs explicit state, for example received, processing, processed, retry_scheduled, failed, dead_letter, and reconciled. Useful metadata includes attempt count, last error, next retry time, event type, provider event ID, business reference, application release, and completion time.

This lets the team distinguish “never arrived” from “accepted but failed” and “completed successfully.”

Red flag 7: Retries repeat destructive side effects

A retry must not repeatedly create an invoice, send the same email, issue credit, decrement stock, provision an account, or create a support ticket.

Idempotence therefore belongs not only at the event boundary but also around downstream side effects. A strong pattern binds side effects to stable business identifiers. An invoice has a unique reference to the payment or order. Provisioning references the same contract. An email job can deduplicate by template plus business event.

That keeps a technical retry technical instead of turning it into another business action.

Red flag 8: There is no reconciliation process

Even a well-built webhook system can miss events. GitHub, for example, documents that failed deliveries are not automatically redelivered. Stripe provides automatic retries and tools for handling undelivered events. Still, a critical business process should not rely on every push notification arriving perfectly.

Reconciliation means periodically comparing local state with the source of truth. Examples include comparing provider payments with local paid orders, active subscriptions with local entitlements, missing GitHub deliveries with delivery history, or CRM synchronization against timestamps and cursors.

Webhooks provide speed. Reconciliation provides completeness control.

A resilient webhook pattern

For many integrations, this baseline architecture works well.

1. Ingestion layer

Use HTTPS, signature verification, payload size limits, allowlisted event types, extract the provider event ID, and store the payload deliberately.

2. Deduplication

Use a unique constraint on the event ID, return a safe 2xx for an already processed duplicate, and do not start a second business execution.

3. Durable queue

Enqueue the event or an internal job ID durably, do not acknowledge successful receipt when queue persistence fails, and define backoff and maximum attempts.

4. Worker

Load current business state, fetch missing provider data where needed, apply idempotent state transitions, deduplicate side effects, and store the outcome.

5. Dead-letter and escalation

Preserve repeatedly failing events visibly, alert according to business impact, allow controlled replay, and record root cause and resolution.

6. Reconciliation

Compare critical source systems on a schedule, detect missed or inconsistent state, and provide a repair path that does not require improvised database edits.

Seven tests that reveal more than “the webhook arrived”

1. Send the same event twice. There must be no duplicate business effect.
2. Send events in the wrong order. The final state must still be correct.
3. Crash the worker after acceptance. The event must be recoverable.
4. Fail the queue during ingestion. The endpoint must not report false success.
5. Send an invalid signature. No business processing should start.
6. Make a downstream API return 500 temporarily. Retry and dead-letter behavior must be visible.
7. Deliberately omit one event. Reconciliation should eventually detect the mismatch.

Automating these tests measures the real reliability of the integration, not only its happy path.

What Website-Pflichtencheck would inspect

A technical webhook and integration review can examine publicly reachable webhook endpoints, HTTPS, secrets and signature verification, raw-body handling, deduplication and unique constraints, provider-specific retry behavior, asynchronous processing and queue durability, event ordering and state machines, idempotent side effects, dead-letter queues, monitoring, manual redelivery, reconciliation, and regression tests for duplicate, delayed, reordered, and missing events.

The goal is not to make every integration unnecessarily complex. A simple contact form may not need an event bus. A payment, provisioning, account, or contract workflow needs more than an endpoint that sometimes says 200.

A successful webhook is not merely a request that arrived. It is a business process that still produces the correct effect exactly once from the user’s perspective, despite retries, duplicates, reordering, and temporary failure.