Skip to content
← All articlesEngineering

Webhooks Explained: Design and Reliability Guide for Teams

A practical webhooks guide for product teams: how webhooks work, signature verification, retries, idempotency and monitoring for reliable integrations.

Sooner or later, almost every product needs to tell another system that something happened: a payment succeeded, an order shipped, a document was signed. Webhooks are the standard way to do it. They look simple, since it is just an HTTP request, yet they are one of the most common sources of silent data problems in integrations.

This guide explains how webhooks work and what a reliable implementation needs, whether you are sending webhooks to your customers or receiving them from a provider.

What a webhook is

A webhook is an HTTP callback. When an event occurs in system A, it sends a POST request with a small payload to a URL that system B registered in advance. System B responds with a success status code, and that is the end of the exchange.

The alternative is polling, where system B repeatedly asks system A whether anything has changed.

Webhooks Polling
Latency Near real time Depends on poll interval
Wasted requests Few Many (most polls find nothing)
Complexity for the receiver Needs a public endpoint and security Simple client code
Failure handling Sender must retry Receiver simply asks again

Webhooks are usually the right default for event notifications. Polling remains useful as a safety net, which we come back to below.

Receiving webhooks: the essentials

If your product consumes webhooks from a payment provider, CRM or logistics service, treat the endpoint as a public door into your system.

  • Verify the signature. Most providers sign the raw request body with a shared secret. Recompute the signature and compare it using a constant-time comparison. Reject anything that does not match. Never trust a payload just because it arrived at the right URL.
  • Check the timestamp. If the provider includes one, reject requests that are too old to limit replay attacks.
  • Respond quickly. Validate, store the event, return a 2xx status, and do the real work afterwards. Providers often time out after a few seconds and will retry, which creates duplicates.
  • Process asynchronously. Put the event on a queue and let a worker handle it. Our guide to background jobs and message queues covers this pattern.
  • Use HTTPS only, and keep the signing secret in a proper secret store, not in source code.

Expect duplicates and out-of-order delivery

Almost every webhook system offers at-least-once delivery. That means the same event can arrive twice, and event B can arrive before event A. Your code must cope with both.

Idempotency. Store the event ID (or a hash of the payload if there is none) in a table with a unique constraint. If you have already processed that ID, acknowledge it and do nothing. Make the handler’s effects safe to repeat, for example “set status to paid” rather than “add payment”.

Ordering. Do not assume arrival order reflects the order events happened. Use the event’s own timestamp or version number, and ignore updates older than what you already hold. For critical state, fetch the current object from the provider’s API when the webhook arrives and treat the webhook as a signal to look, not as the source of truth.

Sending webhooks: what your customers will expect

If you offer webhooks to customers or partners, you inherit the provider’s responsibilities. A good webhook feature includes:

  1. Signed payloads, with documented verification steps and code samples.
  2. Automatic retries with exponential backoff, and a sensible limit after which the delivery is marked failed.
  3. A unique event ID and event type in every payload, plus a stable, versioned schema.
  4. A delivery log that customers can view, with request, response status and a “resend” button.
  5. Automatic disabling of endpoints that fail for a long period, with a notification to the owner.
  6. Protection against SSRF. Customer-supplied URLs must not be allowed to point at your internal network or cloud metadata addresses. Validate and resolve URLs carefully and send requests from an isolated network segment.
  7. Clear documentation of event types, payload examples and expected response codes.

Keep payloads small and stable. Many teams send a minimal event with an ID and let the receiver fetch details through the API, which also reduces the risk of leaking sensitive data in transit.

Reliability checklist

Use this list when reviewing an existing webhook integration:

  • Signature verification is enforced and tested
  • Handlers are idempotent and backed by a unique event ID
  • The endpoint returns quickly and defers heavy work to a queue
  • Failed events are retained for inspection and replay
  • Alerts fire when delivery failure rate or queue age rises
  • A scheduled reconciliation job compares your data with the provider’s, catching events that never arrived
  • Secrets can be rotated without downtime

That reconciliation job is the polling safety net mentioned earlier. Even well-run providers have outages, and a nightly comparison is far cheaper than discovering missing orders from a customer complaint.

Testing and monitoring

Test webhooks like any other integration. Replay recorded payloads in your test suite, including duplicates, out-of-order sequences and malformed bodies. For third-party providers, use their sandbox and a tunnelling tool in development, but run a full end-to-end test in a staging environment too. Our API testing guide outlines how to structure these checks.

In production, track delivery success rate, processing latency, retry counts and the age of the oldest unprocessed event. A dashboard with those four numbers will catch most problems before users do.

Common mistakes

  • Doing slow database or third-party calls inside the request handler, causing timeouts and retries
  • Skipping signature checks “because the URL is secret”
  • Assuming events arrive exactly once and in order
  • Returning a 200 status before the event is safely stored
  • Having no way to replay events after a bug fix

Getting help with integrations

Reliable webhooks are a mix of backend engineering, security and operational discipline. If you need a team to design or harden an event-driven integration, our web development services cover API design, third-party integrations and ongoing monitoring. If a live integration is already dropping events, our support and maintenance engineers can investigate and stabilise it. Contact us to discuss your project.

Let’s build what’s next.

Tell us about your platform, your product idea or the systems you want to modernise. Someone from our Chennai engineering team will get back to you personally.