Reliable Webhook Receivers for Small Applications — Verify, Record, Acknowledge, Then Work

A webhook endpoint can look almost trivial: accept an HTTP POST, decode some JSON, update a record, and return 200. The difficult question appears when any step is interrupted. What if the request is forged, the provider retries it, the process crashes after replying, or two related events arrive in an unexpected order?

Those are not edge cases that can be hidden behind a larger server. They are boundaries between independent systems, and neither side can observe the other perfectly. A reliable receiver therefore needs more than a route and a JSON parser. It needs a small protocol for deciding what to trust, what to preserve, when to acknowledge, and how to repeat work safely.

This article develops that receiver-side protocol for a small application. It does not assume a particular framework or provider, because signature formats, headers, retry schedules, and timeout limits differ. The goal is a portable design, followed by the places where provider documentation must override general advice.

A delivery is not the same as completed work

A webhook is commonly described as an HTTP callback: one service sends an event to another service's endpoint. The Standard Webhooks specification calls this a kind of “reverse API.” The producer initiates the request, while the consumer receives it.

That HTTP exchange and the application's business work are related, but they are not the same transaction. A 2xx response says that the HTTP request succeeded according to the applicable semantics and provider contract. It cannot prove that a later email was sent, an account was updated, or an external API call eventually succeeded. RFC 9110 defines HTTP semantics; it does not turn a response into a guarantee about an unobserved background workflow.

This distinction suggests two separate stages:

1. Admission: authenticate the request, decide whether it is acceptable, and preserve it durably.
2. Processing: apply the intended business effect, retry recoverable failures, and record the outcome.

Combining both stages in one long request can work at very small scale, but it creates an awkward failure boundary. If processing takes too long, the provider may time out and retry even while the first attempt continues. GitHub, for example, documents a ten-second response window and recommends asynchronous processing; that number is a GitHub rule, not a universal webhook timeout. Stripe likewise tells receivers to return a successful response before complex logic.

The receiver sequence: preserve, verify, record, acknowledge, work

A useful receiver can be described as a short sequence:

1. Read the raw request body and the required headers without changing the body.
2. Verify the provider's signature, signed timestamp, and any other required metadata.
3. Reject unsupported event types or clearly invalid payloads.
4. Insert an inbox record into durable storage under a scoped unique key.
5. Return the success response expected by the provider.
6. Let a worker claim the inbox record and apply the business operation.
7. Record success or a controlled failure that can be retried or inspected.

The order matters. Verifying after an update is too late. Replying before durable storage creates a gap in which the process can acknowledge an event and then lose its only copy. Doing expensive work before replying makes the producer's delivery attempt depend on the latency of every downstream dependency.

“Durable” does not necessarily mean a large message broker. For a small application, an inbox table in the same database may be enough. What matters is that a successful acknowledgement follows a committed record, rather than an entry held only in process memory. A dedicated queue can also satisfy this role if its acknowledgement really means the message has been persisted under the guarantees the application needs.

Verify the bytes that were actually sent

A public webhook route is a public input. HTTPS protects the connection, but it does not by itself establish that the request came from the claimed provider. GitHub and Stripe both recommend a webhook secret and signature verification. Standard Webhooks specifies signed message metadata and payloads as well.

One easy mistake is to parse JSON first and then verify a newly serialized version. Whitespace, escaping, and key formatting can change even when the parsed data means the same thing. Both Stripe's documentation and the Standard Webhooks specification warn that verification can depend on the original body bytes. The safe rule is to preserve the raw body until verification is complete and follow the exact signed-content construction documented by the provider.

There is no responsible universal code snippet for this step. Providers differ in header names, encodings, signed fields, secret rotation, and timestamp rules. A maintained provider SDK is often preferable to handwritten cryptography. If manual verification is unavoidable, the provider's current algorithm must be followed exactly, including constant-time signature comparison where required.

A signed timestamp and a stable identifier solve different problems. A freshness window can reject an old captured request, reducing replay risk. An event or delivery ID can identify a request the application has already accepted. The receiver usually needs both checks, with the identifier scoped to the provider, account or endpoint, and event namespace described by that provider. Assuming every vendor's ID is globally unique is unnecessary and unsafe.

Use a durable inbox as the duplicate boundary

Duplicate delivery is normal enough that both GitHub and Stripe document identifiers or techniques for recognizing repeated events. HTTP POST itself is not inherently idempotent under RFC 9110. If receiving the same payload twice must not charge, publish, or provision twice, that property has to come from the application.

A minimal inbox record might contain:

• a local record ID;
• provider and endpoint or account scope;
• the provider's event or delivery ID;
• event type and provider timestamp;
• receipt time and processing status;
• an attempt count and a bounded error summary;
• the payload, or a protected reference to it, only when retention is justified.

A database uniqueness constraint on the scoped provider ID is stronger than a “check, then insert” sequence in application code. Two concurrent deliveries can both pass a prior lookup; only the storage constraint makes their race resolve at one boundary. The duplicate path should normally confirm that the existing event was accepted and return the provider's expected success response, rather than performing the business effect again.

Payload retention deserves restraint. A full payload may help recovery and auditing, but it may also contain personal or sensitive data. A thin event may require a later API request, while a snapshot can preserve the state as sent. Standard Webhooks discusses this tradeoff rather than declaring one shape universally correct. The receiver should keep only what its recovery, audit, and legal requirements justify, protect it like other application data, and avoid placing secrets or complete payloads in routine logs.

Idempotent receipt is not universal exactly-once processing

The inbox can ensure that one provider event maps to one local admission record. It cannot erase every failure boundary.

Suppose a worker reads an accepted event, calls an external email or billing service, and crashes before marking the inbox row complete. On retry, it cannot know from the local row alone whether the external service applied the first call. The next layer therefore needs its own strategy: an idempotency key accepted by the downstream API, a local state transition that can be checked transactionally, or reconciliation against the authoritative system.

This is why “exactly once” is too strong as a casual promise. A more honest target is:

• accept each identified event into one durable local record;
• make each business transition safe to retry;
• detect outcomes that remain ambiguous;
• provide a controlled way to reconcile or replay them.

Some operations are naturally easy to repeat, such as setting a row to a known state. Others, such as incrementing a counter or calling a system without idempotency support, need additional design. The event ID can be part of that design, but merely storing it does not make every downstream effect idempotent.

Do not quietly assume event order

Retries, parallel workers, and network delays can change arrival and completion order. Stripe explicitly documents that its events are not guaranteed to arrive in generation order. Other providers may offer different contracts, so their documentation remains authoritative.

Where order matters, an event handler can use a resource version, event sequence supplied by the provider, or a fresh read from the provider's API. None is automatically best. Fetching current state can collapse several old events into one reconciliation step, but it adds a network dependency and may lose historical detail. Strict sequence processing preserves history, but it needs a defined response to missing events. The correct choice depends on whether the application cares about the latest state, every transition, or both.

The important part is to make the assumption explicit. A handler that silently treats arrival order as truth may look correct until the first retry overtakes a newer event.

Acknowledgement should be fast, but not premature

“Return quickly” is sometimes interpreted as “reply before storing anything.” That trades timeout risk for data-loss risk. The more useful boundary is: perform only the bounded admission work in the request, commit it, then respond.

Admission should avoid slow email, image processing, broad API calls, or multi-step business workflows. It may still include signature verification, basic schema and event-type checks, and one short durable write. If that write fails, returning success would tell the producer to stop retrying even though the receiver preserved nothing. A failure response may cause another delivery, which is precisely what the deduplication boundary is prepared to handle.

Status handling is provider-specific. Standard Webhooks treats 2xx as delivery success and describes other responses as failures, while individual providers document their own timeout, redirect, and retry behavior. The receiver should implement that contract deliberately. A success response means “this receiver has accepted responsibility,” not necessarily “all consequences are already complete.”

Keep failure visible and replay controlled

An asynchronous worker makes the endpoint responsive, but it can also move failures out of sight. A useful inbox needs observable states such as pending, processing, completed, and failed, plus timestamps and bounded attempt counts. A record stuck in processing needs a recovery rule after a worker crash.

Retry only failures that may improve with time. A temporary network error is different from a payload whose event type the application does not understand. Unlimited immediate retries can turn one bad event into a permanent load problem. Backoff, an attempt limit, and a dead-letter or review state keep the failure finite. The Standard Webhooks specification also recommends visibility into failed deliveries and a manual replay mechanism.

Manual replay should not bypass verification history, uniqueness rules, or audit records. It should answer who requested the replay, which stored event was used, and what the new processing attempt did. Observability is not only a dashboard: a small application can begin with structured logs, a query for old pending rows, and an alert when failures exceed a meaningful threshold.

A compact receiver checklist

• Expose the endpoint over valid HTTPS and keep signing secrets outside source code.
• Preserve the raw request body and verify it according to the provider's current documentation.
• Check signed timestamps, expected event types, request size, and schema before business processing.
• Scope a durable uniqueness constraint to the provider's identifier semantics.
• Commit accepted events before returning the provider's success response.
• Move slow work to a worker and make each effect safe to retry where possible.
• Do not assume ordering unless the provider guarantees it and the design enforces it.
• Track pending, completed, failed, and ambiguous outcomes without logging unnecessary secrets or payload data.
• Test valid events, invalid signatures, stale timestamps, concurrent duplicates, worker crashes, and downstream timeouts.
• Document retention, replay, secret rotation, and reconciliation procedures.

Conclusion

A reliable webhook receiver is less about accepting JSON and more about choosing a truthful boundary. Before that boundary, the application is still deciding whether a request is authentic and acceptable. After a durable inbox commit, it has accepted responsibility and can acknowledge the producer. Business processing then becomes a visible, retryable workflow rather than hidden work inside an HTTP request.

This design does not make distributed failures disappear, and it should not be marketed as automatic exactly-once processing. It does something more practical: it gives duplicates, crashes, delays, and ambiguous side effects an explicit place in the system. For a small application, one carefully constrained inbox table and worker may be enough. The next question is not how much infrastructure to add, but which failure the application still cannot detect or safely repeat.

References

• Standard Webhooks, “Standard Webhooks specification,” version 1.0.0.
• GitHub Docs, “Best practices for using webhooks”.
• Stripe Docs, “Receive Stripe events in your webhook endpoint”.
• R. Fielding, M. Nottingham, and J. Reschke, eds., RFC 9110: HTTP Semantics, June 2022.