Webhook debugging often starts in the application logs. The handler returned 200 OK. The event still never showed up. The team checks the next few lines, then the proxy, then the provider’s delivery report. By that point it is no longer clear which request anyone is talking about: the one the provider marked as delivered, the one your server accepted, or a retry that arrived a few minutes later.
A webhook looks like an ordinary HTTP request. The difficulty is that delivery is distributed and there is no shared record of what happened. The provider, the network, the receiver, the queue, the handler, and the side effects all live in different systems. Each one answers a different question.
Provider → network → intake → delivery → handler → side effect → HTTP response
A failure can occur at any of those steps. More awkwardly, several of them can look successful at the same time. That is why webhook debugging is harder than checking a single status code.
Why webhook debugging is harder than it looks
In a synchronous API, the client and the server share one request. If the operation failed, the same response usually says so. A webhook does not work that way. The provider sends the event and moves on. Your application learns about it later, often on another machine, sometimes through a retry, and sometimes only after a customer has already seen the wrong order status.
By the time you investigate, the original evidence may already be gone. The provider kept a delivery log. The load balancer kept an HTTP status. The application kept whatever it managed to write after parsing the body. If those records are not tied together by a stable event ID, the team reconstructs the story from timestamps and roughly similar payloads. Two similar payments start to look like one retry. One retry starts to look like two events.
Observability is not a set of charts. It is the ability to answer specific questions:
Did the request leave the provider at all?
Did your public URL accept it?
What body and headers actually arrived?
Which destination was it sent to?
How did each attempt end?
Is this a new event, a retry of the same delivery, or an intentional Replay?
If those answers have to be assembled after an incident, debugging is guesswork. The five cases below are where a 200 OK log line is especially easy to misread.
Payload lost: the request vanished, and it is unclear where
“The webhook was lost” is too broad. A payload can fail to arrive for several different reasons, and each one leaves a different trail.
The request may never have left the provider: a wrong URL, a sender-side error, or an event that never entered its queue. It may have been dropped on the network before it reached you. It may have been rejected on intake because of the method, size, an account limit, or an unavailable receiver. It may have been accepted and then never delivered to the handler. It may have reached the application and disappeared there: the process restarted before the write, the application queue never accepted the message, and the raw body never made it into ordinary logs.
Without a stored intake record, those cases are almost impossible to tell apart. The provider says, “we sent it.” The application says, “we never got it.” Both can be right about their own boundary.
Rejected requests are especially unforgiving. If the receiver refused the HTTP message before it was fully recorded, the system may retain only the fact of rejection and its reason—not the body, headers, or path. At that point the argument over “what payload arrived” cannot be settled from data. The content was never kept, and it cannot be recovered.
Expired retention is a separate trap. Even an accepted request cannot be inspected if it is already gone by the time you look. Webhook observability exists only while the request is still stored. Missing data cannot be added after the fact.
The first diagnostic question is therefore not “why didn’t the handler run?” It is “was the request accepted, and what exactly was in it?” Until that is answered, fixing business logic is premature.
Timeout: no response arrived, but the work may already be done
A timeout looks like a simple failure: the recipient did not answer in time. For webhooks, it is one of the most ambiguous errors you can get.
The handler may never have received the request. It may have received it, started work, and failed to respond in time. It may have finished the operation—created an order, credited a balance, sent an email—while the response was lost or arrived after the timeout. From the sender’s point of view, all three look the same: delivery was not confirmed.
That is why a timeout is not proof that no side effect occurred. It only means the expected HTTP response was not received in time. The next attempt may repeat work that already succeeded.
Timeouts also sit poorly next to ordinary access logs. The application log may show a successful run. The sender’s log may show a timeout. Both can be accurate. Without an attempt history that includes duration, attempt number, and a final result, you cannot tell whether you are looking at one request or already at a retry.
Diagnosis here needs timing as well as status codes: when the attempt started, how long it ran, and whether it stopped at DNS, TLS, the connection, or after the body had already been sent. The word “timeout” is too coarse until you can see where the request stopped.
Duplicated delivery: repeats are a normal part of delivery
A second delivery is often treated as an infrastructure bug. In a distributed system it is a normal consequence of timeouts, lost responses, a restarted consumer, or a retry policy.
The same logical payment can arrive twice because the first attempt already changed state, while the successful 200 OK never reached the sender. It can arrive twice because the provider and an intermediary retried independently. It can arrive twice after a manual retry that someone triggered without noticing that the handler had already done the work.
Observability does not replace idempotency. It makes repeats visible. Without an attempt history, the team sees two orders and cannot tell whether they came from two events or from two deliveries of one event. With a history, the difference is visible: one Request and several attempts with the same delivery identity—or two Requests, meaning two separate intakes.
Comparing request bodies is not enough. Two independent invoices can share the same amount and currency. A retry can differ only in incidental headers. A stable provider event ID, plus an explicit idempotency key for a given delivery, is a much safer basis. We covered that boundary in Why webhooks need idempotent processing.
The observability point is narrower. If the system cannot show which attempt went to which destination, and in which order, idempotency has to be reconstructed from circumstantial evidence. That is slow, and it is easy to get wrong.
Partial failures: part of the path succeeded, and that hides the rest
A partial failure is especially easy to miss if you only have one 200 OK line.
The direct case is fan-out. One incoming webhook goes to a CRM, a billing service, and an internal notifier. The CRM returns 200 OK, billing returns 500, and notifications are still waiting for a connection. If you look only at the CRM log, the event looks processed. Billing is still incomplete. Notifications have not even started.
There is a quieter version inside a single handler. The application recorded the payment, failed to send the email, then crashed before it stored the idempotency key or returned an HTTP response. Transport sees an error or a timeout. The data already changed. A later attempt may “succeed” and send a second email—or skip the work while leaving part of the original action unfinished.
A partial result cannot be collapsed into one status without losing meaning. You need a separate history for each destination and a separate look at the application’s own state. A transport success for one Destination does not cancel a failure for another, and it does not prove that the business operation completed.
That is why intake status and delivery status should stay separate. A Request can be accepted and still sit in partial: some routes finished, some are still in retry, and some have exhausted their attempts.
Provider retries: the sender has its own policy
On top of your retries, the provider almost always has its own. It decides how many times to send the event, how long to wait, and when to give up. One service may retry for hours. Another may not retry at all.
If the provider posts directly to your handler, those policies mix with application behavior. A slow handler exceeds the sender’s timeout, and the same payload arrives again. The handler returns 500, and another series follows. The team sees a burst of similar requests and cannot immediately tell whether they are new events, provider retries, or retries from the delivery layer.
If intake and delivery are separated, the picture is clearer. The provider gets a fast response from the public receiver and usually stops there. Later retries belong to delivery toward the Destination. The history then shows one accepted Request and several delivery attempts—not several independent intakes that only look like the same payment.
Separation does not eliminate provider retries. They return whenever the public URL errors, takes too long, or is unreachable. Useful debugging therefore needs both edges: what the sender was told at intake, and what happened afterward for each Destination.
Until those two retry loops are visible as separate histories, a burst of identical payloads looks like a single incident. In practice they can be two completely different mechanisms.
Why a 200 OK in the logs is not enough
A 200 OK answers a narrow question: some HTTP server returned a successful status at some point. That is not enough for a webhook.
First, it is unclear whose success you are looking at. The provider log, the edge proxy, the Server in Adal, the Destination, application middleware, and a background queue can all record different events. Success at one boundary does not transfer to the others.
Second, a status code does not include the payload. It does not show signature headers, an idempotency key, an event ID, the path, or the query string. For webhooks those fields often matter more than the response code: they are how you tell a retry from a new event, and how you check that the body did not change in transit.
Third, one successful response does not tell you which attempt it was. The first delivery may have timed out after the work was done. The second may have returned 200 OK on an idempotent path. In application logs both can look like success, even though the side effect happened once—or, worse, twice.
Fourth, a 2xx confirms that the expected HTTP response was received. It does not confirm that the business operation finished. The handler may have returned success after enqueueing work and then crashed. It may have returned 200 OK before writing to the database. It may have completed only some Destinations.
Fifth, ordinary logs are a poor archive for webhooks. Requests can contain signatures, tokens, personal data, and payment details. Those should not be copied into a shared error log, analytics pipeline, or alert. An access log is incomplete by design. If that incomplete log is the only source of truth, there is nothing left to reconstruct after an incident.
In short, 200 OK is a transport signal. Webhook observability starts when that signal is tied to a specific intake, a specific destination, an attempt number, and the original request.
What you should be able to see for every webhook
The failures above cannot be diagnosed from a single status line. They need a connected history.
At intake, you should be able to see whether the request was accepted or rejected, when that happened, in which region, and with which method, path, headers, and body. If it was rejected, at least the reason should remain. Otherwise a missing payload cannot be distinguished from a sender-side failure.
Delivery needs a separate record for each Destination. Success on one route must not hide failure on another. Each attempt should show its number, timestamp, duration, and either an HTTP status or a network error. That is how you tell DNS from TLS, a timeout from a 500, and a CLI waiting for a connection from a delivery that already started and failed.
Three actions that look similar to an application also need to stay distinct:
Automatic retry → another attempt of the same delivery
Manual retry → a new attempt of an existing delivery
Replay → a new Request created from the stored original
Without that distinction, the team cannot tell why the handler saw “the same” webhook again: the system issued a retry, or someone started a Replay.
A delivery layer also does not replace application logs. It covers intake and transport. Whether the business operation finished is visible only in the receiving system’s own state. Honest observability draws that line instead of hiding it behind one green status.
How this looks in Adal
Adal separates intake, storage, and delivery so these questions can be answered from recorded data, not from what people remember after an incident.
External service → Server in Adal → Request → Delivery → Destination
An external service sends the webhook to the stable public HTTPS URL of a Server in Adal. An accepted webhook becomes a Request: the dashboard shows the method, path, query parameters, headers, body, received time, and related deliveries. That is the direct answer to payload lost. You can check whether the request arrived and what it contained, even if the handler was unavailable at the time.
Adal keeps a separate Delivery for each Destination. One Request can succeed on the first route, wait for the CLI on the second, and sit in retry on the third. Request statuses in the dashboard are received, delivering, success, partial, failed, and rejected. The aggregate status partial is exactly this case: some deliveries finished, others did not. That is an observable partial failure, not one shared 200 OK.
Each attempt is recorded: number, time, status, HTTP response code, duration, and, on failure, details of a network, DNS, TLS, or other delivery error. A timeout stops being a word in a ticket and becomes a specific attempt with a known stopping point.
Automatic and manual retries become visible as well. An automatic retry belongs to an existing Delivery and does not create a new Request. A manual retry repeats a specific delivery to one selected Destination. Replay creates a new Request. If a Destination is configured with the X-Adal-Idempotency header, automatic and manual retries keep the same key; Replay gets a new one, because it is a new operation inside Adal.
That split also untangles provider retries from delivery retries. Once a Server in Adal has accepted the request, the provider usually receives a successful intake response and stops. Later delivery to the application stays in the Request history. If the request never appears in the dashboard, the cause may be the sender, the network, or rejection on intake. For a Request with status rejected, Adal keeps only minimal troubleshooting data—no body and no headers.
The model has explicit limits, and they should not be overstated. A successful Delivery means Adal received a 2xx HTTP response from the Destination. It does not prove that the application finished the business operation. Adal does not store Destination response bodies or response headers. Request retention is limited by the plan and Server settings; after deletion, the history cannot be inspected or sent again. Adal does not provide exactly-once delivery. The handler still needs to be idempotent and, when the provider includes one, should rely on a stable event ID.
For outbound webhooks, Adal Outbound answers the same class of questions: the application hands over a message, and the queue, retries, and attempt history remain visible after 202 Accepted. In both directions, one rule holds: acceptance is not delivery.
Webhook observability does not come from an access log that says 200 OK. It comes from a stored intake record, a separate history for each destination, and a readable timeline of attempts. Then payload lost, timeout, duplicated delivery, partial failures, and provider retries stop being one vague “the webhook broke” and become distinct, inspectable cases.