What is a webhook black hole?

The provider says delivery successful. Your handler logs are empty, no database record was created, and the expected business action never happened. The team checks the URL, restarts the application, and sends the event again—sometimes creating a duplicate instead of solving the problem.

This kind of incident is often described as a webhook black hole: the request appears to enter the system and then vanish. In reality, webhooks do not disappear into one abstract “network.” There are several boundaries between the sender and the business operation, and success can mean something different at each one.

To find a missing webhook, start by identifying the last boundary it is known to have crossed. This article looks at what a green delivery status may actually represent and how to investigate five common causes and edge cases: DNS, TLS, firewalls, a timeout near the HTTP response boundary, and an application crash after an early acknowledgement.

What does delivery successful actually mean?

There is no universal definition of a successful webhook delivery. Depending on the provider, success may mean that:

• the event was created inside the provider’s system;

• the message was accepted into its delivery queue;

• one delivery attempt received a 2xx HTTP response;

• the latest attempt succeeded after one or more earlier failures;

• the receiving service acknowledged the request but has not completed the business operation yet.

A 202 Accepted response, for example, normally says only that the request was accepted for asynchronous processing. Even 200 OK and 204 No Content confirm an HTTP outcome, not the application state behind it. They do not prove that a payment was recorded, an order was updated, an event reached a durable queue, or a transaction committed.

So the first question for a provider—or for your own delivery system—should not be “Why does this say success?” A more useful question is:

What exact event marks a delivery as successful, and which attempt does this status describe?

If the answer is “the configured URL returned 2xx,” the search space becomes much smaller. A complete DNS failure, failed TLS handshake, or direct firewall block could not prevent that same attempt from reaching the same final application while also allowing the sender to receive its HTTP response. Either a different address answered, an intermediary generated the response, or the application replied before it had safely accepted responsibility for the event.

DNS: the request reached the wrong place

A DNS problem does not always look like NXDOMAIN or a resolver error. When name resolution fails completely, the delivery attempt fails as well because the sender cannot establish an HTTP connection.

The more deceptive case is successful resolution to the wrong address:

• an old DNS record survived a migration;

• resolvers still return different values because older cached records have not expired;

• the record points to an old load balancer or environment;

• IPv4 and IPv6 lead to different configurations;

• the webhook URL has the right domain but the wrong region, subdomain, or path;

• a redirect sends the request to another hostname.

An old server may still return 200 OK. The provider then shows a successful delivery, while the current backend has no record of the request because it never received it.

Do not verify only the domain name you expect to be configured. Reconstruct the route taken by the actual attempt:

1. Copy the webhook URL from the provider’s settings, not from documentation or a local .env file.

2. Check whether the sender follows redirects and which URL it ultimately used.

3. Compare DNS answers from multiple networks or regions with the addresses of the current load balancer.

4. Inspect both A and AAAA records, their TTLs, and recent DNS changes.

5. Search the access logs of every host that might still serve the old address.

A stable event identifier is especially useful here. Time alone is a weak correlation key: clocks may differ, and a retry might happen several minutes after the original attempt.

TLS: the secure connection ended somewhere else

If the TLS handshake with the configured URL fails, the HTTP request has not started yet. Common causes include an expired certificate, a hostname mismatch, an incomplete certificate chain, an unsupported TLS version, incorrect SNI, or a mutual TLS configuration problem. A useful delivery log should show that attempt as failed, not as delivery successful.

In many production architectures, however, TLS does not terminate in the application itself:

The external TLS connection can work perfectly while the load balancer is unable to connect to the backend. If the edge component generates its own 2xx response—for example, after accepting the request into an internal queue—the sender sees success even though the application did not receive the request.

Treat these as two separate connections:

• TLS between the sender and the public endpoint;

• TLS or plain HTTP between the edge component and the application.

Verify the public certificate, hostname, SNI, and certificate chain, then inspect the load balancer’s upstream errors separately. A successful external handshake says nothing about the availability of the internal service.

Firewalls, WAFs, and load balancers: the edge accepted it, the backend did not

A firewall that directly blocks the sender’s IP address will usually cause a connection timeout or connection refused. The blocked application cannot return 2xx in that same attempt.

But a firewall is rarely the only component in the path. A CDN, WAF, API gateway, ingress controller, or reverse proxy may sit in front of the application, creating two distinct boundaries:

The public edge may remain available while a security group, network policy, or firewall rule blocks the internal route. Other possibilities include:

• the WAF matched a rule against the webhook payload;

• the load balancer selected the wrong target group;

• the ingress configuration does not recognize the requested host or path;

• a health check passes even though the webhook handler on that instance is broken;

• the edge replies before the request has been handed off safely;

• an IP allowlist contains outdated provider egress addresses.

A normal WAF block is more likely to return 4xx than produce a green delivery status. If the dashboard remains green, success probably refers to another attempt, represents a different stage of delivery, or hides an error that an intermediary converted into a successful response.

Correlate the stable event ID or delivery ID, along with request and trace IDs when they are propagated consistently through every layer. Follow those identifiers across the CDN or WAF, load balancer, reverse proxy, and application logs. If the event appears at the edge but not at the next hop, the black hole is somewhere between those two components. The provider’s overall status adds little at that point.

Be particularly careful with rules that turn an internal failure into a successful HTTP response. This kind of fallback can keep a dashboard green while hiding an upstream outage.

A timeout near the response boundary

Timeouts create one of the most ambiguous delivery outcomes. The application may have received the webhook and completed the operation, but the sender did not receive the complete HTTP response before its deadline.

If the dashboard shows only the final result, the team sees successful even though the first attempt timed out. Searching only around the timestamp of the successful attempt may miss the business operation because it happened earlier. Conversely, manually resending the “missing” webhook may execute an action that already completed.

A timeout therefore does not mean “the backend received nothing.” It means only that the sender did not receive the expected response in time. Review every attempt, including its number, start time, duration, and stable event ID. Before triggering a manual retry, check whether the operation was already recorded in a database, queue, or external system.

Idempotent processing is the main protection against this class of failure. Receiving the same event again should not create a second payment, order, or notification. We cover the principle in more detail in Why webhooks need idempotent processing.

An application crash after ACK

This is the most direct explanation for “successful, but no operation occurred.” The handler returned 2xx and only then tried to make the event durable:

If the process crashes after the acknowledgement but before the durable write, the sender will not retry. From its point of view, delivery has already succeeded. The event was lost inside the receiving application.

The same failure pattern appears when a handler:

• starts a background goroutine, promise, or task without a durable queue;

• writes the event to a buffer that is flushed asynchronously;

• acknowledges the request before the database transaction commits;

• ignores an error while publishing to an internal queue;

• returns 2xx from a finally block, recovery middleware, or a shared error handler;

• responds after validation but before taking responsibility for further processing.

The safe ACK boundary does not have to come after the entire business operation. It does need to come after the event has been accepted durably. An application can atomically store the webhook or publish it to a durable queue, then return 2xx quickly. Processing remains asynchronous, but restarting the process no longer destroys the only copy of the event.

The practical rule is simple:

Return 2xx only after the system has genuinely accepted responsibility for the event.

What counts as acceptance depends on the architecture: a committed database write, a confirmed publication to a durable queue, or another operation that survives a process crash.

How to find a missing webhook

Do not start by sending it again. First collect the identifiers and reconstruct one specific delivery attempt.

1. Identify the event

Collect the provider’s event ID, delivery ID, exact timestamp with timezone, event type, and URL from the actual configuration. If there is no stable identifier, record a hash of the payload and a few fields that distinguish this event from similar ones.

2. Define success

Find the HTTP status, attempt number, duration, final URL after redirects, and any earlier failures. A status that means “accepted into the sender’s queue” is not evidence that the receiver got the request.

3. Find the last confirmed layer

• A DNS error means the HTTP request never reached the public endpoint.

• A TLS error means the connection stopped before HTTP.

• The edge logged the request but the reverse proxy did not: inspect the route between them.

• The proxy logged 2xx but the application did not see the request: determine which component generated the response and which upstream received the request.

• The application logged the request but there is no business result: inspect the acknowledgement boundary, transaction, queue, and process lifecycle.

4. Correlate every attempt

Do not stop at the latest green row. The first attempt may have changed state and then timed out, while a retry merely confirmed an event that had already been processed.

5. Retry only after those checks

Before resending, verify the handler’s idempotency and the current state of the business operation. A retry is a recovery and diagnostic tool, not a button that is safe by default.

How Adal makes delivery boundaries visible

In the inbound flow, Adal separates receiving a webhook from delivering it to an application:

The external service first sends the webhook to the public HTTPS URL of a Server in Adal. Adal stores an accepted request as a Request, so the dashboard can show its method, path, query parameters, headers, body, and received time independently of the state of the final handler.

Adal keeps a separate delivery attempt history for every Destination. It includes the attempt number and time, status, response status code, duration, and—when a delivery fails—details about a network, DNS, TLS, or other error. This makes “Adal could not resolve the hostname” visibly different from “the Destination returned 500” and from “the Destination returned 2xx, but its internal operation did not happen.”

The boundary is explicit: a successful Delivery means that Adal received a 2xx HTTP response from the Destination. It does not prove that the application completed its business operation, stored the data, or processed the payload correctly. Adal does not schedule an automatic retry after a successful response, so the application should acknowledge the webhook only after it has accepted responsibility for the event durably.

For applications that send webhooks to their own customers, Adal Outbound keeps a history of outbound attempts. Its diagnostic data separates DNS resolution, connection setup, TLS, time to first byte, total request time, redirects, and the final HTTP status. A 202 Accepted response from Outbound means only that Adal accepted the message into its delivery queue; the actual result appears later in the attempt history.

Adal does not replace the receiving application’s logs, durable queue, or monitoring. It makes the transport part of the path observable and draws a clear boundary around what was received, where it was sent, and how the Destination responded. That turns a webhook black hole from a disagreement—“we sent it” versus “we never got it”—into a sequence of testable hops.

A green status needs a precise boundary

Webhook delivery successful is not the end of the investigation. It is useful only when you know which component responded, which URL was called, which attempt succeeded, and what the system defines as success.

DNS and TLS can stop a request before HTTP begins. A firewall or routing mistake can leave it between the edge and the backend. A timeout can hide an operation that already completed and trigger a duplicate attempt. An early ACK can confirm an event seconds before a process crash destroys its only non-durable copy.

Do not look for one universal black hole. Find the last confirmed boundary, correlate every attempt with a stable event ID, and continue from there. Once the path is divided into observable stages, “successfully delivered, but nothing received” stops being a paradox and becomes a specific technical incident with a cause you can verify.