Webhook failures are rarely caused by one problem alone: a request may be rejected because of an invalid signature, delayed by a transient outage, delivered more than once, or processed out of order. This guide provides a repeatable webhook debugging workflow, including signature verification, delivery tracing, retry analysis, idempotency, replay testing, and a maintenance cadence for keeping integrations reliable.
Overview
A webhook is an HTTP request sent by one system to notify another system that an event has occurred. The receiving endpoint may create a record, update an account, start a deployment, or trigger another API call. Because the sender and receiver operate independently, delivery is normally asynchronous and cannot be treated like a single, guaranteed function call.
When a webhook appears to be missing or incorrect, begin by separating the problem into four stages:
- Emission: Did the source system create the event and attempt delivery?
- Transport: Did the request reach the expected URL and network boundary?
- Authentication: Did the endpoint validate the signature and other request requirements?
- Processing: Did the application handle the event successfully and safely?
This separation prevents a common debugging mistake: changing application logic when the request never arrived, or changing network settings when the endpoint rejected a valid request. For status-code context, keep a searchable HTTP status codes API debugging reference nearby.
Design with two assumptions unless a provider explicitly documents stronger guarantees: webhook delivery can be retried, and events can arrive out of order. A robust consumer acknowledges valid requests quickly, records enough evidence to investigate them, and makes business effects idempotent.
What to track
Validate the endpoint before inspecting application code
Use this endpoint checklist for each environment:
- Confirm the public URL, HTTP method, and environment are correct.
- Verify that DNS, TLS, firewall, gateway, and authentication settings allow the sender to connect.
- Record the raw request body before parsing or normalizing it.
- Check that the endpoint accepts the sender's content type and required headers.
- Return a success response only after the request passes the checks required by your integration.
- Keep the handler fast; move slow work to a queue or background worker where appropriate.
- Log failures without storing access tokens, secret keys, or unnecessary personal data.
A useful first test is a controlled request to a non-production endpoint. Compare the provider's request with your application logs byte for byte, especially the body, timestamp header, event identifier, and signature header.
Verify webhook signatures correctly
Signature verification usually involves a shared secret and a message authentication code, often based on HMAC. The exact header names, digest format, signed fields, timestamp rules, and supported algorithms vary by provider, so follow the integration's current documentation rather than assuming a universal format.
A safe, provider-neutral verification sequence is:
- Read the request body as raw bytes or as the exact string received.
- Extract the signature, timestamp, and any key identifier from the headers.
- Reconstruct the signed message exactly as specified, such as a timestamp joined to the raw body.
- Compute the expected digest using the configured secret and algorithm.
- Compare the expected and received values using a constant-time comparison.
- Apply a reasonable timestamp or replay window when the provider supports one.
- Only then parse the payload and perform business actions.
Parsing and reserializing JSON before verification can change whitespace, escaping, or property formatting and cause a valid request to fail. Also check for accidental encoding changes, proxy-added transformations, and use of the wrong secret between development and production. A decoder can help inspect token-shaped values, but never paste live secrets or sensitive payloads into an untrusted browser tool. For related authentication troubleshooting, see this JWT decoder and token claims guide.
Capture an investigation-ready event record
Store structured delivery data rather than relying on a single text log line. Useful fields include:
- Internal request ID and provider delivery ID
- Event ID, event type, and event creation time
- Received time, processing start time, completion time, and duration
- Endpoint and deployment version
- HTTP method, response status, and response latency
- Signature-validation result and a reason category for failure
- Attempt number, retry indicator, and queue message ID
- Idempotency result, such as processed, duplicate, or in progress
- Redacted error class and downstream dependency status
These fields let you distinguish a provider retry from an internal queue retry and an actual duplicate event. Keep the event identifier searchable across ingress logs, queues, workers, and database writes.
Cadence and checkpoints
Webhook reliability should be checked on a schedule, not only during an incident. A lightweight monthly review is suitable for stable integrations; use a shorter review interval for new, high-volume, or business-critical connections.
Monthly checkpoint
- Review delivery success and failure counts by endpoint and event type.
- Look for rising latency, timeout responses, signature failures, and application errors.
- Inspect the oldest unprocessed queue item and any growing dead-letter queue.
- Confirm that duplicate-event handling is producing expected outcomes.
- Test one recent event in a safe environment using a redacted payload or provider-supported replay.
- Check that logs still include correlation IDs and useful failure categories.
Quarterly checkpoint
- Review secrets, key rotation procedures, and access permissions.
- Confirm that the endpoint URL, TLS configuration, firewall rules, and deployment configuration remain correct.
- Test a complete replay from receipt through database write and downstream action.
- Review retry and backoff behavior against current provider documentation.
- Verify that schema changes are backward-compatible or safely versioned.
- Remove obsolete endpoints, test credentials, and unused event subscriptions.
Keep a small incident timeline for each significant failure: first observed time, affected event types, delivery attempts, response codes, mitigation, and permanent fix. This turns future webhook troubleshooting into comparison rather than guesswork. If retries call another API, align the design with a documented rate-limiting and retry-safe client workflow.
How to interpret changes
Use response outcomes as clues
| Observed outcome | Likely area to inspect | Next action |
|---|---|---|
| No delivery record | Subscription, provider event creation, or routing | Confirm the event was emitted and the correct endpoint is subscribed. |
| Connection or timeout error | DNS, TLS, firewall, gateway, or handler latency | Check edge logs, reachability, and processing time. |
| 4xx response | Authentication, signature, schema, URL, or deliberate application rejection | Compare headers and raw payload with the endpoint contract. |
| 5xx response | Unhandled application or dependency failure | Trace the request through logs, queues, and downstream services. |
| Repeated successful deliveries | Missing or ineffective idempotency | Compare event IDs and inspect the deduplication store. |
| Events processed out of order | Independent delivery attempts or asynchronous workers | Use event timestamps, versions, or state checks before applying changes. |
Do not automatically classify every repeated event as a provider defect. A timeout can occur after the receiver completes the database write but before the sender sees the response, causing a legitimate retry. Likewise, a queue worker may retry after a process crash. The remedy is the same: make the side effect safe to repeat.
Apply idempotency at the business boundary
Before performing an irreversible action, record a provider event ID or another stable deduplication key in durable storage. Enforce uniqueness at the database level where possible. A simplified pattern is:
begin transaction
insert event_id into processed_events
if duplicate:
commit and return success
apply business change
commit
return successThe insert and business change should be coordinated so that a crash cannot mark an event complete while leaving the actual operation unfinished. For external side effects, such as sending email or charging an account, use an outbox or job record with its own idempotency key. If ordering matters, compare event versions or timestamps and reject stale updates rather than trusting arrival order alone.
Replay safely
Replay testing should use a sandbox, staging account, or isolated test data. Redact sensitive fields, preserve the original event ID when testing deduplication, and use a new test identifier when testing first-time processing. Verify both outcomes: the expected action occurs once, and a repeated delivery does not create a second action.
A replay is most useful when it captures the complete path: ingress, signature validation, queueing, worker execution, database transaction, and downstream request. Avoid testing only the controller, because many failures occur in the boundaries around it.
When to revisit
Revisit this guide whenever a webhook integration is added, an endpoint moves, a secret is rotated, a payload schema changes, or a queue and worker architecture is introduced. It should also become part of incident review after missed deliveries, duplicate business actions, unexplained signature failures, or a change in provider retry behavior.
Use the monthly and quarterly checkpoints as a practical maintenance schedule. Update the runbook when your provider changes signature algorithms, header formats, event identifiers, replay controls, or delivery policies. Keep provider-specific details in a separate integration record so the core troubleshooting method remains stable.
For an actionable next step, create one dashboard or saved query per endpoint showing delivery attempts, response classes, signature failures, processing latency, duplicate events, and oldest unprocessed item. Then perform one controlled replay and confirm that your logs answer five questions without manual reconstruction: what event arrived, when it arrived, whether it was authentic, what attempt number it was, and whether the business effect happened exactly once. That small checklist provides an ongoing baseline for webhook debugging and makes the next incident easier to contain.