Design a reliable webhook delivery platform

Question

## Scenario

Design a multi-tenant service that delivers customer events to externally managed HTTP endpoints. Customer endpoints may be slow, unavailable for hours, or return ambiguous failures. The platform must protect its own availability while giving customers secure, observable, and replayable delivery.

Assume high sustained event volume, bursty tenants, multiple endpoints per tenant, and payloads that must be delivered in order for some event streams.

## Functional requirements

- Register, verify, rotate secrets for, pause, and delete webhook endpoints.
- Accept an event once and fan it out to every eligible endpoint.
- Sign each request so customers can authenticate its origin and prevent replay attacks.
- Retry transient failures with bounded exponential backoff and jitter.
- Stop or quarantine deliveries after a configurable terminal policy.
- Let customers inspect attempts and replay a chosen event or time window.
- Enforce tenant isolation, quotas, payload limits, and retention rules.

## Start with these APIs

```text
POST /endpoints
POST /events
GET  /deliveries/{delivery_id}
POST /deliveries/{delivery_id}/replay
POST /endpoints/{endpoint_id}/rotate-secret
```

Refine the request and response contracts, including idempotency keys and pagination.

## Design discussion

Address:

1. Durable event ingestion, fanout, queue partitioning, and worker ownership.
2. Delivery state machine and the boundary between retryable and terminal failures.
3. At-least-once delivery, customer-side deduplication, and why arbitrary HTTP endpoints prevent a general exactly-once guarantee.
4. Per-endpoint ordering without allowing one failing destination to block unrelated tenants.
5. Request signing, secret rotation, timestamp tolerance, and SSRF defenses.
6. Backpressure, noisy-neighbor protection, observability, and capacity planning.
7. Dead-letter retention and safe replay semantics.

## Follow-ups

- One large customer’s endpoint is unavailable for twelve hours while its event rate stays high.
- A request times out after the customer processed it but before the platform received the response.
- Customers need strict order for each object but parallelism across objects.
- A secret is compromised and must be rotated without dropping in-flight deliveries.