179: System Design - Reliability Patterns

Learning outcomes

reason about retries, idempotency, and duplicate work;
explain replication, partitioning, and rate limiting;
design failure isolation and useful observability.

Practice

Add an asynchronous notification worker to the task service. Define an idempotency key, retry policy, dead-letter behavior, backpressure, metrics, and user-visible status.

Checkpoint

Explain why retries can amplify an outage and how the design prevents a duplicated notification.

Failure matrix

For each dependency record timeout, retry limit, fallback, user-visible result, metric, and operator action. Exponential backoff without a cap can make recovery slow; unlimited retries can exhaust workers. Idempotency keys must have a retention policy and must bind to the authenticated operation, not just a client-chosen string.

Worked policy

For notification delivery, acknowledge the task write after the outbox commit, not after email delivery. A worker leases an event, retries transient failures with bounded exponential backoff and jitter, then moves it to a dead-letter queue. Permanent validation failures go directly to dead letter. The consumer stores (eventId, destination) as a unique key or uses a provider idempotency key, making at-least-once delivery observable rather than pretending it is exactly once.

Protect the dependency with per-tenant rate limits, a bounded queue, circuit breaking, and a concurrency limit. Metrics should include queue age, retry count, dead-letter count, delivery latency, and provider errors. Expose “queued,” “sent,” or “failed” status without exposing secrets. Define replay permissions and prevent a replay from bypassing tenant authorization.

Scoring and edge cases

Score 2 points each for retry classification, idempotency scope/retention, backpressure, dead-letter recovery, observability, and user-facing semantics. Walk through provider timeout after acceptance, duplicate worker lease, poison message, queue full, clock skew, and a tenant exceeding quota. Include an operator action for every alert.