The Circuit Breaker: Failing Fast So You Can Recover

A dependency starts timing out. Every request to it now takes the full 30-second timeout before failing.

Your service keeps calling it, because that is what the code says to do. Each call occupies a worker for 30 seconds. Within a minute every worker is blocked waiting on a service that is not going to answer, and your service is now down too — not because it broke, but because it patiently waited for something that had.

Meanwhile the struggling dependency is receiving your full traffic plus your retries, which is precisely the load preventing it from recovering.

That is cascading failure, and it is what a circuit breaker exists to stop.

Three states

Closed — normal. Calls pass through, outcomes are recorded.

Open — the failure rate crossed the threshold. Calls fail immediately, without being attempted. No thread is held, no timeout is waited on, no load reaches the dependency. Your service degrades in milliseconds instead of hanging.

Half-open — after a cooldown, let one or a few calls through. If they succeed, close. If not, open again and wait longer.

        failures > threshold
Closed ─────────────────────► Open
   ▲                            │ after cooldown
   │ trial succeeds             ▼
   └──────────────────── Half-open
              trial fails  ──────┘

The half-open state is the important one. Without it you either stay broken forever or flood the recovering service the moment the timer expires.

Getting the threshold right

Use a rate over a rolling window, with a minimum volume.

A raw count — “open after 5 failures” — misbehaves at both extremes. On a low-traffic endpoint, five failures might be every call you received all hour, and the circuit opens on what was really a rounding error. On a high-traffic one, five failures out of ten thousand is normal and the circuit opens constantly.

A sane default shape:

• Open at 50% failure rate
• Over a rolling window of 20+ calls (below that, do nothing)
• Cooldown of 10–30 seconds before half-open
• 1–3 trial calls in half-open

Count timeouts as failures, and do not count business errors. A 404 or a 422 means the dependency is working perfectly and told you something. Opening a circuit because users submitted invalid data is a self-inflicted outage. Count connection failures, timeouts and 5xx responses only.

It only works with timeouts

A circuit breaker measures failures. A call that hangs forever never fails, so it never gets counted, and the breaker never trips while every worker is stuck.

The timeout is what makes the breaker possible. Set it from the dependency’s real p99 plus headroom — not from a comfortable-sounding round number. If p99 is 200 ms, a 30-second timeout means you wait 150× longer than the service ever legitimately takes before admitting something is wrong.

The full set works together:

• Timeout — bound how long one call can take
• Retry with jittered backoff — absorb genuine transient blips
• Circuit breaker — stop calling when failure is sustained
• Bulkhead — cap concurrent calls per dependency so one cannot consume every worker

Retry and break are not alternatives. Retry handles the single dropped packet; the breaker handles the dependency being down. And retries must be jittered — synchronised retries from many clients recreate the thundering herd you were avoiding.

What to do when it is open

An open circuit means you answer immediately. What you answer is a product decision, and it should be made deliberately rather than defaulting to a 500:

• Serve stale. A cached previous value, with a note that it may be out of date. Often the best answer.
• Degrade the feature. Hide recommendations rather than failing the page.
• Queue it. If the operation can be asynchronous, accept it and process later — with an outbox so it cannot be lost.
• Fail cleanly. Sometimes correct. A checkout that cannot reach payments should say so quickly, not hang.

Deciding this per dependency in advance is most of the value of adopting the pattern at all.

Where to put it

Per dependency, not per service. One breaker for “all outbound calls” means a failing analytics endpoint stops your payment calls. Each downstream gets its own.

Sometimes finer: an endpoint that is slow and one that is fine on the same service may deserve separate breakers.

If you run a service mesh, breakers are available at the proxy layer — no application code, consistent policy, and metrics for free. In-process libraries (Resilience4j, Polly, opossum) give you finer control and per-call fallbacks. Either is fine; having none is not.

Instrument it

A breaker that opens silently converts an outage into a mystery. Emit state transitions as events and alert on them — “circuit to payments opened” is one of the highest-signal alerts you can have, because it fires at the moment of failure and names the dependency.

Track time spent open per dependency. That number is the honest measure of which downstream is costing you availability.

What you have actually built

Not a way to keep working when a dependency is down — nothing does that except not needing the dependency.

You have built a fast, bounded failure instead of a slow, spreading one. The feature that needed the broken service is still broken. Everything else keeps serving, your workers stay free, and the struggling dependency gets the quiet it needs to come back. That is the entire goal: contain the blast radius to the thing that actually failed.