The CAP Theorem: You Only Choose During a Partition

The CAP theorem is the most cited and most misquoted idea in distributed systems. The popular version — “consistency, availability, partition tolerance: pick two” — makes it sound like a menu with three dishes. It is not, and the accurate version is both narrower and far more useful.

Partition tolerance was never optional

A partition is when the network splits, and nodes that should be able to talk to each other cannot. Packets get dropped, a switch dies, a data centre link goes down.

You do not get to opt out of that. If your system runs on more than one machine, partitions will happen — the only question is what it does when one does. So “give up partition tolerance” is not a design choice, it is a design that breaks during an event you were guaranteed to encounter.

Which collapses the menu. The theorem is really:

When a partition occurs, you must choose between consistency and availability.

What the choice looks like

Two nodes, and the link between them is down. A write arrives at node A. Node B cannot be told about it. A read now arrives at node B.

Choose consistency (CP). Node B refuses to answer, because it cannot know whether its copy is current. The client gets an error or a timeout. Nothing wrong is ever returned — but the system is partly down.

Choose availability (AP). Node B answers with what it has, which is now out of date. The client gets a response every time — and sometimes that response is wrong.

That is the whole theorem. Everything else is consequences.

Which is right depends on the cost of being wrong

There is no generally correct answer, only a domain-specific one.

Pick consistency when a stale answer causes harm. Account balances, stock levels for the last item, permissions and access control, anything where two users acting on stale data creates a state you cannot unwind. An error message is recoverable; a double-spend is not.

Pick availability when a stale answer is merely imperfect. Product descriptions, a social feed, view counts, a recommendation list, cached search results. Nobody is harmed by a like count that is thirty seconds behind, and being unreachable costs real money.

The instinct to say “we need consistency” for everything is worth resisting — most data in most applications is the second kind, and paying availability for consistency you did not need is how systems become fragile for no benefit.

The part CAP leaves out

CAP only describes behaviour during a partition, which might be minutes a year. It says nothing about the other 99.99% of the time — and that is where the trade-off you actually feel lives.

PACELC fills the gap: if Partition, choose Availability or Consistency; Else, choose Latency or Consistency.

Strong consistency is not free when the network is healthy either. Confirming a write on a majority of replicas means waiting for round trips, possibly across regions. That is latency you pay on every single request, in exchange for correctness during an event that may never happen this quarter.

This is usually the more important decision, and CAP alone will never surface it.

Where real systems sit

The labels are coarser than the systems, and most are configurable:

• PostgreSQL, MySQL — a single node is not distributed and CAP does not apply. With synchronous replication, CP. With async replication, replica reads can be stale.
• Cassandra, DynamoDB — AP by default, but tunable per query. Cassandra’s consistency levels and DynamoDB’s optional strongly-consistent reads let you choose per operation.
• etcd, ZooKeeper, Consul — firmly CP. They exist to hold configuration and leadership state where a wrong answer is worse than no answer, so a minority partition stops serving.
• MongoDB — CP for writes to the primary; secondary reads can be stale unless you ask otherwise.

The useful observation is that the choice is usually per-operation, not per-database. Modern systems let one query demand strong consistency and the next accept staleness, which is closer to how applications actually work: the balance check and the marketing banner do not need the same guarantees.

What you have actually decided

Not an architecture style, and not a database brand.

You have decided what your system says when it does not know the answer. Consistency says “I would rather tell you nothing than tell you something wrong.” Availability says “here is my best guess, quickly.” Both are defensible. Choosing without saying which one you picked, per operation, is what produces systems nobody can reason about at 3 a.m.