Not "Kafka has brokers, topics, partitions." Instead: two Principal Architects in a design review, where nothing is allowed to be a definition — every concept has to earn its place by solving a problem the previous decision created.
I don't want Kafka. I want a distributed event streaming platform. Convince me we even need one.
An e-commerce company generates orders, payments, inventory updates, notifications, analytics events, audit logs — and every service wants these events. How would you connect them? Form your own answer before revealing.
Option 1: every service directly calls every other service.
Order |------ Payment |------ Analytics |------ Email |------ Inventory |------ Fraud |------ Recommendation
What's wrong with that? Don't say "tight coupling." Everyone says that. Explain.
Order service produces one event. Today, 5 consumers. Tomorrow, 50. Now Order service is suddenly responsible for retries, failures, back pressure, consumer availability, delivery guarantees. The producer has accidentally become a messaging platform — that violates single responsibility.
So producers shouldn't know consumers. Why?
Producers create events. Consumers decide what to do with events. Those responsibilities evolve independently. If the producer knows its consumers, every new consumer requires a producer deployment. That's architectural debt.
So introduce a broker. Why?
A broker separates time. Producer says "I've produced something." Consumer says "I'll process it whenever I'm ready." Neither waits for the other.
Producer --> [Broker] --> Consumer Producer doesn't wait for consumer. Consumer doesn't wait for producer. Time coupling disappears.
Good. But you've only solved decoupling. Now convince me this isn't just RabbitMQ.
What is the primary optimization goal? Choose only one: latency, memory, CPU, storage, network, throughput, durability, consistency, availability. You cannot optimize everything.
Pick one. Then justify from hardware, not from a feature list, why that architecture naturally follows.
Throughput. Not latency.
Why? Latency sounds more important.
At 1 million messages/sec, saving 2 milliseconds is meaningless if your architecture can't sustain the load in the first place. Kafka was designed for massive event pipelines, not request-response systems. Throughput dominates; latency becomes secondary.
If throughput is king, what architecture naturally gives maximum throughput? Don't say partitions. Think lower.
Sequential writes.
Why? Explain from hardware.
RANDOM WRITE: find location -> move disk head/pointer -> update metadata -> maybe touch indexes (expensive, unpredictable) SEQUENTIAL WRITE: [existing data][NEW DATA APPENDED HERE] (just append — cheap, predictable)
Hard disks, SSDs, OS page cache, CPU caches — everything prefers sequential access. Modern SSDs are also far more efficient with sequential writes, due to lower write amplification and simpler internal management.
Then why does every database perform random writes?
Because databases optimize queries — "how do I find a record quickly?" Kafka optimizes appends — "how do I write the next record as fast as possible?" Two different goals.
Suppose every message is stored forever: M1, M2, M3, M4, M5. Why not update messages instead of just appending?
What's fundamentally different about an "event" compared to a row in a database table?
Events represent history. History should never change. "Order placed," "payment completed," "refund issued" — those are facts. Facts should be immutable.
Why is immutability important?
Because history becomes reproducible — analytics, auditing, reprocessing, disaster recovery, machine learning, all of it depends on replaying the exact same history. If history changes, truth changes.
So Kafka stores history, not state.
DATABASE: stores current truth Account balance: ₹40,000 <- one row, overwritten each update KAFKA: stores how truth evolved [Deposit ₹10,000][Withdraw ₹5,000][Deposit ₹35,000] <- the whole story, never overwritten
Exactly. Databases store current truth. Kafka stores how truth evolved.
Events are facts about what happened. Facts don't get updated. If you design a system that needs to "correct" an event, you're not modeling events — you're modeling state, and you need a different tool.
Consumer finishes reading — delete the message. Simple. Why doesn't Kafka do that?
If deleting-after-read seems obviously wrong, name the exact scenario that breaks it.
Question back: which consumer? Analytics? Audit? Fraud? Backup? A new consumer joining tomorrow?
Event: "Order Placed"
|
-----+------+------+---------+
| | | | |
Analytics Audit Fraud Backup (new consumer, joins next month)
Different consumers move at different speeds. Deleting after the first read destroys history for everyone else.
So who owns progress?
The consumer. Not the broker. Broker owns data. Consumer owns position. That's why offsets belong to consumers, not to Kafka.
That decision changes everything.
One log. One machine. Can we scale forever?
What exactly does "partitions" solve — and what's the hidden cost of having too many of them?
No. Eventually CPU, disk, NIC, memory all saturate. We need horizontal scaling.
Most people immediately say "partitions." I don't want that answer. Tell me what problem partitions actually solve.
BEFORE: AFTER: 1 log 100 logs 1 writer 100 writers Concurrency appears naturally — each log is independent.
Now challenge yourself. If partitions increase parallelism, why not create 100,000 partitions?
Because partitions aren't free. Every partition carries:
per partition: - metadata - leader election state - replication state - network connections - open files - page cache usage - memory overhead 100,000 partitions -> 100,000x this overhead Eventually coordination cost EXCEEDS throughput gain.
Exactly. Architecture is always balancing one bottleneck against another.
More partitions means more parallelism — up to a point. Beyond that point, you're paying coordination cost for throughput you'll never see. Picking the right partition count is a capacity decision, not a "bigger is better" default.
The CEO says "I want perfect ordering." Can you give it?
Can you have both perfect global ordering and massive parallel throughput? If not, why not — in one sentence?
Yes — one partition.
Can you still do 5 million messages/sec?
No. One ordered stream means one serialization point.
GLOBAL ORDER: MASSIVE PARALLELISM: [1 log][1 writer] [log][log][log]...[log] perfectly ordered 100 independent writers throughput capped no single global order You get ONE of these. Not both.
Exactly. That's the first unavoidable trade-off in Kafka. You don't "implement" your way around it — you choose one side.
Why replicate? Don't say "high availability." Everyone says that.
If three replicas exist, who should be allowed to accept writes — all three, or one? Justify it.
Replication exists because hardware eventually fails. The real question isn't if a broker dies — it's when. The architecture has to treat failure as normal, not exceptional.
Good. Three replicas — who should accept writes? All three?
ALL THREE ACCEPT WRITES: Replica 1: [A][B] <- network partition happens Replica 2: [A][C] <- diverged history! Replica 3: [A][B] Which one is "correct"? No good answer once histories diverge. ONE LEADER ACCEPTS WRITES: Leader: [A][B][C] <- one authoritative order Follower 1: [A][B][C] <- copies leader's order Follower 2: [A][B][C] <- copies leader's order
If multiple replicas independently accept writes, they can diverge during a network partition — reconciling conflicting histories while preserving one ordered log becomes extremely hard. A single leader gives one authoritative write order; followers just replicate it.
What did we sacrifice?
Write availability during leader transitions. We traded some availability for a consistent, ordered log.
Producer sends a message. Leader writes it. Leader crashes. Producer times out. Producer retries. Did we create a duplicate?
If the answer is "possibly," what's the minimum piece of information the broker needs to tell a retry apart from a genuinely new message?
Possibly.
How do we fix it?
The broker must recognize that a retry is the same logical send, not a new one.
Producer sends "Order Created" -----> Leader writes it -----> *** crashes before ACK *** Producer times out -----> retries -----> "is this new, or the same one again?" Fix: give the producer an IDENTITY + a SEQUENCE NUMBER producer_id=100, sequence=55 --> broker remembers this pair retry arrives with the SAME (100,55) --> broker recognizes it, discards the duplicate
Now you've entered the realm of exactly-once semantics. Notice how we arrived there naturally, from failure analysis — not because it was a feature on a slide.
At-least-once delivery is what you get by default when retries exist. Exactly-once requires the consumer to be idempotent. You don't get to skip that design step — you just choose when to do it.
You have a leader and two followers. One follower is on a slow disk and consistently lags. Should the leader wait for it before acking a producer with acks=all?
If "wait for everyone" is too slow and "wait for no one" is too risky, what's the actual rule that should decide who counts?
No — not forever. If the leader always waits for every replica, one permanently slow disk caps the throughput of the entire partition for everyone.
So drop the requirement to replicate to it at all?
No — that's the other extreme, and it silently weakens durability. The real answer is a dynamic set: replicas that are currently caught up count; the ones that fall behind temporarily stop counting, without being abandoned.
Leader: offset 105
Follower A: offset 105 <- caught up
Follower B: offset 60 <- far behind (slow disk)
ISR (In-Sync Replicas) = { Leader, Follower A }
acks=all only waits for THESE two, not for B
Follower B keeps replicating in the background.
Once it catches up -> rejoins the ISR -> counts again.So ISR isn't a fixed replication factor — it's a moving target?
Exactly. Replication factor is "how many copies exist, eventually." ISR is "how many of those copies can I actually trust right now to define durability." Those are two different numbers, and conflating them is a common mistake.
The leader dies. Follower A was in the ISR. Follower B was not — it was lagging. Who becomes the new leader?
What breaks if you let the lagging follower become leader? What breaks if you refuse to?
Follower A — the one that was actually in the ISR. Electing B risks losing every message B never replicated.
Leader (dead): [100][101][102][103][104][105] Follower A (ISR): [100][101][102][103][104][105] <- fully caught up Follower B (lagging): [100][101][102] <- missing 103,104,105 Elect A --> no data lost, those messages survive Elect B --> messages 103,104,105 are GONE, silently
So you'd never elect B, under any circumstances?
That's the "clean" vs "unclean" leader election choice. Clean election refuses to elect anyone outside the ISR — if no ISR member is available, the partition simply goes offline until one comes back. Unclean election allows electing an out-of-ISR replica to restore availability, accepting silent data loss.
Which one does a payments system choose?
Clean, always. An offline partition is a visible, debuggable outage. Silently losing transactions is a hidden one — and hidden failures are far more dangerous in a financial system than visible ones.
unclean.leader.election.enable), not a fixed architectural answer, precisely because different topics can reasonably choose differently.You've now built ISR and leader election. Given that, why does Kafka expose three different acks settings instead of just always waiting for the full ISR?
Name one real workload where waiting for the full ISR would be actively the wrong choice.
Because not every message is worth the latency cost of waiting for the ISR. A UI click-tracking event and a money-transfer event have completely different loss tolerances.
acks=0: producer fires and forgets, doesn't even check (fastest, riskiest) acks=1: leader writes locally, acks immediately (leader-durable only) acks=all: leader waits for the full ISR before acking (slowest, safest) click-tracking event --> acks=0 or 1, losing one is fine money-transfer event --> acks=all, non-negotiable
So this isn't a broker setting — it's a producer decision, made per message?
Per producer, typically scoped per topic. The broker enforces a floor (min.insync.replicas) — the producer chooses how hard to push against that floor. Neither side alone decides the durability contract; it's negotiated.
Multiple consumers read the same topic in parallel, each owning a subset of partitions. A consumer crashes. Who decides how to reassign its partitions — and why does that question even need a dedicated answer?
What goes wrong if the surviving consumers negotiate the reassignment among themselves, with no single decision-maker?
Consumer A: "I think I own partition 2 now" Consumer C: "No, I already claimed partition 2" Consumer A: "Are you sure?" Two consumers reading the same partition -- double processing. Or: nobody claims it -- partition goes unread.
Without one decision-maker, you get exactly this — a distributed agreement problem, solved badly, ad hoc, by the consumers themselves.
So centralize it. Doesn't that become a bottleneck?
Only if it sat on the hot path — and it doesn't. It only acts on join, leave, or crash. Those are rare events. The actual message flow (produce, fetch) never touches this coordinator at all.
Hot path (constant): Producer --> Broker --> Consumer Rare path (occasional): Consumer join/leave/crash --> Coordinator reassigns
What's the actual risk, then?
Not centralization — a rebalance storm: if consumers join and leave faster than assignment can stabilize, the group spends more time reassigning than actually reading. That's a tuning problem (session timeouts, static membership), not a reason to abandon centralization.
Why does the consumer ask the broker for messages, instead of the broker pushing messages to the consumer as they arrive?
What happens to a slow consumer under a push model that can't happen under a pull model?
PUSH MODEL: Broker --> fires messages at consumer as fast as they arrive Slow consumer --> messages pile up somewhere (broker buffer? consumer memory?) Broker now has to guess every consumer's capacity, per consumer PULL MODEL: Consumer --> "give me the next batch, when I'm ready" Broker --> just serves what's asked, never guesses Slow consumer simply pulls less often -- no special handling needed
Under push, the broker has to actively manage backpressure for every consumer — a genuinely hard, per-consumer problem. Under pull, backpressure is just... the natural consequence of the consumer asking less often. It's not solved; it's avoided.
Doesn't pull mean higher latency — the consumer has to keep asking?
Only if it's a naive poll-in-a-tight-loop. Kafka's fetch is a long-poll — the broker holds the request open until either data arrives or a timeout hits, so pull doesn't have to mean "ask constantly." You get pull's simplicity without paying the naive-polling latency tax.
You've said "sequential writes" three times now. That explains fast writes. What makes reads fast — a consumer fetching data that was written five minutes ago?
Where does data actually live between "written to disk" and "sent over the network to a consumer" — and how many times does it get copied along the way?
Two things: the OS page cache, and zero-copy transfer.
NAIVE read path (multiple copies):
Disk -> kernel buffer -> application memory -> socket buffer -> NIC
(copy 1) (copy 2) (copy 3)
Every copy costs CPU cycles and memory bandwidth -- for data
the application never actually needed to touch or transform.So avoid the copies?
ZERO-COPY read path (sendfile-style):
Disk -> kernel buffer -----------------------> NIC
(data never enters application memory at all)
Kafka doesn't deserialize, transform, or touch the bytes on the read path — it just moves them. That means the OS can transfer straight from the page cache to the network socket, skipping the application-memory hop entirely.
And the page cache — why does that matter separately from zero-copy?
Recently-written segments are still sitting in the OS's page cache in RAM — most "reads" for recent data never actually touch the physical disk at all. Combined with zero-copy, a consumer reading data from a few seconds ago is close to a pure memory-to-network operation, not a disk operation.
Last question. If you were starting today, with everything you now know, what's the one decision from this conversation you'd revisit?
This is deliberately open. Pick one stage's decision — ordering, ISR, leader election, pull vs. push, anything — and argue what you'd change and what you'd give up by changing it. A real answer defends a trade-off; it doesn't just criticize.
Problem → Constraint → Trade-off → Design Decision → New Problem → Next Design Decision— not
Feature → Definition → Next Feature. That's the actual difference between memorizing Kafka and understanding it. Your answer to this last question is the real test of which one you did.