Not "a load balancer distributes traffic across servers." Instead: two Principal Architects in a design review, trading sharp, checkable arguments — where nothing is allowed to be a definition, and every mechanism has to earn its place by defeating the decision before it.
I don't want a load balancer. I want a reason. You have one server handling mutual fund order placement. Convince me a second server even helps — before you tell me how traffic reaches it.
If you add a second identical server, what problem does that alone solve — and what new problem does it immediately create?
One server has a hard ceiling — CPU, memory, open connections. Past that ceiling, requests queue, then time out. A second identical server doubles the ceiling.
Then just tell every client "here are two IPs, pick one." Why do you need anything sitting in front of them?
Because now the decision of which server lives on the client — thousands of clients, each guessing independently, with no idea what the other clients just chose.
CLIENT-SIDE GUESSING: Client 1 --> "I'll pick Server A" Client 2 --> "I'll pick Server A" <- no coordination Client 3 --> "I'll pick Server A" Server A: overloaded. Server B: idle.
So the real problem was never "one server isn't enough." Say the actual problem.
The real problem is that capacity and the decision of how to use it must live in the same place. Splitting them — capacity on the servers, decision on the clients — guarantees the two drift apart.
Pick exactly one: fairness, latency, or throughput. Not all three — pick one and defend it from hardware up, not from a slide.
If two backend servers have wildly different current load, does "fair" distribution and "fast" distribution point in the same direction, or opposite ones?
Latency, not fairness. Fairness sounds virtuous, but a request doesn't care how evenly work is spread — it cares how fast it comes back.
Prove it. Give me the case where "fair" and "fast" disagree.
Round-robin is perfectly fair — every server gets the same number of requests. But if one server is mid-way through a slow NSDL callback and the other is idle, round-robin will still send the next request to the busy one, purely to keep the count even.
FAIR (round-robin): LATENCY-AWARE: A: [busy][busy][NEW]--queued A: [busy][busy] B: [idle] B: [idle][NEW]--served instantly count is equal, request waits count is unequal, request is fast
So fairness is actually the wrong goal dressed up as a good one?
Fairness optimizes the load balancer's own bookkeeping, not the user's experience. Optimizing for latency means the algorithm has to know something about current server state — not just take turns.
Fairness optimizes the load balancer's metrics. Latency optimizes the user's experience. When they conflict, optimize for the user. A balanced count that routes requests to a busy server is not actually fair.
Round-robin: request 1 to server A, request 2 to server B, request 3 to A, and so on forever. Simple, deterministic, no state to maintain. Sell me the case against it.
Round-robin assumes every request costs the same amount of work. Name one real request type in an order-placement system where that assumption is false.
An IPO allotment status check is a 5-millisecond read. A mutual fund redemption that has to call out to MOFSL and wait for a settlement confirmation can take 2 full seconds. Round-robin treats both as one unit of "a request."
Round-robin sees: [req][req][req][req][req] -- all equal Reality is: [5ms][5ms][2000ms][5ms][5ms] -- wildly unequal Send three slow ones to the same server in a row by pure bad luck, and it's drowning while its neighbor sits idle.
"By pure bad luck" — that's not an argument, that's a shrug. Make it precise.
Round-robin has no memory of what it already sent where. It can't see that server A is still processing the last three heavy redemptions — it just knows "A's turn." The moment request cost varies, blind turn-taking stops being fair even by its own definition, because it's fair on count, not on work.
Fine — track something. What, exactly? And don't say "load." Load is not a number until you define it.
Between "number of open connections per server" and "average response time per server," which one can lie to you, and under what condition?
Least-connections: route to whichever server currently has the fewest open connections. It's a live number, not a guess — the load balancer already terminates every connection, so it already knows this for free.
Server A: 42 open connections Server B: 11 open connections Next request --> Server B (fewer open, presumed less loaded)
Where does "fewest connections" lie to you?
When connections aren't equal cost. A server with 11 open connections, all of them slow MOFSL redemption calls, can be more loaded than a server with 42 open connections that are all fast, idle-keep-alive reads.
So connection count is a proxy for load, not load itself. What's the honest fix?
Weighted least-connections, or better, least response time — factor in each server's recent average latency alongside its open connection count. It costs more to compute, but it stops mistaking "many cheap connections" for "overloaded."
Least-connections assumes every server in your list is alive. One crashes at 2 a.m. Your algorithm keeps sending it requests forever, because as far as it knows, that server has zero open connections — which looks like the best choice, not the worst.
A crashed server looks identical to an idle, healthy one from a pure connection-count view. What's the minimum extra piece of information that breaks that tie?
You're right — "zero connections" is ambiguous between "wonderfully idle" and "catastrophically dead." The load balancer needs an independent signal of liveness, separate from the load metric entirely.
CONNECTION COUNT ALONE: + HEALTH SIGNAL: Server C: 0 connections Server C: 0 connections, but --> "route here, it's free!" last health check: FAILED --> requests vanish into a dead server --> excluded from routing pool
So poll every server on a timer and ask "are you alive?" What's wrong with that, stated plainly?
Nothing's wrong with it — it's necessary. But it's not sufficient by itself, because a server can pass a shallow "are you up" ping while its actual dependency — say, its connection to NSDL — is the part that's actually broken.
"Zero connections" looks like the best server to send traffic to — until the server is dead. Health and load are two separate signals. Never use one as a proxy for the other.
Active health checks — the load balancer pings every server every few seconds, on its own schedule. What does that cost you that nobody mentions in the marketing?
If you check every server every 2 seconds and have 200 backend instances, what are you actually spending, and does that cost scale with your real traffic or with something else entirely?
Active checks cost a fixed drumbeat of traffic that scales with server count, not with real user load. 200 servers, checked every 2 seconds, is 100 health-check requests a second — every second, forever, even at 3 a.m. when nobody's placing an order.
ACTIVE: LB --ping--> every server, every 2s, regardless of traffic
cost = f(server count), NOT f(actual usage)So drop active checks and just watch real traffic instead?
Passive checks — marking a server unhealthy after N real requests fail in a row — cost nothing extra, but they only fire when real traffic happens to hit the broken server. At 3 a.m. with near-zero traffic, a dead server can sit undetected for a long time.
PASSIVE: only learns a server is dead when a REAL request fails
cost = free, but detection delay depends on traffic volumeYou've just described two different failure modes, not a winner. Which do you actually ship?
Both, deliberately layered — active checks for baseline liveness at low frequency to keep cost bounded, passive observation on real traffic for fast reaction during actual load. Neither one alone is honest about the trade-off; together they cover each other's blind spot.
Active checks find failures before traffic hits them. Passive checks find failures faster once traffic is flowing. Neither alone is enough — use both, and be honest about each one's blind spot.
Right now you're routing on connection counts and health — you never look inside the request. Is that a limitation, or a design decision? Defend whichever one you claim.
If the load balancer read the URL path of every request before routing it, what could it do that it can't do blind — and what would that cost it?
Blind routing — Layer 4, just TCP/IP, no request inspection — is fast precisely because it's ignorant. It forwards packets without ever assembling or reading the HTTP request. That ignorance is the source of its speed.
L4 (blind): packet in --> pick server by connection --> packet out
never parses HTTP, never sees the URL path
L7 (aware): request in --> PARSE full HTTP --> read path "/insurance/policybazaar/..."
--> route THIS path only to the insurance-journey poolSo why would anyone give that speed up?
Because a fintech platform with 80 microservices doesn't have one homogeneous backend — it has an insurance-journey pool, a demat pool, an IPO-allotment pool. L4 can't tell those apart; it would happily send a PolicyBazaar request to a server that only knows how to handle NSDL calls.
Then L7 is strictly better?
Strictly more capable, not strictly better — parsing every request costs CPU that L4 never spends, and it's a trade you make deliberately, per route, not a free upgrade you take everywhere by default.
A user starts a KYC video-verification flow on server A — server A is holding the in-progress session in memory. Your least-connections algorithm sends their next request to server B, which has never heard of this user. What happens?
If the fix is "always send this user back to the same server," does that fight against everything stage 3 argued for, or fit alongside it? Be specific.
The flow breaks. Server B has no idea the user was three steps into KYC — from its point of view, this is a brand-new session starting from zero.
STATEFUL FLOW, NO STICKINESS: Req 1 --> Server A (session created, step 1 of 4) Req 2 --> Server B (least-loaded right now) --> "who is this user?"
So pin the user to server A for the whole flow. Doesn't that directly contradict everything you argued in stage 3 about routing to whichever server has the least load?
It's a direct, honest trade-off, not a contradiction I can hide from — for the lifetime of that session, correctness overrides load-awareness. A slightly slower correct response beats an instantly-fast broken one.
What if server A becomes overloaded mid-session, precisely because it's now stuck holding every sticky user it was ever assigned?
That's the honest cost of stickiness — it can concentrate load exactly where you don't want it. The real fix isn't clever routing; it's removing the need to stick in the first place, by moving session state out of the server and into a shared store both A and B can read.
Walk through this slowly. You built a load balancer so that no single backend server's death takes the platform down. What happens the moment the load balancer itself dies?
If every server behind it is healthy, and the load balancer is the only thing that's failed, is the platform up or down? Say why in one sentence.
Down. Completely. Every backend can be perfectly healthy and it doesn't matter — nothing can reach them.
BEFORE: clients --> [Server A] (dies) --> platform down for A's users only AFTER: clients --> [Load Balancer] (dies) --> platform down for EVERYONE
So you've concentrated risk, not removed it. That's a serious accusation against your own design. Defend it or fix it.
Fix it — run a standby load balancer, both sharing one virtual IP through a protocol like VRRP. The standby watches heartbeats from the primary; the instant they stop, it takes over the virtual IP itself. Clients never knew there were two.
VIRTUAL IP (VIP): 10.0.0.1 Primary LB --- heartbeat ---> Standby LB (owns VIP) (silent, watching) Primary dies --> heartbeats stop --> Standby claims the VIP --> traffic resumes
How long is that gap — the moment between "primary dies" and "standby has fully taken over"?
Not zero — heartbeat interval plus failover time, typically a few seconds. That's an honest, bounded cost, and it's a vastly better number than "unbounded, until someone gets paged and manually intervenes."
Primary and standby both run health checks independently. A network blip means the primary sees server C as healthy, and at the exact same moment the standby sees server C as dead. Which one is right?
Is there a version of "who's healthy" that's an objective fact both load balancers can agree on, or is it always somebody's local opinion? What follows from your answer?
Neither is "right" in any absolute sense. Health is observed locally, from wherever the checker sits — a network blip between the standby and server C is a real event from the standby's vantage point, even if the primary's path to C is perfectly fine.
Primary LB --path X--> Server C : reachable --> "C is healthy" Standby LB --path Y--> Server C : unreachable --> "C is dead" Both are telling the truth about their OWN observation.
Then don't say "who's right." Say what the system does when they disagree.
Only the active load balancer's opinion counts, ever — the standby's health view is informational until the moment it takes over the VIP, at which point its view becomes authoritative and the old primary's view is simply irrelevant because it's no longer serving traffic.
What if, during a failover, both briefly think they're the active one — a split brain?
That has to be structurally impossible, not just unlikely — the VIP can only be claimed by whichever node currently holds the failover protocol's lock or lease. Two nodes routing live traffic under the same identity at once isn't a health-check disagreement anymore, it's a correctness bug.
You deploy a new build to server A. Right now, at this exact second, it's mid-way through processing a mutual fund redemption for a real user. You kill the process to deploy. What happens to that user?
What's the difference between "stop sending server A new requests" and "stop server A"? Why does that distinction matter more here than anywhere else in this design?
Their request dies mid-flight. Best case, they see a connection-reset error. Worst case, the redemption partially completed on our side but the confirmation never reaches them — now the user and the ledger disagree about what happened.
HARD KILL: Server A: [processing redemption] --SIGKILL--> gone User: waiting for a response that will never arrive
So don't kill it. What's the actual sequence?
Connection draining. Tell the load balancer to stop routing new requests to server A immediately, but let its existing, in-flight requests finish naturally before the process is ever touched.
DRAIN: t=0 Server A marked "draining" --> LB stops sending NEW requests t=0..N existing in-flight requests on A finish normally t=N A has zero active connections --> now safe to deploy
What if one request never finishes — a client that opened a connection and went silent, holding server A open forever?
Drain needs a hard ceiling — a maximum drain timeout. Past that, the remaining stragglers get force-closed anyway; you're trading "never break anyone" for "break the smallest possible number, on purpose, with a bound," because waiting forever isn't actually safer, it's just a deploy that never finishes.
IPO allotment day. Traffic is 40 times normal. Your active-standby pair — one machine actually serving traffic at a time — is pegged at 100% CPU just terminating TCP connections. Adding backend servers doesn't help anymore. Why not?
You already solved "one server isn't enough" once, in stage 0. Does that exact same argument apply one layer up, to the load balancer itself?
It doesn't help because the bottleneck moved. Backend capacity was never the constraint anymore — the single active load balancer's own CPU, socket count, and NIC throughput are the ceiling now. You solved stage 0's problem and reintroduced it one layer up.
BEFORE (backend-bound): NOW (LB-bound): [1 LB] --> [A][B][C]... [1 LB, pegged] --> [A][B][C]...[Z], all idle adding servers helps adding servers does NOTHING
So run multiple active load balancers. Who balances the load across the load balancers?
Nothing needs to sit above them if you use DNS round-robin or ECMP — multiple load balancer IPs advertised at once, and routers or resolvers spread client connections across all of them before traffic ever reaches any single one.
DNS returns: LB1, LB2, LB3, LB4 (rotated per query) Clients spread themselves across all four --> no new single choke point
Haven't you just recreated the exact client-side-guessing problem from stage 0, one layer up?
Partially, and deliberately — the trade-off is acceptable here because DNS-level imbalance is coarse and slow-moving, while each individual load balancer still makes a precise, real-time decision underneath it. You've traded a hard, single choke point for a soft, self-correcting one — not eliminated the problem, but moved it somewhere its cost is tolerable.
IPO allotment traffic exceeds total backend capacity — not one server, all of them. Your load balancer is faithfully finding "the least-loaded" server. What's wrong with that sentence?
If every option is bad, does picking the "least bad" one actually help the user, or does it just guarantee everyone gets a slow, degraded response instead of some getting a fast one?
"Least loaded" implicitly assumes at least one option is actually good. When every server is overloaded, routing to the "least bad" one just spreads the overload evenly — everyone gets a slow, degraded response instead of some users getting a fast one and others getting none.
LOAD BALANCING UNDER OVERLOAD: All servers at 130% capacity --> "least loaded" picks the 128% one --> still overloaded --> EVERY request now degrades, none fail cleanly
So what's the alternative — reject some requests on purpose?
Yes — load shedding. Past a defined capacity threshold, the load balancer starts rejecting a portion of incoming requests immediately, with a clear "try again shortly," rather than accepting all of them and letting every single one degrade.
WITHOUT SHEDDING: 1000 requests --> all 1000 degrade badly WITH SHEDDING: 1000 requests --> 700 served well, 300 rejected fast, cleanly
Isn't rejecting a paying customer's request during an IPO allotment exactly the outcome you built this whole system to avoid?
It's the better of two bad outcomes, and the honest one — a fast, clear rejection lets the client retry or the user see an explicit "system is busy" message. A silent, system-wide slowdown gives everyone the illusion that it's about to work, right up until it times out anyway, having wasted everyone's time on the way there.
You now run backend clusters in Mumbai and Chennai, for disaster recovery. A user in Chennai gets routed to Mumbai by simple round-robin across data centers. What did that decision just cost them, and can your load balancer even see it?
Everything you've built so far balances load between servers that are roughly equidistant from the user. What breaks the instant that stops being true?
Physical distance adds real, unavoidable network latency — a Chennai user routed to Mumbai pays a round-trip penalty on every single request, no matter how idle or healthy the Mumbai cluster is. Every algorithm we've built so far assumed distance was a constant; here it's a variable, and it's the dominant one.
Chennai user --> Chennai cluster: ~5ms round trip Chennai user --> Mumbai cluster: ~25ms round trip, EVERY request "Least loaded" picked Mumbai correctly by load, but wrongly by the metric the user actually feels.
So load balancing needs a geography input now. Where does that decision have to be made — inside one of your existing load balancers, or somewhere else entirely?
Somewhere else entirely, and earlier — at the DNS layer, before the connection even opens. Global server load balancing resolves the domain name itself to the nearest healthy cluster's IP, based on the resolver's location, so the Chennai user never even learns the Mumbai IP exists unless Chennai is down.
GSLB (DNS-level): Chennai resolver asks --> "Chennai cluster is healthy" --> returns Chennai IP Mumbai resolver asks --> "Mumbai cluster is healthy" --> returns Mumbai IP Chennai cluster DOWN --> Chennai resolver now gets Mumbai IP instead
You've just added another layer above the one you built in stage 11. Justify that it's not turtles all the way down.
It bottoms out because each layer is solving a genuinely different-shaped problem: GSLB picks a region, coarse and slow-changing; the regional load balancer picks a server, precise and real-time. Once you're choosing a server within a region, distance is roughly constant again — which is exactly the assumption this whole design started with, back at stage 1.
Last question. If you were starting today, with everything you now know, what's the one decision from this conversation you'd revisit — and what would you knowingly give up to change it?
This is deliberately open. Pick one stage's decision — the algorithm, sticky sessions, load shedding, GSLB, anything — and argue what you'd change and what it costs. A real answer defends a trade-off; it doesn't just criticize.
Problem → Constraint → Trade-off → Design Decision → New Problem → Next Design Decision— not
Feature → Definition → Next Feature. That's the real difference between reciting a load balancer's feature list and understanding why it has to look the way it does. Your answer to this last question is the actual test of which one you did.