Below are 26 compact revision points designed to preserve the key judgment calls without turning this into a huge set of notes.
┌─────────────────────────────────────────────────────────────────┐
│ L4 vs L7 │
├───────────────────────────────┬─────────────────────────────────┤
│ L4 (TCP/UDP) │ L7 (HTTP/gRPC) │
├───────────────────────────────┼─────────────────────────────────┤
│ • Sees IP, port, connections │ • Sees host, path, headers │
│ • Forwards packets/connections│ cookies, method │
│ • Blind to request content │ • Can make per-request decisions│
│ • Fast, cheap, simple │ • Rich, content-aware, heavier │
│ • Balances CONNECTIONS │ • Can balance REQUESTS │
└───────────────────────────────┴─────────────────────────────────┘
Client ──TCP conn──► LB ──TCP conn──► Backend
Client ──HTTP req──► LB (reads path/headers) ──► BackendKey insight: Choosing the algorithm before choosing the layer is the wrong starting point.
1 TCP Connection
┌─────────────────────────────────────────────┐
│ Request 1 ──► Server A │
│ Request 2 ──► Server A │
│ Request 3 ──► Server A │
│ ... │
│ Request 10,000 ──► Server A │
└─────────────────────────────────────────────┘
Server A: ████████████████████████████████████ (overloaded)
Server B: ████ (idle)
Server C: ████ (idle)This isn't an LB bug; it's a consequence of balancing at connection level.
┌─────────────────┐
│ Client │
└────────┬────────┘
│
▼
┌─────────────────┐
│ L4 LB │ ← Fast edge, connection-level
│ (VIP/Anycast) │ Absorbs DDoS, connection flood
└────────┬────────┘
│
▼
┌─────────────────┐
│ L7 LB │ ← Smart HTTP routing
│ (HAProxy/NGINX)│ Reads host/path/headers
└────────┬────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Service │ │ Service │ │ Service │
│ A │ │ B │ │ C │
└─────────┘ └─────────┘ └─────────┘Question: Don't ask "L4 or L7 for everything?" Ask: "Which layer should own which responsibility?"
┌──────────────────────────────────────────────────────────────┐
│ TRAFFIC ENTRY MODELS │
├──────────────────────────────────────────────────────────────┤
│ 1. DNS-based: client ──DNS lookup──► LB IP │
│ (coarse, TTL-limited failover) │
│ │
│ 2. Anycast: same IP announced from many locations │
│ (network routes to nearest healthy) │
│ │
│ 3. Virtual IP (VIP): floating IP moves to healthy LB node │
│ (fast failover, transparent to clients) │
│ │
│ 4. ECMP + multiple LB nodes: router spreads flows │
│ (active-active, scale-out) │
└──────────────────────────────────────────────────────────────┘
Client ──► DNS/Anycast/VIP/ECMP ──► LB ──► Backend Pool DNS Failover:
┌─────────┐ ┌─────────┐
│ Client │──────►│ DNS │ "LB IP = 10.0.0.1"
└─────────┘ └─────────┘
│
▼ TTL = 300s
┌─────────┐ ┌─────────┐
│ Client │──────►│ LB │ 10.0.0.1 dies
└─────────┘ └─────────┘ Client keeps trying dead IP
│ until cache expires!
▼
Death + 300s = failover
VIP Failover:
┌─────────┐ ┌─────────┐
│ Client │──────►│ LB1 │ 10.0.0.1 (VIP) dies
└─────────┘ └─────────┘
│
▼ (seconds, not minutes)
┌─────────┐ ┌─────────┐
│ Client │──────►│ LB2 │ VIP moves to LB2
└─────────┘ └─────────┘ Client IP never changesWhy VIP/Anycast wins: Network-level failover is more transparent/faster than DNS failover.
STATIC CONFIG DYNAMIC DISCOVERY
┌─────────────────┐ ┌─────────────────┐
│ Load Balancer │ │ Load Balancer │
│ │ │ │
│ S1, S2, S3 │ │ ──► Registry │
│ (hardcoded) │ │ ──► Control │
└─────────────────┘ │ Plane │
│ └─────────────────┘
▼ │
Simple but stale Complex but reflects reality
Human edits Autoscaling, rolling deploys
Can't keep up Self-registration possibleDiscovery answers: "Does this backend exist?" It does NOT automatically answer: "Can it serve traffic now?"
┌─────────────────────────────────────────────────────────────┐
│ Discovery vs Health — Different Questions │
├─────────────────────────────────────────────────────────────┤
│ │
│ Registry says: "S1 exists" │
│ │ │
│ ▼ │
│ Health Check says: "S1 is ready" │
│ │ │
│ ▼ │
│ LB Pool says: "S1 is eligible for traffic" │
│ │
└─────────────────────────────────────────────────────────────┘
BEFORE TRAFFIC: Registered ──► Warming Up ──► Ready
(exists) (not ready) (eligible)Key point: A warming-up server shouldn't receive traffic just because it registered. Stale registry can be tolerated with eventual consistency, but may delay discovering/removing capacity.
┌─────────────────────────────────────────────────────────────────────┐
│ SELECTION ALGORITHMS │
├─────────────────────┬───────────────────────────────────────────────┤
│ ROUND ROBIN │ R1→S1, R2→S2, R3→S3, R4→S1, R5→S2... │
│ │ Simple, assumes similar request cost │
├─────────────────────┼───────────────────────────────────────────────┤
│ LEAST CONNECTIONS │ S1=200, S2=50, S3=120 → new req → S2 │
│ │ Useful when connections ≈ work │
├─────────────────────┼───────────────────────────────────────────────┤
│ WEIGHTED │ S1:5, S2:2, S3:1 │
│ │ S1 S1 S1 S1 S1 S2 S2 S3 │
│ │ Useful for heterogeneous capacity │
├─────────────────────┼───────────────────────────────────────────────┤
│ HASH-BASED │ hash(client_id) → backend │
│ │ A→S2, B→S1, C→S2 (stable mapping) │
├─────────────────────┼───────────────────────────────────────────────┤
│ LATENCY-AWARE │ Routes to fastest responding backend │
│ │ Needs damping to avoid oscillation │
└─────────────────────┴───────────────────────────────────────────────┘Senior question: "What resource is actually scarce on my backends?"
┌─────────────────────────────────────────────────────────────────┐
│ Round Robin assumes every request is equal │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Request 1: GET /health (1ms) ──► Server A │
│ Request 2: GET /health (1ms) ──► Server B │
│ Request 3: POST /report (30s) ──► Server C ◄── overloaded!│
│ Request 4: GET /health (1ms) ──► Server A │
│ Request 5: GET /health (1ms) ──► Server B │
│ Request 6: POST /report (30s) ──► Server A ◄── overloaded!│
│ │
│ "Fair" request count ≠ Fair work distribution │
└─────────────────────────────────────────────────────────────────┘The trap: 1 lightweight GET ≠ 1 heavy report-generation POST. The server can receive its "fair" number of requests while receiving an unfair amount of actual work.
┌─────────────────────────────────────────────────────────────────────┐
│ Least-connections ≠ Least-load │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Server A: 1,000 idle WebSocket connections │
│ CPU: 5% Memory: 20% (plenty of headroom) │
│ │
│ Server B: 50 active HTTP connections │
│ CPU: 85% Memory: 90% (near overload) │
│ │
│ LB sees: A has 1,000 connections, B has 50 │
│ LB sends: New request to B (least connections) ◄── wrong! │
│ │
└─────────────────────────────────────────────────────────────────────┘Lesson: Connection count is useful only if it correlates with real load (CPU, memory, active work).
┌─────────────────────────────────────────────────────────────────┐
│ OSCILLATION LOOP │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. A is fast ──► LB sends traffic to A │
│ │ │
│ ▼ │
│ 2. A receives more load ──► A slows down │
│ │ │
│ ▼ │
│ 3. B now looks faster ──► LB swings traffic to B │
│ │ │
│ ▼ │
│ 4. B receives more load ──► B slows down │
│ │ │
│ ▼ │
│ 5. Back to Step 1 ──► Continuous oscillation! │
│ │
└─────────────────────────────────────────────────────────────────┘Solutions: Smoothing, randomness, thresholds, damping factors. Aggressive reaction can create herd behavior that destabilizes the system.
┌─────────────────────────────────────────────────────────────────────┐
│ HEALTH CHECK TYPES │
├─────────────────────┬───────────────────────────────────────────────┤
│ TCP Check │ Can a TCP connection open? │
│ │ Proves: port is listening │
│ │ Does NOT prove: app can serve requests │
├─────────────────────┼───────────────────────────────────────────────┤
│ HTTP Check │ Does GET /health return 200? │
│ │ Proves: HTTP handler responds │
│ │ Does NOT prove: dependencies work │
├─────────────────────┼───────────────────────────────────────────────┤
│ Deep Check │ Does app + DB + dependencies work? │
│ │ Proves: the whole request path │
│ │ Risk: can overload dependencies │
└─────────────────────┴───────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ HEALTH CHECK LIES: │
│ /health returns 200 OK ──► LB thinks: "healthy" │
│ But real request fails ──► auth down / DB timeout / cache miss │
└─────────────────────────────────────────────────────────────────────┘Key point: 200 OK may only mean process alive, not real user request succeeds.
┌─────────────────────────────────────────────────────────────────────┐
│ HEALTH DEPTH vs COST OF BEING WRONG │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Low Risk (static content, read-only APIs) │
│ ─────────────────────────────────── │
│ Shallow HTTP check ✓ (fast, cheap, sufficient) │
│ │
│ Medium Risk (user profiles, product catalog) │
│ ─────────────────────────────────────── │
│ Check cache + DB read ✓ (moderate depth) │
│ │
│ High Risk (payments, orders, settlements) │
│ ────────────────────────────────────────── │
│ Deep dependency check ✓ (expensive, but cost of failure │
│ with rate-limiting/caching) │
│ │
└─────────────────────────────────────────────────────────────────────┘Senior judgment: Define explicitly what "healthy" means for each service. Don't make every health probe expensive; rate-limit and cache deep checks.
┌─────────────────────────────────────────────────────────────────────┐
│ HEALTH STATE MACHINE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ failures >= 3 ┌────────────┐ │
│ │ HEALTHY │ ────────────────────► │ UNHEALTHY │ │
│ └─────────┘ └────────────┘ │
│ ▲ │ │
│ │ │ │
│ │ successes >= 2 │ │
│ └───────────────────────────────────┘ │
│ │
│ Failure threshold = 3 (remove after 3 consecutive failures) │
│ Success threshold = 2 (re-add after 2 consecutive successes) │
│ │
└─────────────────────────────────────────────────────────────────────┘
Too sensitive:
┌─┐ ┌─┐ ┌─┐ ┌─┐ ┌─┐ ┌─┐ ──► Constant flapping
│H│ │U│ │H│ │U│ │H│ │U│ Routing instability
Too lax:
H H H H H H H H H U U U U ──► Broken server keeps receiving trafficTune based on: Cost of false removal vs cost of missed detection.
PROXY MODEL (Terminating):
┌────────┐ conn1 ┌────────┐ conn2 ┌────────┐
│ Client │──────────────►│ LB │──────────────►│Backend │
└────────┘ └────────┘ └────────┘
│ │ │
│◄─── TLS ──────────────►│◄─── New TLS/Plain ───►│
│ │ │
│ LB terminates client TLS, creates new │
│ connection to backend │
PASS-THROUGH:
┌────────┐ ┌────────┐ ┌────────┐
│ Client │──────────────►│ LB │──────────────►│Backend │
└────────┘ └────────┘ └────────┘
│ │ │
│◄───────────── TLS passthrough ─────────────────►│
│ │ │
│ LB forwards at lower layer, │
│ no application awareness │Trade-off: Termination gives visibility, TLS offload, header manipulation, L7 routing. Pass-through preserves end-to-end encryption but sacrifices content-based routing.
BEFORE TLS TERMINATION:
┌────────┐ TLS ┌────────┐
│ Client │───────────────►│Backend │
└────────┘ └────────┘
│ │
└─────── Trust ──────────┘
AFTER TLS TERMINATION:
┌────────┐ TLS ┌────────┐ plain/ ┌────────┐
│ Client │────────►│ LB │──────────►│Backend │
└────────┘ └────────┘ new TLS └────────┘
│ │ │
│◄── Trust ────────►│◄── Trust ─────────►│
│ │ │
│ Two separately-trusted hops │Consequences:
HTTP/1.1: 1 connection = 1 request at a time
┌──────────────────────────────────────────────┐
│ Client ──► 1 connection ──► 1 request │
└──────────────────────────────────────────────┘
HTTP/2: 1 connection = MANY concurrent streams
┌──────────────────────────────────────────────────────────┐
│ 1 Connection │
│ ┌────────────────────────────────────────────────┐ │
│ │ Stream 1 │ Stream 2 │ Stream 3 │ ... │ │
│ └────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────┐
│ TRAP: Balancing per HTTP/2 connection │
│ │
│ 1 connection = 100 concurrent streams │
│ LB sends all 100 streams to Server A │
│ │
│ Recreates the exact L4 connection-fairness problem │
│ Just one abstraction layer higher! │
└──────────────────────────────────────────────────────────┘Modern design question: "Balance per connection or per stream/request?"
┌─────────────────────────────────────────────────────────────────────┐
│ L7 ROUTING RULES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ IF host = api.shop.com AND path = /checkout │
│ ──► Checkout service pool │
│ │
│ IF host = api.shop.com AND path starts /users │
│ ──► User service pool │
│ │
│ IF host = api.shop.com AND path starts /orders │
│ ──► Order service pool │
│ │
│ IF host = static.shop.com │
│ ──► Static content service pool │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ DANGER: Bad routing rule │
│ IF host = api.shop.com AND path starts /orders │
│ ──► User service pool ◄── WRONG! │
│ │
│ Result: All /orders traffic gets 404 or errors │
│ Blast radius: Entire order flow is blackholed │
└─────────────────────────────────────────────────────────────────────┘Key point: Routing config deserves code-like review/change control.
┌─────────────────────────────────────────────────────────────────────┐
│ LB vs API Gateway │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Load Balancer: │
│ ─────────────── │
│ • Route by host/path │
│ • Distribute traffic │
│ • Health checks │
│ • TLS termination │
│ │
│ API Gateway (additional responsibilities): │
│ ─────────────────────────────── │
│ • Authentication/Authorization │
│ • Request/response transformation │
│ • Rate limiting per tenant │
│ • Payload validation │
│ • Business policy enforcement │
│ • URL rewriting │
│ │
└─────────────────────────────────────────────────────────────────────┘Why keep the LB "dumb"? Concentrating more logic at the LB = more logic every request depends on = larger blast radius.
WITH STICKINESS:
┌────────┐ ┌────────┐
│ User A │────────►│ Server │
└────────┘ │ S2 │
│ └────────┘
│ ▲
▼ │
┌────────┐ ┌────────┐
│ User A │────────►│ Server │
└────────┘ │ S2 │
│ └────────┘
│ ▲
▼ │
┌────────┐ ┌────────┐
│ User A │────────►│ Server │ (always the same backend)
└────────┘ │ S2 │
└────────┘
METHODS:
┌─────────────────┬─────────────────────────────────────────┐
│ Cookie-based │ session123 → S2 │
│ Client-IP hash │ hash(IP) → S3 │
│ Session-ID hash │ hash(session_id) → S1 │
│ Custom key │ hash(user_id) → S2 │
└─────────────────┴─────────────────────────────────────────┘Useful when: Local cache, in-memory session, warm context. Bad as a global default because: It creates hotspots. Prefer: Route-specific stickiness (only where needed).
SCENARIO: User A is stuck to S2
┌────────┐ ┌────────┐
│ User A │────────►│ Server │
└────────┘ │ S2 │
└────────┘
│
▼
S2 dies!
│
▼
┌────────┐ ┌────────┐
│ User A │────────►│ Server │ ← Remapped to healthy backend
└────────┘ │ S3 │
└────────┘
BUT:
┌──────────────────────────────────────────────────────────────┐
│ • Sticky state (session, cache) is lost │
│ • Application must rebuild/recover local state │
│ • Cache warm-up, session reload, reconnect logic │
└──────────────────────────────────────────────────────────────┘Key principle: Health always overrides stickiness. If stickiness is required for correctness, failure becomes much harder.
RETRY RISK MATRIX:
┌──────────────────────┬─────────────────────────────────────────────┐
│ REQUEST TYPE │ RETRY SAFE? │
├──────────────────────┼─────────────────────────────────────────────┤
│ GET (read) │ ✓ Yes — idempotent by design │
│ HEAD (read) │ ✓ Yes — no state change │
│ POST (write) │ ✗ DANGER — duplicate if partially executed │
│ PUT (idempotent) │ ✓ Yes — if designed as idempotent │
│ DELETE (idempotent) │ ✓ Yes — if designed as idempotent │
│ PATCH (partial) │ ✗ DANGER — can cause inconsistent state │
└──────────────────────┴─────────────────────────────────────────────┘
RETRY SCENARIO:
┌────────────┐
│ POST /order│ ────► Backend times out ────► Might have succeeded
└────────────┘
│
▼
┌────────────┐
│ RETRY │ ────► Backend processes again ────► DUPLICATE ORDER!
└────────────┘Golden rule: Idempotency key before retry on important write paths. Never make retry a blind universal LB feature.
HEALTH CHECK SAYS:
┌─────────────┐
│ /health │ ──► 200 OK ──► Healthy! ──► Keep sending traffic
└─────────────┘
BUT REAL PERFORMANCE:
┌─────────────────────────────────────────────────────────────────────┐
│ Response time: 15 seconds ◄── Users are experiencing │
│ Error rate: 12% timeout failures │
│ Queue depth: Growing │
└─────────────────────────────────────────────────────────────────────┘
SOLUTION: Watch performance signals alongside binary health
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ Signal │ What to check │
│ ──────────────────── │ ───────────────────────────────────────── │
│ p95/p99 latency │ Has it tripled? │
│ Error rate │ Is it rising? │
│ Queue depth │ Is it growing? │
│ Active streams │ Is it saturated? │
│ Retry counts │ Is the LB compensating for slowness? │
└─────────────────────────────────────────────────────────────────────┘Prefer: Reducing traffic to a degraded backend rather than waiting for total failure.
DRAIN LIFE CYCLE:
┌────────────┐ ┌────────────┐ ┌────────────┐
│ ACTIVE │────►│ DRAINING │────►│ REMOVED │
└────────────┘ └────────────┘ └────────────┘
│ │ │
▼ ▼ ▼
Receives No new traffic, Completely
traffic existing work removed
finishes
SCALING LB NODES:
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ Clients ──► [ LB1 ][ LB2 ][ LB3 ][ LB4 ] ──► Backend Pool │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ Each LB node operates independently │
│ • Shared config from control plane │
│ • Tolerate small eventual inconsistency │
│ • Avoid synchronizing every request │
│ │
└─────────────────────────────────────────────────────────────────────┘Why no tight coordination: Adds latency, creates single point of contention.
SCENARIO:
┌─────────────────────────────────────────────────────────────────────┐
│ Incoming traffic = 100,000 RPS │
│ Safe capacity = 60,000 RPS │
│ │
│ Problem: 40,000 RPS more than backends can handle │
└─────────────────────────────────────────────────────────────────────┘
WHAT "FAIR" DISTRIBUTION DOES:
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Server A │ │ Server B │ │ Server C │
│ 33,333 │ │ 33,333 │ │ 33,334 │
└────────────┘ └────────────┘ └────────────┘
│ │ │
▼ ▼ ▼
All overloaded ──► All fail together!
WHAT OVERLOAD PROTECTION DOES:
┌─────────────────────────────────────────────────────────────────────┐
│ Incoming 100k RPS │
│ │ │
│ ▼ │
│ LB detects saturation (queue depth, latency, CPU) │
│ │ │
│ ├──► Queue briefly (buffer) │
│ ├──► Rate limit (allow only safe capacity) │
│ ├──► Shed load (return 503 to slow clients) │
│ └──► Add capacity (scale out) │
│ │
│ Result: 60k RPS processed safely, remaining 40k shed/queued │
└─────────────────────────────────────────────────────────────────────┘Fair distribution cannot create capacity. Protection preserves the system.
HEALTH STATE MACHINE:
┌────────────┐ ┌────────────┐ ┌────────────┐
│ HEALTHY │───►│ DEGRADED │───►│ UNHEALTHY │
└────────────┘ └────────────┘ └────────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ DRAINING │───►│ REMOVED │ │ │
└────────────┘ └────────────┘ └────────────┘
OUR PLATFORM IN NUMBERS:
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ • L4 + L7 layered │
│ • 80+ microservices │
│ • Mixed health depth (shallow + deep) │
│ • Selective stickiness (route-specific) │
│ • TLS at edge boundary │
│ • Rate-limit + 503 overload posture │
│ │
└─────────────────────────────────────────────────────────────────────┘
DOMAIN-SPECIFIC RISKS:
┌─────────────────────────────────────────────────────────────────────┐
│ MF redemption → Retry risk → Idempotency key │
│ IPO allotment → Burst traffic → Queue/overload protection │
│ NSDL demat → Slow-but-alive → Latency + deep checks │
│ Insurance → Trust boundary → Header verification/WAF │
│ LAMF → Session pinning → Scoped stickiness │
└─────────────────────────────────────────────────────────────────────┘
LEADERSHIP QUESTION:
Not just "How do we fix this incident?"
But:
┌─────────────────────────────────────────────────────────────────────┐
│ • Why does this configuration exist? │
│ • Who owns the decision? │
│ • Does the original justification still exist? │
│ • What's the governance gap that let this happen? │
│ • What's the business/compliance exposure? │
└─────────────────────────────────────────────────────────────────────┘LAYER
↓
TRAFFIC ENTRY
↓
DISCOVERY
↓
HEALTH
↓
SELECTION
↓
CONNECTION / TLS
↓
L7 ROUTING / STICKINESS
↓
FAILURE + RETRY
↓
DRAIN / SCALE / OVERLOAD
↓
OBSERVE STATES
↓
SECURITY BOUNDARY
↓
PRODUCTION + LEADERSHIP JUDGMENTDon't start with "Which LB algorithm?"
Start with:
1. What exactly are we balancing? (connections? requests? streams?)
2. What can fail? (LB node? backend? network? dependency?)
3. What signal tells us it's failing? (health? latency? error rate?)
4. What is the cost of being wrong? (duplicate order? dropped request? customer impact?)
| # | Topic | Key Takeaway |
|---|---|---|
| 1 | L4 vs L7 | Decide the layer before the algorithm |
| 2 | L4 unfairness | Balances connections, not requests |
| 3 | L4 + L7 coexist | Each layer owns different responsibilities |
| 4 | Traffic entry | Separate availability problem from backend balancing |
| 5 | DNS vs VIP | Network failover > DNS failover |
| 6 | Backend discovery | Exists ≠ ready for traffic |
| 7 | Discovery + health | Two different questions, combine both |
| 8 | Selection algorithms | Balance the resource that's actually scarce |
| 9 | Round robin | Counts requests, not |
Note: the source document's reference table ends mid-row at item 9 — preserved exactly as supplied, without completion.