Core principle: Don't add architecture because it is fashionable. Add it because a measurable bottleneck or reliability requirement demands it.
Before designing anything, establish what the system must do and how well it must perform.
User | +-- Register +-- Login +-- Browse products +-- Add to cart +-- Checkout +-- Payment +-- View order
10 million users 1 million DAU 100K requests/sec peak p99 latency < 300 ms 99.99% availability RPO < 5 minutes RTO < 30 minutes
Before adding infrastructure, estimate the workload.
Users = 10M Daily active users = 1M Requests/user/day = 100 Daily requests = 1M × 100 = 100M requests/day Average RPS = 100M / 86,400 ≈ 1,157 RPS Assume peak is 10× average: Peak RPS ≈ 11,570
Capacity estimates give quantitative justification for architectural decisions.
Internet
|
v
DNS
|
v
+-------------------+
| Single Server |
| |
| Web Application |
| Database |
| Static Files |
+-------------------+
Everything runs on one machine. This is simple and can be perfectly reasonable for small workloads.
SINGLE POINT OF FAILURE
|
+-----------+-----------+
| | |
App DB Storage
|
v
Server dies
|
v
EVERYTHING DOWN
Small Server
|
v
Bigger Server
|
v
Even Bigger Server
Increase CPU, RAM, disk, or network capacity.
Internet
|
v
+-----------+
| App Server|
+-----+-----+
|
v
+-----------+
| Database |
+-----------+
Application and database workloads often need different resources. The application may be CPU-heavy, while the database may need RAM and fast storage.
Users
|
+--------+--------+
| |
v v
Main App Admin App
| |
v v
Main DB Admin DB
Partition by feature or user type when workloads have different traffic, performance, or reliability requirements.
Users
|
v
+---------------+
| Load Balancer |
+-------+-------+
|
+-----------+-----------+
| | |
v v v
App-1 App-2 App-3
When one application server cannot handle traffic, introduce multiple servers behind a load balancer.
Request 1 → App-1 Request 2 → App-2 Request 3 → App-3 Request 4 → App-1 Request 5 → App-2
App-1 → weight 5 App-2 → weight 3 App-3 → weight 2
User A ─────────→ App-1
^
|
User A always -----+
Sticky sessions keep a user attached to one server, but create uneven load and make failures disruptive.
App-1 crashes | v User session disappears
A better approach is to externalize session state.
App-1 ----+ App-2 ----+----> Redis / DB App-3 ----+
Stateless means application instances do not own critical user state locally. Any instance can process any request.
Load Balancer
|
+---- App-1 → /health → 200 OK
|
+---- App-2 → /health → 500
|
+---- App-3 → /health → 200 OK
Traffic should stop going to unhealthy instances.
Load Balancer
|
+----------+----------+
| | |
App App App
\ | /
\ | /
+--------+--------+
|
v
Single DB
Eventually database query latency, CPU, I/O, locking, and connection pressure become the bottleneck.
Application
|
+---------+---------+
| |
WRITE READ
| |
v v
+-----------+ +-----------+
| Primary |----->| Replica |
| DB | | DB |
+-----------+ +-----------+
|
v
Replica-2
Writes go to the primary; reads can be distributed to replicas.
100,000 requests/sec 90,000 = reads 10,000 = writes 10K → Primary 30K → Replica-1 30K → Replica-2 30K → Replica-3
Read replication increases read capacity while preserving a primary write path.
Client
|
| WRITE
v
Primary
|
| replication
+-----> Replica
|
| lag
v
older data
Replicas may temporarily contain stale data. This creates a consistency trade-off.
POST /payment
|
v
Primary
|
v
GET /payment/status
|
v
Primary
For a short period after a write, route dependent reads to the primary. Other techniques include session affinity, replica-lag awareness, or consistency tokens.
Primary
X
|
v
FAILED
Replica-1
|
v
Promoted to Primary
Failover requires failure detection, promotion, connection reconfiguration, retry handling, and explicit data-loss/RPO decisions.
Application
|
v
Cache
|
+---- HIT ----> Return
|
+---- MISS
|
v
DB
|
v
Cache
Use a cache to avoid repeatedly executing expensive database reads. Common technologies include Redis and Memcached.
value = cache.get(key)
if value exists:
return value
value = database.get(key)
cache.set(key, value)
return value
The application explicitly reads the cache first and populates it after a miss.
DB: Balance = ₹10,000 Cache: Balance = ₹10,000 DB updated → ₹8,000 Cache → ₹10,000
Common strategies: TTL, explicit invalidation, write-through, and write-behind. The correct choice depends on consistency and durability requirements.
Cache DOWN
|
v
All requests → DB
|
v
DB overloaded
|
v
Cascading failure
Mitigations include TTL jitter, request coalescing, locks, rate limiting, circuit breakers, cache warming, and graceful degradation.
Eviction policies determine which entries leave when cache capacity is exhausted.
Good candidates are frequently read, rarely changed, or expensive to calculate.
Be careful with highly sensitive, frequently changing, or strongly consistent financial state unless the consistency model is explicitly designed.
Database
↓
System of Record
Cache
↓
Performance Optimization
The cache normally serves performance; the database remains the authoritative source unless the architecture explicitly defines otherwise.
Application
|
+-----------+-----------+
| | |
v v v
Shard-1 Shard-2 Shard-3
Users Users Users
1-1M 1M-2M 2M-3M
Sharding horizontally partitions data across multiple database instances.
hash(user_id) % 4 User 101 → Shard 1 User 102 → Shard 3 User 103 → Shard 0 User 104 → Shard 2
The shard key determines where a record lives and how requests are routed.
Shard 1: user_id 1 - 1M Shard 2: user_id 1M - 2M Shard 3: user_id 2M - 3M
Range sharding is simple but can create hotspots when new or popular data concentrates in one range.
hash(user_id)
|
v
+-----+-----+-----+-----+
| S0 | S1 | S2 | S3 |
+-----+-----+-----+-----+
Hashing generally distributes records more evenly, but changing shard count can cause substantial data movement.
Consistent Hash Ring
S1
/ \
S4 S2
\ /
S3
Consistent hashing reduces the amount of data that must move when nodes are added or removed. It is particularly important in distributed caches and some distributed data systems.
Transaction | +---- Shard 1 | +---- Shard 2
Distributed transactions are significantly harder than a transaction within one database.
A good shard key should provide:
Common candidates include user_id, account_id, or tenant_id depending on the workload.
User | v Load Balancer | v Application | v Disk | v image.jpg
Images, videos, CSS, JavaScript, fonts, and documents do not necessarily need to be served by application servers.
Application
|
v
Object Storage
|
+-- images
+-- videos
+-- documents
+-- JS
+-- CSS
Object storage offloads static file serving from application servers and is designed for scalable storage and delivery.
User
|
v
CDN
|
+-----+-----+
| |
CACHE CACHE
|
MISS
|
v
Object Storage
A CDN caches content at geographically distributed edge locations, reducing latency and origin load.
India User
|
v
India / nearby Edge
|
v
Image
When content is already at the edge, the request does not need to travel to the origin.
User
|
v
Edge
|
+--- HIT ---> Return immediately
|
+--- MISS
|
v
Origin
|
v
Edge caches object
|
v
User
Important HTTP caching mechanisms include:
Cache-Control: public, max-age=86400
A strong strategy is versioned or content-hashed assets.
app.js becomes app.v123.js or app.8f32a1.js
Changing the URL makes new content naturally distinct and reduces dependence on global cache invalidation.
API Gateway
|
+-------------+-------------+
| | |
v v v
User Service Order Service Payment Service
| | |
v v v
DB-1 DB-2 DB-3
Microservices decompose a large application into independently deployable services aligned with business/domain boundaries.
For example, a payment service processing 10K TPS may need very different scaling characteristics from a notification service processing 100 TPS.
Service A | | network v Service B | | network v Service C
Microservices introduce network calls, distributed tracing, service discovery, retries, timeouts, partial failures, consistency problems, deployment complexity, and operational overhead.
Order Service
|
DB1
Payment Service
|
DB2
User Service
|
DB3
Database ownership by service reduces tight coupling but creates distributed data and transaction challenges.
Place Order 1. Create order 2. Charge payment 3. Reduce inventory Order | +---- Payment | +---- Inventory
A single database transaction cannot normally span independent service databases without additional distributed transaction machinery.
Create Order
|
v
Reserve Inventory
|
v
Charge Payment
|
v
Confirm Order
If a later step fails, compensating actions undo or counteract earlier business actions.
Payment succeeded Inventory failed → Refund payment → Cancel order
A → B → C → response
If B is slow, A waits.
A | v Message Queue | +----> B | +----> C
Asynchronous communication decouples producers and consumers and is useful when immediate completion is not required.
Producer
|
v
+---------+
| Queue |
+----+----+
|
+----> Consumer-1
|
+----> Consumer-2
|
+----> Consumer-3
Useful for email, notifications, image processing, reports, analytics, order events, and background jobs.
Benefits include decoupling, buffering, asynchronous processing, load smoothing, and independent scaling.
Producer = 100K msg/sec Consumer = 20K msg/sec Queue | v Backlog ↑↑↑
Systems need rate limiting, backpressure, autoscaling, load shedding, and dead-letter queues when producers can outpace consumers.
Payment request
|
v
Payment service
|
| timeout
v
Client retries
The original operation may have succeeded even though the response timed out.
Attempt 1 → ₹1,000 charged Attempt 2 → ₹1,000 charged again
Critical operations therefore require idempotency.
Idempotency-Key: ABC123 ABC123 → Payment Result
If the same request arrives again with the same key, return the previously recorded result rather than executing the business operation twice.
Especially important for payments, orders, money transfers, bookings, and other financial transactions.
Global Users
/ | \
/ | \
India Europe USA
| | |
v v v
Region-A Region-B Region-C
GeoDNS can route users toward an appropriate geographic region.
Global Users
|
GeoDNS
|
+---------------+---------------+
| | |
v v v
India Europe USA
| | |
+----+----+ +----+----+ +----+----+
| | | | | |
App DB App DB App DB
Multi-region design introduces the question: where is the source of truth, and what consistency model is required?
India ───────┐ Europe ──────┼──→ Primary DB USA ─────────┘
Simple consistency model, but cross-region latency exists.
India → India DB Europe → Europe DB USA → USA DB
This reduces write latency but requires conflict resolution and replication strategy.
Users
|
v
Primary Region
|
v
Secondary Region
The secondary is standby. If the primary fails, the secondary can be promoted.
Users
|
+--------+--------+
| |
v v
Region A Region B
ACTIVE ACTIVE
| |
+--------+--------+
|
Replication
Both regions actively serve traffic. This can improve availability and latency but makes data consistency and conflict resolution harder.
Global User
|
v
Edge
|
+---------+---------+
| |
Cached data Dynamic request
|
v
Origin
Edge infrastructure can perform routing, TLS termination, authentication-related processing, rate limiting, WAF rules, A/B testing, and lightweight computation closer to users.
User | v DNS | v CDN / Edge | v WAF | v Load Balancer | v Application
A Web Application Firewall helps protect internet-facing applications from common malicious HTTP traffic patterns, including SQL injection and XSS.
Client | v API Gateway | +---- User Service +---- Order Service +---- Payment Service +---- Portfolio Service
An API gateway can centralize authentication, authorization, rate limiting, routing, TLS termination, request transformation, observability, and API versioning.
Do not turn the gateway into a giant business-logic layer.
Client A → 100 requests/sec
Client B → 100 requests/sec
Client C → 100 requests/sec
Client D → 100,000 requests/sec
|
v
overloaded
Common algorithms include Token Bucket, Leaky Bucket, Fixed Window, and Sliding Window.
At scale, failures are expected.
Architecture must assume server, database, network, region, dependency, queue, cache, and deployment failures can occur.
CLOSED
|
failures
v
OPEN
|
reject requests
|
timeout
v
HALF-OPEN
|
test request
/ \
success failure
| |
CLOSED OPEN
Circuit breakers prevent repeated calls to a failing dependency from creating cascading failures.
Observability
/ | \
Logs Metrics Traces
User | v Gateway | v Order Service | v Payment Service | v Database
Propagate a trace ID such as ABC123 through the request path so the complete distributed transaction can be reconstructed.
Requirement: p99 < 300 ms Gateway 20 ms Order 50 ms Inventory 50 ms Payment 80 ms DB 40 ms Network 30 ms ---------------------- Total 270 ms
Every distributed hop consumes part of the end-to-end latency budget.
| Availability | Approx. annual downtime |
|---|---|
| 99% | 3.65 days |
| 99.9% | 8.76 hours |
| 99.99% | 52.6 minutes |
| 99.999% | 5.26 minutes |
The higher the availability requirement, the more redundancy, automation, operational maturity, and cost are generally required.
Network Partition
|
+---------+---------+
| |
Consistency Availability
CAP is specifically about behavior during a network partition: a distributed system cannot simultaneously guarantee both strong consistency and availability under the partition while also tolerating the partition.
WRITE | v Read immediately | v Latest committed value
WRITE | v Some replicas updated | v Temporary stale reads | v Eventually converge
Eventual consistency can be appropriate for social likes, search indexing, analytics, recommendations, and similar workloads. Strong consistency is generally more important for account balances, money transfers, inventory reservations, and other critical financial state.
INTERNET
|
v
DNS
|
v
+---------------+
| GeoDNS / GSLB |
+-------+-------+
|
+----------+----------+
| | |
v v v
Region A Region B Region C
| | |
v v v
CDN CDN CDN
| | |
v v v
WAF WAF WAF
| | |
v v v
+--------------------------------+
| Load Balancer |
+--------------------------------+
|
+------+------+
| |
v v
Gateway Gateway
|
+-----+-----+----------------+
| | |
v v v
User Order Payment
Service Service Service
| | |
v v v
Redis Redis Redis
| | |
v v v
DB DB DB
| | |
+-----------+----------------+
|
Event Bus / Kafka
|
+----------+----------+
| | |
v v v
Analytics Search Notifications
| Stage | Problem | Solution |
|---|---|---|
| 1 | Small workload | Single server |
| 2 | Hardware limits | Vertical scaling |
| 3 | Different workloads | Separate concerns |
| 4 | App server overloaded | Load balancer |
| 5 | Session dependency | Stateless applications |
| 6 | DB read bottleneck | Read replicas |
| 7 | Repeated expensive reads | Cache |
| 8 | DB write/storage limits | Sharding |
| 9 | Static content load | Object storage |
| 10 | Global static latency | CDN |
| 11 | System/organizational complexity | Microservices |
| 12 | Async workloads | Message queues |
| 13 | Global users | GeoDNS |
| 14 | Regional resilience | Multi-region |
| 15 | Internet attacks | WAF |
| 16 | Service failures | Circuit breakers/timeouts |
| 17 | Operational complexity | Observability |
Requirements
↓
Capacity estimation
↓
Simple architecture
↓
Identify bottleneck
↓
Scale that bottleneck
↓
Identify NEW bottleneck
↓
Add next architectural component
↓
Analyze failure modes
↓
Analyze consistency
↓
Analyze observability
↓
Analyze security
↓
Analyze cost
↓
Multi-region / disaster recovery