Design a Distributed Transaction System from scratch

A single database transaction requires a single database. The PSP is a separate company's system. NSDL is a separate, regulated registry. Neither exposes "join my transaction" as an API — they expose HTTP calls that either succeed or fail, on their own clock, with no shared commit log.

ONE DB TRANSACTION:                   REALITY:
  BEGIN
  debit balance                        Our ledger  -- our DB
  mark lien                            NSDL        -- their registry, HTTP API
  disburse loan                        PSP         -- their gateway, HTTP API
  COMMIT (all or nothing, one disk)    THREE separate systems, THREE clocks,
                                        no shared commit log between them

No — it's a sequence with no undo. If the lien-marking call to NSDL succeeds and the loan-disbursement call to the bank then times out, the customer's units are now frozen and no money moved. Nothing automatically reverses the first step because there was never a shared transaction to roll back.

The gap is this: the moment a business operation crosses a network boundary, there is no longer any mechanism, by default, that guarantees all its steps happen or none of them do. That guarantee has to be built — it doesn't come free the way it does inside one database engine.

Atomicity. Not speed, not availability. A slow success is an inconvenience. A half-finished money movement — units locked, cash not moved, or the reverse — is an incident that needs a human, a reconciliation team, and possibly a regulatory disclosure.

Something precise, not weaker: a guarantee that the system will always know, and eventually enforce, that every step either completed or was undone — even if "undone" isn't instant. I'm not promising NSDL and the bank commit in lockstep. I'm promising the overall business operation never gets stuck half-true.

Then you're stuck with exactly the same unresolved-state problem, one level deeper — except now it's hidden inside a catch block that everyone assumes already "handled" it.

TRY:
  mark lien on NSDL         -- SUCCEEDS
  disburse loan via bank    -- FAILS, times out
CATCH:
  undo lien on NSDL         -- also FAILS (network blip)
  -- now what? the catch block has no plan B.

Retrying blindly, in-process, dies the moment the service itself crashes or restarts mid-retry — the loop, and the memory of "I still owe an undo," disappears with the process. Whatever tracks "this operation is incomplete" has to survive the failure of the code that's trying to fix it.

Exactly — try/catch treats failure as a momentary exception to handle inline. What we actually need is a durable record of "this multi-step operation is in progress," one that outlives the process, so something can pick it back up no matter what crashes.

Split it into two rounds. Round one — PREPARE — ask every participant "could you commit this, if asked?" without actually committing. Each one locks its resources and replies yes or no, but takes no irreversible action yet.

PHASE 1 -- PREPARE:
  Coordinator --> NSDL:  "could you mark this lien?"  --> NSDL: "yes, locked"
  Coordinator --> Bank:  "could you disburse this?"   --> Bank: "yes, locked"
  Nothing IRREVERSIBLE has happened yet on either side.

Then round two never becomes COMMIT — it becomes ABORT, sent to everyone, and every participant releases its lock. Nothing was ever committed, so there's nothing to undo. That's the entire trick: defer the irreversible step until every participant has already agreed.

PHASE 2, all said YES:              PHASE 2, one said NO:
  Coordinator --> ALL: COMMIT          Coordinator --> ALL: ABORT
  every participant commits           every participant releases locks
  for real, now                       nothing irreversible ever happened

Every other operation waiting on that same lien or that same account balance is now blocked, for the full round-trip time of the slowest participant. We've bought atomicity by spending availability — and that trade is about to get much worse.

None. That's the honest, uncomfortable answer. NSDL is stuck holding the lock, unable to safely guess either direction, until the coordinator comes back — however long that takes.

NSDL, prepared, waiting:
  Guess COMMIT --> if bank actually said NO, we just committed alone. WRONG.
  Guess ABORT  --> if bank actually already COMMITTED, we're now inconsistent. WRONG.
  Only safe move: WAIT. Lock stays held. Nobody else can touch this resource.

Yes. That's not an implementation bug I can engineer away — it's a structural property of the protocol. The instant you require unanimous agreement before anyone acts, you've made every participant's progress depend on hearing back from one single coordinator. There's no clever fix that removes that dependency while keeping true atomicity.

Inside a single data center, with a fast, reliable network and a coordinator that restarts quickly from a durable log, the blocking window is small and rare enough to tolerate. Across an external registry like NSDL and an external bank, on the open internet, that same blocking window becomes routine, not rare — which is exactly why it stops being the right answer here.

Nothing changes for them in the moment. The durable log guarantees the coordinator, once it recovers, will remember and correctly re-announce COMMIT to everyone. But until that recovery happens, NSDL and the bank are exactly as blocked as before — the log protects the coordinator's memory, not the participants' wait time.

Coordinator log: "DECISION = COMMIT"  -- written to durable disk
Coordinator crashes.
NSDL:  still blocked, holding lock, no idea the decision exists
Bank:  still blocked, holding lock, no idea the decision exists
Coordinator recovers, reads its own log, NOW tells everyone.
The gap between crash and recovery was pure blocked time.

They're genuinely different problems. Amnesia is "does the coordinator remember what it decided." Blocking is "how long do participants wait to find out." A durable log fixes the first completely. It does nothing for the second — recovery time is still whatever it takes to restart the coordinator process and reconnect to every participant.

A saga: a sequence of independent local transactions, each one committing for real, immediately, with no cross-service lock ever held. If a later step fails, you don't roll back the earlier ones in the database sense — you run a separate compensating action that semantically undoes them.

SAGA (no locks held across steps):
  Step 1: mark lien on NSDL      --> COMMITS immediately, for real
  Step 2: disburse loan via bank --> FAILS
  Step 3 (compensation): release lien on NSDL --> a NEW, separate transaction
  No lock was EVER held waiting on step 2's outcome.

Isolation, and true rollback. Between step 1 committing and the compensation running, the lien is real and visible — anyone checking that customer's units during that window sees a locked state that will, shortly, be reversed. A saga trades "never visibly inconsistent" for "never permanently stuck."

There isn't one. You can send a follow-up "disregard our earlier message" — but that's damage control, not undo. The honest answer is that a notification is semantically irreversible, and no compensating transaction changes what already happened in the customer's inbox.

REVERSIBLE:                        IRREVERSIBLE:
  lock a resource --> unlock it        send an email --> "un-send" it?
  reserve funds  --> release them      the customer already read it.

Irreversible steps — notifications, external settlement confirmations, anything that leaves the system's boundary permanently — must be ordered last in the saga, only after every reversible step has already succeeded. If it can't be undone, it doesn't get to run until everything undoable already has.

SAGA ORDERING RULE:
  [reversible][reversible][reversible] ... [irreversible, LAST]
  by the time the irreversible step runs, every other step
  has already proven it will succeed or the saga never got this far.

Then it retries, because failing to send is recoverable — the failure mode we can't compensate for is the message actually arriving and then needing to un-arrive. A failed send is just a send that hasn't succeeded yet.

Two shapes, and they fail differently. Choreography — each service publishes an event when it finishes, and listens for events to know when to act — has no single point of failure, but nobody, at any moment, can answer "what's the overall status of this loan disbursement?"

CHOREOGRAPHY:
  Payment svc  --publishes--> "lien.marked"
  Loan svc     --listens, then publishes--> "loan.disbursed"
  Notify svc   --listens, then publishes--> "customer.notified"
  Nobody holds the full picture. Debugging means reconstructing
  the story from scattered logs across three services.

Orchestration — one saga coordinator explicitly calls each step in order and explicitly triggers compensations on failure. It's fully visible and debuggable, but that coordinator is now a piece of shared infrastructure every saga depends on — a new component that can itself have bugs, and needs its own durability story.

ORCHESTRATION:
  Saga Coordinator: step1 --> OK --> step2 --> FAILS
                    --> explicitly calls compensate(step1)
  ONE place holds "here's exactly where this saga is."

Orchestration, because auditability isn't optional here — a regulator asking "what exactly happened to this loan and in what order" needs one authoritative timeline, not a reconstruction exercise across five services' logs. Choreography's decoupling is a genuine strength, just not the one this specific workload needs most.

Every step and every compensation needs a unique, stable idempotency key — tied to the saga instance and the step, not generated fresh on each call. The bank's API stores which keys it's already processed and returns the original result instead of disbursing again.

Orchestrator calls disburse(saga_id=S991, step=2, idem_key="S991-step2")
  Call 1: times out on the response, but the bank actually processed it
  Call 2 (retry): SAME idem_key --> bank recognizes it, returns cached result
  Money moves ONCE, no matter how many times the orchestrator retries.

Same treatment, for the same reason — a retried "release the lien" compensation without its own idempotency key could double-release a lien that a different, later saga had already legitimately re-acquired. Idempotency isn't just a forward-path concern; every action a saga can take, forward or backward, needs it.

They see their units as locked against a loan that, as far as they can tell, doesn't exist yet — no disbursed cash, no visible loan account. That's not a bug in the read path. It's the exact isolation gap stage 6 admitted to when it chose sagas over 2PC.

Mid-saga, real customer state:
  NSDL: units LOCKED (lien marked, step 1 done)
  Loan ledger: no loan record yet (step 2 not done)
  Customer sees: "why are my units locked with no loan to show for it?"

Hiding the lock would be worse — it would let the customer place another order against units that are, in fact, already encumbered. The honest fix is a semantic lock: the read path shows the units as "pending — loan application in progress," an explicit, named intermediate state, not silence and not a lie.

WITHOUT semantic state:      WITH semantic state:
  "units: LOCKED" (why?)        "units: pending loan disbursement"
  customer confused             customer informed, and other systems
                                 know not to also touch these units

Informs, not stops — a "pending" label is just a status field unless something enforces it. Without an explicit rule, the redemption saga could run its own local checks, see units that appear present, and proceed, completely unaware the loan saga has already claimed them.

WITHOUT enforcement:
  Loan saga:       step1 marks lien --> units "pending loan"
  Redemption saga: reads units --> sees them, proceeds to redeem
  BOTH saw a valid-looking state. BOTH proceeded. Conflict.

No — that just reintroduces stage 4's blocking problem one layer up. The fix is a semantic lock that any saga touching a given resource must acquire as its own first, reversible step, before doing anything else — the resource itself refuses a second claim while one is already active, the same way a database row lock would, but scoped to the saga's lifetime instead of a transaction's.

Loan saga:       step0: ACQUIRE semantic lock on units --> granted
Redemption saga: step0: ACQUIRE semantic lock on units --> DENIED, already held
                 --> redemption saga fails FAST, at step 0, cleanly
                 -- no wasted work, no silent conflict

The difference is scope and ownership, not the existence of a lock — 2PC held locks across a coordinator waiting on unanimous agreement from strangers; here, the resource itself owns and enforces its one lock, released the instant its own saga finishes or compensates, with no external coordinator anyone else has to wait on. Locking didn't disappear from this design. It got pushed down to exactly the one place — the contested resource — where it actually belongs.

No exemption — I nearly gave myself one by accident. The database write and the event publish are two separate systems, exactly like NSDL and the bank were. A crash between them leaves the database saying "lien marked" while the event that should trigger step 2 never goes out. The saga silently stalls.

DB write "lien marked" --> COMMITS
*** process crashes ***
publish "lien.marked" event --> NEVER SENT
Saga is now stuck, forever, with no step 2 ever triggered.

No — this specific case has a narrower fix, because both writes are ours. The outbox pattern: write the event into an "outbox" table in the same local database transaction as the state change itself. One database, one real ACID transaction, so both rows commit together or neither does.

ONE local DB transaction:
  UPDATE lien SET status='marked'
  INSERT INTO outbox (event='lien.marked', payload=...)
  COMMIT -- both rows exist, or NEITHER does. Real ACID, same DB.

Separate background process reads the outbox table,
publishes each row to the queue, marks it published.

Yes — and that's fine, because we already solved duplicates in stage 9. The publisher can safely re-publish an outbox row it's unsure about, because every downstream consumer is required to be idempotent. The outbox guarantees at-least-once delivery; idempotency, already built, turns that into effectively-once processing.

If state lived only in memory, it knows nothing — every in-flight saga simply vanishes from the orchestrator's awareness, even though NSDL still has a lien marked and is waiting for a next step that will now never come.

In-memory orchestrator state:
  saga S991: step2 done, step3 pending  -- lives in RAM
  *** process crashes and restarts ***
  saga S991: -- GONE. Orchestrator has no memory this saga ever existed.
  NSDL is still holding a lien for a saga nobody is driving anymore.

It's the same shape of problem, with a materially different consequence — a crashed orchestrator with durable state can restart, read its own saga log, and resume exactly where it left off, driving every step forward or backward on its own. Nothing else in the system is blocked waiting for it in the meantime, because sagas, unlike 2PC participants, never held a lock waiting on the coordinator's word.

Durable saga log (its own DB table):
  saga S991: step1=DONE, step2=DONE, step3=PENDING
  Orchestrator crashes, restarts, reads the log:
  "resume S991 at step3" -- picks up exactly where it stopped.
  Meanwhile: NOTHING else was blocked waiting for this recovery.

Correct, and that's an honest final cost, not a hidden one — the saga log needs the same replication and durability discipline as any other critical store in this platform. The difference from 2PC is what happens during the gap before that durability kicks in: participants keep making local, independent progress instead of freezing in place.