Lost transactions? How to design Event-Driven architectures

Why traditional transactions break in the cloud

When building monolithic applications, relying on relational databases and ACID transactions seems like the ultimate solution. In practice, when we split this system into microservices scattered across different servers, the traditional model breaks down. In the cloud, networks fail, instances crash, and immediate consistency gives way to eventual consistency, where data takes a few seconds to synchronize across services.

Event-driven architectures solve scaling issues by allowing services to communicate asynchronously, notifying each other when something happens. However, this freedom brings a dangerous side effect: the real risk of losing transactions midway or processing the same message twice. To avoid chaos, engineers must abandon the illusion that distributed environments work just like a single local server.

The eventual consistency challenge in microservices

Imagine a customer making an online purchase. The payment service processes the credit card, but the inventory service crashes right after before reserving the item. In a traditional database, everything would roll back instantly. In a distributed system, the money has left the account, but inventory remains untouched, creating a severe financial discrepancy.

Eventual consistency means data isn't synchronized all the time, but it will align shortly. To make this work smoothly, software engineering relies on design patterns that accept temporary failures and build automatic recovery mechanisms. This is where strategies like the Sagas pattern come in to coordinate steps without locking the main database.

How the Sagas pattern works in practice

The Sagas pattern replaces a giant transaction with a sequence of local transactions in each participating microservice. Each step updates its own database and emits a new event for the next service in line. If everything goes well, the workflow finishes successfully. But what if the third service fails? Instead of a global rollback that doesn't exist in the cloud, the system executes compensating transactions to undo previous steps.

If inventory fails after payment approval, the Saga automatically triggers a routine to refund the customer. In practice, this requires every write operation to have a well-designed logical inverse. It is not an instant process, but it guarantees that the entire system returns to a coherent state without freezing other customers who are still shopping.

The invisible danger of duplicate messages

Another ghost in event-driven architectures is duplicate message delivery. Due to network instability, a message broker like Kafka or RabbitMQ might deliver the same billing event twice to the same microservice. If the microservice is not prepared, the customer will be charged twice for the same order.

Defense against this problem is called idempotency — a mathematical property ensuring that executing the same operation ten times produces the exact same result as executing it once. In practice, this is implemented by recording a unique transaction identifier in a control table. If the message arrives again with the same ID, the system simply discards the duplicate and returns the previous success.

Ensuring reliable delivery with the Outbox pattern

To prevent an event from being lost between the moment the database is updated and the moment the message is sent to the queue, we use the Transactional Outbox Pattern. Instead of publishing the event directly to the network, the application writes the data and the event into the same local transactional table. A secondary process reads this table and safely dispatches messages to the event bus.

This strategy guarantees no event gets left behind, even if the service crashes right at dispatch. By integrating this practice with a CQRS-based read architecture, we clearly separate data writes from queries, drastically improving overall system performance. To explore these architectural concepts further in production, see our related technical analysis on AI hallucinating? How to build LlamaIndex guardrails.

Final thoughts on distributed resilience

Designing event-driven architectures requires a profound mindset shift: stopping the attempt to prevent all failures and instead building systems that gracefully survive them. By combining eventual consistency, compensating transactions via Sagas, and strict idempotency checks, your engineering gains the robustness needed to scale without dropping data along the way.