The Problem
There is a lot of confusion and debate about whether or not messaging-based distributed systems can guarantee exactly-once delivery (most of the industry seems to be stuck at at-least-once delivery - it suffices to google "exactly-once delivery" to check for yourself). In this post we will show how it can be done, plus how simple it is with Atomikos. Let's start by defining the relevant actors on the scene, based on the lifecycle of a single message (which will be delivered - as you can probably guess - exactly-once):
The Producer
The producer is responsible for sending a message as the result of some prior processing.
The Broker
The (message) broker is responsible for keeping the message until the relevant consumer picks it up. On pick-up, the message is "delivered".
The Consumer
The consumer is responsible for picking up the message from the broker and processing it, presumably by updating its database state and/or publishing related messages to the broker.
The Solution
Producer Architecture
The producer should do its processing and send its message as part of a JTA/XA transaction. This ensures a message is sent if and only if the transaction commits. Any failures will result in rollback of the JTA/XA transaction - and no message will be sent. This means that failures can be safely retried until they succeed, without sending the resulting message more than once.
Broker Architecture
Assuming that the broker supports XA, any message sent to it in a JTA/XA transaction will only really be sent if the transaction commits. Likewise, any message delivered to a transactional consumer will only really be delivered (i.e. "removed" from the broker an no longer redelivered) if the consumer's transaction commits.
Consumer Architecture
The consumer should pick up its message from the broker as part of a JTA/XA transaction. If the consumer fails, the transaction will rollback and the message will be kept in the broker for later (re)delivery. Note that if this happens then the database will rollback too (as per JTA/XA semantics) so when the message is redelivered the database will not have seen it yet. This implies that even if (technically speaking) the message is delivered twice (or more), the effect is the same as if there were exactly-once delivery.
Why Does This Work?
The producer sends exactly once - if and when the producer transaction commits. The broker keeps and delivers the message until the consumer transaction commits. The message is removed from the broker if and only if the consumer transaction commits. Whereas the industry standard seems to be stuck at "at-least-once", our solution ensures "exactly-once" delivery: in no event will the consumer's state (database or other) process or see the same message twice. From at-least-once, this brings us to exactly-once.
Caveat
You may have noticed that the consumer might consistently fail to process the message, for instance if the payload cannot be parsed due to a bug. In that case, the message will not be processed, so we don't "even" reach at-least-once semantics. However, this also happens in all of the other proposed solutions out there so it would be unfair to consider this a weakness of our approach. Correct messages should eventually be processed, and exactly once. Goal achieved!
Comments
Add a comment