Idempotent Jms Consumer Pitfalls

A common and perceived alternative to JTA/XA transactions in messaging is the so-called Idempotent Consumer or Idempotent Receiver pattern.

Idempotent Consumer Logic

The basic flow goes like this:

1. The receiver gets the next message from the queue
2. The receiver processes the message and updates the database accordingly
3. The receiver commits the DB changes
4. The receiver returns any response (where applicable)
5. The receiver commits the queue (removing the message)

Idempotent Consumer Pitfalls

This works fine if there are no problems. However, a crash between steps 3 and 5 will effectively lead to duplicate message consumption (see Reliable JMS with Transactions for details). A trivial solution here is to use JTA/XA and, say, TransactionsEssentials -- and get improved performance at the same time.

However, surprisingly many people choose to hack around as follows:

1. The receiver gets the next message from the queue
2. The receiver checks if the message was already processed
3. The receiver processes the message and updates the database accordingly
4. The receiver commits the DB changes
5. The receiver returns any CACHED response (where applicable)
6. The receiver commits the queue (removing the message)

The new step 2 here ensures that messages are never processed twice, but it comes with a significant cost:

• a lot of complexity to hack around
• very hard to get right - requiring some kind of request store and dirty reads with NO locking to work in a clustered environment (by the way, how long are you going to keep those requests stored in your database?)
• very hard to test because of the above
• more important: this is not scalable

Step 5 requires you to store your responses in case the request was processed before, but the caller failed to receive the answer. How are you going to do that, and how long do you keep those responses around?

Also, scalability is limited because all the messages would probably have to be tracked with inserts in a database table of prior messages. The implications are severe:

• This table is shared among all receivers on the queue
• It becomes a hot-spot that makes linear scalability hard

Performance

The idempotent consumer pattern is slow. It may come as a surprise, but the idempotent consumer tends to be much slower than XA - so it is hardly worth the extra complexity. The only case where it may be useful is if you have an incomplete broker that does not support XA.

Solution

Want to avoid the hassle of implementing the idempotent consumer and get high performance message consumers? Try this instead:

FREE download: TransactionsEssentials