Re­view: Un­der­stand­ing Mes­sage Bro­kers - Kaf­ka ver­sus Ac­tiveMQ

Our re­view of the ebook: "Un­der­stand­ing Mes­sage Bro­kers" by Jakub Korab...

This ar­ti­cle is our re­view of the ebook "Un­der­stand­ing Mes­sage Bro­kers" pub­lished by O'Reil­ly. It's a bi­ased re­view be­cause we ap­proach mes­sag­ing from the per­spec­tive of trans­ac­tion­al ex­act­ly-once de­liv­ery - but if you're in­ter­est­ed in ex­act­ly-once de­liv­ery then this ar­ti­cle is prob­a­bly a must-read.

In­tro­duc­tion

The au­thor of the ebook has lots of past ex­per­tise with Ac­tiveMQ (and Camel) and re­cent­ly start­ed work­ing on the Kaf­ka project. The ebook gives his per­spec­tives on both these mes­sage bro­kers, by dis­cussing each one in turn:

Ac­tiveMQ

As the au­thor ex­plains, a lot of the Ac­tiveMQ ar­chi­tec­ture was dic­tat­ed by the JMS spec­i­fi­ca­tion. The pow­er of the JMS spec­i­fi­ca­tion is a stan­dard way of in­ter­fac­ing with mes­sage bro­kers - and for this rea­son JMS has been nom­i­nat­ed as one of the best Java en­ter­prise APIs.

A big part of JMS is con­cerned about re­li­able, per­sis­tent mes­sag­ing with­out putting too much bur­den on the mes­sag­ing clients. For in­stance, it is the mes­sage bro­ker's re­spon­si­bil­i­ty to keep mes­sages un­til the con­sumers have processed them. This makes the bro­ker a bit more com­plex, and (at least for Ac­tiveMQ) this makes per­for­mance de­grade as the num­ber of con­sumers goes up.

Con­se­quent­ly, JMS mes­sage bro­kers like Ac­tiveMQ are less suit­ed for the "uni­ver­sal data pipeline" pat­tern - in fact it's con­sid­ered an anti-pat­tern for the JMS bro­ker ar­chi­tec­ture. And be­cause the bro­ker keeps mes­sages for the con­sumers, too many con­sumers that go of­fline can con­sume a bit more disk space (if this makes you think Kaf­ka uses less disk space: read on!). Note that many JMS bro­kers al­low set­ting a mes­sage life­time, af­ter which un­con­sumed mes­sages can be dis­card­ed - so in prac­tice the prob­lem may not be as big as you might ex­pect.

Kaf­ka

The de­sign goal of Kaf­ka was (re­port­ed­ly) to en­able the "uni­ver­sal data pipeline" at LinkedIn. This re­quired a dif­fer­ent bro­ker - be­cause it's an anti-pat­tern in JMS for the rea­sons ex­plained above. So where JMS and Ac­tiveMQ are tuned for re­li­able per­sis­tent mes­sag­ing (and there­fore can't sup­port data pipelines very well), Kaf­ka's de­sign fo­cus­es on ex­act­ly these data pipelines. It does of­fer per­sis­tence, but it's not as guar­an­teed as with JMS-based bro­kers.

To achieve its goals, Kaf­ka uses a "uni­fied des­ti­na­tion mod­el" called "top­ic" (some­thing in be­tween the no­tion of a JMS top­ic and a JMS queue). Mes­sages are kept for a while (and can be con­sumed more than once via re­set­table point­ers if de­sired). How­ev­er, mes­sage per­sis­tence is lim­it­ed in time and it is the con­sumer's re­spon­si­bil­i­ty to con­sume rel­e­vant mes­sages be­fore the bro­ker deletes them. So: mes­sages can be lost (in ad­di­tion to be­ing de­liv­ered more than once).

Over­all, the de­sign of the Kaf­ka bro­ker in­creas­es disk us­age by as much as 1000 times what Ac­tiveMQ (or JMS bro­kers) need, with­out any guar­an­tees for con­sumers that fail to pick up their mes­sages be­fore the bro­ker de­cides to delete them. How­ev­er, it's re­al­ly fast and can han­dle up to mil­lions of mes­sages per sec­ond.

Author's note: I found sev­er­al dis­cus­sions sug­gest­ing that per­sis­tence is not guar­an­teed in Kaf­ka be­cause it does not im­me­di­ate­ly flush to disk. If this is true, then there is ac­tu­al­ly no per­sis­tence guar­an­tee at all - only best-ef­fort. This prob­a­bly ex­plains why it can be very fast - at the cost of pos­si­ble mes­sage loss.

Con­clu­sions

Our take

Where­as the book claims that ex­act­ly-once de­liv­ery does not ex­ist, we have shown how easy it is in this ar­ti­cle. That is: it's easy if you use JMS/XA ca­pa­ble bro­kers like Ac­tiveMQ. In the case of Kaf­ka, it's quite dif­fer­ent: be­cause Kaf­ka del­e­gates the bur­den of ex­act­ly-once to the mes­sage con­sumer, you're bound to en­counter the pit­falls of the idem­po­tent con­sumer. Mes­sage loss is also pos­si­ble. Need­less to say, XA is not sup­port­ed by Kaf­ka.

We rec­om­mend us­ing Kaf­ka for high­er-per­for­mance mon­i­tor­ing use cas­es where mes­sage loss is not im­por­tant, such as di­ag­nos­tic log­ging events, per­for­mance met­rics, or oth­er sta­tis­ti­cal event types. But if you care about ex­act­ly-once de­liv­ery, if your mes­sages are valu­able or if you don't need a "uni­ver­sal data pipeline" then it's best to stick to a clas­si­cal bro­ker like Ac­tiveMQ.

Without Kaf­ka

Without Kaf­ka, you ben­e­fit from the in­fra­struc­ture log­ic of­fered by your DBMS and/or mes­sage bro­ker:

For reg­u­lar ap­pli­ca­tions, this is prob­a­bly what you want.

With Kaf­ka

With Kaf­ka you only have log files, mean­ing you have to do every­thing else your­self in your code:

If you're LinkedIn or any oth­er In­ter­net gi­ant then this may be ac­cept­able to you. Al­ter­na­tive­ly, if man­ag­ing log files is your thing then this is what you want. In all oth­er cas­es, you may want to stick to what is out there al­ready...

Next steps?

Down­load now to start ex­act­ly-once pro­cess­ing with Ac­tiveMQ and XA:

FREE down­load: Trans­ac­tion­sEssen­tials

(In­cludes work­ing sam­ples that use Ac­tiveMQ for ex­act­ly-once de­liv­ery)

Ref­er­ences

• The ebook can be down­loaded from here.
• The ebook au­thor's web­site can be found here.