Ex­act­ly-Once De­liv­ery With Kaf­ka

Ex­act­ly-once de­liv­ery (pro­cess­ing) in Kaf­ka? It's not as easy as claimed - read this to find out about the pit­falls...

The Prob­lem

There is a lot of en­thu­si­asm con­cern­ing Kaf­ka as a high-speed bro­ker for your mes­sages. How­ev­er, so far Kaf­ka does not sup­port ex­tend­ed trans­ac­tions. De­spite what the doc­u­men­ta­tion sug­gests, this im­plies the fol­low­ing:

Mes­sages sent to Kaf­ka can be lost if there is a crash or a bug

Sup­pose you have your data stored in some reg­u­lar DBMS (not Kaf­ka). When you up­date the data, you want to pub­lish an event to Kaf­ka. You can do this as fol­lows:

1. First up­date the DBMS (i.e., com­mit the changes)
2. Then pub­lish to Kaf­ka

How­ev­er, if a crash hap­pens in be­tween 1 and 2 then you will have up­dat­ed the DBMS, but the mes­sage for Kaf­ka will have been lost. You may not want this.

Al­ter­na­tive­ly, let's try the oth­er way around:

1. Pub­lish a mes­sage to Kaf­ka
2. Then up­date (com­mit) the DBMS changes

This does not help: if a crash hap­pens in be­tween 1 and 2 then you will have pub­lished a mes­sage, but the as­so­ci­at­ed data will have been lost (no com­mit).

Mes­sages re­ceived from Kaf­ka can be lost if there is a crash or a bug

On the oth­er end, there is like­ly some con­sumer of the mes­sages you pub­lish. As­sum­ing that the con­sumers use their own DBMS stor­age (not Kaf­ka), how do you con­sume mes­sages re­li­ably? Let's try:

1. Take a mes­sage from Kaf­ka
2. Up­date the DBMS
3. Save the off­set of your Kaf­ka mes­sages read so far - so you don't get the same mes­sage again

A crash be­tween 1 and 2 will do noth­ing. A crash be­tween 2 and 3 will re­de­liv­er the mes­sage again to the up­date DBMS, so you need to de­tect du­pli­cates. This means you need to im­ple­ment the idem­po­tent con­sumer pat­tern, whose pit­falls are known and doc­u­ment­ed.

If your de­vel­op­ers don't think hard about fail­ure sce­nar­ios (it's hard to think about every­thing when you are un­der pres­sure to de­liv­er work­ing code) then they might get the or­der wrong and the fol­low­ing can hap­pen in­stead:

1. Take a mes­sage from Kaf­ka
2. Save the off­set of your Kaf­ka mes­sages read so far - so you don't get the same mes­sage again
3. Up­date the DBMS

In this case, a crash be­tween 2 and 3 will again lose mes­sages.

The So­lu­tion

So how do you pre­vent data in­con­sis­ten­cy with Kaf­ka? Is it at all pos­si­ble?

Yes, it is pos­si­ble! You need to store every­thing in­side Kaf­ka. You see, Kaf­ka is based on a log­ging ar­chi­tec­ture so it can "log" every­thing, and its logs can be used to re­store the lat­est state of your do­main ob­jects via "event re­play". In that sense, the Kaf­ka team's ar­ti­cle on ex­act­ly-once in Kaf­ka is cor­rect. How­ev­er, be aware that it is ac­tu­al­ly in­suf­fi­cient if you use oth­er stor­age tech­nolo­gies as out­lined in the above...

Lim­i­ta­tions

If you chose this kind of so­lu­tion, you have to be aware of the con­se­quence: you are mak­ing it hard to mi­grate away from your ex­ist­ing stor­age, and you will ef­fec­tive­ly be locked into Kaf­ka be­cause mi­grat­ing to a new, fu­ture DBMS is like­ly im­pos­si­ble as long as Kaf­ka does not sup­port XA. To un­der­stand why, check out our DBMS mi­gra­tion post. While it may seem a safe choice to go for Kaf­ka to­day, con­sid­er what hap­pens if cor­po­rate poli­cies change af­ter, say, a merg­er or ac­qui­si­tion - when you find your­self with a dif­fer­ent tech­nol­o­gy stack to in­te­grate.