The CAP the­o­rem (Con­sis­ten­cy, Avail­abil­i­ty, Par­ti­tion­ing) has been re­ceiv­ing quite a lot of in­ter­est late­ly, just to men­tion one of the many ref­er­ences.

What is CAP about?

First let me give cred­its here: I am de­riv­ing my in­spi­ra­tion from the the­o­ret­i­cal in­sights found in this pa­per co-au­thored by one of my fa­vorite woman sci­en­tists, Nan­cy Lynch from MIT. If you get a chance to read this pa­per, go ahead it will bring you some very use­ful fun­da­men­tal un­der­stand­ing...

The CAP the­o­rem is es­sen­tial­ly a lim­i­ta­tion on what you can do with clus­tered (web) ser­vices in the fash­ion­able con­text of SOA.

The word 'clus­ter' is im­por­tant here since that is what it is all about. In par­tic­u­lar, the the­o­rem states that you can't have all three prop­er­ties (Con­sis­ten­cy, Avail­abil­i­ty, Par­ti­tion­ing) in one and the same sys­tem (read: ser­vice). This im­plies that there is no per­fect so­lu­tion to build­ing a high-through­put pop­u­lar ser­vice, or is there? Let's first ex­plore what each thing means...

Con­sis­ten­cy

By con­sis­ten­cy, the the­o­rem refers to the prop­er­ty that changes (up­dates) to the ser­vice back-end are vis­i­ble to lat­er queries. Sim­pli­fy­ing: if you add some­thing to your shop­ping bas­ket then it will ap­pear there next time you re­trieve your bas­ket sta­tus. That sounds triv­ial, but it is not if the bas­ket is spread over mul­ti­ple phys­i­cal serv­er process­es... Con­sis­ten­cy is com­mon­ly en­sured (be­tween process­es) by hav­ing some sort of dis­trib­uted trans­ac­tion co­or­di­na­tor, or (as­sum­ing a cen­tral back-end) a sin­gle cen­tral­ized data­base.

Avail­abil­i­ty

The Lynch pa­per uses a very sim­ple but suf­fi­cient de­f­i­n­i­tion of "avail­abil­i­ty": a sys­tem is avail­able if every re­quest to it re­turns. In oth­er words: there is no in­fi­nite block­ing.

Par­ti­tion­ing

Par­ti­tion­ing means the cut-off be­tween two seg­ments of the clus­ter. In oth­er words, one or more nodes be­come un­reach­able for at least some time.

What is the The­o­rem say­ing?

You can't have all three of the above qual­i­ties, pe­ri­od. How­ev­er, you can com­bine any two of them if you like. This is proven in the pa­per by Lynch et al. Also (and this is im­por­tant) you can ap­ply dif­fer­ent com­bi­na­tions of qual­i­ties to parts of your sys­tem. Mean­ing: you can stress con­sis­ten­cy in one part, avail­abil­i­ty in an­oth­er part, and so on. For in­stance, or­der pro­cess­ing or pay­ment pro­cess­ing can be done con­sis­tent­ly and avail­able (sac­ri­fic­ing par­ti­tion tol­er­ance) where­as query­ing the prod­uct cat­a­log can be done dif­fer­ent­ly (stress­ing par­ti­tion tol­er­ance in fa­vor of con­sis­ten­cy).

Does this con­tra­dict or in­val­i­date Atomikos?

Not at all, quite the con­trary: it makes Atomikos (and its third gen­er­a­tion of TP mon­i­tors) all the more rel­e­vant. Why? Be­cause Atomikos prod­ucts can help you in mak­ing those parts con­sis­tent when you want them to be.

Vir­tu­al­ly achiev­ing all three qual­i­ties

If you em­brace asyn­chro­nous mes­sag­ing (a la JMS or email) and ex­treme trans­ac­tion pro­cess­ing (XTP) then it is pos­si­ble to as­ymp­tot­i­cal­ly re­al­ize all three qual­i­ties (con­sis­ten­cy, avail­abil­i­ty, par­ti­tion-tol­er­ance) pro­vid­ed that you do use a call­back mech­a­nism to com­mu­ni­cate re­sults (e.g., by send­ing a con­fir­ma­tion email). Here is how:

• Queue re­quests in JMS.
• Pro­cess each re­quest trans­ac­tion­al­ly (so fail­ures will leave the re­quest queued for re­tries).
• The process that di­gests each re­quest can be ar­bi­trar­i­ly com­plex and use trans­ac­tions (con­sis­ten­cy) and re­turn when­ev­er it likes (thanks to the queu­ing, no re­ply is ex­pect­ed with­in a pre­set time frame).
• Any lack of avail­abil­i­ty of the pro­cess­ing is re­cov­ered by the queues: failed re­quests will stay queued un­til the process in the back-end is in fact avail­able again.

Now did I just break the CAP im­pos­si­bil­i­ty? More on this in a next post...

Sup­pose you want to de­vel­op a high-vol­ume trans­ac­tion pro­cess­ing sys­tem in Java/J2EE. How would you do it? Most peo­ple would say: don't use JTA/XA trans­ac­tions be­cause they kill per­for­mance. Wrong. And they would also say: use an appserv­er to scale. Again, they couldn't be more wrong.

Here is the mag­ic recipe on how we build sys­tems with vir­tu­al­ly un­lim­it­ed scal­a­bil­i­ty at Atomikos:

• Kick out your appserv­er as soon as you can, as ex­plained here. J2EE is not lim­it­ed to an appserv­er. J2EE is a set of APIs. The appserv­er ties these APIs to a pro­gram­ming mod­el that al­most no­body needs. Con­clu­sion: drop the lat­ter.
• Use a per­sis­tent JMS queue to store trans­ac­tion re­quests. This al­lows easy load-bal­anc­ing and pro­vides crash re­silience for on­go­ing re­quests. It also de-cou­ples the clients from the trans­ac­tion pro­cess­ing sys­tem.
• Use Ex­tremeTrans­ac­tions to process the re­quests (stored in JMS). This al­lows for re­li­able, ex­act­ly-once mes­sage pro­cess­ing as out­lined here. Make sure to use the sup­plied JMS and JDBC dri­vers!
• To add more pow­er, just add a sec­ond VM (process) on a sep­a­rate CPU.
• Re­peat un­til per­for­mance is high enough.

You will reach the re­quired per­for­mance be­cause of the in­tra-VM na­ture of each process you add. The only po­ten­tial bot­tle­necks are your own data­base or JMS back­end. So scal­ing comes down to scal­ing your back­ends, which is much sim­pler than scal­ing your ap­pli­ca­tion it­self (which has al­ready been done in a nat­ur­al way as out­lined above).

So don't let any­body fool you: trans­ac­tions do scale - even with­out lim­its!.

Ex­tra: Elas­tic scal­ing

Our com­mer­cial prod­uct Ex­tremeTrans­ac­tions in­cludes elas­tic scal­ing fea­tures for cloud de­ploy­ments. You can ap­ply for a free tri­al be­low...

Try Ex­tremeTrans­ac­tions For Free

I have talked to a num­ber of peo­ple who claim to be do­ing SOA, when in the end all they do is loose­ly-cou­pled de­sign. Let me ex­plain what I mean by an ex­am­ple.

A team of en­ter­prise ar­chi­tects was de­sign­ing an SOA in­fra­struc­ture for a bank I know. The sys­tem they were build­ing would be based on in­ter­faces, so that it would be pos­si­ble to de­ploy parts of the sys­tem as sep­a­rate in­stances lat­er on. This was their no­tion of SOA...

The good thing about it is that there are in­ter­faces in their de­sign, mean­ing it is like­ly to be loose­ly-cou­pled. The bad news is that this is not SOA, at least not in my view: one of the biggest ad­van­tages of SOA - reuse in place - is nev­er re­al­ized in this way. So, where­as this ap­proach to 'SOA' may be loose­ly cou­pled in de­sign, it is not loose­ly cou­pled in de­ploy­ment (which is at least as im­por­tant).

The con­se­quence? When­ev­er a 'ser­vice' is up­grad­ed, they will need to up­grade all the de­pen­dent ser­vices and re­de­ploy them. This is be­cause each 'ser­vice' is re­al­ly an em­bed­ded mod­ule in­side oth­er parts of the sys­tem.

I guess this also holds for the de­bate on cloud vs grid com­put­ing: in my view, a cloud is more loose­ly cou­pled than a grid in its de­ploy­ment.

Is BPEL a good tool for im­ple­ment­ing com­pen­sa­tion? It re­al­ly de­pends, and you re­al­ly have to know what you are do­ing - which (with all re­spect) doesn't seem the case for most peo­ple (not even BPEL spe­cial­ists). So if not even those ex­perts know, how can we ex­pect the rest of us to know? Hence this blog en­try.

For in­stance, on re­peat­ed oc­ca­sions I have heard renowned BPEL and work­flow ex­perts men­tion that com­pen­sat­ing trans­ac­tions are "per­haps" best mod­eled at the busi­ness log­ic lev­el. This, by the way, in­cludes Bill Burke in the case of JBoss/jBPM - see here. Note that I em­pha­sized the word "per­haps": this in­di­cates the shade of mis­un­der­stand­ing usu­al­ly present in the ar­gu­ments.

I have been say­ing this here and there in the past (and in fine de­tail in this ar­ti­cle), but I want to re­peat it again: BPEL, nor work­flow nor WS-BA are ide­al for com­pen­sa­tion un­less the com­pen­sat­ing par­ty doesn't care whether it needs to com­pen­sate even­tu­al­ly. In oth­er words, if the com­pen­sa­tion is busi­ness as usu­al to the provider of the com­pen­sa­t­able ser­vice then BPEL might be OK (though cer­tain­ly not de­sir­able - see be­low).

Why is that? Put your­self in the place of a ser­vice that is asked to com­pen­sate by a BPEL en­gine some­where. Also sup­pose that you are in a B2B ecosys­tem where you don't nec­es­sar­i­ly trust the par­ty that owns the BPEL en­gine. Now what would you rather do: trust the BPEL to com­pen­sate - even­tu­al­ly (which might be nev­er!) or rather deal with com­pen­sa­tion your­self, say af­ter a time­out? I would def­i­nite­ly choose the lat­ter. I don't want some­one else to de­cide when I need to com­pen­sate. I want to de­cide for my­self, and the Atomikos TCC mod­el al­lows for that. BPEL and jBPM don't.

So BPEL is ruled out for me - at least as far as com­pen­sa­tion goes. What about WS-BA? It is a step in the right di­rec­tion, but un­for­tu­nate­ly it is a bloat­ed pro­to­col, very in­ef­fi­cient and loaded with ap­pli­ca­tion-lev­el mes­sages that pol­lute the com­pen­sat­ing part. Even worse, it also suf­fers in a large part from the lack of time­out and de­pends on the BPEL to at least trig­ger com­pen­sa­tion.

Also, WS-BA doesn't al­low for ap­pli­ca­tion log­ic on close - I won't go and both­er you with the en­tire spec de­tails but it is like a try..catch...fi­nal­ly where the ex­cep­tion is raised by the client (ugly!) and where the fi­nal­ly block can only be emp­ty! Again, Atomikos TCC is far su­pe­ri­or, more ef­fi­cient and more el­e­gant. It is also more nat­ur­al for com­pen­sa­tion than any BPEL en­gine will ever be.

One last note on BPEL and this sup­posed "mod­el­ing the com­pen­sa­tion in the busi­ness process": I was talk­ing to an IBM ar­chi­tect the oth­er day. He said that they were do­ing a large tel­co project with BPEL to co-or­di­nate things. One of the things he com­plained about was ex­act­ly this: they have to mod­el the com­pen­sa­tion and er­ror log­ic as ex­plic­it work­flow paths, and it was lit­er­al­ly over­load­ing every­thing with com­plex­i­ty. More­over, this com­plex­i­ty is hard to test. As he cor­rect­ly put it, they were im­ple­ment­ing a trans­ac­tion man­ag­er at the busi­ness log­ic (BPEL) lev­el, over and again in every process mod­el. In ad­di­tion, this was also hard to test he said and that it was vir­tu­al­ly killing the project - es­pe­cial­ly if there were change re­quests to con­sid­er. I be­lieve him:-) I gave him the URL to our TCC ar­ti­cle above.

Atomikos and TCC al­low you to fo­cus on the hap­py path of your work­flow mod­els. We take care of the rest. Now imag­ine what a re­duc­tion in com­plex­i­ty that is, and how much more re­li­able things get! So no, com­pen­sa­tion should NOT be mod­eled at the busi­ness lev­el. Ex­cept on rare oc­ca­sions maybe.

When­ev­er I see a pre­sen­ta­tion on REST I am im­pressed by its sim­plic­i­ty. With just four op­er­a­tions (GET, POST, PUT, DELETE) it seems to ac­com­plish a sim­ple mod­el for ser­vice-ori­ent­ed ar­chi­tec­tures, where every busi­ness re­source has a URL.

With this sim­plic­i­ty, REST also lever­ages the ubiq­ui­tous HTTP pro­to­col as the un­der­ly­ing mech­a­nism. More and more peo­ple seem to like this, in­clud­ing me.

How­ev­er, the big ques­tion for me is: how do you make this re­li­able? Imag­ine that you in­te­grate 4 sys­tems in a REST style. You would be us­ing HTTP and a syn­chro­nous in­vo­ca­tion mech­a­nism for each ser­vice. Now comes the ques­tion: how re­li­able is this? The an­swer: less than the least re­li­able sys­tem that you are us­ing! More pre­cise­ly, avail­abil­i­ty goes down quick­ly be­cause your ag­gre­gat­ed ser­vice fails as soon as one of the ser­vices fails...

With trans­ports like JMS you can im­prove re­li­a­bil­i­ty, but how do you do REST of JMS, giv­en its close re­la­tion­ship with HTTP and URLs? That is the prob­lem with REST for me.