7 Mi­croser­vice Data Con­sis­ten­cy Fail­ures That Hide Be­hind the Hap­py Path

Your mi­croser­vices work per­fect­ly when every­thing goes ac­cord­ing to plan. But what hap­pens when a data­base com­mits, a mes­sage dis­ap­pears, or a net­work time­out leaves you guess­ing? Here are sev­en data con­sis­ten­cy fail­ures that can qui­et­ly com­pro­mise your sys­tem.

Your mi­croser­vices work per­fect­ly when every­thing goes ac­cord­ing to plan. But what hap­pens when a data­base com­mits, a mes­sage dis­ap­pears, or a net­work time­out leaves you guess­ing? Here are sev­en data con­sis­ten­cy fail­ures that can qui­et­ly com­pro­mise your sys­tem.

Mi­croser­vices make it eas­i­er to build, de­ploy, and scale ap­pli­ca­tions in­de­pen­dent­ly. But they also in­tro­duce a chal­lenge that is easy to un­der­es­ti­mate: keep­ing data con­sis­tent across ser­vices, data­bas­es, and mes­sage bro­kers.

In a mono­lith­ic ap­pli­ca­tion, sev­er­al re­lat­ed up­dates can of­ten be han­dled with­in a sin­gle data­base trans­ac­tion. Either all changes com­mit, or none do.

In a mi­croser­vices ar­chi­tec­ture, those same op­er­a­tions may in­volve mul­ti­ple in­de­pen­dent sys­tems:

• A data­base stor­ing an or­der
• A mes­sage bro­ker dis­trib­ut­ing an event
• A pay­ment ser­vice pro­cess­ing a trans­ac­tion
• Another data­base main­tain­ing in­ven­to­ry

Each sys­tem may suc­cess­ful­ly com­plete its own op­er­a­tion. But that doesn't guar­an­tee that the over­all busi­ness trans­ac­tion suc­ceeds.

And that's where things get in­ter­est­ing.

Most mi­croser­vice data con­sis­ten­cy prob­lems don't ap­pear dur­ing nor­mal op­er­a­tion. They emerge when some­thing un­ex­pect­ed hap­pens: a process crash­es, a net­work con­nec­tion fails, a mes­sage is re­de­liv­ered, or a ser­vice restarts at pre­cise­ly the wrong mo­ment.

The re­sult? Lost mes­sages, du­pli­cate up­dates, in­con­sis­tent data­bas­es, and busi­ness op­er­a­tions that ap­pear suc­cess­ful in one sys­tem but fail in an­oth­er.

Let's ex­am­ine sev­en com­mon fail­ure sce­nar­ios, why they hap­pen, and what you can do to pre­vent them.

1. Lost Mes­sages: The Database Com­mits, but the Event Nev­er Ar­rives

Imag­ine an or­der-pro­cess­ing ser­vice that per­forms two op­er­a­tions:

• Saves a new or­der in its data­base.
• Pub­lish­es an OrderCreat­ed event to a mes­sage bro­ker.

The de­vel­op­er de­cides to com­mit the data­base trans­ac­tion first and pub­lish the mes­sage af­ter­ward.

That seems rea­son­able. After all, why pub­lish an event be­fore the or­der ac­tu­al­ly ex­ists?

But con­sid­er this se­quence:

• The ap­pli­ca­tion saves or­der #1234.
• The data­base trans­ac­tion com­mits suc­cess­ful­ly.
• The ap­pli­ca­tion crash­es be­fore pub­lish­ing OrderCreat­ed.
• The ser­vice restarts.

The or­der ex­ists in the data­base, but no event was pub­lished.

Down­stream ser­vices may nev­er learn that the or­der was cre­at­ed. In­ven­to­ry re­mains un­changed. Ship­ping is nev­er ini­ti­at­ed.

From the data­base's per­spec­tive, every­thing worked. From the busi­ness per­spec­tive, the work­flow is in­com­plete.

Why re­tries don't nec­es­sar­i­ly solve this

A retry can help only if the ap­pli­ca­tion knows that an event still needs to be pub­lished.

After a crash, the orig­i­nal process may have lost that in­for­ma­tion. Un­less the ap­pli­ca­tion durably records the pend­ing pub­li­ca­tion, there may be noth­ing to retry.

How to pre­vent lost mes­sages

Sev­er­al ap­proach­es can ad­dress this prob­lem:

• Trans­ac­tion­al out­box: Store the busi­ness up­date and an out­go­ing event record in the same lo­cal data­base trans­ac­tion. A sep­a­rate process pub­lish­es the event af­ter­ward.
• Trans­ac­tion co­or­di­na­tion: Use a dis­trib­uted trans­ac­tion to co­or­di­nate data­base and mes­sage-bro­ker op­er­a­tions when both re­sources sup­port a com­pat­i­ble trans­ac­tion­al pro­to­col.
• Rec­on­cil­i­a­tion: De­tect miss­ing down­stream up­dates and re­pair them, al­though this gen­er­al­ly pro­vides even­tu­al rather than im­me­di­ate con­sis­ten­cy.

For Java ap­pli­ca­tions us­ing XA-ca­pa­ble JDBC data­bas­es and JMS bro­kers, a trans­ac­tion man­ag­er such as Atomikos can co­or­di­nate the op­er­a­tions so they share a com­mon com­mit out­come.

The im­por­tant dis­tinc­tion is that two suc­cess­ful in­di­vid­ual op­er­a­tions are not au­to­mat­i­cal­ly one re­li­able busi­ness trans­ac­tion.

2. Ghost Mes­sages: The Event Ex­ists, but the Database Up­date Doesn't

What if you re­verse the or­der?

In­stead of com­mit­ting the data­base first, the ap­pli­ca­tion pub­lish­es the mes­sage be­fore com­mit­ting the data­base trans­ac­tion.

That might seem to solve the lost-mes­sage prob­lem.

Un­for­tu­nate­ly, it cre­ates the op­po­site fail­ure.

Con­sid­er the same or­der-pro­cess­ing ser­vice:

• The ap­pli­ca­tion re­ceives an or­der re­quest.
• It pub­lish­es an OrderCreat­ed event.
• Another ser­vice re­ceives the event and starts pro­cess­ing it.
• The orig­i­nal data­base trans­ac­tion fails or rolls back.

Now the event ex­ists, but the or­der doesn't.

This is a ghost mes­sage: an event an­nounc­ing a busi­ness change that nev­er ac­tu­al­ly com­mit­ted.

Why ghost mes­sages are dan­ger­ous

A ghost mes­sage can trig­ger real busi­ness con­se­quences.

A ship­ping ser­vice might pre­pare an or­der that doesn't ex­ist. A billing ser­vice might ini­ti­ate pay­ment pro­cess­ing. An in­ven­to­ry ser­vice might re­serve prod­ucts un­nec­es­sar­i­ly.

The re­ceiv­ing ser­vices have no rea­son to as­sume that the pub­lished event rep­re­sents an un­com­mit­ted op­er­a­tion.

Why chang­ing com­mit or­der doesn't help

Com­mit­ting the data­base first risks lost mes­sages.

Pub­lish­ing the mes­sage first risks ghost mes­sages.

Nei­ther or­der­ing guar­an­tees atom­ic­i­ty be­cause the data­base and bro­ker are in­de­pen­dent trans­ac­tion­al re­sources.

How to pre­vent ghost mes­sages

The trans­ac­tion­al out­box pat­tern avoids pub­lish­ing an event be­fore the cor­re­spond­ing busi­ness up­date com­mits.

Al­ter­na­tive­ly, when sup­port­ed, XA trans­ac­tion co­or­di­na­tion can en­sure that the data­base up­date and mes­sage pub­li­ca­tion par­tic­i­pate in one co­or­di­nat­ed trans­ac­tion.

The les­son: Com­mit or­der alone can­not guar­an­tee con­sis­ten­cy across in­de­pen­dent sys­tems.

3. Du­pli­cate Mes­sages: One Busi­ness Event Gets Pub­lished Twice

Distrib­uted sys­tems fre­quent­ly en­counter sit­u­a­tions where an op­er­a­tion suc­ceeds, but the caller nev­er re­ceives con­fir­ma­tion.

Imag­ine that an or­der ser­vice pub­lish­es an event:

• The ap­pli­ca­tion sends OrderCreat­ed.
• The bro­ker ac­cepts and per­sists the mes­sage.
• The ap­pli­ca­tion crash­es be­fore it marks the mes­sage as sent.
• The ap­pli­ca­tion as­sumes pub­li­ca­tion failed.
• It re­tries the pub­li­ca­tion.

Now two copies of the same busi­ness event may ex­ist.

The ap­pli­ca­tion in­tend­ed to pub­lish once. The bro­ker re­ceived it twice.

Why du­pli­cates mat­ter

Du­pli­cate mes­sages aren't al­ways harm­less.

Sup­pose a down­stream con­sumer process­es each event by cre­at­ing a ship­ment, up­dat­ing an ac­count bal­ance, or is­su­ing a cus­tomer no­ti­fi­ca­tion.

Without pro­tec­tion, du­pli­cate events can pro­duce du­pli­cate busi­ness ef­fects.

Why re­tries can make the prob­lem worse

Retries im­prove avail­abil­i­ty when op­er­a­tions fail tem­porar­i­ly.

But when the out­come is un­known, retry­ing can re­peat an op­er­a­tion that al­ready suc­ceed­ed.

This is one of the fun­da­men­tal chal­lenges of dis­trib­uted sys­tems: an ac­knowl­edge­ment fail­ure is not proof that the orig­i­nal op­er­a­tion failed.

How to pre­vent du­pli­cate mes­sages

Pos­si­ble so­lu­tions in­clude:

• Idem­po­tent con­sumers: En­sure pro­cess­ing the same busi­ness event mul­ti­ple times pro­duces only one in­tend­ed busi­ness ef­fect.
• Dedu­pli­ca­tion: Track processed event iden­ti­fiers and re­ject du­pli­cates. Often used to achieve “idem­po­ten­cy” like above.
• In gen­er­al, these ap­proach­es re­quire sig­nif­i­cant com­plex­i­ty in ALL down­stream con­sumers of a mes­sage. The com­plex­i­ty be­comes vi­ral.
• Trans­ac­tion­al mes­sag­ing: Co­or­di­nate pub­li­ca­tion with the rel­e­vant trans­ac­tion­al re­sources where sup­port­ed. This avoids an ex­plo­sion of down­stream com­plex­i­ty so it pays off many times.

An out­box alone does not guar­an­tee that a mes­sage will nev­er be pub­lished twice. A pub­lish­er may crash af­ter suc­cess­ful­ly send­ing an event but be­fore mark­ing it as de­liv­ered.

This is why re­li­able pub­li­ca­tion and re­li­able con­sump­tion must be con­sid­ered to­geth­er.

4. Lost Con­sumer Up­dates: The Mes­sage Dis­ap­pears Without Chang­ing the Database

The pre­vi­ous ex­am­ples fo­cused on mes­sage pro­duc­ers.

But con­sumers face sim­i­lar prob­lems.

Imag­ine an in­ven­to­ry ser­vice re­ceiv­ing an OrderCreat­ed event.

Its job is to up­date the in­ven­to­ry data­base and ac­knowl­edge the mes­sage to the bro­ker.

Sup­pose it ac­knowl­edges the mes­sage first.

• The in­ven­to­ry ser­vice re­ceives OrderCreat­ed.
• It ac­knowl­edges the mes­sage.
• The bro­ker con­sid­ers the mes­sage han­dled.
• The in­ven­to­ry ser­vice at­tempts to up­date the data­base.
• The data­base up­date fails.

The mes­sage has been ac­knowl­edged, but in­ven­to­ry was nev­er up­dat­ed.

Depend­ing on the bro­ker's ac­knowl­edge­ment se­man­tics, the mes­sage may no longer be avail­able for re­de­liv­ery.

Why this is dif­fi­cult to de­tect

The bro­ker may re­port suc­cess­ful mes­sage con­sump­tion.

The data­base may show no cor­re­spond­ing up­date.

Nei­ther sys­tem nec­es­sar­i­ly iden­ti­fies the mis­match as a dis­trib­uted trans­ac­tion fail­ure.

Mon­i­tor­ing mes­sage queues alone won't re­veal the miss­ing busi­ness ef­fect.

How to pre­vent lost con­sumer up­dates

Where sup­port­ed, co­or­di­nate the data­base trans­ac­tion and bro­ker ac­knowl­edge­ment in one dis­trib­uted trans­ac­tion.

For ex­am­ple, a Java ap­pli­ca­tion us­ing a com­pat­i­ble JMS bro­ker and XA-ca­pa­ble JDBC data­base can use Atomikos to co­or­di­nate both re­sources.

If trans­ac­tion­al co­or­di­na­tion isn't avail­able, ap­pli­ca­tions need a care­ful­ly de­signed con­sump­tion strat­e­gy, durable state track­ing, and rec­on­cil­i­a­tion.

The goal is to en­sure that a mes­sage is not per­ma­nent­ly con­sid­ered processed be­fore its re­quired data­base ef­fects are safe­ly com­mit­ted.

5. Du­pli­cate Mes­sage Pro­cess­ing: One Mes­sage Creates Two Busi­ness Ef­fects

What if the con­sumer up­dates the data­base first and ac­knowl­edges the mes­sage af­ter­ward?

This avoids ac­knowl­edg­ing mes­sages be­fore the busi­ness up­date com­mits.

But it in­tro­duces an­oth­er fail­ure win­dow.

Con­sid­er the fol­low­ing:

• The in­ven­to­ry ser­vice re­ceives an or­der event.
• It re­duces the avail­able in­ven­to­ry by one unit.
• The data­base trans­ac­tion com­mits.
• The ap­pli­ca­tion crash­es be­fore ac­knowl­edg­ing the mes­sage.
• The bro­ker re­de­liv­ers the mes­sage.
• The ser­vice re­duces in­ven­to­ry again.

One or­der has now re­duced in­ven­to­ry twice.

The bro­ker be­haved cor­rect­ly ac­cord­ing to its re­de­liv­ery se­man­tics. The data­base also com­mit­ted both up­dates suc­cess­ful­ly.

Yet the busi­ness out­come is wrong.

Isn't at-least-once de­liv­ery enough?

At-least-once de­liv­ery is de­signed to avoid los­ing mes­sages, but it per­mits re­de­liv­ery.

That means the ap­pli­ca­tion must be pre­pared to en­counter the same mes­sage more than once.

If the con­sumer isn't idem­po­tent, re­de­liv­ery can cre­ate du­pli­cate ef­fects.

How to pre­vent du­pli­cate pro­cess­ing

One com­mon ap­proach is to store a unique mes­sage iden­ti­fi­er along­side the busi­ness up­date, in the same lo­cal data­base trans­ac­tion.

Be­fore pro­cess­ing, the con­sumer checks whether the iden­ti­fi­er has al­ready been han­dled.

If it has, the du­pli­cate can be ig­nored.

For com­pat­i­ble trans­ac­tion­al re­sources, XA co­or­di­na­tion pro­vides an­oth­er ap­proach by cou­pling the data­base up­date with the bro­ker's trans­ac­tion­al mes­sage con­sump­tion.

How­ev­er, no trans­ac­tion man­ag­er can au­to­mat­i­cal­ly make ar­bi­trary non-trans­ac­tion­al side ef­fects, such as an ex­ter­nal HTTP call, ex­e­cute ex­act­ly once.

Ex­act­ly-once busi­ness pro­cess­ing re­quires guar­an­tees across the en­tire rel­e­vant op­er­a­tion, not just the mes­sage bro­ker.

6. Par­tial and Du­pli­cate Re­mote Com­mits: A Time­out Doesn't Tell You What Hap­pened

Not every mi­croser­vice ar­chi­tec­ture uses asyn­chro­nous mes­sag­ing.

Many ap­pli­ca­tions com­mu­ni­cate through syn­chro­nous REST or SOAP calls.

Th­ese in­ter­ac­tions in­tro­duce a dif­fer­ent con­sis­ten­cy prob­lem: the caller may not know whether a re­mote op­er­a­tion com­mit­ted.

Imag­ine a pay­ment ser­vice re­ceiv­ing a re­quest to charge a cus­tomer €100.

• The or­der ser­vice sends the pay­ment re­quest.
• The pay­ment ser­vice process­es the charge.
• The pay­ment trans­ac­tion com­mits.
• The net­work con­nec­tion fails be­fore the re­sponse reach­es the or­der ser­vice.
• The or­der ser­vice re­ceives a time­out.

Did the pay­ment fail?

No. The pay­ment may have suc­ceed­ed.

But the caller doesn't know.

The dan­ger of retry­ing

If the or­der ser­vice re­tries the re­quest, the pay­ment ser­vice might process a sec­ond charge.

A sin­gle time­out can there­fore lead to a du­pli­cate re­mote com­mit.

The op­po­site prob­lem is also pos­si­ble: the re­mote op­er­a­tion may have failed be­fore com­mit­ting, leav­ing the caller un­cer­tain about what hap­pened.

How to han­dle un­known out­comes

The right so­lu­tion de­pends on the re­mote sys­tem's ca­pa­bil­i­ties.

Com­mon tech­niques in­clude:

• Idem­po­ten­cy keys: As­so­ci­ate re­peat­ed re­quests with one log­i­cal busi­ness op­er­a­tion.
• Sta­tus queries: Check the re­mote op­er­a­tion's out­come be­fore retry­ing.
• Trans­ac­tion-aware in­te­gra­tion: Use co­or­di­nat­ed trans­ac­tion­al re­sources where the un­der­ly­ing pro­to­cols and sys­tems sup­port them.
• Rec­on­cil­i­a­tion: Com­pare records across sys­tems and re­solve dis­crep­an­cies.

Cir­cuit break­ers can pre­vent re­peat­ed calls to an un­healthy ser­vice, but they don't au­to­mat­i­cal­ly re­solve an un­cer­tain trans­ac­tion out­come.

A cir­cuit break­er can tell you to stop call­ing a ser­vice. It can­not tell you whether the pre­vi­ous busi­ness op­er­a­tion com­mit­ted.

This dis­tinc­tion is es­pe­cial­ly im­por­tant for pay­ments, trans­fers, reser­va­tions, and oth­er op­er­a­tions where du­pli­cate ex­e­cu­tion has real con­se­quences.

7. In­con­sis­tent Re­cov­ery: Your Databas­es Are Re­stored, but Your Busi­ness Data Isn't

Mi­croser­vice ap­pli­ca­tions of­ten use mul­ti­ple data­bas­es.

An or­der ser­vice might use Post­greSQL. An in­ven­to­ry ser­vice might use an­oth­er re­la­tion­al data­base. A re­port­ing ser­vice might use a doc­u­ment data­base.

This ap­proach, of­ten called poly­glot per­sis­tence, al­lows teams to choose stor­age tech­nolo­gies that suit their re­quire­ments.

But it in­tro­duces a sub­tle dis­as­ter-re­cov­ery prob­lem.

The re­cov­ery sce­nario

Imag­ine a busi­ness op­er­a­tion that up­dates two data­bas­es.

At 10:00:00, data­base A records a new or­der.

At 10:00:01, data­base B records the cor­re­spond­ing in­ven­to­ry change.

Now imag­ine the data­bas­es are backed up in­de­pen­dent­ly.

• Database A is re­stored to a point af­ter the or­der was com­mit­ted.
• Database B is re­stored to a point be­fore the in­ven­to­ry change.

Both data­bas­es re­store suc­cess­ful­ly.

But the com­bined sys­tem is in­con­sis­tent.

The or­der ex­ists, while the cor­re­spond­ing in­ven­to­ry up­date does not.

Why suc­cess­ful back­ups aren't enough

A valid back­up of each in­di­vid­ual data­base doesn't guar­an­tee a trans­ac­tion­al­ly con­sis­tent snap­shot of the en­tire ap­pli­ca­tion.

The prob­lem is not nec­es­sar­i­ly back­up cor­rup­tion.

It's the ab­sence of a com­mon re­cov­ery bound­ary across sys­tems.

How to im­prove re­cov­ery con­sis­ten­cy

Pos­si­ble ap­proach­es in­clude co­or­di­nat­ed re­cov­ery pro­ce­dures, trans­ac­tion-log re­ten­tion, con­sis­tent snap­shot mech­a­nisms, and ap­pli­ca­tion-lev­el rec­on­cil­i­a­tion.

Where com­pat­i­ble XA re­sources par­tic­i­pate in dis­trib­uted trans­ac­tions, trans­ac­tion re­cov­ery mech­a­nisms can also help re­solve trans­ac­tions that were in­com­plete dur­ing a fail­ure.

Atomikos pro­vides trans­ac­tion co­or­di­na­tion for Java ap­pli­ca­tions, while ca­pa­bil­i­ties such as LogCloud sup­port trans­ac­tion re­cov­ery in dis­trib­uted en­vi­ron­ments.

How­ev­er, trans­ac­tion re­cov­ery and dis­as­ter re­cov­ery are not iden­ti­cal. Re­cov­er­ing an in-doubt trans­ac­tion doesn't au­to­mat­i­cal­ly make in­de­pen­dent­ly cap­tured data­base back­ups mu­tu­al­ly con­sis­tent.

A com­plete re­cov­ery strat­e­gy must ac­count for both trans­ac­tion out­comes and the con­sis­ten­cy of re­stored data.

Sum­ma­ry: Seven Mi­croser­vice Data Con­sis­ten­cy Fail­ures

Th­ese sev­en fail­ures share a com­mon theme: in­de­pen­dent sys­tems can each be­have cor­rect­ly while the over­all busi­ness process be­comes in­con­sis­tent.

Fail­ure	Ex­am­ple out­come	Un­der­ly­ing prob­lem
1. Lost pro­duc­er mes­sage	Order com­mits, event nev­er ap­pears	Database and bro­ker com­mit in­de­pen­dent­ly
2. Ghost mes­sage	Event is pub­lished, or­der rolls back	Publi­ca­tion pre­cedes busi­ness com­mit
3. Du­pli­cate pro­duc­er mes­sage	One event is pub­lished twice	Un­known pub­li­ca­tion out­come fol­lowed by retry
4. Lost con­sumer up­date	Mes­sage ac­knowl­edged, data­base un­changed	Ac­knowl­edge­ment and data­base com­mit are sep­a­rate
5. Du­pli­cate pro­cess­ing	One event caus­es two in­ven­to­ry up­dates	Rede­liv­ery re­peats a com­mit­ted busi­ness ef­fect
6. Par­tial or du­pli­cate re­mote com­mit	Pay­ment suc­ceeds de­spite time­out, then runs again	Caller can­not de­ter­mine re­mote com­mit out­come
7. In­con­sis­tent re­cov­ery	Re­stored data­bas­es dis­agree about an or­der	Re­cov­ery points lack a shared con­sis­ten­cy bound­ary

Why Mi­croser­vice Data Con­sis­ten­cy Is Harder Than It Looks

The cen­tral prob­lem isn't that data­bas­es, mes­sage bro­kers, or HTTP APIs are in­her­ent­ly un­re­li­able.

It's that they pro­vide dif­fer­ent trans­ac­tion­al guar­an­tees.

A data­base trans­ac­tion can pro­tect changes in­side one data­base.

A mes­sage bro­ker can pro­vide durable de­liv­ery and ac­knowl­edge­ment mech­a­nisms.

An HTTP API can re­port whether a re­quest re­ceived a suc­cess­ful re­sponse.

But none of these guar­an­tees au­to­mat­i­cal­ly makes a mul­ti-sys­tem busi­ness op­er­a­tion atom­ic.

This is why choos­ing the right con­sis­ten­cy mech­a­nism mat­ters.

Com­mon ap­proach­es to mi­croser­vice con­sis­ten­cy

Idem­po­ten­cy and dedu­pli­ca­tion help pre­vent re­peat­ed re­quests or mes­sage de­liv­er­ies from pro­duc­ing du­pli­cate busi­ness ef­fects. They are es­pe­cial­ly use­ful when at-least-once de­liv­ery is un­avoid­able.

The trans­ac­tion­al out­box pat­tern cou­ples a busi­ness data­base up­date with durable event cre­ation. It can elim­i­nate the gap be­tween com­mit­ting busi­ness data and record­ing an out­go­ing event, al­though pub­li­ca­tion and down­stream pro­cess­ing still re­quire ap­pro­pri­ate du­pli­cate han­dling.

Saga pat­terns co­or­di­nate a se­quence of lo­cal trans­ac­tions through events or com­mands, of­ten us­ing com­pen­sat­ing ac­tions when lat­er steps fail. They can be ap­pro­pri­ate for long-run­ning work­flows, but com­pen­sa­tion is not equiv­a­lent to atom­ic roll­back and will al­ways in­tro­duce ad­di­tion­al ap­pli­ca­tion com­plex­i­ty.

XA trans­ac­tion co­or­di­na­tion en­ables com­pat­i­ble trans­ac­tion­al re­sources to par­tic­i­pate in a dis­trib­uted trans­ac­tion, typ­i­cal­ly us­ing two-phase com­mit. This can be par­tic­u­lar­ly use­ful when a Java ap­pli­ca­tion must co­or­di­nate JDBC data­base op­er­a­tions and JMS mes­sag­ing with­out im­ple­ment­ing cus­tom com­pen­sa­tion or dedu­pli­ca­tion for those co­or­di­nat­ed op­er­a­tions.

No sin­gle ap­proach fits every ar­chi­tec­ture.

For ex­am­ple, XA can­not di­rect­ly co­or­di­nate ar­bi­trary REST APIs or every Kaf­ka de­ploy­ment. And even when XA is avail­able, ap­pli­ca­tions must ac­count for re­source com­pat­i­bil­i­ty, avail­abil­i­ty, re­cov­ery, and op­er­a­tional trade-offs.

The key is to se­lect a con­sis­ten­cy mod­el based on the ac­tu­al fail­ure sce­nar­ios your ap­pli­ca­tion must han­dle.

Where Atomikos Fits

For Java teams work­ing with trans­ac­tion­al data­bas­es and mes­sag­ing sys­tems, Atomikos pro­vides a light­weight al­ter­na­tive to man­ag­ing dis­trib­uted trans­ac­tions through heavy­weight ap­pli­ca­tion servers.

Atomikos is an XA-com­pli­ant trans­ac­tion man­ag­er that in­te­grates with Java tech­nolo­gies such as JDBC, JMS, Spring Boot, and Hiber­nate.

It helps co­or­di­nate sup­port­ed re­sources so re­lat­ed data­base and mes­sag­ing op­er­a­tions can com­mit or roll back as one trans­ac­tion.

This can elim­i­nate en­tire class­es of lost-mes­sage, ghost-mes­sage, and du­pli­cate-pro­cess­ing fail­ure win­dows as­so­ci­at­ed with in­de­pen­dent­ly com­mit­ted op­er­a­tions.

Un­like ap­proach­es that re­quire de­vel­op­ers to im­ple­ment com­pen­sa­tion work­flows across mul­ti­ple ser­vices, trans­ac­tion co­or­di­na­tion can keep con­sis­ten­cy log­ic clos­er to the in­fra­struc­ture lay­er when the par­tic­i­pat­ing re­sources sup­port it.

Atomikos also sup­ports cloud-na­tive Java de­ploy­ments with­out re­quir­ing a tra­di­tion­al ap­pli­ca­tion serv­er, with LogCloud pro­vid­ing ca­pa­bil­i­ties for de­cen­tral­ized trans­ac­tion re­cov­ery.

For teams build­ing fi­nan­cial sys­tems, or­der-pro­cess­ing plat­forms, or oth­er ap­pli­ca­tions where data con­sis­ten­cy is crit­i­cal, this can sig­nif­i­cant­ly re­duce the amount of cus­tom re­li­a­bil­i­ty log­ic they need to main­tain.

Di­ag­nos­tic Check­list: Is Your Mi­croser­vice Ar­chi­tec­ture at Risk?

Be­fore as­sum­ing your sys­tem is re­li­able, ask these ques­tions:

• Does any ser­vice up­date a data­base and pub­lish a mes­sage as sep­a­rate op­er­a­tions?
• Can a mes­sage be ac­knowl­edged be­fore the cor­re­spond­ing data­base trans­ac­tion com­mits?
• Can a con­sumer process the same busi­ness event more than once?
• Does your ap­pli­ca­tion retry op­er­a­tions af­ter a time­out with­out check­ing whether the orig­i­nal re­quest suc­ceed­ed?
• Do you rely on com­mit or­der to guar­an­tee con­sis­ten­cy be­tween in­de­pen­dent re­sources?
• Are your data­base back­ups cre­at­ed and re­stored with­out check­ing cross-sys­tem con­sis­ten­cy?
• Have you test­ed what hap­pens when a ser­vice crash­es be­tween two re­lat­ed op­er­a­tions?
• Do you know which parts of your ap­pli­ca­tion pro­vide at-most-once, at-least-once, or ef­fec­tive­ly ex­act­ly-once busi­ness pro­cess­ing?

If you an­swered yes to any of the first sev­en ques­tions, or can­not an­swer the last one con­fi­dent­ly, your ar­chi­tec­ture de­serves a clos­er con­sis­ten­cy re­view.

The most im­por­tant test isn't whether your ap­pli­ca­tion works when every com­po­nent is healthy.

It's whether the sys­tem re­mains cor­rect when some­thing fails halfway through a busi­ness op­er­a­tion.

Fi­nal Thoughts: Don't Let the Hap­py Path De­fine Reli­a­bil­i­ty

Mi­croser­vices are de­signed to make ap­pli­ca­tions more mod­u­lar, scal­able, and in­de­pen­dent­ly de­ploy­able.

But dis­trib­ut­ing an ap­pli­ca­tion also dis­trib­utes its fail­ure modes.

Lost mes­sages, ghost events, du­pli­cate pro­cess­ing, un­cer­tain re­mote com­mits, and in­con­sis­tent re­cov­ery aren't nec­es­sar­i­ly the re­sult of poor pro­gram­ming.

They're pre­dictable con­se­quences of co­or­di­nat­ing op­er­a­tions across in­de­pen­dent sys­tems with­out suf­fi­cient trans­ac­tion­al guar­an­tees.

The good news is that these fail­ures can be ad­dressed.

Depend­ing on your ar­chi­tec­ture, the so­lu­tion may in­volve trans­ac­tion­al mes­sag­ing, idem­po­ten­cy, an out­box, re­cov­ery co­or­di­na­tion, or XA trans­ac­tions.

What mat­ters is un­der­stand­ing the fail­ure win­dows be­fore they af­fect pro­duc­tion.

Be­cause re­li­able mi­croser­vices aren't just about mak­ing in­di­vid­ual op­er­a­tions suc­ceed. They're about keep­ing the en­tire busi­ness trans­ac­tion con­sis­tent—even when things go wrong.

Build More Reli­able Java Mi­croser­vices with Atomikos

Are you co­or­di­nat­ing data­base up­dates and mes­sag­ing in a Java mi­croser­vices ar­chi­tec­ture?

Dis­cov­er how Atomikos helps you sim­pli­fy dis­trib­uted trans­ac­tion man­age­ment, pre­vent in­con­sis­tent up­dates, and co­or­di­nate sup­port­ed JDBC and JMS re­sources with­out a heavy­weight ap­pli­ca­tion serv­er.

Ex­plore Atomikos trans­ac­tion man­age­ment →

Or down­load our Mi­croser­vice Data Con­sis­ten­cy Pit­falls cheat sheet for a quick ref­er­ence to the sev­en fail­ure sce­nar­ios cov­ered in this ar­ti­cle.

Send me the PDF!

I have read and ac­cept the pri­va­cy pol­i­cy.