Bug190537

Al­low Ses­sionHan­dleS­tate reuse af­ter clos­ing a pooled con­nec­tion with re­cy­cleAc­tiveCon­nec­tion­sInTrans­ac­tion=true

Sever­i­ty:	4
Af­fect­ed ver­sion(s):	5.0.104

De­scrip­tion

When set­ting re­cy­cleAc­tiveCon­nec­tion­sInTrans­ac­tion=true, you can now reuse con­nec­tions more flex­i­bly.

Tech­ni­cal de­tails

Con­sid­er the fol­low­ing use case with re­cy­cleAc­tiveCon­nec­tion­sInTrans­ac­tion en­abled:

Method foo():

1. Open con­nec­tion c1
2. Do SQL with c1
3. Call method bar()
4. Do more SQLs with con­nec­tion c1
5. Close c1

Method bar():

1. Open con­nec­tion c2
2. Do SQL with c2
3. Close c2

With re­cy­cleAc­tiveCon­nec­tion­sInTrans­ac­tion=true, c1 will be the same con­nec­tion in­stance as c2.

So af­ter method bar() clos­es c2, c1 will also be closed and this caused er­rors like these in step 4 of method foo():

The underlying XA session is closed
                at com.atomikos.jdbc.internal.AtomikosSQLException.throwAtomikosSQLException(AtomikosSQLException.java:29)
                at com.atomikos.jdbc.internal.AtomikosJdbcConnectionProxy.enlist(AtomikosJdbcConnectionProxy.java:108)
                at com.atomikos.jdbc.internal.AtomikosJdbcConnectionProxy.updateTransactionContext(AtomikosJdbcConnectionProxy.java:61)
                at com.atomikos.jdbc.internal.AbstractJdbcConnectionProxy.prepareStatement(AbstractJdbcConnectionProxy.java:64)
                at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at com.atomikos.util.DynamicProxySupport.callProxiedMethod(DynamicProxySupport.java:162)
                at com.atomikos.util.DynamicProxySupport.invoke(DynamicProxySupport.java:116)
                at com.sun.proxy.$Proxy801.prepareStatement(Unknown Source)

Changes im­pact­ing client API

None.

Fea­ture190375

Al­low re­cy­cling of ac­tive con­nec­tions with­in the same trans­ac­tion

You can now al­low re­cy­cling of ac­tive JDBC/XA pooled con­nec­tions with­in the same trans­ac­tion, be­fore they are "closed" by the ap­pli­ca­tion. This means that cer­tain dead­lock sce­nar­ios can be avoid­ed.

Tech­ni­cal de­tails

Imag­ine the fol­low­ing use case:

1. start a JTA trans­ac­tion
2. get a JDBC/XA con­nec­tion (c1) from the pool via getCon­nec­tion()
3. do some SQL on c1
4. get a sec­ond JDBC/XA con­nec­tion (c2) from the pool via getCon­nec­tion()

Be­fore this fea­ture, step 4 would re­turn a dif­fer­ent phys­i­cal con­nec­tion from the pool. This would trig­ger a new XA branch, with un­spec­i­fied iso­la­tion (lock­ing) be­hav­iour with re­spect to any up­dates per­formed via the con­nec­tion in step 2. This could even cause dead­locks.

There­fore, peo­ple have asked us to al­low for step 4 to reuse the same con­nec­tion, c1. You can now en­able this new be­hav­iour by call­ing setRe­cy­cleAc­tiveCon­nec­tion­sInTrans­ac­tion(true) on the AtomikosDataSourceBean.

Changes im­pact­ing client API

A new, op­tion­al set­ter on our data­source. The de­fault is false for back­ward com­pat­i­bil­i­ty.

Read this for our ex­pe­ri­ence with run­ning on JDK 11...

Bug189921

Avoid that ex­cep­tions (when writ­ing a check­point) need­less­ly cor­rupt the trans­ac­tion log

Sever­i­ty:	2
Af­fect­ed ver­sion(s):	5.0.x, 4.0.x

De­scrip­tion

You now no longer get "Log cor­rupt­ed - restart JVM" ex­cep­tions af­ter you in­ter­rupt a thread that is writ­ing to the trans­ac­tion log file, or af­ter any oth­er ex­cep­tion that make a log check­point fail.

Tech­ni­cal de­tails

Any ex­cep­tions dur­ing a check­point (such as when a thread was in­ter­rupt­ed dur­ing trans­ac­tion log file I/O) would lead to a gener­ic ex­cep­tion han­dling block in our com.atom­ikos.re­cov­ery.fs.CachedRe­pos­i­to­ry class, leav­ing the in­stance in an in­valid state:

2021-03-01 16:15:56.662 ERROR 41669 --- [pool-1-thread-1] c.a.recovery.fs.FileSystemRepository     : Failed to write checkpoint

java.nio.channels.ClosedByInterruptException: null
   at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[na:1.8.0_192]
   at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:392) ~[na:1.8.0_192]
   at com.atomikos.recovery.fs.FileSystemRepository.writeCheckpoint(FileSystemRepository.java:196) ~[transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.recovery.fs.CachedRepository.performCheckpoint(CachedRepository.java:84) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.recovery.fs.CachedRepository.put(CachedRepository.java:77) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.recovery.fs.OltpLogImp.write(OltpLogImp.java:46) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.persistence.imp.StateRecoveryManagerImp.preEnter(StateRecoveryManagerImp.java:51) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:164) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:251) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:284) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CoordinatorStateHandler.commitFromWithinCallback(CoordinatorStateHandler.java:346) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.ActiveStateHandler$6.doCommit(ActiveStateHandler.java:273) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CoordinatorStateHandler.commitWithAfterCompletionNotification(CoordinatorStateHandler.java:587) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.ActiveStateHandler.commit(ActiveStateHandler.java:268) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CoordinatorImp.commit(CoordinatorImp.java:550) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:682) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:279) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:168) [transactions-jta-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:428) [transactions-jta-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.jta.UserTransactionManager.commit(UserTransactionManager.java:160) [transactions-jta-5.0.9-SNAPSHOT.jar:na]
   at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1035) [spring-tx-5.2.5.RELEASE.jar:5.2.5.RELEASE]
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743) [spring-tx-5.2.5.RELEASE.jar:5.2.5.RELEASE]
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711) [spring-tx-5.2.5.RELEASE.jar:5.2.5.RELEASE]
   at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:152) [spring-tx-5.2.5.RELEASE.jar:5.2.5.RELEASE]
   at com.example.atomikos.AtomikosApplicationTests.lambda$4(AtomikosApplicationTests.java:78) [test-classes/:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_192]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_192]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_192]
   at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_192]

Later re­quests try­ing to read from the trans­ac­tion logs would get sys­tem­at­ic cor­rup­tion er­rors like this:

com.atomikos.recovery.LogReadException: Log corrupted - restart JVM
   at com.atomikos.recovery.fs.CachedRepository.assertNotCorrupted(CachedRepository.java:137) ~[transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.recovery.fs.CachedRepository.findAllCommittingCoordinatorLogEntries(CachedRepository.java:145) ~[transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.recovery.fs.RecoveryLogImp.getExpiredPendingCommittingTransactionRecordsAt(RecoveryLogImp.java:52) ~[transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.RecoveryDomainService.performRecovery(RecoveryDomainService.java:76) ~[transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.icatch.imp.RecoveryDomainService$1.alarm(RecoveryDomainService.java:55) [transactions-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:101) [atomikos-util-5.0.9-SNAPSHOT.jar:na]
   at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:88) [atomikos-util-5.0.9-SNAPSHOT.jar:na]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_192]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_192]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192]

This has now been fixed.

Changes im­pact­ing client API

None.

Bug190034

Spring Boot JDBC meta­da­ta: im­prove getAc­tive method

Sever­i­ty:	3
Af­fect­ed ver­sion(s):	5.0.x

De­scrip­tion

The method getAc­tive() in the DataSourceBeanMe­ta­da­ta class­es of mod­ule trans­ac­tions-spring­boot2 now no longer re­turns the to­tal num­ber of open con­nec­tions, but rather the num­ber of con­nec­tions that are cur­rent­ly be­ing used by the ap­pli­ca­tion.

Tech­ni­cal de­tails

Due to a mis­un­der­stand­ing of Spring Boot's se­man­tics, this method re­turned the wrong re­sult: the to­tal num­ber of open con­nec­tions in the pool, rather than the num­ber of con­nec­tions be­ing used. This has now been fixed.

Changes im­pact­ing client API

None.

Bug190035

Spring Boot JDBC meta­da­ta: sup­port wrapped data­sources

Sever­i­ty:	3
Af­fect­ed ver­sion(s):	5.0.x

De­scrip­tion

You can now re­trieve mean­ing­ful DataSourcePoolMe­ta­da­ta in Spring Boot, even if one of our data­sources is used in wrapped or prox­ied mode in your Spring Boot run­time.

Tech­ni­cal de­tails

We used to re­turn meta­da­ta in the fol­low­ing style:

if (dataSource instanceof AtomikosDataSourceBean) {
         return new AtomikosDataSourceBeanMetadata((AtomikosDataSourceBean) dataSource);
}

(and sim­i­lar for our AtomikosNonXADataSourceBean class)

This would not work if the dataSource pre­sent­ed is wrapped or prox­ied. So we now use the built-in Spring Boot DataSourceUn­wrap­per.un­wrap to han­dle those cas­es.

Changes im­pact­ing client API

None.

Is­sue

Post­greSQL: XARe­source ig­nores trans­ac­tion time­out

Sever­i­ty:	2
Af­fect­ed ver­sion(s):	5.0.x, 4.0.x, 3.9.x

De­scrip­tion

The XA im­ple­men­ta­tion of Post­greSQL ig­nores the trans­ac­tion time­out, which means that you may have long-lived or­phaned SQL ses­sions in your data­base serv­er.

Tech­ni­cal de­tails

The XA spec­i­fi­ca­tion al­lows a trans­ac­tion man­ag­er to in­form the XARe­source back­end of trans­ac­tion time­outs, so this in­for­ma­tion can be used to ter­mi­nate (roll­back) pend­ing or long-lived trans­ac­tions. How­ev­er, Post­greSQL seems to ig­nore this in­for­ma­tion (see the source code on GitHub - which some­times leads to pend­ing SQL ses­sions that ex­ceed the trans­ac­tion time­out.

Pos­si­ble workarounds

The fol­low­ing workarounds are avail­able:

Set the queryTime­out on your JDBC State­ment ob­jects, or try set­ting a serv­er-lev­el time­out like this:

SET SESSION idle_in_transaction_session_timeout = '5min’;

If you have any oth­er so­lu­tion then please let us know - thanks!

Fea­ture189601

Al­low dis­abling retry on heuris­tic haz­ard par­tic­i­pants

You can now choose to dis­able retry­ing com­mit or roll­back for heuris­tic haz­ard trans­ac­tions.

Tech­ni­cal de­tails

Heuris­tic haz­ard trans­ac­tions can arise out of net­work con­nec­tiv­i­ty is­sues dur­ing the com­mit phase: if a re­source gets a pre­pare re­quest and sub­se­quent­ly be­comes un­reach­able dur­ing com­mit or roll­back then the trans­ac­tion will go into "heuris­tic haz­ard" mode. This es­sen­tial­ly means that com­mit will be re­tried a num­ber of times, even if com.atom­ikos.icatch.olt­p_­max_re­tries is set to zero. The ra­tio­nale be­ing: it is bet­ter to ter­mi­nate pend­ing in-doubt trans­ac­tions soon­er rather than lat­er be­cause of the pend­ing locks they may be hold­ing on to.

If you don't want this be­hav­iour then you can now dis­able this, and rely on the re­cov­ery process in the back­ground to take care of it (which also works, but will hap­pen only pe­ri­od­i­cal­ly). To dis­able, just set this new prop­er­ty to false:

com.atomikos.icatch.retry_on_heuristic_hazard=false

Changes im­pact­ing client API

A new start­up prop­er­ty that can op­tion­al­ly be set. If not present, it will de­fault to true to pre­serve com­pat­i­bil­i­ty with ex­ist­ing be­hav­iour.

Fea­ture189603

API ex­ten­sion to al­low trig­ger­ing re­cov­ery by the ap­pli­ca­tion

You can now ex­plic­it­ly trig­ger re­cov­ery in your ap­pli­ca­tion, via our API.

Tech­ni­cal de­tails

Re­cov­ery al­ready hap­pens pe­ri­od­i­cal­ly, in the back­ground. For big­ger clus­ters that con­nect to the same data­base (or oth­er shared re­source) this can cause a high load on the back­end, be­cause of many such back­ground threads hit­ting the back­end at the same time. This is es­pe­cial­ly true if most clus­ter nodes start up at the same time with the same con­fig­u­ra­tion for re­cov­ery, and are NOT us­ing LogCloud Doc­u­men­ta­tion. To al­le­vi­ate this, you can now have a bit more con­trol over when re­cov­ery hap­pens, like this:

import com.atomikos.icatch.RecoveryService;
import com.atomikos.icatch.config.Configuration;

boolean lax = true; //false to force recovery, true to allow intelligent mode
RecoveryService rs = Configuration.getRecoveryService();
rs.performRecovery(lax);

In or­der for this to work, make sure to set (in jta.prop­er­ties):

# set to Long.MAX_VALUE so background recovery is disabled
com.atomikos.icatch.recovery_delay=9223372036854775807L 

Changes im­pact­ing client API

We have added meth­ods on an ex­ist­ing API in­ter­face, which does not break ex­ist­ing clients.

Bug189602

Don't call XA re­cov­ery on the XARe­source on Heuris­ticHazard

Sever­i­ty:	2
Af­fect­ed ver­sion(s):	5.0.x

De­scrip­tion

From now on we no longer sys­tem­at­i­cal­ly call XARe­source.re­cov­er() when fail­ures hap­pen dur­ing the reg­u­lar com­mit or roll­back, so the over­head for the back­end is re­duced.

Tech­ni­cal de­tails

For his­tor­i­cal rea­sons we used to call the XA re­cov­ery rou­tine on the backed when­ev­er com­mit or roll­back failed. The most com­mon cause is net­work glitch­es, mean­ing that big clus­ters with a short net­work prob­lem would sud­den­ly hit the back­ends with re­cov­ery for all ac­tive trans­ac­tions. Since re­cov­ery can be an ex­pen­sive op­er­a­tion, this would re­sult in need­less load on the back­ends.

The ra­tio­nale be­hind this was to avoid need­less com­mit re­tries (based on the val­ue of com.atom­ikos.icatch.olt­p_­max_re­tries), but the over­head does not jus­ti­fy the pos­si­ble ben­e­fit.

From now on we no longer do this, since it is ei­ther the re­cov­ery process (in the back­ground) or the ap­pli­ca­tion (via our API) that con­trols when re­cov­ery hap­pens.

Worst case, this can lead to need­less com­mit re­tries, in which case the back­end should re­spond with er­ror code XAER_NOTA and our code will han­dle this grace­ful­ly. How­ev­er, we have his­tor­i­cal records where some old­er ver­sion of Ac­tiveMQ did not be­have like this. This would re­sult in er­rors in the Ac­tiveMQ log files, in turn lead­ing to alerts for the op­er­a­tions team.

Changes im­pact­ing client API

If you ex­pe­ri­ence is­sues with this, then it suf­fices to set com.atom­ikos.icatch.olt­p_­max_re­tries to zero. That will dis­able reg­u­lar com­mit re­tries and del­e­gate to the re­cov­ery back­ground process.

Is­sue189886

Avoid us­ing 0 for the max­i­mum trans­ac­tion time­out

Sever­i­ty:	2
Af­fect­ed ver­sion(s):	5.0.x

De­scrip­tion

For re­leas­es 5.0 or high­er, the max­i­mum time­out should not be set to 0 or re­cov­ery will in­ter­fere with reg­u­lar ap­pli­ca­tion-lev­el com­mits.

Tech­ni­cal de­tails

The 5.0 re­lease has a new re­cov­ery work­flow that is in­com­pat­i­ble with com.atom­ikos.icatch.max_­time­out be­ing zero. That is be­cause re­cov­ery de­pends on the max­i­mum time­out to per­form roll­back of pend­ing (or­phaned) pre­pared trans­ac­tions in the back­ends. If the max­i­mum time­out is zero then re­cov­ery (in the back­ground) will roll­back pre­pared trans­ac­tions that are con­cur­rent­ly be­ing com­mit­ted in your ap­pli­ca­tion. This will re­sult in heuris­tic ex­cep­tions and in­con­sis­tent trans­ac­tion out­comes.

Keep in mind that the max­i­mum time­out is also in­dica­tive of max­i­mum lock du­ra­tion in your data­bas­es, so choose it wise­ly! If you are / were de­pend­ing on an un­lim­it­ed max­i­mum time­out then you are also al­low­ing un­lim­it­ed lock times.