Connection pool and parallel deployment problem

Discussion:

Gilles SCHLIENGER

2018-11-21 11:00:15 UTC

Hi all,

We are using Tomcat 9 and parallel deployment.

I use a connection pool defined in the xml context (myApp##1.xml, myApp##2.xml in my exemple)

I have the following problem :
- I have myApp##1.war deployed using a connection pool (configured in myApp##1.xml)
- I deploy myApp##2.war (using a connection pool defined in myApp##2.xml)
- when the last session in myApp##1 expires, myApp##1 is automatically undeployed (I have undeployOldVersions="true" in server.xml for the Host) but the connections opened by myApp##1 are not closed.

I used the Tomcat configuration from the example in :
https://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP_2)_Configurations

<Resource name="jdbc/integrationGG" auth="Container" type="javax.sql.DataSource"
maxTotal="100" maxIdle="30" maxWaitMillis="10000" destroy-method="close"
username="postgres" password="password" driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/postgres?stringtype=unspecified"/>

During undeploy, I get the following messages :

21-Nov-2018 11:42:54.795 AVERTISSEMENT [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [myApp##1] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

...

21-Nov-2018 11:42:54.800 AVERTISSEMENT [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [myApp##1] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
***@10.0.1/java.lang.Object.wait(Native Method)
***@10.0.1/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
***@10.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
***@10.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
***@10.0.1/java.lang.Thread.run(Thread.java:844)

I have tested adding < destroy-method=close > but it did not change anything

Is this a bug ?

Thanks for any help !

Regards
Gilles

HeeGu Lee

2018-11-21 12:23:58 UTC

Permalink

I think it is good for your mental health not to use this feature.
You just have to think of it as an experimental function like google
chrome's about://flags.

JMX has subscribe model.
However, Tomcat is no ability to receive notifications.
And most JDBC implementations are not interested in such features.
Both Tomcat and DB vendors are not going to fix it.

It's not a tragedy, it's just a comedy.

Post by Gilles SCHLIENGER
Hi all,
We are using Tomcat 9 and parallel deployment.
I use a connection pool defined in the xml context (myApp##1.xml,
myApp##2.xml in my exemple)
- I have myApp##1.war deployed using a connection pool (configured in myApp##1.xml)
- I deploy myApp##2.war (using a connection pool defined in myApp##2.xml)
- when the last session in myApp##1 expires, myApp##1 is automatically
undeployed (I have undeployOldVersions="true" in server.xml for the Host)
but the connections opened by myApp##1 are not closed.
https://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP_2)_Configurations
<Resource name="jdbc/integrationGG" auth="Container"
type="javax.sql.DataSource"
maxTotal="100" maxIdle="30" maxWaitMillis="10000" destroy-method="close"
username="postgres" password="password"
driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/postgres?stringtype=unspecified"/>
21-Nov-2018 11:42:54.795 AVERTISSEMENT
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The
web application [myApp##1] registered the JDBC driver
[org.postgresql.Driver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
...
21-Nov-2018 11:42:54.800 AVERTISSEMENT
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [myApp##1] appears to have started a thread named
[Abandoned connection cleanup thread] but has failed to stop it. This is
/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
I have tested adding < destroy-method=close > but it did not change anything
Is this a bug ?
Thanks for any help !
Regards
Gilles

Gilles SCHLIENGER

2018-11-21 16:33:48 UTC

Permalink

Are you talking about parallel deployment or connection pools ?

Parallel deployment is in Tomcat since Tomcat 7

Gilles

-----Message d'origine-----
De : HeeGu Lee [mailto:***@gmail.com]
Envoyé : mercredi 21 novembre 2018 13:24
À : Tomcat Users List
Objet : Re: Connection pool and parallel deployment problem

I think it is good for your mental health not to use this feature.
You just have to think of it as an experimental function like google
chrome's about://flags.

JMX has subscribe model.
However, Tomcat is no ability to receive notifications.
And most JDBC implementations are not interested in such features.
Both Tomcat and DB vendors are not going to fix it.

It's not a tragedy, it's just a comedy.

Т��ХF�V�7V'67&�&R�R��âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F��6��G2�R�

Mark Thomas

2018-11-21 13:18:07 UTC

Permalink

That is a warning you should be able to fix by unregistering the JDBC
driver in a ServletContextListener.

Post by Gilles SCHLIENGER
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)

That is a thread started by the MySQL driver. There should be an API
provided by the MySQL driver that you can call to stop the thread (again
in a ServletContextListener) although I'd expect that to happen
automatically as part of unloading the driver.

If there is no way to stop the thread then that would be a bug in the
MySQL driver.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Gilles SCHLIENGER

2018-11-23 10:07:21 UTC

Permalink

Thanks Mark for your answer

Here is what I found in case someone has the same problem.

When you use parallel deployment, you should not use a connexion pool in the context.xml file
Once the connexions are opened, they stay opened until Tomcat shuts down, whatever configuration you try.
So everytime you deploy a new version of your webapp, new connexions are being opened.
So we will configure our connexion pools directly in our Spring webapp. This way, the connexions are closed when the webapp is undeployed.

The warning/error messages are not actually linked to the connexions not closed.
To ovoid these messages, you can:
- move the jars (connexion pool, drivers...) into TOMCAT/lib
- Have a ServletContextListener that calls AbandonedConnectionCleanupThread.checkedShutdown();

Regards
Gilles

-----Message d'origine-----
De : Mark Thomas [mailto:***@apache.org]
Envoyé : mercredi 21 novembre 2018 14:18
À : Tomcat Users List; Gilles SCHLIENGER
Objet : Re: Connection pool and parallel deployment problem

That is a warning you should be able to fix by unregistering the JDBC
driver in a ServletContextListener.

Post by Gilles SCHLIENGER
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)

Christopher Schultz

2018-11-24 16:18:34 UTC

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Gilles,

Post by Gilles SCHLIENGER
Thanks Mark for your answer
Here is what I found in case someone has the same problem.
When you use parallel deployment, you should not use a connexion
pool in the context.xml file

Why not?

Post by Gilles SCHLIENGER
Once the connexions are opened, they stay opened until Tomcat
shuts down, whatever configuration you try.

I would expect that any connection pool created for a context (or
instance of a context, when parallel deployments are being done) would
be shut-down when the application is undeployed. Are you actually
undeploying the old context? It doesn't happen automatically IIRC.

Post by Gilles SCHLIENGER
So everytime you deploy a new version of your webapp, new
connexions are being opened.

This is expected. How could it work, otherwise?

Post by Gilles SCHLIENGER
So we will configure our connexion pools directly in our Spring
webapp. This way, the connexions are closed when the webapp is
undeployed.

That's one way to solve the issue, but not the only way.

Post by Gilles SCHLIENGER
The warning/error messages are not actually linked to the
connexions not closed. To ovoid these messages, you can: - move the
jars (connexion pool, drivers...) into TOMCAT/lib - Have a
ServletContextListener that calls
AbandonedConnectionCleanupThread.checkedShutdown();

The Connector/J devs haven't been able to understand how ClassLoaders
work, and have never really fixed that long-standing bug in a
satisfying way as far as I know. But you should always use a
ServletContextListener to attempt to shut-down the
AbadonedConnectionCleanuopThread.

BTW that thread does not work as documented. It's claimed that it
doesn't start unless you issue a query with a timeout, but it always
starts whether or not you use queries with timeouts. It's frustrating.

- -chris

Post by Gilles SCHLIENGER
-----Message d'origine----- De : Mark Thomas
À : Tomcat Users List; Gilles SCHLIENGER Objet : Re: Connection
pool and parallel deployment problem

Post by Gilles SCHLIENGER
Hi all,
We are using Tomcat 9 and parallel deployment.
I use a connection pool defined in the xml context (myApp##1.xml,
myApp##2.xml in my exemple)
I have the following problem : - I have myApp##1.war deployed
using a connection pool (configured in myApp##1.xml) - I deploy
myApp##2.war (using a connection pool defined in myApp##2.xml) -
when the last session in myApp##1 expires, myApp##1 is
automatically undeployed (I have undeployOldVersions="true" in
server.xml for the Host) but the connections opened by myApp##1
are not closed.
https://tomcat.apache.org/tomcat-9.0-doc/jndi-datasource-examples-how

to.html#Database_Connection_Pool_(DBCP_2)_Configurations
<Resource name="jdbc/integrationGG" auth="Container"
type="javax.sql.DataSource"