"Too Many Connections" exceptions after moving to Tomcat 8

Hi jerry ,

It's probably issue with the application itself,
You might want to check your application for connections leaks.
I usually use this https://github.com/javamelody/javamelody/wiki
To monitor the application in both production and test environment.
It's easy to use and very efficient tool.
Hope it helps.

With regards
Prabhu

-----Original Message-----
From: Jerry Malcolm [mailto:***@malcolms.com]
Sent: Wednesday, 29 July, 2015 1:39 PM
To: Tomcat Users List <***@tomcat.apache.org>
Subject: "Too Many Connections" exceptions after moving to Tomcat 8

I am trying to move my client to Tomcat 8 from 7. We've been on 7 for a year or two and have not had any major issues. I read through the 7-->8 migration instructions. As far as I can tell, I have renamed all of the resource config parameters that changed (maxTotal, maxWaitMillis, removedAbandonedOnBorrow). But after running for a couple of hours, I start getting "Too Many Connections" exceptions when connecting to the db. Granted, there's a chance I have some connection leaks in the code, and I'll look into those. But this hasn't been an issue on 7 for the past year or two. And shouldn't removeAbandoned clean those up? I've got logAbandoned set to 'true', and nothing is showing up in the logs.
Is this exception referring to running out of available connections in the pool?

Is there something else I need to change in the configuration to move to Tomcat 8 that I've overlooked?

Thanks.

Jerry

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Konstantin Kolinko

2015-07-29 08:49:39 UTC

Post by Jerry Malcolm
I am trying to move my client to Tomcat 8 from 7.

Tomcat version = ?

Post by Jerry Malcolm
We've been on 7 for a
year or two and have not had any major issues. I read through the 7-->8
migration instructions. As far as I can tell, I have renamed all of the
resource config parameters that changed (maxTotal, maxWaitMillis,
removedAbandonedOnBorrow).

There are two pool implementations in Tomcat 7 as well as in Tomcat 8.
Which one are you using? (Your factory = ?)

Post by Jerry Malcolm
But after running for a couple of hours, I start
getting "Too Many Connections" exceptions when connecting to the db.
Granted, there's a chance I have some connection leaks in the code, and I'll
look into those. But this hasn't been an issue on 7 for the past year or
two. And shouldn't removeAbandoned clean those up? I've got logAbandoned
set to 'true', and nothing is showing up in the logs. Is this exception
referring to running out of available connections in the pool?

Describe "this exception" in more detail.

What component logs that message? (class, method name that accompany
the log message)
Stack trace?

Post by Jerry Malcolm
Is there something else I need to change in the configuration to move to
Tomcat 8 that I've overlooked?

Read official documentation for Apache Commons DBCP pool
http://commons.apache.org/proper/commons-dbcp/configuration.html

Maybe you will notice some discrepancies with your configuration.

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Jerry Malcolm

2015-07-29 13:47:20 UTC

Post by Jerry Malcolm
I am trying to move my client to Tomcat 8 from 7.

Tomcat version = ?

version 8.0.15

There are two pool implementations in Tomcat 7 as well as in Tomcat 8.
Which one are you using? (Your factory = ?)

I looked at my <resource> tags, and I'm not specifying a factory
attribute at all. The doc says: "...with the only difference being that
you have to specify the |factory| attribute and set the value to
|org.apache.tomcat.jdbc.pool.DataSourceFactory" Is this new in TC 8 or
have I just been misconfigured for years? I assume from the doc that I
need to go in and add that to all resource tags, correct?
What has the default been in TC7 and TC8 without a factory being
specified? Could leaving this off actually be the cause of running out
of connections?
|

Describe "this exception" in more detail.
What component logs that message? (class, method name that accompany
the log message)
Stack trace?

Stack trace at the very bottom of this note

Post by Jerry Malcolm
Is there something else I need to change in the configuration to move to
Tomcat 8 that I've overlooked?

Read official documentation for Apache Commons DBCP pool
http://commons.apache.org/proper/commons-dbcp/configuration.html
Maybe you will notice some discrepancies with your configuration.

I'll keep looking. But I'm very curious that even though I might have
connection leaks, what is preventing Tomcat from abandoning them and
returning them to the pool as I believe I have it configured to do?
I've got logAbandoned set to true. But I'm not seeing a single entry in
the logs about connections being abandoned.

Off topic, but relevant.... it sure would be nice if the xml parser for
<resource> tags and other config items would flag with errors any
parameter misspellings such as using the old obsolete 'removeAbandoned'
instead of the new 'removeAbandonedOnBorrow', etc. or even just typo
misspells.

Post by Konstantin Kolinko
Best regards,
Konstantin Kolinko

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Could not create connection to database server. Attempted reconnect 3
times. Giving up.
at
sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870)
at
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2144)
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2069)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
at
sun.reflect.GeneratedConstructorAccessor53.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
at
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:204)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:860)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
at jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:117)
at jwm.servletdb.ServletDBData.getInstance(ServletDBData.java:101)
at jwm.wgbe.PageRouterFilter.doFilter(PageRouterFilter.java:151)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at
org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:291)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
at
org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:831)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1556)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1513)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Data source rejected establishment of connection, message from server:
"Too many connections"
at
sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1037)
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
at
com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2085)
... 38 more

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Christopher Schultz

2015-07-29 15:11:56 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Jerry,

2015-07-29 8:38 GMT+03:00 Jerry Malcolm

Post by Jerry Malcolm
I am trying to move my client to Tomcat 8 from 7.

Tomcat version = ?

version 8.0.15

Any reason not to upgrade to 8.0.24? 8.0.15 is nearly a year out of
date and there have been a bunch of performance, security, and
stability fixes made. It shouldn't have any effect on your current
problem, but upgrading would be worth your time.

Post by Jerry Malcolm
We've been on 7 for a year or two and have not had any major
issues. I read through the 7-->8 migration instructions. As
far as I can tell, I have renamed all of the resource config
parameters that changed (maxTotal, maxWaitMillis,
removedAbandonedOnBorrow).

There are two pool implementations in Tomcat 7 as well as in
Tomcat 8. Which one are you using? (Your factory = ?)

I looked at my <resource> tags, and I'm not specifying a factory
attribute at all.

Then you are using the default commons-DBCP-based connection pool. The
configuration is different for each one, so it's important we know
which one you have.

Post by Jerry Malcolm
What has the default been in TC7 and TC8 without a factory being
specified?

commons-DBCP for both, though TC7 used DBCP 1.2 and TC 8 uses DBCP 2.
It's possible you have found a bug in DBCP2 but at this point, it's
unlikely since nobody else is really having this kind of problem.

Post by Jerry Malcolm
Could leaving this off actually be the cause of running out of
connections?

No, it just chooses the connection pool factory.

Post by Jerry Malcolm
But after running for a couple of hours, I start getting "Too
Many Connections" exceptions when connecting to the db.
Granted, there's a chance I have some connection leaks in the
code, and I'll look into those. But this hasn't been an issue
on 7 for the past year or two. And shouldn't removeAbandoned
clean those up? I've got logAbandoned set to 'true', and
nothing is showing up in the logs. Is this exception referring
to running out of available connections in the pool?

Describe "this exception" in more detail.
What component logs that message? (class, method name that
accompany the log message) Stack trace?

Stack trace at the very bottom of this note

Post by Jerry Malcolm
Is there something else I need to change in the configuration
to move to Tomcat 8 that I've overlooked?

Read official documentation for Apache Commons DBCP pool
http://commons.apache.org/proper/commons-dbcp/configuration.html
Maybe you will notice some discrepancies with your
configuration.

Do you have a test environment you can instrument? I highly recommend
setting maxTotal="1" (or maxActive="1" for DBCP 1.2 for Tomcat 7) and
using whatever spelling of logAbandoned="true" for your DBCP version.
That will find leaks pretty quickly, especially the ones that can
cause deadlocks in production if you aren't careful.

http://blog.christopherschultz.net/index.php/2009/03/16/properly-handlin
g-pooled-jdbc-connections/

Post by Jerry Malcolm
I've got logAbandoned set to true. But I'm not seeing a single
entry in the logs about connections being abandoned.

What's the abandoned timeout?

Post by Jerry Malcolm
Off topic, but relevant.... it sure would be nice if the xml parser
for <resource> tags and other config items would flag with errors
any parameter misspellings such as using the old obsolete
'removeAbandoned' instead of the new 'removeAbandonedOnBorrow',
etc. or even just typo misspells.

It does:

Jul 02, 2015 10:17:25 AM
org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
ObjectInstance
WARNING: Name = diagnosis Property maxActive is not used in DBCP2, use
maxTotal instead. maxTotal default value is 8. You have set value of
"1" for "maxActive" property, which is being ignored.

...

WARNING: Name = diagnosis Property removeAbandoned is not used in
DBCP2, use one or both of removeAbandonedOnBorrow or
removeAbandonedOnMaintenance instead. Both have default value set to
false. You have set value of "true" for "removeAbandoned" property,
which is being ignored.

(I have a context.xml that I want to use between both Tomcat 7 and
Tomcat 8, so I actually have both spellings of those attributes in my
XML.)

What does your <Resource> element actually look like? (Remember to
remove sensitive information such as passwords.)

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVuO08AAoJEBzwKT+lPKRYlNgQAI/MQUBV45kvHxv8Ma8Kj8EV
xQ3njp1FoCXmdqzfSptrJ+eQh7EkJZ4dJ5sqvTysfl71JgetgwdleD3muaJfW0ES
xmQysVfTzaj88bj18P0OE8vps6T/4+V/mAB/qqttsdvviY9pnFzMzKUq2PQNaoIn
H0L2SW6frseKVGSh8hJ63JvC42d1HGjPpLy4Mmx19/J3c+TSckj7ywyKMeTr90yd
tm1xlmX+EW2DFebmfJ+ywCgugt2um+Zr9G5c9pmmKyeVGAvhDE5+iygFjBWkwvhj
FXL25vmOxa3bGnuakEC5LAr7oet11IlYeN2/CARsRKeNXGEJTfC1AZ+GK96quFxS
DD5gakEn2m9WFQNW+kvZRPG/7Jr7b2zVoz/H0H4ahzDWv/jyOf2YSQxfb23+curh
V3sWt8LOnJNX0hetoEvnkLn+nWIxKpTFSVmT0dc9PbRRcKb0vZmOzcgPqOBeQa/c
dARLaaVUXNIciW+4W/V5N8THVTZ75Lk2E42r7A8eSvUssgsEVtDhv3tZKZCKV5B4
ry977SiEoDJICebADCQArtpQGaGh6oPk3Xl2qeAXahvoiQzYhIbN92vfJOVqOg4H
Z0vG28icUv7kLWvDV3gv1DeZ6wnjIjOoZ9icBl8+AdsNQNGqRDanTaBcQGwsxHbV
F/3q9SdAkwxjqIpGxD7O
=Tyk6
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Jerry Malcolm

2015-07-29 16:06:55 UTC

This post might be inappropriate. Click to display it.

Christopher Schultz

2015-07-29 18:15:13 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Jerry,

I guess I could move up to the latest 8. But it'll require some
client down time, and they aren't thrilled with that at this
point.

Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?

Post by Jerry Malcolm
Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.

Jul 02, 2015 10:17:25 AM
org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory get
ObjectInstance WARNING: Name = diagnosis Property maxActive is
not used in DBCP2, use maxTotal instead. maxTotal default value
is 8. You have set value of "1" for "maxActive" property, which
is being ignored.
...
WARNING: Name = diagnosis Property removeAbandoned is not used
in DBCP2, use one or both of removeAbandonedOnBorrow or
removeAbandonedOnMaintenance instead. Both have default value set
to false. You have set value of "true" for "removeAbandoned"
property, which is being ignored.
(I have a context.xml that I want to use between both Tomcat 7
and Tomcat 8, so I actually have both spellings of those
attributes in my XML.)

This is really strange. Are the above log entries in stdout?

Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.

I had 'removeAbandoned' in my resource statements until yesterday.
But I am not seeing anything in stdout like the statement above.
And after changing removeAbandoned to the correct spelling, I'm
also not seeing any abandoned log entries either.

Hmm.

Is there some sort of 'log level' that may be throttling back my
log entries? If my code has leaks, and I'm running out of
connections but not seeing any abandoned entries in the log, that
should be a concern, right?

Possibly.

Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.

Post by Christopher Schultz
What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)

<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"

validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.

Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.

auth="Container" type="javax.sql.DataSource" maxTotal="300"

That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?

maxIdle="30" maxWaitMillis="10000" removeAbandonedOnBorrow="true"
removeAbandonedOnMaintenance="true" removeAbandonedTimeout="30"
logAbandoned="true" username="xxxxxxx" password="xxxxxxxxx"
driverClassName="org.gjt.mm.mysql.Driver"
url="jdbc:mysql://localhost:3306/wgbe?autoReconnect=true" />

Do you have the 'manager' web application installed? Can you check to
see what applications are actually deployed? If you have 2 copies
deployed, you might see 600 connections open.

Also, do you have any stray context.xml files hanging around in
conf/*/* that you weren't expecting? If you are making changes to your
configuration and they don't appear to be having any effect, it's
possible that your new configuration isn't being applied because
another config is masking it.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVuRgxAAoJEBzwKT+lPKRY/2YQALiweWD9Le3Ip3LeyFL/c43d
ngd50y3iqYcPGbA33IOAy5ThDmcrRDiSbs5cMG4hhAo3rFIU8QmrCJdSFrhLIXu4
K0ZcZPtlmPrEL0CUUE2D6HIX4mg0FMiNB7c0N4XjB6iicd16VGEMBVs4Ck7Sr0DM
A2zTFEYhH/YXlWoOIkTE8gduNfHNNM7y6aaspIgRy9gqAMRFvD4HZaXONCrXPyKD
hh2o8dInaSZY/jv3HrGcZd0SG9LzggKUynN195SOhc9cy6tsvOeIRFiOZWKckYSz
NpKg77+2q9SYFVM5GSKPdrFlzbcjavmsoAP3EvCtWV5HvFE7MHo/ajvI/rSb47Uy
3jYD2b7S9NZdvPu/d8EPlqL29gbUhK2a1ISIY0S3Dgyi/Junzps/sztmOfgib71X
IRDa3KH6gww4IwOX9K8YmMfHi9On06M2NF9NOVYiLqEN4CkHPAQ8YhzpnQz7/LSg
DsxLQ9D5lDdaBjE3kvL17kd9h969SgM2SJKHfHay2xOTYjOHQ9vHY1EjF/C+3smd
BzChYmUmjy0CBWjSCfYxJTjShNkLvyRyenAE+5WAFZhpOGTGW2DxSCXf9zQ2rqlv
vSk7xVf6AMrZXQEZ376MsbRXGlUipZ9ZlufXLj9Pdqmcad7WoLHGaKNNwIyJvdWy
AVhayMcrSTj5Y/xBeSXQ
=xsp7
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Jerry Malcolm

2015-07-29 19:25:10 UTC

Well, it appears that we are slowly getting to the bottom of this. But
with every answer, I get a few more questions....

First, I installed the latest TC8 on my laptop, copied my server.conf
and conf/catalina folder to it and started it up just to see what errors
I got. After changing out an obsolete listener, it came up. I found
all of the <resource> parm exceptions in stderr. So that question is
cleared up. Thanks for clarifying where to find that.

The site is a wedding vendor advertiser site that spans two major
cities. There is no user login. Simply a very huge online catalog.
I'm certain it's deployed only once. Whether I need that many
connections is a valid question. As far as I know, I haven't hit the
limit in normal operation until now. Could possibly reduce the count if
I collect statistics.

I've been monitoring the production server logs all day watching to be
sure connection pool doesn't dry up again. About an hour ago, there was
a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
They showed connections from 1am right after my last bounce of the
server thru 1:35pm. The good news is with the stack trace on one of
them I was able to see the bug causing the leak. But why did it decide
to wait over 12 hours accumulating abandoned connections before dumping
them back in the pool? I realize from now knowing the code bug that the
leak is a slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has returned to
silence. Since leaks are occurring regularly and would be timing out
regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
in stdout instead of a big dump every 12 hours?

It's going to take a day or two to fix the leak, test, and deploy. If
indeed abandoned connections are now correctly being returned to the
pool, then I presume we are back to working the way it did on TC7.
Still not sure why it started working now. But I guess once I get the
leak fixed and if TC8 is now configured to handle abandoned connections,
I'm good. Still would like to know about the mega-dump vs. trickle of
abandoned connections being logged.

Thanks again for all of the info.

Jerry

Post by Christopher Schultz
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jerry,

I guess I could move up to the latest 8. But it'll require some
client down time, and they aren't thrilled with that at this
point.

Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?

Post by Jerry Malcolm
Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.

This is really strange. Are the above log entries in stdout?

Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.

Hmm.

Possibly.
Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.

Post by Christopher Schultz
What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)

<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"

validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.
Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.

auth="Container" type="javax.sql.DataSource" maxTotal="300"

That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?

Do you have the 'manager' web application installed? Can you check to
see what applications are actually deployed? If you have 2 copies
deployed, you might see 600 connections open.
Also, do you have any stray context.xml files hanging around in
conf/*/* that you weren't expecting? If you are making changes to your
configuration and they don't appear to be having any effect, it's
possible that your new configuration isn't being applied because
another config is masking it.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVuRgxAAoJEBzwKT+lPKRY/2YQALiweWD9Le3Ip3LeyFL/c43d
ngd50y3iqYcPGbA33IOAy5ThDmcrRDiSbs5cMG4hhAo3rFIU8QmrCJdSFrhLIXu4
K0ZcZPtlmPrEL0CUUE2D6HIX4mg0FMiNB7c0N4XjB6iicd16VGEMBVs4Ck7Sr0DM
A2zTFEYhH/YXlWoOIkTE8gduNfHNNM7y6aaspIgRy9gqAMRFvD4HZaXONCrXPyKD
hh2o8dInaSZY/jv3HrGcZd0SG9LzggKUynN195SOhc9cy6tsvOeIRFiOZWKckYSz
NpKg77+2q9SYFVM5GSKPdrFlzbcjavmsoAP3EvCtWV5HvFE7MHo/ajvI/rSb47Uy
3jYD2b7S9NZdvPu/d8EPlqL29gbUhK2a1ISIY0S3Dgyi/Junzps/sztmOfgib71X
IRDa3KH6gww4IwOX9K8YmMfHi9On06M2NF9NOVYiLqEN4CkHPAQ8YhzpnQz7/LSg
DsxLQ9D5lDdaBjE3kvL17kd9h969SgM2SJKHfHay2xOTYjOHQ9vHY1EjF/C+3smd
BzChYmUmjy0CBWjSCfYxJTjShNkLvyRyenAE+5WAFZhpOGTGW2DxSCXf9zQ2rqlv
vSk7xVf6AMrZXQEZ376MsbRXGlUipZ9ZlufXLj9Pdqmcad7WoLHGaKNNwIyJvdWy
AVhayMcrSTj5Y/xBeSXQ
=xsp7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Mark Thomas

2015-07-29 20:39:47 UTC

Post by Jerry Malcolm
Well, it appears that we are slowly getting to the bottom of this. But
with every answer, I get a few more questions....
First, I installed the latest TC8 on my laptop, copied my server.conf
and conf/catalina folder to it and started it up just to see what errors
I got. After changing out an obsolete listener, it came up. I found
all of the <resource> parm exceptions in stderr. So that question is
cleared up. Thanks for clarifying where to find that.
The site is a wedding vendor advertiser site that spans two major
cities. There is no user login. Simply a very huge online catalog.
I'm certain it's deployed only once. Whether I need that many
connections is a valid question. As far as I know, I haven't hit the
limit in normal operation until now. Could possibly reduce the count if
I collect statistics.
I've been monitoring the production server logs all day watching to be
sure connection pool doesn't dry up again. About an hour ago, there was
a single huge dump in stdout of approx 2000 'logAbandoned' exceptions.
They showed connections from 1am right after my last bounce of the
server thru 1:35pm. The good news is with the stack trace on one of
them I was able to see the bug causing the leak. But why did it decide
to wait over 12 hours accumulating abandoned connections before dumping
them back in the pool? I realize from now knowing the code bug that the
leak is a slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has returned to
silence. Since leaks are occurring regularly and would be timing out
regularly, shouldn't I see a similar 'slow drip' of logAbandoned entries
in stdout instead of a big dump every 12 hours?
It's going to take a day or two to fix the leak, test, and deploy. If
indeed abandoned connections are now correctly being returned to the
pool, then I presume we are back to working the way it did on TC7.
Still not sure why it started working now. But I guess once I get the
leak fixed and if TC8 is now configured to handle abandoned connections,
I'm good. Still would like to know about the mega-dump vs. trickle of
abandoned connections being logged.

That sounds like a bug that has just been fixed in Pool:
https://issues.apache.org/jira/browse/POOL-300

The fix for that should be in the next Tomcat 8 release.

Mark

Post by Jerry Malcolm
Thanks again for all of the info.
Jerry
Jerry,

I guess I could move up to the latest 8. But it'll require some
client down time, and they aren't thrilled with that at this
point.

Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?

Post by Jerry Malcolm
Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.

This is really strange. Are the above log entries in stdout?

Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.

Hmm.

Possibly.
Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.

Post by Christopher Schultz
What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)

<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"

validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.
Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.

auth="Container" type="javax.sql.DataSource" maxTotal="300"

That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?

Post by Jerry Malcolm
---------------------------------------------------------------------
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Jerry Malcolm

2015-07-29 20:49:27 UTC

Post by Mark Thomas

https://issues.apache.org/jira/browse/POOL-300
The fix for that should be in the next Tomcat 8 release.
Mark

Thanks, Mark..... Yep that would definitely explain why with a smaller
count of abandoned connections, I wasn't seeing any output.

Jerry

Post by Mark Thomas

Post by Jerry Malcolm
Thanks again for all of the info.
Jerry
Jerry,

I guess I could move up to the latest 8. But it'll require some
client down time, and they aren't thrilled with that at this
point.

Understood, though trying various <Resource> configurations is
certainly resulting in downtime, no?

Post by Jerry Malcolm
Off topic, but relevant.... it sure would be nice if the xml
parser for <resource> tags and other config items would flag
with errors any parameter misspellings such as using the old
obsolete 'removeAbandoned' instead of the new
'removeAbandonedOnBorrow', etc. or even just typo misspells.

This is really strange. Are the above log entries in stdout?

Yes, so they typically end up in catalina.out. Those logs are from my
dev system's catalina.out from a recent restart.

Hmm.

Possibly.
Perhaps you are deploying your application more times than you
thought, and are making many more connections than you expected.

Post by Christopher Schultz
What does your <Resource> element actually look like? (Remember
to remove sensitive information such as passwords.)

<Resource name="jdbc/wgbe" testOnBorrow="true"
validationQuery="SELECT 1"

validationQuery might not be necessary anymore. I think DBCP2 does a
lightweight connection test instead issuing an actual query, unless
you override it with a validationQuery.
Also, for Connector/J, you can use "/* ping */SELECT 1" and the driver
itself will perform the lightweight connection test instead of
actually issuing a query. Much faster than going through the SQL
parser, etc.

auth="Container" type="javax.sql.DataSource" maxTotal="300"

That's a LOT of connections IMO. What is your total connection limit?
Is it per-user?

---------------------------------------------------------------------

---------------------------------------------------------------------
-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.6081 / Virus Database: 4392/10332 - Release Date: 07/29/15

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Christopher Schultz

2015-07-30 16:03:43 UTC

This post might be inappropriate. Click to display it.

Phil Steitz

2015-07-31 00:05:15 UTC

Post by Christopher Schultz
Jerry,

If you have an obsolete Listener, you probably copied your server.xml
from Tomcat 7 to Tomcat 8 which, while being less of a disaster than
with previous version-pairs, is not good practice.
Instead, start with the stock server.xml that comes with your Tomcat
version and modify it to suit. These days, you should pretty much only
have to configure the <Executor> and <Connector> elements, unless you
have a particularly exotic <Host> configuration.

Post by Jerry Malcolm
The site is a wedding vendor advertiser site that spans two major
cities. There is no user login. Simply a very huge online
catalog. I'm certain it's deployed only once. Whether I need that
many connections is a valid question. As far as I know, I haven't
hit the limit in normal operation until now. Could possibly reduce
the count if I collect statistics.

Our user load is roughly 250 concurrently logged-in users per Tomcat
node, and we have maxTotal="20". I never get alarms about hitting that
maximum. Your requirements may be different.

Post by Jerry Malcolm
I've been monitoring the production server logs all day watching to
be sure connection pool doesn't dry up again. About an hour ago,
there was a single huge dump in stdout of approx 2000
'logAbandoned' exceptions. They showed connections from 1am right
after my last bounce of the server thru 1:35pm.

It looks like your startup process (likely loading and caching stuff
from the db on launch) is leaky. That can run-up your connection could
quite quickly.

Post by Jerry Malcolm
The good news is with the stack trace on one of them I was able to
see the bug causing the leak.

Good.

Post by Jerry Malcolm
But why did it decide to wait over 12 hours accumulating abandoned
connections before dumping them back in the pool?

I was about to say the following, but markt says it might be a bug in
DBCP.

The bug is in Commons Pool (POOL-300). It is not flushing its
abandoned object log. That means abandoned traces won't appear (in
the default System.out configuration) until some have been
accumulated.

Post by Christopher Schultz
DBCP 2 looks like it only checks for abandoned connections "on borrow"
so it might only log their abandonment when you see a flurry of
connection-checkouts occurring, not when the connections are returned
to the pool. DBCP 1 would complain pretty much immediately when the
timeout was reached and the connection hadn't been returned.

When DBCP checks for abandoned connections depends in its
configuration properties. There are two relevant properties:
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.

Post by Jerry Malcolm
I realize from now knowing the code bug that the leak is a slow
drip that is continually leaking on a regular basis. But since that
last 12-hour accumulated dump, the abandoning has returned to
silence. Since leaks are occurring regularly and would be timing
out regularly, shouldn't I see a similar 'slow drip' of
logAbandoned entries in stdout instead of a big dump every 12
hours?
It's going to take a day or two to fix the leak, test, and deploy.

For testing, set maxTotal="1". You'll find your leaks *very quickly*
that way, because everything will come grinding to a halt when you try
to fetch that second connection from the pool.

Post by Jerry Malcolm
If indeed abandoned connections are now correctly being returned to
the pool, then I presume we are back to working the way it did on
TC7. Still not sure why it started working now. But I guess once I
get the leak fixed and if TC8 is now configured to handle abandoned
connections, I'm good. Still would like to know about the
mega-dump vs. trickle of abandoned connections being logged.

You should be able to run in testing with an upgraded DBCP 2. You
might have to build it from trunk, though. I'm not sure if you are
okay with that, but it might help you with your testing.

The thing to swap out is Commons Pool. There is a release VOTE in
progress now for an RC including a fix for POOL-300.

A workaround that should work is to get a reference to the
BasicDataSource instance, say, bds and do

bds.setAbandonedLogWriter(new PrintWriter(System.out, true));

before using the pool.

Not sure if this will work correctly to get the output properly
directed under tomcat; but it is worth a try.

Phil

Post by Christopher Schultz
Glad you are rooting-out some problems with your code.
http://findbugs.sourceforge.net/
It automatically detects stuff like this in your code, as well as a
whole bunch of other things. It sometimes finds false positives, but
you can knock-up a configuration file to exclude those things in very
specific cases if need be.
-chris
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Christopher Schultz

2015-07-31 15:16:49 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Phil,

Post by Christopher Schultz
Jerry,

If you have an obsolete Listener, you probably copied your
server.xml from Tomcat 7 to Tomcat 8 which, while being less of a
disaster than with previous version-pairs, is not good practice.
Instead, start with the stock server.xml that comes with your
Tomcat version and modify it to suit. These days, you should
pretty much only have to configure the <Executor> and <Connector>
elements, unless you have a particularly exotic <Host>
configuration.

Post by Jerry Malcolm
The site is a wedding vendor advertiser site that spans two
major cities. There is no user login. Simply a very huge
online catalog. I'm certain it's deployed only once. Whether I
need that many connections is a valid question. As far as I
know, I haven't hit the limit in normal operation until now.
Could possibly reduce the count if I collect statistics.

Our user load is roughly 250 concurrently logged-in users per
Tomcat node, and we have maxTotal="20". I never get alarms about
hitting that maximum. Your requirements may be different.

Post by Jerry Malcolm
I've been monitoring the production server logs all day
watching to be sure connection pool doesn't dry up again.
About an hour ago, there was a single huge dump in stdout of
approx 2000 'logAbandoned' exceptions. They showed connections
from 1am right after my last bounce of the server thru 1:35pm.

It looks like your startup process (likely loading and caching
stuff from the db on launch) is leaky. That can run-up your
connection could quite quickly.

Post by Jerry Malcolm
The good news is with the stack trace on one of them I was able
to see the bug causing the leak.

Good.

Post by Jerry Malcolm
But why did it decide to wait over 12 hours accumulating
abandoned connections before dumping them back in the pool?

I was about to say the following, but markt says it might be a
bug in DBCP.

The bug is in Commons Pool (POOL-300). It is not flushing its
abandoned object log. That means abandoned traces won't appear
(in the default System.out configuration) until some have been
accumulated.

Thanks for the correction.

Post by Christopher Schultz
DBCP 2 looks like it only checks for abandoned connections "on
borrow" so it might only log their abandonment when you see a
flurry of connection-checkouts occurring, not when the
connections are returned to the pool. DBCP 1 would complain
pretty much immediately when the timeout was reached and the
connection hadn't been returned.

When DBCP checks for abandoned connections depends in its
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.

Right. I think most people don't use the "maintenance" mode, so I was
being sloppy. I haven't read the code, but the configuration options
make it sound like the connection isn't checked until it's borrowed
again from the pool, which could be a very long time after it would be
expected to have been "abandoned".

Post by Jerry Malcolm
I realize from now knowing the code bug that the leak is a
slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has
returned to silence. Since leaks are occurring regularly and
would be timing out regularly, shouldn't I see a similar 'slow
drip' of logAbandoned entries in stdout instead of a big dump
every 12 hours?
It's going to take a day or two to fix the leak, test, and
deploy.

For testing, set maxTotal="1". You'll find your leaks *very
quickly* that way, because everything will come grinding to a
halt when you try to fetch that second connection from the pool.

Post by Jerry Malcolm
If indeed abandoned connections are now correctly being
returned to the pool, then I presume we are back to working the
way it did on TC7. Still not sure why it started working now.
But I guess once I get the leak fixed and if TC8 is now
configured to handle abandoned connections, I'm good. Still
would like to know about the mega-dump vs. trickle of abandoned
connections being logged.

You should be able to run in testing with an upgraded DBCP 2.
You might have to build it from trunk, though. I'm not sure if
you are okay with that, but it might help you with your testing.

The thing to swap out is Commons Pool.

Yep, thanks for the clarification.

Post by Phil Steitz
There is a release VOTE in progress now for an RC including a fix
for POOL-300.

Good news!

Post by Phil Steitz
A workaround that should work is to get a reference to the
BasicDataSource instance, say, bds and do
bds.setAbandonedLogWriter(new PrintWriter(System.out, true));
before using the pool.
Not sure if this will work correctly to get the output properly
directed under tomcat; but it is worth a try.

Definitely.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVu5FhAAoJEBzwKT+lPKRY4iYQAJyk0anccACg2gINK7RomL7G
mG/q3ZCIfOTFaY+GJAJHk2iHnN8rzA+yoJ+od6xrJyF+TZDtu4X+AdUXsJLKzYYL
oKt7SGk1hBT1mUd2TxXhdyL/+i+dw4Knz2TImgB5EPknPiHQDc4w6yppvuz7LNdc
BnADCoqbW6HO6htzqhkuRDoMC8rErJgMRVBW+U9yXdejWF1x1vIWG4zLQu0GJF75
2IlwNjO4n8jHHfBqrXw/OkSTF448pJSomXUGq1MtwFk+D5/e1n7IIuz31xnx/TIF
rXcyPMW1ukYpSUH2lHHKOKqf+LCQOp1UlJwpCpFm3QvgI6dDwmfP1WUrzi80y+33
m+E/D/WfRZ7saIN2qFtGFo7S88YGeP0eJBJ9UNeuNLFpXGgaYdv+iuxBqd4isRPN
KfwL0J8Jg4xFO7IhV3CWRpPe37C6nsK3H2iTX4IFQ/fqWXICAm3UJgCz43OwpSWE
3vt7TLDicP0+wsFWmLCFELGcJvvpmS1MKD6I8zXV5ryt7uun8j5FdIwt6gJSkpCV
Vhm33vhC59L1ZqkQWo/s1TSKgQ0EMxJsgRE97+onOE9oL63bimlbXS1QZM6inSEC
xsC0V6FVf7kp0IonTsOLCqUUXW4yNlYrFYEZvqA5GR7xMRs8NPSoINyQ+57Ehvjn
6T4ANOg/gcdtd5Bni5cH
=npUe
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Phil Steitz

2015-08-01 15:12:20 UTC

Post by Christopher Schultz
Jerry,

If you have an obsolete Listener, you probably copied your
server.xml from Tomcat 7 to Tomcat 8 which, while being less of a
disaster than with previous version-pairs, is not good practice.
Instead, start with the stock server.xml that comes with your
Tomcat version and modify it to suit. These days, you should
pretty much only have to configure the <Executor> and <Connector>
elements, unless you have a particularly exotic <Host>
configuration.

Post by Jerry Malcolm
The site is a wedding vendor advertiser site that spans two
major cities. There is no user login. Simply a very huge
online catalog. I'm certain it's deployed only once. Whether I
need that many connections is a valid question. As far as I
know, I haven't hit the limit in normal operation until now.
Could possibly reduce the count if I collect statistics.

Our user load is roughly 250 concurrently logged-in users per
Tomcat node, and we have maxTotal="20". I never get alarms about
hitting that maximum. Your requirements may be different.

Post by Jerry Malcolm
I've been monitoring the production server logs all day
watching to be sure connection pool doesn't dry up again.
About an hour ago, there was a single huge dump in stdout of
approx 2000 'logAbandoned' exceptions. They showed connections
from 1am right after my last bounce of the server thru 1:35pm.

It looks like your startup process (likely loading and caching
stuff from the db on launch) is leaky. That can run-up your
connection could quite quickly.

Post by Jerry Malcolm
The good news is with the stack trace on one of them I was able
to see the bug causing the leak.

Good.

Post by Jerry Malcolm
But why did it decide to wait over 12 hours accumulating
abandoned connections before dumping them back in the pool?

I was about to say the following, but markt says it might be a
bug in DBCP.

The bug is in Commons Pool (POOL-300). It is not flushing its
abandoned object log. That means abandoned traces won't appear
(in the default System.out configuration) until some have been
accumulated.

Thanks for the correction.

Post by Christopher Schultz
DBCP 2 looks like it only checks for abandoned connections "on
borrow" so it might only log their abandonment when you see a
flurry of connection-checkouts occurring, not when the
connections are returned to the pool. DBCP 1 would complain
pretty much immediately when the timeout was reached and the
connection hadn't been returned.

When DBCP checks for abandoned connections depends in its
removeAbandonedOnBorrow and removeAbandonedOnMaintenance.

Sorry the docs are not that clear. The problem is that the config
properties work together to determine behavior and documenting them
individually makes it hard to put the whole picture together.
Improvement patches most welcome :) In any case, here is how it works:

Connections are evaluated for abandonment when they are out in
circulation - checked out to clients. If you have set
timeBetweenEvictionRunsMillis to a positive value, pool maintenance
runs every timeBetweenEvictionRunsMillis milliseconds. If you have
removeAbandonedOnMaintenance set to true, each time maintenance runs
the pool removes abandoned connections. If you have
removeAbandonedOnBorrow set to true, the pool removes abandoned
connections if it is nearing depletion when a borrow request
arrives. In both cases, the pool looks at the statistics that it
maintains on the list of all objects checked out by clients to
determine which ones appear to be abandoned. To appear abandoned
means to be checked out by a client but not used for longer than the
removeAbandonedTimeout. For DBCP, "used" means the connection has
been used.

Phil

Post by Jerry Malcolm
I realize from now knowing the code bug that the leak is a
slow drip that is continually leaking on a regular basis. But
since that last 12-hour accumulated dump, the abandoning has
returned to silence. Since leaks are occurring regularly and
would be timing out regularly, shouldn't I see a similar 'slow
drip' of logAbandoned entries in stdout instead of a big dump
every 12 hours?
It's going to take a day or two to fix the leak, test, and
deploy.

For testing, set maxTotal="1". You'll find your leaks *very
quickly* that way, because everything will come grinding to a
halt when you try to fetch that second connection from the pool.

Post by Jerry Malcolm
If indeed abandoned connections are now correctly being
returned to the pool, then I presume we are back to working the
way it did on TC7. Still not sure why it started working now.
But I guess once I get the leak fixed and if TC8 is now
configured to handle abandoned connections, I'm good. Still
would like to know about the mega-dump vs. trickle of abandoned
connections being logged.

You should be able to run in testing with an upgraded DBCP 2.
You might have to build it from trunk, though. I'm not sure if
you are okay with that, but it might help you with your testing.

The thing to swap out is Commons Pool.

Yep, thanks for the clarification.

Post by Phil Steitz
There is a release VOTE in progress now for an RC including a fix
for POOL-300.

Good news!

Definitely.
-chris
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Christopher Schultz

2015-08-02 01:14:17 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Phil,

Post by Phil Steitz
Sorry the docs are not that clear. The problem is that the config
properties work together to determine behavior and documenting
them individually makes it hard to put the whole picture together.
Connections are evaluated for abandonment when they are out in
circulation - checked out to clients. If you have set
timeBetweenEvictionRunsMillis to a positive value, pool
maintenance runs every timeBetweenEvictionRunsMillis milliseconds.
If you have removeAbandonedOnMaintenance set to true, each time
maintenance runs the pool removes abandoned connections. If you
have removeAbandonedOnBorrow set to true, the pool removes
abandoned connections if it is nearing depletion when a borrow
request arrives. In both cases, the pool looks at the statistics
that it maintains on the list of all objects checked out by clients
to determine which ones appear to be abandoned. To appear
abandoned means to be checked out by a client but not used for
longer than the removeAbandonedTimeout. For DBCP, "used" means the
connection has been used.

So when during all that does the abandoned connection get "logged"?

If I have a pool with two connections in it, and the first one gets
leaked (after, lets say, 10 seconds) but the second one is returned
properly, and the application continues to borrow and return the
second connection, what happens?

Are you only checking connections for "abandonment" after they are
returned and then when a client attempts to borrow that specific
connection? Or, are all currently-borrowed connections checked for
abandonment whenever any borrow operation is requested?

Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVvW7pAAoJEBzwKT+lPKRYY0UQAIBXTrkBRNC9j7KBYw8DZFcJ
UY+U1WrHi7osxEseupCjSjP/xmmOHoC16LkEXqtil5k8rd5yQ9D5I1imPKYZSjiE
GBBkkjrbvlwDq/sRp3fxN8La6ueTed4yyh96J5DFfx0opcTtX06YMnpkJDTN2iTt
R3GUKbBo4dNZKIFNGhicR4j9JbSwerxbfJDVpEtb0igO7fQu1dRTDQh7gijyUsZ7
PPa0owq9WOBl5dr4ebsoipirArbLbB4WZqCtteOj5QpafEOYyGuKAjFs6PhodVAR
A2BlikA9+s6AUcKVnqP+CabZFNYO/1raZWl61q7vSWNR9YMrSSI6TapCsBgzNDQW
p8KloJYqjaZFNGtVW23eDtRjTehABnQqoXrB4OL2CEFqgZVjnfpCDl1K0b8wc01y
QNnA0eo40462XkwsRSyMOwtunLu02mGayOrzrn7blJ0f2Kj594+/U6HKqAttRlOX
0Sr2VnYJTaBRACHVjycKjjQh1A1lIIfRoIQ0enMBiw8kZ4rVpQsCvwk/5h54/oap
5osEzulFO7zwqhAxWaHigFG9hCeHJHGr7eT4XUXfUqw+/zg6vI//r5rLExIwSi4D
p/R214PYSkEMiz+i4uSu+sUOYpOJbaAnJH0R6kXSKkoDmaaylTyLuHujPvLKPF6b
ZV739LJI+k5cgcDEWmjv
=PKZ/
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Phil Steitz

2015-08-02 01:34:19 UTC

So when during all that does the abandoned connection get "logged"?

The log message is written when the abandoned connection is closed
by the pool. (The pool bug was basically delaying the effective
write; but the stack trace goes to the buffer when the close happens)

Post by Christopher Schultz
If I have a pool with two connections in it, and the first one gets
leaked (after, lets say, 10 seconds) but the second one is returned
properly, and the application continues to borrow and return the
second connection, what happens?

That depends on the settings. See below.

Post by Christopher Schultz
Are you only checking connections for "abandonment" after they are
returned and then when a client attempts to borrow that specific
connection? Or, are all currently-borrowed connections checked for
abandonment whenever any borrow operation is requested?

It is the latter, assuming removeAbandonedOnBorrow is true. Removal
on borrow also requires that the pool be near depletion when the
borrow happens (see the javadoc for precise description of
conditions). There are two things that can trigger abandoned object
removal - a borrow request and pool maintenance. In each case, if
the config setting to enable the check on that event is on, all
checked out objects are examined. The ones that appear abandoned
are destroyed (in DBCP that means physical connections are closed).

Phil

Post by Christopher Schultz
Thanks,
-chris
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Christopher Schultz

2015-08-04 21:03:27 UTC

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Phil,

Post by Phil Steitz
Sorry the docs are not that clear. The problem is that the
config properties work together to determine behavior and
documenting them individually makes it hard to put the whole
picture together. Improvement patches most welcome :) In any
Connections are evaluated for abandonment when they are out in
circulation - checked out to clients. If you have set
timeBetweenEvictionRunsMillis to a positive value, pool
maintenance runs every timeBetweenEvictionRunsMillis
milliseconds. If you have removeAbandonedOnMaintenance set to
true, each time maintenance runs the pool removes abandoned
connections. If you have removeAbandonedOnBorrow set to true,
the pool removes abandoned connections if it is nearing
depletion when a borrow request arrives. In both cases, the
pool looks at the statistics that it maintains on the list of
all objects checked out by clients to determine which ones
appear to be abandoned. To appear abandoned means to be
checked out by a client but not used for longer than the
removeAbandonedTimeout. For DBCP, "used" means the connection
has been used.

So when during all that does the abandoned connection get
"logged"?

The log message is written when the abandoned connection is closed
by the pool.

So if the client code never calls close(), does that mean the
amandonment is never logged? IIRC, DBCP 1.x would log the stack trace
as soon as removeAbandonedTimeout elapsed if the connection hadn't
been returned.

Post by Phil Steitz
(The pool bug was basically delaying the effective write; but the
stack trace goes to the buffer when the close happens)

Post by Christopher Schultz
If I have a pool with two connections in it, and the first one
gets leaked (after, lets say, 10 seconds) but the second one is
returned properly, and the application continues to borrow and
return the second connection, what happens?

That depends on the settings. See below.

Post by Christopher Schultz
Are you only checking connections for "abandonment" after they
are returned and then when a client attempts to borrow that
specific connection? Or, are all currently-borrowed connections
checked for abandonment whenever any borrow operation is
requested?

It is the latter, assuming removeAbandonedOnBorrow is true.
Removal on borrow also requires that the pool be near depletion
when the borrow happens (see the javadoc for precise description
of conditions). There are two things that can trigger abandoned
object removal - a borrow request and pool maintenance. In each
case, if the config setting to enable the check on that event is
on, all checked out objects are examined. The ones that appear
abandoned are destroyed (in DBCP that means physical connections
are closed).

Why is there no option to dump a warning as soon as the timeout
elapses? I suppose if the pool is being used slightly more than
/never/, the abandoned notice will happen pretty quickly, but for
testing purposes, it means that a single-user will have to perform
some operation, wait, and then perform another operation in order to
get the feedback about the abandonment. That's surprising behavior if
you ask me.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVwSifAAoJEBzwKT+lPKRYoaAP/jLgMP3BWm3j5OTvAFw55Sbm
GyDQel0X0LxlXC1Nn/YETjTvUGh3d1sv0FE0zyuNFfrmEGi35hX6vEIFA+aAx5wb
1LjUR4gUTbMR/4mYGbVyHb65Pt+7CY/oMuGbp22o+2DwXEy/+1sS7lHAXK/xKkbb
F+HcLN7bU2khLp/1JuSp7NfmSEpBKkpuitkTUyGR27ijorYRRqpLfM80/aJV8aTh
MSMhgFuJ6igkJv29oTMqTXY3k+xIfZeIpJuYQ0yFLFwKmZuxPgSP+9hSfOyQAbDs
fev2bcwE9yYmypWOJrPBFR+T7xt61BSLzi1LwUixuTDP6eZEOYrYDvXd0lrB+Ohv
jS5Md182lWiAwbF+8grMwALDTUCE4cCfq/BqO1gSWNaZSDJoFhmm0QmPPgB2iVU+
ZEX9rkwCfY0SKal2x7uJzMwCJYgtPlXq88Fr2aX+Wvg90nXKqwVucAMBf+zOyaP3
tEIuw4H20ug6OPgbfUmdKS1qYLyjS5V7XzP/LoZCNjzDK+pqrmdkRneNdSiIlTzv
2HIQKtIYFZXHEMOmnBGiMWrGfF8lUrAk/rpmFerXk2RI/e6H9ufAExSIz7xhKZEG
AToWrGhb7DOCBOAsABoGnU4R9qE27FuWBcceKdDmSRZzqThmT+9o2tLjXEGJfcc1
Bmt9TxMpWJYKboZkaj/w
=eTpv
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Phil Steitz

2015-08-04 22:38:21 UTC

Post by Phil Steitz
Sorry the docs are not that clear. The problem is that the
config properties work together to determine behavior and
documenting them individually makes it hard to put the whole
picture together. Improvement patches most welcome :) In any
Connections are evaluated for abandonment when they are out in
circulation - checked out to clients. If you have set
timeBetweenEvictionRunsMillis to a positive value, pool
maintenance runs every timeBetweenEvictionRunsMillis
milliseconds. If you have removeAbandonedOnMaintenance set to
true, each time maintenance runs the pool removes abandoned
connections. If you have removeAbandonedOnBorrow set to true,
the pool removes abandoned connections if it is nearing
depletion when a borrow request arrives. In both cases, the
pool looks at the statistics that it maintains on the list of
all objects checked out by clients to determine which ones
appear to be abandoned. To appear abandoned means to be
checked out by a client but not used for longer than the
removeAbandonedTimeout. For DBCP, "used" means the connection
has been used.

So when during all that does the abandoned connection get
"logged"?

The log message is written when the abandoned connection is closed
by the pool.

The behavior is the same in 2.x as 1.x with the exception that in
2.x, pool maintenance can also trigger abandoned object removal. In
1.x, it is necessary to attempt a borrow to get abandoned
connections to be removed. Both 1.x and 2.x essentially remove
abandoned connections in batches, closing the connections and
logging the traces when the batches run. In 1.x, the only way to
trigger an abandoned cleanup batch is to request a connection when
the pool is near depletion. In 2.x, if removeAbandonedOnMaintenance
is true, an abandoned cleanup batch will run every
timeBetweenEvictionRunsMillis ms.