Tomcat running out of threads.

[Philip Wigg]

Hi,

I have several Tomcat web applications that run in separate JVMs. I
have one Apache instance that connects to the Tomcat instances via
mod_jk.

I am using:-

RHEL ES 4.0
Java 1.5.0_14
mod_jk-1.2.25-httpd-2.0.59.so
Tomcat 6.0.14.

Usually the sites work fine and are very responsive. The problem is
that I am intermittently seeing the following error in catalina.out:-

SEVERE: All threads (200) are currently busy, waiting. Increase
maxThreads (200) or check the servlet status

Tomcat never recovers from this needs to be restarted. The error does
seem to occur more often, but not exclusively, during periods of heavy
traffic. The thing is that I can see that even during very busy times
the number of busy threads does not normally exceed about 50. It seems
that sometimes the number of busy threads spikes very rapidly and the
system gets into some kind of state whereby threads are not released
for some reason and the system just runs out within about 10 minutes.
I tried increasing the maxThreads to 500, but it still ran out when
this problem occurred.

I am a sysadmin but the developers say that they don't think it is
related to code and that the three applications are different in the
way that they're written.

I have taken a thread dump after this problem occurs, and the threads
are all in the following state:-

"TP-Processor191" daemon prio=1 tid=0x0000002b698b8000 nid=0x3a39
runnable [0x000000004d727000..0x000000004d727bb0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
- locked <0x0000002adf95e878> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:558)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:595)

I have a very simple workers.properties file which looks like so:-

worker.list=rt-worker,fs-worker,at-worker,tg-worker,gw-worker,gf-worker,pp-worker,bm-worker

worker.rt-worker.type=ajp13
worker.rt-worker.host=localhost
worker.rt-worker.port=8300

worker.fs-worker.type=ajp13
worker.fs-worker.host=localhost
worker.fs-worker.port=8301

[...] (other workers defined as above)

I was wondering if anyone had experienced similar issues or if anyone
could provide some guidance on how to investigate the cause. We've
tried to replicate the issue in our load-testing environment but we've
had no luck so far. Please let me know if there's any other
information you'd like me to provide.

Kind regards,
Philip Wigg.

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Rainer Jung]

Post by Philip Wigg
Hi,
I have several Tomcat web applications that run in separate JVMs. I
have one Apache instance that connects to the Tomcat instances via
mod_jk.
I am using:-
RHEL ES 4.0
Java 1.5.0_14
mod_jk-1.2.25-httpd-2.0.59.so
Tomcat 6.0.14.
Usually the sites work fine and are very responsive. The problem is
that I am intermittently seeing the following error in catalina.out:-
SEVERE: All threads (200) are currently busy, waiting. Increase
maxThreads (200) or check the servlet status
Tomcat never recovers from this needs to be restarted. The error does
seem to occur more often, but not exclusively, during periods of heavy
traffic. The thing is that I can see that even during very busy times
the number of busy threads does not normally exceed about 50. It seems
that sometimes the number of busy threads spikes very rapidly and the
system gets into some kind of state whereby threads are not released
for some reason and the system just runs out within about 10 minutes.
I tried increasing the maxThreads to 500, but it still ran out when
this problem occurred.
I am a sysadmin but the developers say that they don't think it is
related to code and that the three applications are different in the
way that they're written.
I have taken a thread dump after this problem occurs, and the threads
are all in the following state:-
"TP-Processor191" daemon prio=1 tid=0x0000002b698b8000 nid=0x3a39
runnable [0x000000004d727000..0x000000004d727bb0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
- locked <0x0000002adf95e878> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:558)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:595)

There should be at least one TP-Processor thread doing an accept on a
socket.

How many threads with the above stack do you have? This stack usually
means, that the threads are connected to httpd and are waiting for the
next request.

Post by Philip Wigg
I have a very simple workers.properties file which looks like so:-
worker.list=rt-worker,fs-worker,at-worker,tg-worker,gw-worker,gf-worker,pp-worker,bm-worker
worker.rt-worker.type=ajp13
worker.rt-worker.host=localhost
worker.rt-worker.port=8300
worker.fs-worker.type=ajp13
worker.fs-worker.host=localhost
worker.fs-worker.port=8301

You should at least have a look at the timeouts docs page:

http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Philip Wigg]

Thanks for your help.

Post by Rainer Jung
There should be at least one TP-Processor thread doing an accept on a
socket.

There is this one?

"TP-Processor4" daemon prio=1 tid=0x0000002b69b0e980 nid=0x655f
runnable [0x0000000041768000..0x0000000041768ab0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x0000002aa1914040> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:306)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:660)
at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:595)

Post by Rainer Jung
How many threads with the above stack do you have? This stack usually
means, that the threads are connected to httpd and are waiting for the next
request.

The rest seem to be in either of the two following states:-

"TP-Processor107" daemon prio=1 tid=0x0000002b699b2070 nid=0x3969
runnable [0x00000000482d3000..0x00000000482d3db0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
- locked <0x0000002adf8760b0> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:620)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:558)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:685)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:595)

"TP-Processor103" daemon prio=1 tid=0x0000002b69c0ad10 nid=0x3965 in
Object.wait() [0x0000000047ecf000..0x0000000047ecfbb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x0000002ad0716cc8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
- locked <0x0000002ad0716cc8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)

Post by Rainer Jung
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html

I have spent a while reading this document I wasn't sure if it would
help because the thread seems to be locked at the backend, ie. Tomcat
and not with mod_jk.

I have taken a grab of the Tomcat manager status page after this
problem has occured, one thing I have noticed is that all the
connections seem to be in 'keepalive' state as so:-

jk-8300 - Max threads: 200 Current thread count: 200 Current thread
busy: 199 Max processing time: 1667835 ms Processing time: 20958.227 s
Request count: 131357 Error count: 586 Bytes received: 0.00 MB Bytes
sent: 11208.42 MB

Stage Time B Sent B Recv Client VHost Request
K 5236624 ms ? ? 86.131.34.244 ? ?
K 4534173 ms ? ? 86.144.67.135 ? ?
K 4438805 ms ? ? 195.92.35.134 ? ?
K 4118296 ms ? ? 81.133.90.42 ? ?
K 3631710 ms ? ? 90.195.77.89 ? ?
K 4192041 ms ? ? 82.17.167.106 ? ?
K 3632369 ms ? ? 81.152.26.38 ? ?
K 3867751 ms ? ? 91.84.104.189 ? ?
K 3733813 ms ? ? 86.13.237.138 ? ?
K 3932247 ms ? ? 88.212.143.70 ? ?
K 3811565 ms ? ? 82.2.55.100 ? ?
K 3892483 ms ? ? 91.125.225.165 ? ?
K 4190193 ms ? ? 78.149.64.232 ? ?

Hope that helps.

Kind regards,
Phil.

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Bernardo Cabezas]

Hi Philip

Post by Philip Wigg
jk-8300 - Max threads: 200 Current thread count: 200 Current thread
busy: 199 Max processing time: 1667835 ms Processing time: 20958.227 s
Request count: 131357 Error count: 586 Bytes received: 0.00 MB Bytes
sent: 11208.42 MB
Stage Time B Sent B Recv Client VHost Request
K 5236624 ms ? ? 86.131.34.244 ? ?
K 4534173 ms ? ? 86.144.67.135 ? ?
K 4438805 ms ? ? 195.92.35.134 ? ?
K 4118296 ms ? ? 81.133.90.42 ? ?
K 3631710 ms ? ? 90.195.77.89 ? ?
K 4192041 ms ? ? 82.17.167.106 ? ?
[...]

I'm having exactly the same issue than you.
Our software stack is very similar:

RHEL4
Java 1.5.0_14
mod_jk-1.2.20
tomcat5-5.5.23

Also, it happends only under heavy load scenarios, and not so often (a
couple of times a month).

It seems that at tomcat side, all threads are on keepalive status, while at
mod_jk side, threads are on Busy status. So all stays this way forever, and
no new requests are submited to tomcat.

I've seen that AJP13 connections only can do keepalive when running with APR
(apache portable runtime).
Here are described the APR connector extensions:
http://tomcat.apache.org/tomcat-5.5-doc/apr.html
Currenly I'm doing tests with paramaters: firstReadTimeout,pollTime and
pollerSize to control keepalive status.

Philip, did you find a solution?
¿any one knows why with APR tomcat thread status is keepalive, socket
connection is STABLISHED with mod_jk, but mod_jk believes thread is Busy?

Thanks so much,
Bernardo

[Hari Prasad]

hi.. i too have the same issue..
did anyone find solution?

Post by Bernardo Cabezas
Hi Philip

Post by Philip Wigg
jk-8300 - Max threads: 200 Current thread count: 200 Current thread
busy: 199 Max processing time: 1667835 ms Processing time: 20958.227 s
Request count: 131357 Error count: 586 Bytes received: 0.00 MB Bytes
sent: 11208.42 MB
Stage Time B Sent B Recv Client VHost Request
K 5236624 ms ? ? 86.131.34.244 ? ?
K 4534173 ms ? ? 86.144.67.135 ? ?
K 4438805 ms ? ? 195.92.35.134 ? ?
K 4118296 ms ? ? 81.133.90.42 ? ?
K 3631710 ms ? ? 90.195.77.89 ? ?
K 4192041 ms ? ? 82.17.167.106 ? ?
[...]

I'm having exactly the same issue than you.
RHEL4
Java 1.5.0_14
mod_jk-1.2.20
tomcat5-5.5.23
Also, it happends only under heavy load scenarios, and not so often (a
couple of times a month).
It seems that at tomcat side, all threads are on keepalive status, while
at mod_jk side, threads are on Busy status. So all stays this way forever,
and no new requests are submited to tomcat.
I've seen that AJP13 connections only can do keepalive when running with
APR (apache portable runtime).
http://tomcat.apache.org/tomcat-5.5-doc/apr.html
Currenly I'm doing tests with paramaters: firstReadTimeout,pollTime and
pollerSize to control keepalive status.
Philip, did you find a solution?
¿any one knows why with APR tomcat thread status is keepalive, socket
connection is STABLISHED with mod_jk, but mod_jk believes thread is Busy?
Thanks so much,
Bernardo

[Mark Thomas]

Post by Hari Prasad
hi.. i too have the same issue..
did anyone find solution?

1. JkOptions +DisableReuse
2. Use correct timeouts and CPing/CPong

I'd go with option 1. YMMV.

Mark

[Bernardo Cabezas]

Hello,

According to tomcat documentation, JkOptions +DisableReuse has some
performance penalty, because opens new connection for each request.

In my case, issue was solved by setting the param:
connectionTimeout
on the ajp13 <connector /> entry on tomcat's server.xml

Also look at docs for firstReadTimeout parameter.

Now mod_jk is using a pool of connectors, but tomcat expires old threads on
the pool so issue was solved.

Regards,
Bernardo.

Post by Mark Thomas

Post by Hari Prasad
hi.. i too have the same issue..
did anyone find solution?

1. JkOptions +DisableReuse
2. Use correct timeouts and CPing/CPong
I'd go with option 1. YMMV.
Mark
---------------------------------------------------------------------

[Mark Thomas]

Post by Bernardo Cabezas
Hello,
According to tomcat documentation, JkOptions +DisableReuse has some
performance penalty, because opens new connection for each request.

There is actually quite a lot of disagreement between the Tomcat committers
about the correctness of that statement in the docs. I am aware of fairly
large deployments that actaully saw a significant performance increase by
enabling that option. That said, it may well be that a similar performance
increase could have been obtained by correctly tuning the timeout
parameters. It is certainly the case the YMMV and you need to test in your
environment.

Mark

[Hubert de Heer]

Hello,

We run a setup with 3 Apache servers with mod_jk (2.2.6 / 1.2.26) and 10
tomcat instances running on 5 servers (tomcat 5.5.26 / RHEL4).

We used the combination of connectionTimeout & connection_pool_timeout,
both set to 5min (connectionTimeout="300000",
connection_pool_timeout=300). But we still saw a lot of KeepAlive
connections on the tomcat side staying open (average thread count 400
per tomcat instance).

worker.properties:
worker.server1.type=ajp13
worker.server1.host=xxx.xxx.xxx.xxx
worker.server1.port=8009
worker.server1.socket_keepalive=1
worker.server1.socket_timeout=15
worker.server1.connect_timeout=10000
worker.server1.prepost_timeout=8000
worker.server1.lbfactor=1
worker.server1.connection_pool_timeout=300

server.xml:
<Connector port="8009" maxThreads="750" minSpareThreads="25"
maxSpareThreads="75" connectionTimeout="300000"

This week we started to use JkOptions +DisableReuse and our current
thread busy count dropped to an average of 8.
So we will stick with the DisableReuse and find another use for 2 out of
5 tomcat servers.

Regards,

Hubert de Heer
-----Original Message-----
From: Bernardo Cabezas [mailto:***@bergantells.net]
Sent: 12 January 2009 13:01
To: ***@tomcat.apache.org
Subject: RE: Tomcat running out of threads.


Hello,

According to tomcat documentation, JkOptions +DisableReuse has some
performance penalty, because opens new connection for each request.

In my case, issue was solved by setting the param:
connectionTimeout
on the ajp13 <connector /> entry on tomcat's server.xml

Also look at docs for firstReadTimeout parameter.

Now mod_jk is using a pool of connectors, but tomcat expires old threads
on
the pool so issue was solved.

Regards,
Bernardo.

Post by Mark Thomas

Post by Hari Prasad
hi.. i too have the same issue..
did anyone find solution?

1. JkOptions +DisableReuse
2. Use correct timeouts and CPing/CPong
I'd go with option 1. YMMV.
Mark
---------------------------------------------------------------------