Tomcat and IIS losing their link

[Travis Haagen]

Hello,

I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and during high traffic periods, the connection between IIS and Tomcat is being severed. This has been occurring in the mid-afternoon every day since this server went live and to get it running again, I've had to restart Tomcat. The peak connections to IIS during one of the days was just under 100, so this isn't really all that high of traffic.

The isapi_redirect.log isn't much help. It has only been logging errors after the connection to Tomcat has been broken. Tomcat itself hasn't logged anything of relevance and I don't actually think that Tomcat is affected when the link is broken, but as I mentioned earlier, I have to restart Tomcat to get it to reconnect with IIS.

Most of my configuration settings came from this website:

http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration

Here are the relevant workers.properties settings:

worker.ajp13w.type=ajp13
worker.ajp13w.host=localhost
worker.ajp13w.port=8009
worker.ajp13w.lbfactor=1
worker.ajp13w.connect_timeout=10000
worker.ajp13w.prepost_timeout=10000
worker.ajp13w.socket_timeout=10
worker.ajp13w.connection_pool_timeout=600
worker.ajp13w.connection_pool_size=350

Here are the relevant server.xml settings:

<Connector port="8443" maxHttpHeaderSize="8192"
maxThreads="400" minSpareThreads="50" maxSpareThreads="150"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="https" secure="true"
SSLEngine="on" SSLPassword="removed"
SSLCertificateFile="removed"
SSLCertificateKeyFile="removed" />

<Connector port="8009" redirectPort="8444" protocol="AJP/1.3"
connectionTimeout="600000" enableLookups="false"
maxThreads="400" minSpareThreads="50" maxSpareThreads="150" />

I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's the end of my ideas for fixing this issue. The only other ISAPI filters that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL rewriting). IIS web site connections is set to 'unlimited', connection timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy thing stops working, everything is fine.

Any help will be greatly appreciated.

Thanks,

Travis

[Travis Haagen]

Thanks for the reply Per...

Tomcat and IIS are running on the same machine, with no external load
balancer or firewall between IIS and Tomcat.

When the connection is lost, IIS displays either "service is unavailable" or
a "503 error" message.

I only have maxThreads=400, because I didn't think it would hurt anything. I
was running with far fewer maxThreads and had the same problem. The machine
is a quad-core CPU with 4 gigs of ram and the load is very light.

I'll look into your other suggestions. Also, I noticed in my original post
that the redirectPort was wrong and I fixed that :)

Any other ideas out there?

-Travis


----- Original Message -----
From: "Per Johnsson" <***@omxgroup.com>
To: <***@comcast.net>
Sent: Thursday, December 13, 2007 11:52 PM
Subject: FW: Tomcat and IIS losing their link


(Tomcat maillist bounced my mail so I send it to you directly./per)


Hi!

Some questions:
- Is the tomcat and IIS running on the same machine?
- What's the result when the connection is lost? Service unavaible from
IIS?
- Is there any loadbalancer, firewall between the IIS and tomcat?
- You have maxThreads=400, are there 400 concurrant requests ever during
high load?

We had some similar issues and we didn't find any good solution for it
but a lot of small adjustments made it better like:
- Lower the maxthreads (we had very cpu consuming requests at some
times) we went from like 300 to 20 I beleive.
- Installing java 1.6 which is much more efficient.
- Using the latest APR and Isapi dll.


/Per Jonsson

-----Original Message-----
From: Travis Haagen [mailto:***@comcast.net]
Sent: den 14 december 2007 05:08
To: Tomcat Posts
Subject: Tomcat and IIS losing their link

Hello,

I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and
AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server, and
during high traffic periods, the connection between IIS and Tomcat is
being severed. This has been occurring in the mid-afternoon every day
since this server went live and to get it running again, I've had to
restart Tomcat. The peak connections to IIS during one of the days was
just under 100, so this isn't really all that high of traffic.

The isapi_redirect.log isn't much help. It has only been logging errors
after the connection to Tomcat has been broken. Tomcat itself hasn't
logged anything of relevance and I don't actually think that Tomcat is
affected when the link is broken, but as I mentioned earlier, I have to
restart Tomcat to get it to reconnect with IIS.

Most of my configuration settings came from this website:


http://wiki.jboss.org/wiki/Wiki.jsp?page=OptimalMod_jk1.2Configuration

Here are the relevant workers.properties settings:

worker.ajp13w.type=ajp13
worker.ajp13w.host=localhost
worker.ajp13w.port=8009
worker.ajp13w.lbfactor=1
worker.ajp13w.connect_timeout=10000
worker.ajp13w.prepost_timeout=10000
worker.ajp13w.socket_timeout=10
worker.ajp13w.connection_pool_timeout=600
worker.ajp13w.connection_pool_size=350

Here are the relevant server.xml settings:

<Connector port="8443" maxHttpHeaderSize="8192"
maxThreads="400" minSpareThreads="50" maxSpareThreads="150"
enableLookups="false" disableUploadTimeout="true"
acceptCount="100" scheme="https" secure="true"
SSLEngine="on" SSLPassword="removed"
SSLCertificateFile="removed"
SSLCertificateKeyFile="removed" />

<Connector port="8009" redirectPort="8444" protocol="AJP/1.3"
connectionTimeout="600000" enableLookups="false"
maxThreads="400" minSpareThreads="50" maxSpareThreads="150" />

I just configured IIS 6.0 to run in IIS 5.0 isolation mode and that's
the end of my ideas for fixing this issue. The only other ISAPI filters
that I'm running are ASP.NET_1.1.4322.2307 and ISAPI_Rewrite3 (for URL
rewriting). IIS web site connections is set to 'unlimited', connection
timeout is 120 seconds, HTTP Keep-Alives is enabled. Until the crazy
thing stops working, everything is fine.

Any help will be greatly appreciated.

Thanks,

Travis


---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Rainer Jung]

Post by Travis Haagen
Hello,
I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and
AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server,
and during high traffic periods, the connection between IIS and
Tomcat is being severed. This has been occurring in the mid-afternoon
every day since this server went live and to get it running again,
I've had to restart Tomcat. The peak connections to IIS during one of
the days was just under 100, so this isn't really all that high of
traffic.
The isapi_redirect.log isn't much help. It has only been logging
errors after the connection to Tomcat has been broken. Tomcat itself
hasn't logged anything of relevance and I don't actually think that
Tomcat is affected when the link is broken, but as I mentioned
earlier, I have to restart Tomcat to get it to reconnect with IIS.

Nevertheless at least i would be curious to know, what the exact
contents of the isapi redirector log files are. We don't need all of it,
but the first 100 lines or so could be interesting. What's you log level?

Have a look at the other thread running on this list the last few days
("ISAPI JK2 ran better than JK, how can that be?"). To get a slightly
better idea what's happening when, you could also introduce a load
balancer and a status worker.

Most likely it would help to take Java Thread Dumps and look inside the
JVM, what your request threads are doing, when IIS gets stuck.

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Martin Gainty]

Travis
take a look at this excerpt from Tomcat committer Mladen Turk..
"Just like Apache Web server for Windows, Microsoft IIS maintains a separate
child process and thread pool for serving concurrent client connections. For
non server products like Windows 2000 Professional or Windows XP the number
of concurrent connections is limited to 10. This mean that you can not use
workstation products for production servers unless the 10 connections limit
will fulfil your needs. The server range of products does not impose that 10
connection limit, but just like Apache, the 2000 connections is a limit when
the thread context switching will take its share and slow down the effective
number of concurrent connections. If you need higher load you will need to
deploy additional web servers and use Windows Network Load Balancer (WNLB)
in front of Tomcat servers. "

http://people.apache.org/~mturk/docs/article/ftwai.html
this means that Windows 2000 Pro or Windows XP has an artificial limit of
10..2 solutions
another (more capable) OS
existing OS with WindowsNetworkLoadBalancer

Nota Bene:
Any diagnosis is incomplete without looking at Java Thread Dumps as Rainier
suggested earlier..

Martin-
----- Original Message -----
From: "Rainer Jung" <***@kippdata.de>
To: "Tomcat Users List" <***@tomcat.apache.org>
Sent: Friday, December 14, 2007 12:20 PM
Subject: Re: Tomcat and IIS losing their link

Post by Rainer Jung

Post by Travis Haagen
Hello,
I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and
AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server,
and during high traffic periods, the connection between IIS and
Tomcat is being severed. This has been occurring in the mid-afternoon
every day since this server went live and to get it running again,
I've had to restart Tomcat. The peak connections to IIS during one of
the days was just under 100, so this isn't really all that high of
traffic.
The isapi_redirect.log isn't much help. It has only been logging
errors after the connection to Tomcat has been broken. Tomcat itself
hasn't logged anything of relevance and I don't actually think that
Tomcat is affected when the link is broken, but as I mentioned
earlier, I have to restart Tomcat to get it to reconnect with IIS.

Nevertheless at least i would be curious to know, what the exact
contents of the isapi redirector log files are. We don't need all of it,
but the first 100 lines or so could be interesting. What's you log level?
Have a look at the other thread running on this list the last few days
("ISAPI JK2 ran better than JK, how can that be?"). To get a slightly
better idea what's happening when, you could also introduce a load
balancer and a status worker.
Most likely it would help to take Java Thread Dumps and look inside the
JVM, what your request threads are doing, when IIS gets stuck.
Regards,
Rainer
---------------------------------------------------------------------

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Travis Haagen]

Thanks for the responses thus far. I've been trying to get more information
about the problem, but haven't been very successfull. I can't seem to create
thread dumps, because I have to use Terminal Services. Is there a way to get
a thread dump even though I'm using Terminal Services?

Tomcat keeps locking up every afternoon, during the work week, so its
definitely related to server traffic. Sometimes it locks up twice in an
afternoon.

I found this article about IIS 6.0 application health monitoring and I was
wondering if anyone know if the items listed in the section 'ISAPI
Extensions Can Declare Themselves Unhealthy' are implemented in the ISAPI
redirector?

http://help.alentus.com/article.aspx?id=10958&cNode=7F8F8Y

Here's what the ISAPI redirector logs say (the following errors are simply
repeated until Tomcat is restarted). My log level is currently set to
'WARN':

Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (966):
(ajp13w) can't receive the response message from tomcat, network problems or
tomcat (127.0.0.1:8009) is down (errno=60)
[Mon Dec 17 13:48:34.484 2007] [4340:5744] [error] jk_ajp_common.c (1658):
(ajp13w) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (966):
(ajp13w) can't receive the response message from tomcat, network problems or
tomcat (127.0.0.1:8009) is down (errno=60)
[Mon Dec 17 13:48:38.390 2007] [4340:3340] [error] jk_ajp_common.c (1658):
(ajp13w) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (966):
(ajp13w) can't receive the response message from tomcat, network problems or
tomcat (127.0.0.1:8009) is down (errno=60)
[Mon Dec 17 13:48:47.015 2007] [4340:5744] [error] jk_ajp_common.c (1658):
(ajp13w) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Mon Dec 17 13:48:49.015 2007] [4340:5744] [error] jk_ajp_common.c (2097):
(ajp13w) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port







----- Original Message -----
From: "Rainer Jung" <***@kippdata.de>
To: "Tomcat Users List" <***@tomcat.apache.org>
Sent: Friday, December 14, 2007 9:20 AM
Subject: Re: Tomcat and IIS losing their link

Post by Rainer Jung

Post by Travis Haagen
Hello,
I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and
AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server,
and during high traffic periods, the connection between IIS and
Tomcat is being severed. This has been occurring in the mid-afternoon
every day since this server went live and to get it running again,
I've had to restart Tomcat. The peak connections to IIS during one of
the days was just under 100, so this isn't really all that high of
traffic.
The isapi_redirect.log isn't much help. It has only been logging
errors after the connection to Tomcat has been broken. Tomcat itself
hasn't logged anything of relevance and I don't actually think that
Tomcat is affected when the link is broken, but as I mentioned
earlier, I have to restart Tomcat to get it to reconnect with IIS.

Nevertheless at least i would be curious to know, what the exact
contents of the isapi redirector log files are. We don't need all of it,
but the first 100 lines or so could be interesting. What's you log level?
Have a look at the other thread running on this list the last few days
("ISAPI JK2 ran better than JK, how can that be?"). To get a slightly
better idea what's happening when, you could also introduce a load
balancer and a status worker.
Most likely it would help to take Java Thread Dumps and look inside the
JVM, what your request threads are doing, when IIS gets stuck.
Regards,
Rainer
---------------------------------------------------------------------

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Rainer Jung]

Post by Travis Haagen
Thanks for the responses thus far. I've been trying to get more information
about the problem, but haven't been very successfull. I can't seem to create
thread dumps, because I have to use Terminal Services. Is there a way to get
a thread dump even though I'm using Terminal Services?

There are a couple of not so nice ways of doing that. One example:

- Install a JDK not only a JRE
- enable JMX for your java process (adding the following flags to your
service definition):

-Dcom.sun.management.jmxremote.port=1090
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

(Note: no authentication here)

- search for the jar file FullThreadDump.jar, something like

<JDK_HOME>\demo\management\FullThreadDump\FullThreadDump.jar

- run

java -cp PathToMyJar\FullThreadDump.jar FullThreadDump localhost:1090 >>
C:\temp\ThreadDumps.log

Post by Travis Haagen
Tomcat keeps locking up every afternoon, during the work week, so its
definitely related to server traffic. Sometimes it locks up twice in an
afternoon.
I found this article about IIS 6.0 application health monitoring and I was
wondering if anyone know if the items listed in the section 'ISAPI
Extensions Can Declare Themselves Unhealthy' are implemented in the ISAPI
redirector?
http://help.alentus.com/article.aspx?id=10958&cNode=7F8F8Y

Interesting, but not used at the moment in the isapi redirector.

Post by Travis Haagen
Here's what the ISAPI redirector logs say (the following errors are simply
repeated until Tomcat is restarted). My log level is currently set to

Those unfortunately don't give us more than what we already expect: that
the problem lies inside your Tomcat instance. Try the above recipe for
thread dumps.

Regards,

Rainer

Post by Travis Haagen
----- Original Message -----
Sent: Friday, December 14, 2007 9:20 AM
Subject: Re: Tomcat and IIS losing their link

Post by Rainer Jung

Post by Travis Haagen
Hello,
I've got IIS 6.0 linked with Tomcat 5.5.25 (with JDK 1.5.0_14-b03 and
AJP dll) via the ISAPI connector (JK 1.2.25) on Windows 2003 Server,
and during high traffic periods, the connection between IIS and
Tomcat is being severed. This has been occurring in the mid-afternoon
every day since this server went live and to get it running again,
I've had to restart Tomcat. The peak connections to IIS during one of
the days was just under 100, so this isn't really all that high of
traffic.
The isapi_redirect.log isn't much help. It has only been logging
errors after the connection to Tomcat has been broken. Tomcat itself
hasn't logged anything of relevance and I don't actually think that
Tomcat is affected when the link is broken, but as I mentioned
earlier, I have to restart Tomcat to get it to reconnect with IIS.

Nevertheless at least i would be curious to know, what the exact
contents of the isapi redirector log files are. We don't need all of it,
but the first 100 lines or so could be interesting. What's you log level?
Have a look at the other thread running on this list the last few days
("ISAPI JK2 ran better than JK, how can that be?"). To get a slightly
better idea what's happening when, you could also introduce a load
balancer and a status worker.
Most likely it would help to take Java Thread Dumps and look inside the
JVM, what your request threads are doing, when IIS gets stuck.
Regards,
Rainer
---------------------------------------------------------------------

---------------------------------------------------------------------

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Travis Haagen]

It appears that my problem with IIS "losing its link" with Tomcat was
actually an application problem, as several of you suggested. The JMX thread
dump (very helpful over terminal services) indicated that 100 threads were
locked up with Hibernate trying to get a database connection from DBCP. Upon
further research I learned that Hibernate (and probably everything else)
works better with the c3p0 pooling API, so I switched all my applications
over to c3p0 and all is well (for now).

Thanks for all the help,

Travis Haagen


---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org