Memory behavior: Tomcat versus Jetty

[Christian Stöber]

Hello list,

we are about to migrate our webserver cluster from Jetty 5.1.14 to
Tomcat 6.0.20. Currently there are four servers in our cluster with
simple load balancing and no session clustering.

At the moment we are testing Tomcat on 2 nodes of our cluster (also no
session clustering yet). 2 remains with Jetty. Same WAR file an the same
environment. Our first test ends in an OutOfMemoryError after two days.
The java vm of the Tomcat servers has the same memory parameters as the
server with the Jetty: -Xmx512m. And no other special GC parameters.

Apperently the Tomcat needs more memory than Jetty.

So we are trying to give Tomcat more memory: -Xmx768m. After two days
the Tomcat is still running fine.

Obviously Tomcat needs just a little bit more memory than Jetty. After
these two days we can state the following memory behavior.

Node Mem used Mem reserved
1 TC 400m 530m
2 TC 350m 495m
3 J 230m 350m
4 J 290m 400m

Averagely Tomcat is using round about 110m more memory than Jetty. But why?

I have seen that the Tomcat DefaultServlet may cache content up to 10m
per default. But that are only 10m.
Are there some other things Tomcat may cache and Jetty does not?

Do you have any experiences?

Thanks in advance
Christian

[Mark Thomas]

Post by Christian Stöber
Hello list,
we are about to migrate our webserver cluster from Jetty 5.1.14 to
Tomcat 6.0.20. Currently there are four servers in our cluster with
simple load balancing and no session clustering.
At the moment we are testing Tomcat on 2 nodes of our cluster (also no
session clustering yet). 2 remains with Jetty. Same WAR file an the same
environment. Our first test ends in an OutOfMemoryError after two days.
The java vm of the Tomcat servers has the same memory parameters as the
server with the Jetty: -Xmx512m. And no other special GC parameters.
Apperently the Tomcat needs more memory than Jetty.
So we are trying to give Tomcat more memory: -Xmx768m. After two days
the Tomcat is still running fine.
Obviously Tomcat needs just a little bit more memory than Jetty. After
these two days we can state the following memory behavior.
Node Mem used Mem reserved
1 TC 400m 530m
2 TC 350m 495m
3 J 230m 350m
4 J 290m 400m
Averagely Tomcat is using round about 110m more memory than Jetty. But why?
I have seen that the Tomcat DefaultServlet may cache content up to 10m
per default. But that are only 10m.
Are there some other things Tomcat may cache and Jetty does not?

There are certainly other things Tomcat caches. Whether or not Jetty caches them
I couldn't say. Take some heap dumps and see what the differences are.

Mark

[Tim Funk]

If you use JSP tags where the JSP body does not directly stream but
needs buffered for the tag to finish processing it (using BodyContent) -
then tomcat will allocate and reuse these. If you are creating pages
with large body contents - this can take *A LOT* of memory. The rational
is to reuse these instead of letting them go to the GC. But there is a
way to not have tomcat reuse these.

http://tomcat.apache.org/tomcat-5.5-doc/jspapi/javax/servlet/jsp/tagext/BodyContent.html

-Tim

Post by Christian Stöber
Hello list,
we are about to migrate our webserver cluster from Jetty 5.1.14 to
Tomcat 6.0.20. Currently there are four servers in our cluster with
simple load balancing and no session clustering.
At the moment we are testing Tomcat on 2 nodes of our cluster (also no
session clustering yet). 2 remains with Jetty. Same WAR file an the same
environment. Our first test ends in an OutOfMemoryError after two days.
The java vm of the Tomcat servers has the same memory parameters as the
server with the Jetty: -Xmx512m. And no other special GC parameters.
Apperently the Tomcat needs more memory than Jetty.
So we are trying to give Tomcat more memory: -Xmx768m. After two days
the Tomcat is still running fine.
Obviously Tomcat needs just a little bit more memory than Jetty. After
these two days we can state the following memory behavior.
Node Mem used Mem reserved
1 TC 400m 530m
2 TC 350m 495m
3 J 230m 350m
4 J 290m 400m
Averagely Tomcat is using round about 110m more memory than Jetty. But why?
I have seen that the Tomcat DefaultServlet may cache content up to 10m
per default. But that are only 10m.
Are there some other things Tomcat may cache and Jetty does not?
Do you have any experiences?
Thanks in advance
Christian
---------------------------------------------------------------------

[Christian Stöber]

Tim, thanks for the hint with the BodyContent.
Mark, thanks for your hint to take heap dump.

On this morning the two Tomcat instances were using 200 MB more memory
than the Jetty instances do. So I thought it's a good time to take a heap
dump.

In the heap histogram in jhat I saw a big difference between the
BodyContentImpl (class org.apache.jasper.runtime.BodyContentImpl) counts:

Jetty: 43
Tomcat: 768

Suspicious!

I've take a look at this code and the first thing I've seen was, that
since Jasper 2.1 BodyContents won't reset its buffer by default. (see
BodyContentImpl.clear()). The second thing - what's responsible - is the
change at org.apache.jasper.runtime.JspFactoryImpl. JspFactoryImpl in
Jasper 2.1 now caches the PageContext-objects per thread. A
PageContext-object have an array of BodyContents and so the memory the
buffers of these BodyContents reserved will newer be freed until the
thread will be stopped and garbage collected.

Until today I were using the default connectors with the maxThreads
parameter, which newer shutsdown a thread. So the BodyContent's buffer
were growing and growing, because we have some tags which generates much
output, and would never be cleared.

Now I am using a separate thread pool (executor), which cleans up the
pool when a thread is idle. And now the memory behavior is almost equal
to Jetty.

Christian

Post by Tim Funk
If you use JSP tags where the JSP body does not directly stream but
needs buffered for the tag to finish processing it (using BodyContent)
- then tomcat will allocate and reuse these. If you are creating pages
with large body contents - this can take *A LOT* of memory. The
rational is to reuse these instead of letting them go to the GC. But
there is a way to not have tomcat reuse these.
http://tomcat.apache.org/tomcat-5.5-doc/jspapi/javax/servlet/jsp/tagext/BodyContent.html
-Tim

Post by Christian Stöber
Hello list,
we are about to migrate our webserver cluster from Jetty 5.1.14 to
Tomcat 6.0.20. Currently there are four servers in our cluster with
simple load balancing and no session clustering.
At the moment we are testing Tomcat on 2 nodes of our cluster (also
no session clustering yet). 2 remains with Jetty. Same WAR file an
the same environment. Our first test ends in an OutOfMemoryError
after two days. The java vm of the Tomcat servers has the same memory
parameters as the server with the Jetty: -Xmx512m. And no other
special GC parameters.
Apperently the Tomcat needs more memory than Jetty.
So we are trying to give Tomcat more memory: -Xmx768m. After two days
the Tomcat is still running fine.
Obviously Tomcat needs just a little bit more memory than Jetty.
After these two days we can state the following memory behavior.
Node Mem used Mem reserved
1 TC 400m 530m
2 TC 350m 495m
3 J 230m 350m
4 J 290m 400m
Averagely Tomcat is using round about 110m more memory than Jetty. But why?
I have seen that the Tomcat DefaultServlet may cache content up to
10m per default. But that are only 10m.
Are there some other things Tomcat may cache and Jetty does not?
Do you have any experiences?
Thanks in advance
Christian
---------------------------------------------------------------------

---------------------------------------------------------------------

[Mark Thomas]

Post by Christian Stöber
Now I am using a separate thread pool (executor), which cleans up the
pool when a thread is idle. And now the memory behavior is almost equal
to Jetty.

Thanks for posting your analysis. You might be interested to know that
Tomcat 7 will use executors exclusively.

Mark