OutOfMemory! Tomcat meets sudden spikes of used memory sometimes.

[Li Zhenxing]

Hi folks,
I am puzzled by the situation that Tomcat 4.1.31 meets sudden spikes of used
memory sometimes. Then it will increase to throw OutOfMemory Exception and
tomcat is hanging.

Those memory spikes appear in my Web application for a long time. The odd
thing is that it does not seem to progressively go up but rather spike up
with each next full GC will clean less objects out.

This is three pictures of the gc log.
Loading Image...
Loading Image...
Loading Image...
I used HPjtune tool to watch the gc log.
System using windows 2000 server, 2*CPU, 2G memory, JSDK 1.4.2_05, tomcat
4.1.31, And
set -Xms1250m -Xmx1250m -server -XX:+UseParallelGC -Xloggc:D:\tomcat\logs\gc.log

At first, I doubt if my app program has memory leak. But the application
only runs by JSPs and javabeans.(I abandon using DBCP pool and any thread
operation because of the memory leak.) I think simple JSPs can't cause the
memory spike. I have a monitor.jsp to show my application status.
http://my.sme.cn/jsp/main/monitor.jsp

The WebSite uses normally below 100M memory heap through my monitor.jsp. It
can run a few days, or a few hours before the spike appears. It is
irregular. I read the access_log and want to find answer. But I can't find
any clue.

Then, I think tomcat 4.1.* has some bugs causing the memory leak. I upgraded
my application to tomcat 5.0.18, but I met same thing.

The WebSite's hits is more than 200,000 per day. I try to use OptimizeIt 6
profile to watch memory status, but OptimizeIt always exits after runs 2~3
hours. I guess this program maybe need using progressive memory of system to
log things and analyze those when I notice Optimizeit used 1.6G system
memory.(and OptimizeIt 6 profile only runs under tomcat 4.1.*! So I return
to tomcat 4.1.31)

I don't know how to cause the thing and how to find solution. Any suggestion
is appreciated. Thanks.

Li Zhenxing

[Dale, Matt]

The fact that you have the same symptons in tomcat 4 and in 5 points towards your application as being the culprit.

Is there any reason why you chose tomcat 5.0.18 instead of one of the many newer releases?

I would find another profiler that works with both and try your tests again.

Ta
Matt

-----Original Message-----
From: Li Zhenxing [mailto:***@gmail.com]
Sent: 08 December 2004 13:47
To: tomcat-***@jakarta.apache.org
Subject: OutOfMemory! Tomcat meets sudden spikes of used memory
sometimes.


Hi folks,
I am puzzled by the situation that Tomcat 4.1.31 meets sudden spikes of used
memory sometimes. Then it will increase to throw OutOfMemory Exception and
tomcat is hanging.

Those memory spikes appear in my Web application for a long time. The odd
thing is that it does not seem to progressively go up but rather spike up
with each next full GC will clean less objects out.

This is three pictures of the gc log.
http://my.sme.cn/jsp/main/memoryleak.png
http://my.sme.cn/jsp/main/memoryleak2.png
http://my.sme.cn/jsp/main/memoryleak3.png
I used HPjtune tool to watch the gc log.
System using windows 2000 server, 2*CPU, 2G memory, JSDK 1.4.2_05, tomcat
4.1.31, And
set -Xms1250m -Xmx1250m -server -XX:+UseParallelGC -Xloggc:D:\tomcat\logs\gc.log

At first, I doubt if my app program has memory leak. But the application
only runs by JSPs and javabeans.(I abandon using DBCP pool and any thread
operation because of the memory leak.) I think simple JSPs can't cause the
memory spike. I have a monitor.jsp to show my application status.
http://my.sme.cn/jsp/main/monitor.jsp

The WebSite uses normally below 100M memory heap through my monitor.jsp. It
can run a few days, or a few hours before the spike appears. It is
irregular. I read the access_log and want to find answer. But I can't find
any clue.

Then, I think tomcat 4.1.* has some bugs causing the memory leak. I upgraded
my application to tomcat 5.0.18, but I met same thing.

The WebSite's hits is more than 200,000 per day. I try to use OptimizeIt 6
profile to watch memory status, but OptimizeIt always exits after runs 2~3
hours. I guess this program maybe need using progressive memory of system to
log things and analyze those when I notice Optimizeit used 1.6G system
memory.(and OptimizeIt 6 profile only runs under tomcat 4.1.*! So I return
to tomcat 4.1.31)

I don't know how to cause the thing and how to find solution. Any suggestion
is appreciated. Thanks.

Li Zhenxing

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-***@jakarta.apache.org
For additional commands, e-mail: tomcat-user-***@jakarta.apache.org

[Shapira, Yoav]

Hi,
Besides what Dale said, which is true, I'd like to point out a couple of
other additional things.

Post by Li Zhenxing
Those memory spikes appear in my Web application for a long time. The

odd

When a "spike" appears "for a long time" it's not a spike: it's the
steady state.

Post by Li Zhenxing
This is three pictures of the gc log.
http://my.sme.cn/jsp/main/memoryleak.png
http://my.sme.cn/jsp/main/memoryleak2.png
http://my.sme.cn/jsp/main/memoryleak3.png

Please don't jump to determine it's a memory leak just because more
memory is used.

Post by Li Zhenxing
I used HPjtune tool to watch the gc log.
System using windows 2000 server, 2*CPU, 2G memory, JSDK 1.4.2_05,

tomcat

Post by Li Zhenxing
4.1.31, And
set -Xms1250m -Xmx1250m -server -XX:+UseParallelGC -
Xloggc:D:\tomcat\logs\gc.log

So you're setting the heap size to a constant 1250MB. How can there be
big jumps all over the place?

Post by Li Zhenxing
At first, I doubt if my app program has memory leak. But the

application

Post by Li Zhenxing
only runs by JSPs and javabeans.(I abandon using DBCP pool and any

thread

Post by Li Zhenxing
operation because of the memory leak.) I think simple JSPs can't cause

the

Post by Li Zhenxing
memory spike. I have a monitor.jsp to show my application status.
http://my.sme.cn/jsp/main/monitor.jsp

I bet your app is more likely than Tomcat to have a leak ;) Simple JSPs
can cause memory "spikes," as can simple servlets. DBCP and threads do
NOT necessarily cause increased memory usage.

Post by Li Zhenxing
The WebSite uses normally below 100M memory heap through my

monitor.jsp. It

Post by Li Zhenxing
can run a few days, or a few hours before the spike appears. It is
irregular. I read the access_log and want to find answer. But I can't

find

Post by Li Zhenxing
any clue.

Perhaps it's high load causing the spike? That would be normal and
expected.

Post by Li Zhenxing
The WebSite's hits is more than 200,000 per day. I try to use

OptimizeIt 6

Post by Li Zhenxing
profile to watch memory status, but OptimizeIt always exits after runs

2~3

Post by Li Zhenxing
hours. I guess this program maybe need using progressive memory of

system

Post by Li Zhenxing
to
log things and analyze those when I notice Optimizeit used 1.6G system
memory.(and OptimizeIt 6 profile only runs under tomcat 4.1.*! So I

return

Post by Li Zhenxing
to tomcat 4.1.31)

Yes, OptimizeIt, like other profilers, can routinely use an order of
magnitude more memory than your app itself.

Post by Li Zhenxing
I don't know how to cause the thing and how to find solution. Any
suggestion
is appreciated. Thanks.

Get a test system, put your app on it, run it with a profiler, and
simulate load using a test tool of your choice, e.g. JMeter, ab, wget,
grinder, whatever. Then see where memory is retained during your
"spikes."

Yoav Shapira http://www.yoavshapira.com




This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you.

[Li Zhenxing]

Post by Shapira, Yoav
When a "spike" appears "for a long time" it's not a spike: it's the
steady state.

Yes, it's the steady state. I just aim at the gc figure, the spike.
Maybe this word don't discribes the figure exactly. I am sorry for my
English words.

Post by Shapira, Yoav
Please don't jump to determine it's a memory leak just because more
memory is used.

I think it is a obvious memory leak because the number of request
threads doesn't increase and the app hits just is normal when the
inflexion of gc figure appreas.
The question only is caused by my app, or tomcat.

Post by Shapira, Yoav
So you're setting the heap size to a constant 1250MB. How can there be
big jumps all over the place?

It should been given max memory as possible to avoid JVM increases
heap progressively when the server machine only runs tomcat.
I think all heavy load app should set two parameter equal.

Post by Shapira, Yoav
I bet your app is more likely than Tomcat to have a leak ;) Simple JSPs
can cause memory "spikes," as can simple servlets. DBCP and threads do
NOT necessarily cause increased memory usage.

It is probably that my app has a leak. I agree. But every JSP has been
visited during four days of system in order. Why to appear spike at
fifth day?
I have 2~3 hundreds JSPs that almost be requested everyday.
Does one JSP has memory leak at a special situation? Or one JSP
running at a special situation causes tomcat leak?

Post by Shapira, Yoav
Get a test system, put your app on it, run it with a profiler, and
simulate load using a test tool of your choice, e.g. JMeter, ab, wget,
grinder, whatever. Then see where memory is retained during your
"spikes."

Thanks for the advice.
and thanks for Dale's suggustion.
I will try to profile heap through -Xrunhprof and other tools.

Li Zhenxing

[Shapira, Yoav]

Hi,

Post by Li Zhenxing
I think it is a obvious memory leak because the number of request
threads doesn't increase and the app hits just is normal when the
inflexion of gc figure appreas.

So there is a constant load and constant memory usage for a couple of
days, and then under the same load the memory usage spikes up? No other
external parameters change, e.g. the database going down?

Post by Li Zhenxing
It should been given max memory as possible to avoid JVM increases
heap progressively when the server machine only runs tomcat.
I think all heavy load app should set two parameter equal.

I disagree. Pretty much no performance tuning criterion applies to "all
heavy load" apps. And the practice of setting -Xms = -Xmx is outdated,
as recent JDKs are very good at rapidly allocating large chunks of the
heap on-demand. However, it's your app and your call, and it sounds
like you've given it some thought, which is good.

Post by Li Zhenxing
I have 2~3 hundreds JSPs that almost be requested everyday.

And they don't change, right? So they don't need to be recompiled by
Jasper.

Post by Li Zhenxing
Does one JSP has memory leak at a special situation? Or one JSP
running at a special situation causes tomcat leak?

Can you correlate your access log (assuming it's enabled: if not, enable
it) to the increase in memory usage> Maybe it will show one particular
page being requested when the memory spikes up.

Yoav Shapira http://www.yoavshapira.com




This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you.

[Li Zhenxing]

Post by Shapira, Yoav
So there is a constant load and constant memory usage for a couple of
days, and then under the same load the memory usage spikes up? No other
external parameters change, e.g. the database going down?

I don't find other external parameters change. And the app gets right,
once restart tomcat!

Post by Shapira, Yoav
And they don't change, right? So they don't need to be recompiled by
Jasper.

Yes, they don't change.

Post by Shapira, Yoav
Can you correlate your access log (assuming it's enabled: if not, enable
it) to the increase in memory usage> Maybe it will show one particular
page being requested when the memory spikes up.

I have already correlated all JSP. But find nothing. All the access
JSPs during memory spiking up are normal pages. Those are visited
everyday.
Now I guess maybe some parameters of one JSP are wrong. I don't
correlate every access URL. I am tring to finish this work. You know,
this is big work because the access URL is more than 10 thousands
within a few minutes.

Thank Yoav very much.
I will report my ending for the problem.

Li Zhenxing

[Shapira, Yoav]

Hi,

Post by Li Zhenxing
Thank Yoav very much.
I will report my ending for the problem.
Li Zhenxing

No problem, I'm glad to help. These are the interesting problems, to me
at least. Especially when the person working on them seems to have a
good grip on things, like you do. I look forward to seeing your results
/ solution...

Yoav Shapira http://www.yoavshapira.com




This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you.