Tomcat 8 uses high CPU

[Josep M Beleta]

Hi,

I need some help with a problem we have running Tomcat 8, with Java 8 on a
Windows 2008 R2 Standard with Service Pack 1 running as a virtual machine
on top of ESXi, 5.1.0. The virtual machine has 16 GB of memory and 4 CPUs.
The physical host computer has lots of memory and is not stressed at all.

One week ago our Tomcats, we have two, began to use high CPU. When the
problem started we had no made any change we can remember in the preceding
week, and no Windows update was installed.

We use two Tomcats, one of them with some in-house applications the other
one with Jira. Both began to shows the same symptoms roughly at the same
time.

The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64 Bit, the
other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.

These are the facts we can summarize:

1. Now we run only the Tomcat for our in-house applications.We stopped
the Jira one.
2. We run it as a service.
3. Every time, two hours after launching the Tomcat service, the CPU use
jumps to 70-75% and some time after (one hour more) it reaches 100 %. More
threads are allocated using lots of CPU. The final result is our entire
system being degraded as time goes on.
4. The problems has nothing to do with our applications. We undeployed
them with the same result.
5. We tested versions 8.0.23 and 8.0.33 in our internal Tomcat with the
same results.
6. We tested JVM 1.8.0_66, 1.8.0_71 and 1.8.0_77 with the same results.
7. We have changed some JVM parameters, allocating more heap memory, or
changing GC methods with always the same result, 100% CPU.
8. Using jstack we take several thread dumps that we are enclosing in
this message. Dump75 is whe the CPU was at 75 %, the other one, Dump100,
the CPU was at 100%.
9. I am attaching the Process Explorer images of the Tomcat process
showing also the Tomcat threads. I am including a Tomcat manager screenshot
showing the status of the server.

Any help would be deeply appreciated,

Josep M Beleta

[image: Imatge inserida 1]

[image: Imatge inserida 3]

[image: Imatge inserida 4]

[Mark Thomas]

Post by Josep M Beleta
Hi,
I need some help with a problem we have running Tomcat 8, with Java 8 on
a Windows 2008 R2 Standard with Service Pack 1 running as a virtual
machine on top of ESXi, 5.1.0. The virtual machine has 16 GB of memory
and 4 CPUs. The physical host computer has lots of memory and is not
stressed at all.
One week ago our Tomcats, we have two, began to use high CPU. When the
problem started we had no made any change we can remember in the
preceding week, and no Windows update was installed.
We use two Tomcats, one of them with some in-house applications the
other one with Jira. Both began to shows the same symptoms roughly at
the same time.
The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64 Bit,
the other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.
1. Now we run only the Tomcat for our in-house applications.We stopped
the Jira one.
2. We run it as a service.
3. Every time, two hours after launching the Tomcat service, the CPU
use jumps to 70-75% and some time after (one hour more) it reaches
100 %. More threads are allocated using lots of CPU. The final
result is our entire system being degraded as time goes on.
4. The problems has nothing to do with our applications. We undeployed
them with the same result.
5. We tested versions 8.0.23 and 8.0.33 in our internal Tomcat with the
same results.
6. We tested JVM 1.8.0_66, 1.8.0_71 and 1.8.0_77 with the same results.
7. We have changed some JVM parameters, allocating more heap memory, or
changing GC methods with always the same result, 100% CPU.
8. Using jstack we take several thread dumps that we are enclosing in
this message. Dump75 is whe the CPU was at 75 %, the other one,
Dump100, the CPU was at 100%.
9. I am attaching the Process Explorer images of the Tomcat process
showing also the Tomcat threads. I am including a Tomcat manager
screenshot showing the status of the server.
Any help would be deeply appreciated,

We really need to see those thread dumps. I've checked the list config
and plain text files should be allowed through. I'm attaching a test
file to this message to confirm that. If it comes through, please
provide your thread dumps in plain text form and we can take a look.

Mark

[Josep M Beleta]

Hi Mark, thanks a lot.

here come the dumps:

Josep


*Josep M Beleta*

*Director de Tecnologia*


Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona


Tel: [phone]
Fax: [phone]

*segueix-nos <http://www.linkedin.com/company/carver-advanced-systems-s.l.>*
<http://twitter.com/#%21/carver_as>
<https://plus.google.com/112583630853594851700?prsrc=3>

www.carver-as.com

Post by Mark Thomas

Post by Josep M Beleta
Hi,
I need some help with a problem we have running Tomcat 8, with Java 8 on
a Windows 2008 R2 Standard with Service Pack 1 running as a virtual
machine on top of ESXi, 5.1.0. The virtual machine has 16 GB of memory
and 4 CPUs. The physical host computer has lots of memory and is not
stressed at all.
One week ago our Tomcats, we have two, began to use high CPU. When the
problem started we had no made any change we can remember in the
preceding week, and no Windows update was installed.
We use two Tomcats, one of them with some in-house applications the
other one with Jira. Both began to shows the same symptoms roughly at
the same time.
The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64 Bit,
the other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.
1. Now we run only the Tomcat for our in-house applications.We stopped
the Jira one.
2. We run it as a service.
3. Every time, two hours after launching the Tomcat service, the CPU
use jumps to 70-75% and some time after (one hour more) it reaches
100 %. More threads are allocated using lots of CPU. The final
result is our entire system being degraded as time goes on.
4. The problems has nothing to do with our applications. We undeployed
them with the same result.
5. We tested versions 8.0.23 and 8.0.33 in our internal Tomcat with the
same results.
6. We tested JVM 1.8.0_66, 1.8.0_71 and 1.8.0_77 with the same results.
7. We have changed some JVM parameters, allocating more heap memory, or
changing GC methods with always the same result, 100% CPU.
8. Using jstack we take several thread dumps that we are enclosing in
this message. Dump75 is whe the CPU was at 75 %, the other one,
Dump100, the CPU was at 100%.
9. I am attaching the Process Explorer images of the Tomcat process
showing also the Tomcat threads. I am including a Tomcat manager
screenshot showing the status of the server.
Any help would be deeply appreciated,

We really need to see those thread dumps. I've checked the list config
and plain text files should be allowed through. I'm attaching a test
file to this message to confirm that. If it comes through, please
provide your thread dumps in plain text form and we can take a look.
Mark
---------------------------------------------------------------------

[Josep M Beleta]

I am sorry, I sent the same file twice.

Here is the other one.

*Josep M Beleta*

*Director de Tecnologia*


Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona


Tel: [phone]
Fax: [phone]

*segueix-nos <http://www.linkedin.com/company/carver-advanced-systems-s.l.>*
<http://twitter.com/#%21/carver_as>
<https://plus.google.com/112583630853594851700?prsrc=3>

www.carver-as.com

Post by Josep M Beleta
Hi Mark, thanks a lot.
Josep
*Josep M Beleta*
*Director de Tecnologia*
Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona
Tel: [phone]
Fax: [phone]
*segueix-nos
<http://www.linkedin.com/company/carver-advanced-systems-s.l.>*
<http://twitter.com/#%21/carver_as>
<https://plus.google.com/112583630853594851700?prsrc=3>
www.carver-as.com

Post by Mark Thomas

Post by Josep M Beleta
Hi,
I need some help with a problem we have running Tomcat 8, with Java 8 on
a Windows 2008 R2 Standard with Service Pack 1 running as a virtual
machine on top of ESXi, 5.1.0. The virtual machine has 16 GB of memory
and 4 CPUs. The physical host computer has lots of memory and is not
stressed at all.
One week ago our Tomcats, we have two, began to use high CPU. When the
problem started we had no made any change we can remember in the
preceding week, and no Windows update was installed.
We use two Tomcats, one of them with some in-house applications the
other one with Jira. Both began to shows the same symptoms roughly at
the same time.
The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64 Bit,
the other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.
1. Now we run only the Tomcat for our in-house applications.We stopped
the Jira one.
2. We run it as a service.
3. Every time, two hours after launching the Tomcat service, the CPU
use jumps to 70-75% and some time after (one hour more) it reaches
100 %. More threads are allocated using lots of CPU. The final
result is our entire system being degraded as time goes on.
4. The problems has nothing to do with our applications. We undeployed
them with the same result.
5. We tested versions 8.0.23 and 8.0.33 in our internal Tomcat with the
same results.
6. We tested JVM 1.8.0_66, 1.8.0_71 and 1.8.0_77 with the same results.
7. We have changed some JVM parameters, allocating more heap memory, or
changing GC methods with always the same result, 100% CPU.
8. Using jstack we take several thread dumps that we are enclosing in
this message. Dump75 is whe the CPU was at 75 %, the other one,
Dump100, the CPU was at 100%.
9. I am attaching the Process Explorer images of the Tomcat process
showing also the Tomcat threads. I am including a Tomcat manager
screenshot showing the status of the server.
Any help would be deeply appreciated,

We really need to see those thread dumps. I've checked the list config
and plain text files should be allowed through. I'm attaching a test
file to this message to confirm that. If it comes through, please
provide your thread dumps in plain text form and we can take a look.
Mark
---------------------------------------------------------------------

[Mark Thomas]

Post by Josep M Beleta
I am sorry, I sent the same file twice.
Here is the other one.

Nope. They need to be plain text attachments and no larger than 1MB.

Mark

Post by Josep M Beleta
**
*Josep M Beleta*
/Director de Tecnologia/
Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona
Tel: [phone]
Fax: [phone]
*/segueix-nos
<http://www.linkedin.com/company/carver-advanced-systems-s.l.>/*<http://twitter.com/#%21/carver_as> <https://plus.google.com/112583630853594851700?prsrc=3>
www.carver-as.com <http://www.carver-as.com>
Hi Mark, thanks a lot.
Josep
**
*Josep M Beleta*
/Director de Tecnologia/
Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona
Tel: [phone] <tel:93%20238%205880>
Fax: [phone] <tel:93%20238%205881>
*/segueix-nos
<http://www.linkedin.com/company/carver-advanced-systems-s.l.>/*<http://twitter.com/#%21/carver_as> <https://plus.google.com/112583630853594851700?prsrc=3>
www.carver-as.com <http://www.carver-as.com>

Post by Josep M Beleta
Hi,
I need some help with a problem we have running Tomcat 8, with Java 8 on
a Windows 2008 R2 Standard with Service Pack 1 running as a virtual
machine on top of ESXi, 5.1.0. The virtual machine has 16 GB of memory
and 4 CPUs. The physical host computer has lots of memory and is not
stressed at all.
One week ago our Tomcats, we have two, began to use high CPU. When the
problem started we had no made any change we can remember in the
preceding week, and no Windows update was installed.
We use two Tomcats, one of them with some in-house applications the
other one with Jira. Both began to shows the same symptoms roughly at
the same time.
The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64 Bit,
the other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.
1. Now we run only the Tomcat for our in-house

applications.We stopped

Post by Josep M Beleta
the Jira one.
2. We run it as a service.
3. Every time, two hours after launching the Tomcat service,

the CPU

Post by Josep M Beleta
use jumps to 70-75% and some time after (one hour more) it reaches
100 %. More threads are allocated using lots of CPU. The final
result is our entire system being degraded as time goes on.
4. The problems has nothing to do with our applications. We

undeployed

Post by Josep M Beleta
them with the same result.
5. We tested versions 8.0.23 and 8.0.33 in our internal

Tomcat with the

Post by Josep M Beleta
same results.
6. We tested JVM 1.8.0_66, 1.8.0_71 and 1.8.0_77 with the

same results.

Post by Josep M Beleta
7. We have changed some JVM parameters, allocating more heap

memory, or

Post by Josep M Beleta
changing GC methods with always the same result, 100% CPU.
8. Using jstack we take several thread dumps that we are

enclosing in

Post by Josep M Beleta
this message. Dump75 is whe the CPU was at 75 %, the other one,
Dump100, the CPU was at 100%.
9. I am attaching the Process Explorer images of the Tomcat

process

Post by Josep M Beleta
showing also the Tomcat threads. I am including a Tomcat manager
screenshot showing the status of the server.
Any help would be deeply appreciated,

We really need to see those thread dumps. I've checked the list config
and plain text files should be allowed through. I'm attaching a test
file to this message to confirm that. If it comes through, please
provide your thread dumps in plain text form and we can take a look.
Mark
---------------------------------------------------------------------
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Christopher Schultz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Josep,

Post by Josep M Beleta
I need some help with a problem we have running Tomcat 8, with Java
8 on a Windows 2008 R2 Standard with Service Pack 1 running as a
virtual machine on top of ESXi, 5.1.0. The virtual machine has 16
GB of memory and 4 CPUs. The physical host computer has lots of
memory and is not stressed at all.
One week ago our Tomcats, we have two, began to use high CPU. When
the problem started we had no made any change we can remember in
the preceding week, and no Windows update was installed.
We use two Tomcats, one of them with some in-house applications
the other one with Jira. Both began to shows the same symptoms
roughly at the same time.
The Jira Tomcat version is 8.0.17 and runs with Java 1.8.0_51 64
Bit, the other Tomcat is 8.0.33 and runs over Java 1.8.0_77 64 Bit.
1. Now we run only the Tomcat for our in-house applications.We
stopped the Jira one. 2. We run it as a service. 3. Every time, two
hours after launching the Tomcat service, the CPU use jumps to
70-75% and some time after (one hour more) it reaches 100 %. More
threads are allocated using lots of CPU. The final result is our
entire system being degraded as time goes on. 4. The problems has
nothing to do with our applications. We undeployed them with the
same result.

So if you start Tomcat with no applications deployed and make no
requests, Tomcat starts taking 75% of the CPU after 2 hours?

Mark is right: we need to see those thread dumps. It looks like your
second attempt also didn't work. Please try to re-attach them as plain
text files.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlcNQ2QACgkQ9CaO5/Lv0PDJTwCfWuiSiAqrb8hYw5YQLOxE52uX
isIAnRpR9Vj2GZLjiCMr6+hnFbB7eEPb
=/3nu
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Josep M Beleta]

Again,

[Mark Thomas]

Post by Josep M Beleta
Again,

Thanks. They came thought this time but aren't that helpful as several
key bits of information are missing:
- thread name
- thread ID

There should be an option in Tomcat8w.exe to trigger a thread dump. This
will be written to the log files and should be more complete.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Josep M Beleta]

Mark,

I tested Tomcat 8.5 and experienced the same problem.

I am enclosing a new dump. I hope it will be useful.

Regards

Josep M Beleta

[Josep M Beleta]

I produced a new dump on our production server that runs Tomcat 8.0.

This time I identified the three threads that are consuming high CPU.

I marked them with:

****** This thread uses high CPU ******

All these threads are named "ajp-nio-8009-ClientPoller"

This is the information it lists about one of them:

"ajp-nio-8009-ClientPoller-1" #111 daemon prio=5 os_prio=0
tid=0x000000001d880000 nid=0x9c4 runnable [0x000000003b7df000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000006c96ce138> (a sun.nio.ch.Util$2)
- locked <0x00000006c96ce128> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c96ce0a8> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1034)
at java.lang.Thread.run(Unknown Source)

Locked ownable synchronizers:
- None

Thank you for your help.

Josep

[Josep M Beleta]

I was wrong in my last message. One of the threads is different, is is not
related to AJP. This thread is:

"NioBlockingSelector.BlockPoller-2" #15 daemon prio=5 os_prio=0
tid=0x000000001ce0c800 nid=0x19b8 runnable [0x000000001dfff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000006c016abd0> (a sun.nio.ch.Util$2)
- locked <0x00000006c016abc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c016aa50> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:342)

Locked ownable synchronizers:
- None

All they have in common is that all them are running inside
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0.

Could be a Java problem?

Could I find a workaround?

What could have been the cause that triggered that behavior?

Thanks again.

Josep M Beleta

[Mark Thomas]

Post by Josep M Beleta
I was wrong in my last message. One of the threads is different, is is not
"NioBlockingSelector.BlockPoller-2" #15 daemon prio=5 os_prio=0
tid=0x000000001ce0c800 nid=0x19b8 runnable [0x000000001dfff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000006c016abd0> (a sun.nio.ch.Util$2)
- locked <0x00000006c016abc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c016aa50> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at
org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:342)
- None
All they have in common is that all them are running inside
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0.

Isolating it to the poller threads is very helpful.

Post by Josep M Beleta
Could be a Java problem?

Maybe. A Tomcat bug is more likely at this point.

Post by Josep M Beleta
Could I find a workaround?

We need to figure out the root cause first...

Post by Josep M Beleta
What could have been the cause that triggered that behavior?

Hard to say. Maybe some edge case error condition that puts things into
an unexpected state.

It sounds like the Poller is in a tight loop. The next thing to figure
out is what code path it is following and why. Are you able to configure
a test instance to allow remote debugging? If you can, wait for the
problem to occur, set a break point on NioBlockingSelector.java:342, and
attach the debugger. Check that the poller thread keeps coming back to
this point. If it does, step through the code and record the path followed.

Alternatively, if you find a way to recreate the issue, other people on
this list will be able to help investigate.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Josep M Beleta]

Ok. I will try to debug the issue this weekend.

Josep

*Josep M Beleta*

*Director de Tecnologia*


Carver Advanced Systems, S.L.
París, 184. 4º 08036 - Barcelona


Tel: [phone]
Fax: [phone]

*segueix-nos <http://www.linkedin.com/company/carver-advanced-systems-s.l.>*
<http://twitter.com/#%21/carver_as>
<https://plus.google.com/112583630853594851700?prsrc=3>

www.carver-as.com

Post by Josep M Beleta

Post by Josep M Beleta
I was wrong in my last message. One of the threads is different, is is

not

Post by Josep M Beleta
"NioBlockingSelector.BlockPoller-2" #15 daemon prio=5 os_prio=0
tid=0x000000001ce0c800 nid=0x19b8 runnable [0x000000001dfff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000006c016abd0> (a sun.nio.ch.Util$2)
- locked <0x00000006c016abc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c016aa50> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at

org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:342)

Post by Josep M Beleta
- None
All they have in common is that all them are running inside
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0.

Isolating it to the poller threads is very helpful.

Post by Josep M Beleta
Could be a Java problem?

Maybe. A Tomcat bug is more likely at this point.

Post by Josep M Beleta
Could I find a workaround?

We need to figure out the root cause first...

Post by Josep M Beleta
What could have been the cause that triggered that behavior?

Hard to say. Maybe some edge case error condition that puts things into
an unexpected state.
It sounds like the Poller is in a tight loop. The next thing to figure
out is what code path it is following and why. Are you able to configure
a test instance to allow remote debugging? If you can, wait for the
problem to occur, set a break point on NioBlockingSelector.java:342, and
attach the debugger. Check that the poller thread keeps coming back to
this point. If it does, step through the code and record the path followed.
Alternatively, if you find a way to recreate the issue, other people on
this list will be able to help investigate.
Mark
---------------------------------------------------------------------

[André Warnier (tomcat)]

Post by Josep M Beleta
Could I find a workaround?

Maybe for the meantime, you could try another Connector protocol ?
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
-> protocol

Note: I am not an expert, and not sure that in this particular case it would help.
But it is very quickly done, and it may provide some additional insight into the issue.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Josep M Beleta]

I'll try it. Thanks a lot.

Post by André Warnier (tomcat)

Post by Josep M Beleta
Could I find a workaround?

Maybe for the meantime, you could try another Connector protocol ?
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
-> protocol
Note: I am not an expert, and not sure that in this particular case it would help.
But it is very quickly done, and it may provide some additional insight into the issue.
---------------------------------------------------------------------

[Josep M Beleta]

Following André suggestion I replaced the connector protocols, both for
HTTP and AJP ports, to force NIO.2. Now Tomcat is working for seven hours
without any problem.

My findings until now are:

1. It is not a Tomcat related problem, WildFly 10 has the same behavior.
The WildFly high CPU threads also
calls sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method).
2. When switching from NIO to NIO.2 the problem goes away.
3. It seems clearly a Java 8 for Windows problem that happens in several
versions. I tested 1.8.0_66, 1.8.0_71 and 1.8.0_77.
4. Perhaps the problem is related to Windows 2008 R2 or VMWare ESXi. On
a Windows 10 machine it works like a charm.
5. I cannot imagine what triggered the problem, no change in WIndows or
ESX was made. The only thing that changed was that some new applications
were installed when the problem started to show, but after that I removed
all the applications. In the case of WildFly no application was never
started.

I'll try to fill a bug in the Oracle site if there is not any other
suggestion.

Thanks again to all.

Josep

Post by Josep M Beleta
I'll try it. Thanks a lot.

Post by André Warnier (tomcat)

Post by Josep M Beleta
Could I find a workaround?

Maybe for the meantime, you could try another Connector protocol ?
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
-> protocol
Note: I am not an expert, and not sure that in this particular case it would help.
But it is very quickly done, and it may provide some additional insight into the issue.
---------------------------------------------------------------------

[David Kerber]

Following André suggestion I replaced the connector protocols, both for
HTTP and AJP ports, to force NIO.2. Now Tomcat is working for seven hours
without any problem.
1. It is not a Tomcat related problem, WildFly 10 has the same behavior.
The WildFly high CPU threads also
calls sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method).
2. When switching from NIO to NIO.2 the problem goes away.
3. It seems clearly a Java 8 for Windows problem that happens in several
versions. I tested 1.8.0_66, 1.8.0_71 and 1.8.0_77.
4. Perhaps the problem is related to Windows 2008 R2 or VMWare ESXi. On
a Windows 10 machine it works like a charm.
5. I cannot imagine what triggered the problem, no change in WIndows or
ESX was made. The only thing that changed was that some new applications
were installed when the problem started to show, but after that I removed
all the applications. In the case of WildFly no application was never
started.
I'll try to fill a bug in the Oracle site if there is not any other
suggestion.

My money would be on VMWare. I would think that if it occurred in a
bare metal windows installation, it would have been reported before now.

Thanks again to all.
Josep

Post by Josep M Beleta
I'll try it. Thanks a lot.

Post by André Warnier (tomcat)

Post by Josep M Beleta
Could I find a workaround?

Maybe for the meantime, you could try another Connector protocol ?
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
-> protocol
Note: I am not an expert, and not sure that in this particular case it would help.
But it is very quickly done, and it may provide some additional insight into the issue.
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[André Warnier (tomcat)]

Following André suggestion I replaced the connector protocols, both for
HTTP and AJP ports, to force NIO.2. Now Tomcat is working for seven hours
without any problem.
1. It is not a Tomcat related problem, WildFly 10 has the same behavior.
The WildFly high CPU threads also
calls sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method).
2. When switching from NIO to NIO.2 the problem goes away.
3. It seems clearly a Java 8 for Windows problem that happens in several
versions. I tested 1.8.0_66, 1.8.0_71 and 1.8.0_77.
4. Perhaps the problem is related to Windows 2008 R2 or VMWare ESXi. On
a Windows 10 machine it works like a charm.
5. I cannot imagine what triggered the problem, no change in WIndows or
ESX was made. The only thing that changed was that some new applications
were installed when the problem started to show, but after that I removed
all the applications. In the case of WildFly no application was never
started.
I'll try to fill a bug in the Oracle site if there is not any other
suggestion.

My money would be on VMWare. I would think that if it occurred in a bare metal windows
installation, it would have been reported before now.

Hmm. With the number of Windows servers which must by now be running under Vmware, I think
that this may be a bit of a dangerous assumption.

On the bright side : it does not look like a Tomcat issue, so Mark will be happy. And
there is an easy workaround in Tomcat, so Tomcat users will be happy too.

On the other side : I haven't done the boolean matrix, but it at least looks like the
issue is somewhere ar the murky border between an Oracle JVM, a Microsoft OS, and a Vmware
hypervisor. Delicious. I wish everyone good luck to get it it (a) acknowledged and (b) fixed.

Thanks again to all.
Josep

Post by Josep M Beleta
I'll try it. Thanks a lot.

Post by André Warnier (tomcat)

Post by Josep M Beleta
Could I find a workaround?

Maybe for the meantime, you could try another Connector protocol ?
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Common_Attributes
-> protocol
Note: I am not an expert, and not sure that in this particular case it would help.
But it is very quickly done, and it may provide some additional insight
into the issue.
---------------------------------------------------------------------

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org