Debugging tomcat native connector

[Mark A. Claassen]

Is there a way to debug the native connectors? Specifically, we are having some problems getting the native openssl connector working on Ubuntu.
Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but we don't know why. We were hoping that there was some way to get more information.

Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
E-mail: mailto:***@ocie.net
Voice: [phone]
Fax: [phone]

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.

[Roger Brechbühl]

with which connector do you have problems? nio, nio2 or apr?

I ask because we have problems with nio2-openssl when ssl session is reused
e.g. when a request is proxied with nginx.

kind regards,
Roger

Post by Mark A. Claassen
Is there a way to debug the native connectors? Specifically, we are
having some problems getting the native openssl connector working on Ubuntu.
Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but we
don't know why. We were hoping that there was some way to get more
information.
Thanks,
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
Voice: [phone]
Fax: [phone]
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.

[Mark A. Claassen]

Thanks for the reply. I am using "org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.

The NIO connectors worked fine when we tried them, but the native ones seem to start up, but then don’t work.

Interestingly, we found out that these connectors do eventually start working! We tried to connect a few hours later and they were working fine.
We are now trying to determine exactly how long it takes for them to accept connections in hopes that this will help us figure out what is going on.

Having a way to debug this better would be helpful.


Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:***@ocie.net
Voice: [phone]
Fax: [phone]

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Roger Brechbühl <***@gmail.com>
Sent: Wednesday, October 31, 2018 5:25 PM
To: Tomcat Users List <***@tomcat.apache.org>
Subject: Re: Debugging tomcat native connector

with which connector do you have problems? nio, nio2 or apr?

I ask because we have problems with nio2-openssl when ssl session is reused e.g. when a request is proxied with nginx.

kind regards,
Roger

Post by Mark A. Claassen
Is there a way to debug the native connectors? Specifically, we are
having some problems getting the native openssl connector working on Ubuntu.
Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but
we don't know why. We were hoping that there was some way to get more
information.
Thanks,
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
Voice: [phone]
Fax: [phone]
The opinions provided herein do not necessarily state or reflect those
of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
no legal liability or responsibility for the posting.

Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F����6����G2�R�

[Christopher Schultz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

Post by Mark A. Claassen
Thanks for the reply. I am using
"org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.
The NIO connectors worked fine when we tried them, but the native
ones seem to start up, but then don’t work.
Interestingly, we found out that these connectors do eventually
start working! We tried to connect a few hours later and they were
working fine. We are now trying to determine exactly how long it
takes for them to accept connections in hopes that this will help
us figure out what is going on.
Having a way to debug this better would be helpful.

Are you getting log entries in catalina.out similar to these?

INFO: Starting ProtocolHandler ["ajp-nio-8215"]
Oct 10, 2018 4:23:38 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-127.0.0.1-8217"]
Oct 10, 2018 4:23:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6819 ms

??

If so, there should also be some timing information for initializing
certain things. For example, if the server doesn't have much entropy
in the entropy pool, then you can get a slow-startup while the crypto
engine starts. This may be worse with the APR connector than for the
NIO connectors as the source of randomness may be different by default
(e.g. /dev/urandom for Java, /dev/random for APR). That may not
actually be *true*, but it's a possibility.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvbNXYACgkQHPApP6U8
pFhXkBAAny/RjuglPB38vMd1HZ45+9ifW4zRVgGY+2VjtJLSCJKf8+bouMdr6ZWY
5MbE9ru4WYNnIE7J01gYK7vjGfrotjxNLVAsY52qiCAf9u6Q+hsZb00T5B9Fog+z
9lFm2J4alHUMTe758vBJDtuuCqiB3I3SaY2f5rp5TiCWYVExrbNDQUe50ClbqFmB
+/e0XIybWXMCPsgylfyqQIZLwJ3t/Q+1eocQUIst86+tMrJdv2PWu+b7+J2y752Y
8dy3LqUjLQ218nDaXe1DXnWajunhuwAunK0/y1fL4pqlwJYDf6GTuWbW+AVDkBZh
3Q1o74QmM230FVyBhll2HSh9wi+rOON7MnjQD8Buks8LeTzyHRdGPRnFXoZm4H99
Re5uDXHLsJ0pweucVc4hqfE+2BxW33GT36FHUjGVAijMO6V3NX4CYhiOD7XkSlaz
uvoC+3ibhZZrgmc1AkM122u0YciSLDLb5iqGoVP/cOGJ0JVEwlEPMrpwSSe0ZTXz
BB6mTRu9WzN9j4YboTNmuiYnBx6wDAEWu+f9CjFptw7qGcwrzqrFyLp+/QuYzf4l
3iEVDlXPjF0bFsuJOvTChxfVm7s5NrxUBf2hJfMxmuKjeKKPZElgnWYtyoKokCHW
GNUD+HIGvJmc5bICo+WRzyf7zu1bsVoATcNvzM+K9WF1TBsrS+I=
=7fqB
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Mark A. Claassen]

Wow, thanks a lot. That was exactly what it was.

Something else, just in the event that someone references this email thread in the future, I also realized I could put this in my logging.properties file:
org.apache.coyote.level = FINE

I didn't think to add that until after the entropy thing was figured out, but it is something that may come in handy in similar situations.


Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:***@ocie.net
Voice: [phone]
Fax: [phone]

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Christopher Schultz <***@christopherschultz.net>
Sent: Thursday, November 1, 2018 1:19 PM
To: ***@tomcat.apache.org
Subject: Re: Debugging tomcat native connector

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

Post by Mark A. Claassen
Thanks for the reply. I am using
"org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.
The NIO connectors worked fine when we tried them, but the native ones
seem to start up, but then don’t work.
Interestingly, we found out that these connectors do eventually start
working! We tried to connect a few hours later and they were working
fine. We are now trying to determine exactly how long it takes for
them to accept connections in hopes that this will help us figure out
what is going on.
Having a way to debug this better would be helpful.

Are you getting log entries in catalina.out similar to these?

INFO: Starting ProtocolHandler ["ajp-nio-8215"] Oct 10, 2018 4:23:38 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-127.0.0.1-8217"] Oct 10, 2018 4:23:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6819 ms

??

If so, there should also be some timing information for initializing certain things. For example, if the server doesn't have much entropy in the entropy pool, then you can get a slow-startup while the crypto engine starts. This may be worse with the APR connector than for the NIO connectors as the source of randomness may be different by default (e.g. /dev/urandom for Java, /dev/random for APR). That may not actually be *true*, but it's a possibility.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvbNXYACgkQHPApP6U8
pFhXkBAAny/RjuglPB38vMd1HZ45+9ifW4zRVgGY+2VjtJLSCJKf8+bouMdr6ZWY
5MbE9ru4WYNnIE7J01gYK7vjGfrotjxNLVAsY52qiCAf9u6Q+hsZb00T5B9Fog+z
9lFm2J4alHUMTe758vBJDtuuCqiB3I3SaY2f5rp5TiCWYVExrbNDQUe50ClbqFmB
+/e0XIybWXMCPsgylfyqQIZLwJ3t/Q+1eocQUIst86+tMrJdv2PWu+b7+J2y752Y
8dy3LqUjLQ218nDaXe1DXnWajunhuwAunK0/y1fL4pqlwJYDf6GTuWbW+AVDkBZh
3Q1o74QmM230FVyBhll2HSh9wi+rOON7MnjQD8Buks8LeTzyHRdGPRnFXoZm4H99
Re5uDXHLsJ0pweucVc4hqfE+2BxW33GT36FHUjGVAijMO6V3NX4CYhiOD7XkSlaz
uvoC+3ibhZZrgmc1AkM122u0YciSLDLb5iqGoVP/cOGJ0JVEwlEPMrpwSSe0ZTXz
BB6mTRu9WzN9j4YboTNmuiYnBx6wDAEWu+f9CjFptw7qGcwrzqrFyLp+/QuYzf4l
3iEVDlXPjF0bFsuJOvTChxfVm7s5NrxUBf2hJfMxmuKjeKKPZElgnWYtyoKokCHW
GNUD+HIGvJmc5bICo+WRzyf7zu1bsVoATcNvzM+K9WF1TBsrS+I=
=7fqB
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-m