mod_jk endpoints

[Jerry Malcolm]

Thanks for the responses to my earlier post about websockets/JSPs. 
Http/2 is definitely the right answer for my issue there.

HTTP/2 solved the problem with the throttling the multiple parallel
requests with the browser connection limits.  But now opening that
throttle has now exposed a different problem at the server.  Quick
refresh.... I have an app that does not have many concurrent users at
this time.  However it does periodically have a very large number of
concurrent requests (AJAX/REST) from a single client.

I now realize that with the browser limiting concurrent requests in
HTTP/1.1, mod_jk was not getting overwhelmed.  But now that the
floodgates have opened with HTTP/2, I realize that I have some serious
configuration problems with mod_jk.  The requests are clogging up in
mod_jk, timing out, and logging errors something like "cannot get
endpoint for worker".  It then takes about 5 minutes for the clog to
clear out before mod_jk is functional again for the other virtual hosts
I'm hosting.

Specifics... I'm using Apache 2.4.33, Tomcat 9.0.  I tried to download
the latest mod_jk.so.  The latest mod_jk for httpd on the binaries
download site is from 2016.  All the versions for the past 2 years are
iis only... I assume that's ok???  So I'm using the 2016 1.2.40 version.

My question is.... how do I increase the number of 'endpoints' that
mod_jk has access to?  Is that a TC configuration or workers.properties
setting?  I want to make sure mod_jk has enough resources/endpoints to
pass (i.e.not 'block') all the requests it receives.  Once mod_jk is
passing everything, I realize the flood might hit TC, and I may be back
asking about increasing TC resources.  But for now, I just want to get
mod_jk opened up.

Thanks again.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Greg Huber]

Post by Jerry Malcolm
So I'm using the 2016 1.2.40 version.

Think you have to build mod_jk to use the latest and greatest, this is what
I have in my notes (also from the website docs):

mod_jk.so build
yum install httpd-devel
download tomcat-connectors-x.x.xx-src.tar.gz
extract and cd into the native folder
cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
./configure --with-apxs=/usr/bin/apxs
make
copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
make clean

(I only have a pretty much standard workers.properties so can't help much.)

Post by Jerry Malcolm
Thanks for the responses to my earlier post about websockets/JSPs.
Http/2 is definitely the right answer for my issue there.
HTTP/2 solved the problem with the throttling the multiple parallel
requests with the browser connection limits. But now opening that
throttle has now exposed a different problem at the server. Quick
refresh.... I have an app that does not have many concurrent users at
this time. However it does periodically have a very large number of
concurrent requests (AJAX/REST) from a single client.
I now realize that with the browser limiting concurrent requests in
HTTP/1.1, mod_jk was not getting overwhelmed. But now that the
floodgates have opened with HTTP/2, I realize that I have some serious
configuration problems with mod_jk. The requests are clogging up in
mod_jk, timing out, and logging errors something like "cannot get
endpoint for worker". It then takes about 5 minutes for the clog to
clear out before mod_jk is functional again for the other virtual hosts
I'm hosting.
Specifics... I'm using Apache 2.4.33, Tomcat 9.0. I tried to download
the latest mod_jk.so. The latest mod_jk for httpd on the binaries
download site is from 2016. All the versions for the past 2 years are
iis only... I assume that's ok??? So I'm using the 2016 1.2.40 version.
My question is.... how do I increase the number of 'endpoints' that
mod_jk has access to? Is that a TC configuration or workers.properties
setting? I want to make sure mod_jk has enough resources/endpoints to
pass (i.e.not 'block') all the requests it receives. Once mod_jk is
passing everything, I realize the flood might hit TC, and I may be back
asking about increasing TC resources. But for now, I just want to get
mod_jk opened up.
Thanks again.
Jerry
---------------------------------------------------------------------

[Mark Thomas]

Post by Greg Huber

Post by Jerry Malcolm
So I'm using the 2016 1.2.40 version.

Think you have to build mod_jk to use the latest and greatest,

Correct.

We stopped providing binary builds for anything other than Windows because there were just too many combinations. And it is relatively simple to build on Linux.

We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).

Post by Greg Huber
this is
what
mod_jk.so build
yum install httpd-devel
download tomcat-connectors-x.x.xx-src.tar.gz
extract and cd into the native folder
cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
./configure --with-apxs=/usr/bin/apxs
make
copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
make clean
(I only have a pretty much standard workers.properties so can't help much.)

Post by Jerry Malcolm
Thanks for the responses to my earlier post about websockets/JSPs.
Http/2 is definitely the right answer for my issue there.
HTTP/2 solved the problem with the throttling the multiple parallel
requests with the browser connection limits. But now opening that
throttle has now exposed a different problem at the server. Quick
refresh.... I have an app that does not have many concurrent users at
this time. However it does periodically have a very large number of
concurrent requests (AJAX/REST) from a single client.
I now realize that with the browser limiting concurrent requests in
HTTP/1.1, mod_jk was not getting overwhelmed. But now that the
floodgates have opened with HTTP/2, I realize that I have some

serious

Post by Jerry Malcolm
configuration problems with mod_jk. The requests are clogging up in
mod_jk, timing out, and logging errors something like "cannot get
endpoint for worker". It then takes about 5 minutes for the clog to
clear out before mod_jk is functional again for the other virtual

hosts

Post by Jerry Malcolm
I'm hosting.
Specifics... I'm using Apache 2.4.33, Tomcat 9.0. I tried to

download

Post by Jerry Malcolm
the latest mod_jk.so. The latest mod_jk for httpd on the binaries
download site is from 2016. All the versions for the past 2 years

are

Post by Jerry Malcolm
iis only... I assume that's ok??? So I'm using the 2016 1.2.40

version.

Post by Jerry Malcolm
My question is.... how do I increase the number of 'endpoints' that
mod_jk has access to? Is that a TC configuration or

workers.properties

Post by Jerry Malcolm
setting? I want to make sure mod_jk has enough resources/endpoints

to

Post by Jerry Malcolm
pass (i.e.not 'block') all the requests it receives. Once mod_jk is
passing everything, I realize the flood might hit TC, and I may be

back

Post by Jerry Malcolm
asking about increasing TC resources. But for now, I just want to

get

Post by Jerry Malcolm
mod_jk opened up.

We need to see the configuration for:

- httpd mpm / threads / processes
- workers.proprties
- Tomcat's AJP connector

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Rainer Jung]

Post by Mark Thomas

Post by Greg Huber

Post by Jerry Malcolm
So I'm using the 2016 1.2.40 version.

Think you have to build mod_jk to use the latest and greatest,

Correct.
We stopped providing binary builds for anything other than Windows because there were just too many combinations. And it is relatively simple to build on Linux.
We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).

Post by Greg Huber
this is
what
mod_jk.so build
yum install httpd-devel
download tomcat-connectors-x.x.xx-src.tar.gz
extract and cd into the native folder
cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
./configure --with-apxs=/usr/bin/apxs
make
copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
make clean
(I only have a pretty much standard workers.properties so can't help much.)

Post by Jerry Malcolm
Thanks for the responses to my earlier post about websockets/JSPs.
Http/2 is definitely the right answer for my issue there.
HTTP/2 solved the problem with the throttling the multiple parallel
requests with the browser connection limits. But now opening that
throttle has now exposed a different problem at the server. Quick
refresh.... I have an app that does not have many concurrent users at
this time. However it does periodically have a very large number of
concurrent requests (AJAX/REST) from a single client.
I now realize that with the browser limiting concurrent requests in
HTTP/1.1, mod_jk was not getting overwhelmed. But now that the
floodgates have opened with HTTP/2, I realize that I have some

serious

Post by Jerry Malcolm
configuration problems with mod_jk. The requests are clogging up in
mod_jk, timing out, and logging errors something like "cannot get
endpoint for worker". It then takes about 5 minutes for the clog to
clear out before mod_jk is functional again for the other virtual

hosts

Post by Jerry Malcolm
I'm hosting.
Specifics... I'm using Apache 2.4.33, Tomcat 9.0. I tried to

download

Post by Jerry Malcolm
the latest mod_jk.so. The latest mod_jk for httpd on the binaries
download site is from 2016. All the versions for the past 2 years

are

Post by Jerry Malcolm
iis only... I assume that's ok??? So I'm using the 2016 1.2.40

version.

Post by Jerry Malcolm
My question is.... how do I increase the number of 'endpoints' that
mod_jk has access to? Is that a TC configuration or

workers.properties

Post by Jerry Malcolm
setting? I want to make sure mod_jk has enough resources/endpoints

to

Post by Jerry Malcolm
pass (i.e.not 'block') all the requests it receives. Once mod_jk is
passing everything, I realize the flood might hit TC, and I may be

back

Post by Jerry Malcolm
asking about increasing TC resources. But for now, I just want to

get

Post by Jerry Malcolm
mod_jk opened up.

- httpd mpm / threads / processes
- workers.proprties
- Tomcat's AJP connector

Plus:

mod_http2 uses additional threads per Apache httpd process. Look for
H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
default it seems you get ThreadsPerChild normal http worker threads plus
ThreadsPerChild HTTP/2 worker threads.

So the old-style assumption of mod_jk, that it is enough to provide as
many backend connections per httpd process, as normal httpd worker
threads are configured (ThreadsPerChild), does no longer apply for HTTP/2.

You could try increasing the mod_jk pool size (connection_pool_size) to
2*ThreadsPerChild, but you might also want to consider switching to
mod_proxy.

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Jerry Malcolm]

Post by Rainer Jung
mod_http2 uses additional threads per Apache httpd process. Look for
H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
default it seems you get ThreadsPerChild normal http worker threads
plus ThreadsPerChild HTTP/2 worker threads.
So the old-style assumption of mod_jk, that it is enough to provide as
many backend connections per httpd process, as normal httpd worker
threads are configured (ThreadsPerChild), does no longer apply for HTTP/2.
You could try increasing the mod_jk pool size (connection_pool_size)
to 2*ThreadsPerChild, but you might also want to consider switching to
mod_proxy.
Regards,
Rainer

Rainer,

I've done some reading up on mpm, http/2, etc.  But it's given me more
questions than answers.  I need a bit of education help sorting this out.

Back to the original problem.  I need to understand these mod_jk log
entries (there are hundreds of them...):

[Sun Oct 21 17:15:19 2018] [10116:5456] [warn]
ajp_get_endpoint::jk_ajp_common.c (3229): Unable to get the free
endpoint for worker worker1 from 64 slots
[Sun Oct 21 17:15:19 2018] [10116:5456] [error] jk_handler::mod_jk.c
(2744): Could not get endpoint for worker=worker1
[Sun Oct 21 17:15:19 2018] [10116:5456] [info] jk_handler::mod_jk.c
(2806): Service error=0 for worker=worker1

a) what does a 'free endpoint' represent?  Is it an httpd thread that it
can't get?  Is it a connection to Tomcat?  Something else?

b) it says it has 64 of whatever it's looking for.... what is a 'slot'?

I currently have no mpm enabled.  I am running on WAMP (May2018
download).   From what I can tell after much digging, Apache Windows
does not use prefork, worker, or event mpm and rather has it's own WinNT
mpm.  The default config file has only two parms for WinNT. But this mpm
config file is commented out by default.  I'll enable it.

<IfModule mpm_winnt_module>
    ThreadsPerChild        150
    MaxConnectionsPerChild   0
</IfModule>

You suggested increasing connection_pool_size.  Is there a way to see
what pool size is set to by default?  The docs talk about iis, iPlanet,
etc defaults.  But it's not clear what it's set to for my default
config.  Is there some kind of config dump option for mod_jk?

You also suggested moving to mod_proxy.  You really lost me there. I
found info on how to config proxy, etc.  But I couldn't find anything
about how proxy differs from non-proxy.  What problem does it solve? 
And how would it potentially fix my issue?

Sorry for all the questions.  But I'm very concerned about just shooting
in the dark making all kinds of changes when I really don't know if and
how it's helping or hurting.  Any education to help me understand what
the fundamental problem is with 'endpoints' in the mod_jk log will be
greatly appreciated.

Thx.

Jerry

[Rainer Jung]

Post by Jerry Malcolm

Post by Rainer Jung
mod_http2 uses additional threads per Apache httpd process. Look for
H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
default it seems you get ThreadsPerChild normal http worker threads
plus ThreadsPerChild HTTP/2 worker threads.
So the old-style assumption of mod_jk, that it is enough to provide as
many backend connections per httpd process, as normal httpd worker
threads are configured (ThreadsPerChild), does no longer apply for HTTP/2.
You could try increasing the mod_jk pool size (connection_pool_size)
to 2*ThreadsPerChild, but you might also want to consider switching to
mod_proxy.
Regards,
Rainer

Rainer,
I've done some reading up on mpm, http/2, etc.  But it's given me more
questions than answers.  I need a bit of education help sorting this out.
Back to the original problem.  I need to understand these mod_jk log
[Sun Oct 21 17:15:19 2018] [10116:5456] [warn]
ajp_get_endpoint::jk_ajp_common.c (3229): Unable to get the free
endpoint for worker worker1 from 64 slots

That means the size of the per process connection pool used to connect
to Tomcat "worker1" is 64 and all 64 connections were busy when another
one was needed. Probably this 64 was not configured by you but instead
auto-sized via ThreadsPerChild (MPM sizing). From what I understand
about the HTTP/2 implementation by mod_http2, you should try in doubling
the number of endpoints by setting
worker.worker1.connection_pool_size=128 in workers.properties without
changing ThreadsPerChild. Some info on the param can be found at
http://tomcat.apache.org/connectors-doc/reference/workers.html.

What you need to keep in mind: it is also possible, that the real
culprit is, that already 64 connections from one Apache process to the
backend were busy. That could mean, that the requests became too slow in
Tomcat or more likely in your application running in Tomcat. So I would
suggest you also take thread dumps, in this case of Tomcat worker1, when
the above message happens frequently to find out, what those 64
concurrent requests are doing (that use up all 64 possible connections).

Post by Jerry Malcolm
[Sun Oct 21 17:15:19 2018] [10116:5456] [error] jk_handler::mod_jk.c
(2744): Could not get endpoint for worker=worker1
[Sun Oct 21 17:15:19 2018] [10116:5456] [info] jk_handler::mod_jk.c
(2806): Service error=0 for worker=worker1

These two are just follow-up messages of the above.

Post by Jerry Malcolm
a) what does a 'free endpoint' represent?  Is it an httpd thread that it
can't get?  Is it a connection to Tomcat?  Something else?

Connection from a specific Apache httpd process to Tomcat. These
endpoints=connections are pooled per Apache process. The number of
connections can be configured via worker.WORKER_NAME.connection_pool_size.

Post by Jerry Malcolm
b) it says it has 64 of whatever it's looking for.... what is a 'slot'?

Just a synonym for endpoint. By slot in that message is meant an item in
the pool.

Post by Jerry Malcolm
I currently have no mpm enabled.  I am running on WAMP (May2018
download).   From what I can tell after much digging, Apache Windows
does not use prefork, worker, or event mpm and rather has it's own WinNT
mpm.  The default config file has only two parms for WinNT. But this mpm
config file is commented out by default.  I'll enable it.

Ah, Windows. So unlike Unix/Linux we do only have a single child
process, which does all the work. The number 64 is the default number of
threads in this child for Windows.

Post by Jerry Malcolm
<IfModule mpm_winnt_module>
    ThreadsPerChild        150
    MaxConnectionsPerChild   0
</IfModule>

OK, after that you have 150 threads in the one and only Apache httpd
child process, that does all the work. mod_jk will automatically also
set the number of endpoints (slots) = the connection_pool_size to 150,
if not configured by you otherwise.

Post by Jerry Malcolm
You suggested increasing connection_pool_size.  Is there a way to see
what pool size is set to by default?  The docs talk about iis, iPlanet,
etc defaults.  But it's not clear what it's set to for my default
config.  Is there some kind of config dump option for mod_jk?

Configure a status worker. It has plenty of interesting info. For
monitoring it is enough to configure a read-only status worker. See
http://tomcat.apache.org/connectors-doc/reference/status.html.

Post by Jerry Malcolm
You also suggested moving to mod_proxy.  You really lost me there. I
found info on how to config proxy, etc.  But I couldn't find anything
about how proxy differs from non-proxy.  What problem does it solve? And
how would it potentially fix my issue?

mod_jk and mod_proxy are simply two different ways of connecting Apache
httpd to backend Tomcats. mod_jk uses a special protocol named AJP13
(and an AJP connector in Tomcat which understands that protocol).
mod_proxy simply uses HTTP or HTTPS and the normal HTTP/HTTPS connector
in Tomcat. mod_jk is provided by the Tomcat project and configured
mostly via workers.properties, mod_proxy is provided as part of Apache
httpd and configured via "normal" Proxy... httpd directives.

Post by Jerry Malcolm
Sorry for all the questions.  But I'm very concerned about just shooting
in the dark making all kinds of changes when I really don't know if and
how it's helping or hurting.  Any education to help me understand what
the fundamental problem is with 'endpoints' in the mod_jk log will be
greatly appreciated.

Now that I understand that you are on Windows, the default size of 64
woker threads in total for Apache httpd is not that big. So i suggest

- enable the explicit config with 150 as you already did

- increase connection_pool_size to 300 if you are actually using HTTP/2

- enable a read-only status worker for monitoring. The number of busy
endpoints can be seen under the busy column, the max number since start
or statistics reset under max_busy. There are also the number of
connections and max number of connections since start/statistics reset.
The difference between busy and number of connections is, that a
connection is counted in busy only, if it is currently being used for a
request/response.

- If you are till getting errors take a few thread dumps of your backend
Tomcats during the time you get errors and provide them to the mailing
list, possibly after redacting things there, that you don't want public.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Jerry Malcolm]

Post by Mark Thomas
- httpd mpm / threads / processes
- workers.proprties
- Tomcat's AJP connector
Mark

I'm not sure where to find httpd mpm / threads / processes config.
Nothing like that in my httpd.conf

----------- workers.properties:
worker.list=jk-status
worker.jk-status.type=status
worker.jk-status.read_only=true
worker.list=jk-manager
worker.jk-manager.type=status
worker.list=worker1
# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.socket_keepalive=1
worker.worker1.socket_timeout=300
worker.worker1.reply_timeout=12000000

-------------- httpd (JK section)
LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkShmFile logs/mod_jk.shm
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"



---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Jerry Malcolm]

Post by Mark Thomas
We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).

Mark,

Does you mean that I can use the Windows binary dll that is in the
mod_jk iis binaries download if I'm running Apache and Tomcat on a Win
Server platform instead of using mod_jk.so?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Rainer Jung]

Post by Jerry Malcolm

Post by Mark Thomas
We provide the Windows binaries because the build process is more
involved, most users don't have the necessary toolchain  easily
available and we only need to provide one binary for all Windows
platforms (ok we provide several binaries with different build options
but it is still a small, manageable number).

Mark,
Does you mean that I can use the Windows binary dll that is in the
mod_jk iis binaries download if I'm running Apache and Tomcat on a Win
Server platform instead of using mod_jk.so?

No, IIS binaries can not be used as module binaries for Apache httpd on
Windows. But you typically get module binaries from the same sources
that provide Apache httpd on Windows for you. The problem there is, that
the module and the web server must be build with compatibel compiler
setups. Therefore it is best to get them from the same providers (like
Apache Lounge or others).

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org