Cluster more than 2 tomcat servers

[Tim K]

I'm experimenting with Tomcat 9.0.8 enabling Cluster support. I've
uncommented the line for it within the server.xml. I have 4 separate
servers on different hosts, all with the same config, except unique
jvmRoute values. Upon startup, it appears that server1 and server4 pair up
and server2 and server3 pair up, every time! I'm using the default udp
multicast, again, just uncommenting the single cluster line in server.xml.
I'm expecting that all 4 would see each other and I would have a single
cluster of 4, not 2 clusters of 2. I confirmed each server can see each
other over port 4000. Any help is appreciated.

Regards,
Tim

[Christopher Schultz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Tim,

Post by Tim K
I'm experimenting with Tomcat 9.0.8 enabling Cluster support.
I've uncommented the line for it within the server.xml. I have 4
separate servers on different hosts, all with the same config,
except unique jvmRoute values. Upon startup, it appears that
server1 and server4 pair up and server2 and server3 pair up, every
time! I'm using the default udp multicast, again, just
uncommenting the single cluster line in server.xml. I'm expecting
that all 4 would see each other and I would have a single cluster
of 4, not 2 clusters of 2. I confirmed each server can see each
other over port 4000. Any help is appreciated.

Can you post your configuration(s)? Presumably, all 4 are nearly
identical. Please post one of them (minus any secrets) and describe or
copy/paste the differences for the other notes.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAluEhnEACgkQHPApP6U8
pFjuERAAyo7/4u+/ZXkQ1XAK0YYPey6bNI4m10A+D+7Db+UkPfuP7rNo4g8z6LsL
qroo8ZHw3BCi2gmIA0HkWsoaWnsEnOXU2OdvbOtWZn4Eqpys4w8dpQqXJY9ZwwKK
ivaOiL3i37mqb4o7CTM3fU6MDpHILOkmOiuxT/n/uzW4gWqzrijm23rnG7wxR9Nw
KA7N6mRkUL4AqnTj1MNI3yfrCDk/rvVPMBtC7jvT4hkKrI+FNTEBdlUtFJAw5xSo
lkYrNOKJz73G/0Bje2Bi9vk7UB05WF0oSJgYFsiersrl1/eDTRIkcPy+AokU5P4p
3yFxvurTDGD2Q9GDloKPPxGwMZV/vB3HvY/BwsWzKTjXDy24AksuRqLJAo/mM0Ul
8cXzuHPbkG48MNcBwQJBB3DJmJNVmi5W5/1o++M9z3r55ym8cEvwP359j7cuT6V0
ebGXqk7kCfthjUpn3PQYCMzTpUDISd8N+4+7YH+yyhe8P9whU+Rbe5mkL2AkkBLP
Lt3Pdfvt7utpjxw2OdiVzDaRhmm9735iev9FPOaa5T7VSqW8FMDkx8e2GSDPfzfr
L3hFNW13iZ71HJe3a3J3NA8apyTr615RaPZeKHOr0Gpmpx3rG22KDJxOUAIQGZND
Aa9CGYUwAuNObCzpuI1rEDHRJN6uwUMX00J6T2hBs2NmAlfwzL4=
=A6ss
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Tim K]

Post by Christopher Schultz
Can you post your configuration(s)? Presumably, all 4 are nearly
identical. Please post one of them (minus any secrets) and describe or
copy/paste the differences for the other notes.
- -chris

Here is my cluster config that I got working, all 4 nodes can talk to each
other after I added them as StaticMembers (without this, the mcast appeared
that 2 of them would pair up with each other).

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">

<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/>

<Channel className="org.apache.catalina.tribes.group.GroupChannel">

<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100" selectorTimeout="5000"
maxThreads="6"/>

<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>

</Sender>

<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor"/>

<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>

<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>

<Interceptor
className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">

<Member
className="org.apache.catalina.tribes.membership.StaticMember"
securePort="-1" port="4000" host="X.X.X.90" domain="dblue-cluster"
uniqueId="{1,0,2,3,4,5,6,7,8,9,10,11,12,13,14,15}"/>

<Member
className="org.apache.catalina.tribes.membership.StaticMember"
securePort="-1" port="4000" host="X.X.X.91" domain="dblue-cluster"
uniqueId="{2,0,2,3,4,5,6,7,8,9,10,11,12,13,14,15}"/>

<Member
className="org.apache.catalina.tribes.membership.StaticMember"
securePort="-1" port="4000" host="X.X.X.92" domain="dblue-cluster"
uniqueId="{3,0,2,3,4,5,6,7,8,9,10,11,12,13,14,15}"/>

</Interceptor>

</Channel>

<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>

<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>

</Cluster>

I now was interested in setting these up with SSL (securePorts), so for
both the NioReceiver and the StaticMember(s), I removed port="4000" in
favor of securePort="4000" (I did replace the securePort="-1" in the
StaticMembers). In doing this and starting them all up, it appears to act
the same as it did with mcast without StaticMembers, 2 of them pair up
instead of all 4 joining the single cluster. I don't understand why this
is happening.

[Jäkel, Guido]

Dear Tim,

maybe you can do an "independent" check using the JGroups lib stand-alone, which offers two sets of a cluster demo. This may help to narrow down the issue to the Tomcat setup or the "other" (OS or network configuration) areas.

Please refer to http://www.jgroups.org/tutorial-3.x/html_single/ , section "1.4. Running a Demo Program"

You just need the lib and a JVM, i.e. you can use it out of the box at your Tomcat machines. The "Draw Demo" requires Graphics, but you also may use another Testcase offering a textmode receiver and sender. You may start this receivers on every machine and then one or more senders on demand.

On both demo's, cluster status messages are displayed on stdout.


Here my somewhat older starter scripts:


***@genless /opt/jgroups # cat _receiver
( cd `dirname $0` || exit 1
java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555 $@
)

***@genless /opt/jgroups # cat _sender
( cd `dirname $0` || exit 1
java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555 $@
)

***@genless /opt/jgroups # cat _draw-demo
( cd `dirname $0` || exit 1
java -Djava.net.preferIPv4Stack=true -cp jgroups.jar org.jgroups.demos.Draw $@
)

***@genless /opt/jgroups # tree
.
|-- _draw-demo
|-- _receiver
|-- _sender
|-- _src
| |-- jgroups-2.9.0.GA.jar
| `-- jgroups-3.0.10.Final.jar
`-- jgroups.jar -> _src/jgroups-3.0.10.Final.jar


Yours

Guido
Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âW6W'2�V�7V'67&�&TF��6B�6�R��&pФf�"FF�F����6����G2�R�

[Tim K]

Post by Jäkel, Guido
Dear Tim,
maybe you can do an "independent" check using the JGroups lib stand-alone,
which offers two sets of a cluster demo. This may help to narrow down the
issue to the Tomcat setup or the "other" (OS or network configuration)
areas.
Please refer to http://www.jgroups.org/tutorial-3.x/html_single/ ,
section "1.4. Running a Demo Program"
You just need the lib and a JVM, i.e. you can use it out of the box at
your Tomcat machines. The "Draw Demo" requires Graphics, but you also may
use another Testcase offering a textmode receiver and sender. You may start
this receivers on every machine and then one or more senders on demand.
On both demo's, cluster status messages are displayed on stdout.
( cd `dirname $0` || exit 1
java -cp jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr
)
( cd `dirname $0` || exit 1
java -cp jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr
)
( cd `dirname $0` || exit 1
java -Djava.net.preferIPv4Stack=true -cp jgroups.jar
)
.
|-- _draw-demo
|-- _receiver
|-- _sender
|-- _src
| |-- jgroups-2.9.0.GA.jar
| `-- jgroups-3.0.10.Final.jar
`-- jgroups.jar -> _src/jgroups-3.0.10.Final.jar
Yours
Guido

Thank you Guido.

That appears to be using multicast which I'm not using at the moment, I'm
trying to use static members with the secure ports. Everything works fine
with my config if I don't use the secure ports... So I would think that
would rule out OS or Networking issues...