Discussion:
JVM keystores and CA
Igor Cicimov
2018-10-15 23:49:08 UTC
Permalink
Hi all,

I just want to clarify something that I've been seeing behave differently
on various Java versions during the years. In case we have the following
setting:

-Djavax.net.ssl.trustStore=/keystore/truststore.jks"

in Tomcat's default config file, is JVM suppose to fall back to the global
CA store on the server under /etc/ssl/certs for verification in case *any*
of the certificates returned by a trusted domain are not present in the
above JKS store? By any I mean, all the certs in the chain returned by the
SSL handshake.

For example, lets say we have a situation like this:

DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com

Lets say I have imported the *CN=*.mydomain.com <http://mydomain.com>*
certificate in the truststore.jks, the question is is JVM going to look
under /etc/ssl/certs for RapidSSL in order to validate the CN signature and
then for DigiCert to validate the RapidSSL cert?

As I said I had a mixed luck with this over the years, sometimes it works
as (I) expect it to work i.e. verify the certs by looking at the system CA
store and sometimes this is not the case.

Thanks for any help/insites.

Igor
Jose María Zaragoza
2018-10-16 06:52:10 UTC
Permalink
Hi
Post by Igor Cicimov
Hi all,
I just want to clarify something that I've been seeing behave differently
on various Java versions during the years. In case we have the following
-Djavax.net.ssl.trustStore=/keystore/truststore.jks"
in Tomcat's default config file, is JVM suppose to fall back to the global
CA store on the server under /etc/ssl/certs for verification in case *any*
of the certificates returned by a trusted domain are not present in the
above JKS store? By any I mean, all the certs in the chain returned by the
SSL handshake.
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com
Lets say I have imported the *CN=*.mydomain.com <http://mydomain.com>*
certificate in the truststore.jks, the question is is JVM going to look
under /etc/ssl/certs for RapidSSL in order to validate the CN signature and
then for DigiCert to validate the RapidSSL cert?
As I said I had a mixed luck with this over the years, sometimes it works
as (I) expect it to work i.e. verify the certs by looking at the system CA
store and sometimes this is not the case.
My experience is that if you store a server certificate in the
truststore.jks , don't search anyone more.
This behaviour makes sense for me because you are saying that you
trust in that certificate because you verified ( by other means ) it
before
Even if certificate is expired ( and server send it expired too,
obviously ) , I think that is validated as trusted

Regards
Post by Igor Cicimov
Thanks for any help/insites.
Igor
---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
Igor Cicimov
2018-10-16 09:56:47 UTC
Permalink
Hi Jose,
Post by Igor Cicimov
Hi
Post by Igor Cicimov
Hi all,
I just want to clarify something that I've been seeing behave differently
on various Java versions during the years. In case we have the following
-Djavax.net.ssl.trustStore=/keystore/truststore.jks"
in Tomcat's default config file, is JVM suppose to fall back to the
global
Post by Igor Cicimov
CA store on the server under /etc/ssl/certs for verification in case
*any*
Post by Igor Cicimov
of the certificates returned by a trusted domain are not present in the
above JKS store? By any I mean, all the certs in the chain returned by
the
Post by Igor Cicimov
SSL handshake.
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com
Lets say I have imported the *CN=*.mydomain.com <http://mydomain.com>*
certificate in the truststore.jks, the question is is JVM going to look
under /etc/ssl/certs for RapidSSL in order to validate the CN signature
and
Post by Igor Cicimov
then for DigiCert to validate the RapidSSL cert?
As I said I had a mixed luck with this over the years, sometimes it works
as (I) expect it to work i.e. verify the certs by looking at the system
CA
Post by Igor Cicimov
store and sometimes this is not the case.
My experience is that if you store a server certificate in the
truststore.jks , don't search anyone more.
That's my experience too ... most of the time. But what happens when lets
say the domain returns multiple certificates in the handshake, like the
intermediate and the domain certificate? For the example above I gave:

DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com

what if what is sent back are the Intermediate (RapidSSL TLS RSA CA G1) and
the domain one (*.mydomain.com) cert in the chain? Is Java going to need to
validate the Intermediate cert now too? There is only the domain cert in
the truststore so how is it going to validate the Intermediate in this
case? Is the Intermediate ignored maybe since the domain one validates
anyway?

This behaviour makes sense for me because you are saying that you
Post by Igor Cicimov
trust in that certificate because you verified ( by other means ) it
before
Even if certificate is expired ( and server send it expired too,
obviously ) , I think that is validated as trusted
Regards
Igor Cicimov
2018-10-16 21:03:03 UTC
Permalink
Post by Igor Cicimov
Hi Jose,
Post by Igor Cicimov
Hi
Post by Igor Cicimov
Hi all,
I just want to clarify something that I've been seeing behave
differently
Post by Igor Cicimov
on various Java versions during the years. In case we have the following
-Djavax.net.ssl.trustStore=/keystore/truststore.jks"
in Tomcat's default config file, is JVM suppose to fall back to the
global
Post by Igor Cicimov
CA store on the server under /etc/ssl/certs for verification in case
*any*
Post by Igor Cicimov
of the certificates returned by a trusted domain are not present in the
above JKS store? By any I mean, all the certs in the chain returned by
the
Post by Igor Cicimov
SSL handshake.
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com
Lets say I have imported the *CN=*.mydomain.com <http://mydomain.com>*
certificate in the truststore.jks, the question is is JVM going to look
under /etc/ssl/certs for RapidSSL in order to validate the CN signature
and
Post by Igor Cicimov
then for DigiCert to validate the RapidSSL cert?
As I said I had a mixed luck with this over the years, sometimes it
works
Post by Igor Cicimov
as (I) expect it to work i.e. verify the certs by looking at the system
CA
Post by Igor Cicimov
store and sometimes this is not the case.
My experience is that if you store a server certificate in the
truststore.jks , don't search anyone more.
That's my experience too ... most of the time. But what happens when lets
say the domain returns multiple certificates in the handshake, like the
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 -> CN=*.mydomain.com
what if what is sent back are the Intermediate (RapidSSL TLS RSA CA G1)
and the domain one (*.mydomain.com) cert in the chain? Is Java going to
need to validate the Intermediate cert now too? There is only the domain
cert in the truststore so how is it going to validate the Intermediate in
this case? Is the Intermediate ignored maybe since the domain one validates
anyway?
This behaviour makes sense for me because you are saying that you
Post by Igor Cicimov
trust in that certificate because you verified ( by other means ) it
before
Even if certificate is expired ( and server send it expired too,
obviously ) , I think that is validated as trusted
Regards
To make it more clear, with a custom trusted keystore how do the certs that
are not trusted get validated? Now Java needs to validate the whole chain
so where does it look for the CAs? In its own keystore
$JAVA_HOME/jre/lib/security/cacerts or under /etc/ssl/certs ?
Christopher Schultz
2018-10-18 15:14:54 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Igor,
Post by Igor Cicimov
Post by Igor Cicimov
Hi Jose,
On Tue, Oct 16, 2018 at 5:52 PM Jose María Zaragoza
Hi
El mar., 16 oct. 2018 a las 1:49, Igor Cicimov
Post by Igor Cicimov
Hi all,
I just want to clarify something that I've been seeing
behave
differently
Post by Igor Cicimov
on various Java versions during the years. In case we have
-Djavax.net.ssl.trustStore=/keystore/truststore.jks"
in Tomcat's default config file, is JVM suppose to fall back to the
global
Post by Igor Cicimov
CA store on the server under /etc/ssl/certs for verification in case
*any*
Post by Igor Cicimov
of the certificates returned by a trusted domain are not
present in the above JKS store? By any I mean, all the certs
in the chain returned by
the
Post by Igor Cicimov
SSL handshake.
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 ->
CN=*.mydomain.com
Lets say I have imported the *CN=*.mydomain.com
<http://mydomain.com>* certificate in the truststore.jks, the
question is is JVM going to look under /etc/ssl/certs for
RapidSSL in order to validate the CN signature
and
Post by Igor Cicimov
then for DigiCert to validate the RapidSSL cert?
As I said I had a mixed luck with this over the years,
sometimes it
works
Post by Igor Cicimov
as (I) expect it to work i.e. verify the certs by looking at the system
CA
Post by Igor Cicimov
store and sometimes this is not the case.
My experience is that if you store a server certificate in the
truststore.jks , don't search anyone more.
That's my experience too ... most of the time. But what happens
when lets say the domain returns multiple certificates in the
handshake, like the intermediate and the domain certificate? For
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 ->
CN=*.mydomain.com
what if what is sent back are the Intermediate (RapidSSL TLS RSA
CA G1) and the domain one (*.mydomain.com) cert in the chain? Is
Java going to need to validate the Intermediate cert now too?
There is only the domain cert in the truststore so how is it
going to validate the Intermediate in this case? Is the
Intermediate ignored maybe since the domain one validates
anyway?
This behaviour makes sense for me because you are saying that
you
trust in that certificate because you verified ( by other means
) it before Even if certificate is expired ( and server send it
expired too, obviously ) , I think that is validated as
trusted
Regards
To make it more clear, with a custom trusted keystore how do the
certs that are not trusted get validated? Now Java needs to
validate the whole chain so where does it look for the CAs? In its
own keystore $JAVA_HOME/jre/lib/security/cacerts or under
/etc/ssl/certs ?
Java has no notion of CAs, nor does any trust store, really. A
certificate is trusted if it is present in the trust store, full stop.
It not need be a "CA". The oly thing being a CA gets you is ... in
everyone's default trust stores.

The system property javax.net.ssl.trustStore only sets the default
trust store for the JVM and any components which choose to use it. For
example, if you use HttpURLConnection without any explicit
configuration, it will use that. Same with Apache httpclient.

But both of those can be configured to use a different trust store,
which case they will *not* fall-back to the built-in trust store (the
one in JAVA_HOME/lib/security/cacerts.

When overriding the default trust store for the JVM, the trust store
you specify should be the ONLY trust store consulted. It should not
fall back. I can confirm this is the case on Java 8 - 11, at least the
ones I happen to be using. Any other behavior would be a security proble
m.

The proper way to validate a certificate chain is to perform the
following algorithm:

0. Start with the server's certificate (the leaf)
1. Is the certificate in the trust store?
Yes: chain is valid; stop
2. Is the certificate signed by a cert in the trust store?
Yes: chain is valid; stop
3. Is the certificate signed by the next cert in the chain?
No: chain is invalid; stop
4. Move to the next cert in the chain
5. Go to step 1

So if you use an empty trust store and try to connect to
https://www.google.com, you should find that you get an exception
thrown. Something like this:

Exception in thread "main" javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvIo24ACgkQHPApP6U8
pFgJSBAAw/Q8vRyj5w4cB0cBx8EeNpBamr89jVvNQj5jHeuXOn6htATILjCQZpm9
FL1Ikn5lxkWvMn1Ru+x7br3F3cxTdkVvRWXH4KcksVh6KkPGcp0PEogl7sscjf4X
pSpnYocNKiCBfnlXG3bJV2u4VS4J9m9pXAh9proxnBxY1fg5NgtUp+uq4kF4q1z+
uopesloM9RdKoLREJVjXsZemBXQyF3tKGziOb0+u3Meq1eG+N+P+gSZr7AsQndCx
KQmdlKK5NyD8NhrCi8EGfGR8992JgrsvOG43gPokqfzOJ0IHXZs6vrCU8yrl7KVX
RcKtc+mBI+8biAW+41ut5cPIqM5p7h9ZMCn9KH7vLz26HuFeQYhhE+WOQ8QRt07A
QiNL2uqIbBG1wVaBzmB664wu0ifbWdUfzVq39T56Rzd9HUFDn26GaO7+IO4kfdpt
UdklRvf+Ldw/SCuXYtTI7nArrhfCFN0ALUVBkY81hUcdKNXdWrE7DEp0KLm7LD61
aFl3M/HWNSSJnGvgyT5OJ1Q1Z8wVaisccPe9vBDJ6EmE9u868uPkquY6I6jnsC1X
4Q9wGzf3OsPYQ0vzSNhN1At8kMHj6C/5RRTVAUIXW1M3FE9YiiELy25CWOdaF95f
aFNRSoiEHzRTWcAVh8M5KXb0G3rtq8b6WjWgMCPJ5Kh33uRShWM=
=g3kh
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
Igor Cicimov
2018-10-18 23:09:16 UTC
Permalink
Hi Chris,

On Fri, Oct 19, 2018 at 2:14 AM Christopher Schultz <
Post by Christopher Schultz
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Igor,
Post by Igor Cicimov
Post by Igor Cicimov
Hi Jose,
On Tue, Oct 16, 2018 at 5:52 PM Jose María Zaragoza
Hi
El mar., 16 oct. 2018 a las 1:49, Igor Cicimov
Post by Igor Cicimov
Hi all,
I just want to clarify something that I've been seeing
behave
differently
Post by Igor Cicimov
on various Java versions during the years. In case we have
-Djavax.net.ssl.trustStore=/keystore/truststore.jks"
in Tomcat's default config file, is JVM suppose to fall back to the
global
Post by Igor Cicimov
CA store on the server under /etc/ssl/certs for verification in case
*any*
Post by Igor Cicimov
of the certificates returned by a trusted domain are not
present in the above JKS store? By any I mean, all the certs
in the chain returned by
the
Post by Igor Cicimov
SSL handshake.
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 ->
CN=*.mydomain.com
Lets say I have imported the *CN=*.mydomain.com
<http://mydomain.com>* certificate in the truststore.jks, the
question is is JVM going to look under /etc/ssl/certs for
RapidSSL in order to validate the CN signature
and
Post by Igor Cicimov
then for DigiCert to validate the RapidSSL cert?
As I said I had a mixed luck with this over the years,
sometimes it
works
Post by Igor Cicimov
as (I) expect it to work i.e. verify the certs by looking at the system
CA
Post by Igor Cicimov
store and sometimes this is not the case.
My experience is that if you store a server certificate in the
truststore.jks , don't search anyone more.
That's my experience too ... most of the time. But what happens
when lets say the domain returns multiple certificates in the
handshake, like the intermediate and the domain certificate? For
DigiCert Global Root G2 -> RapidSSL TLS RSA CA G1 ->
CN=*.mydomain.com
what if what is sent back are the Intermediate (RapidSSL TLS RSA
CA G1) and the domain one (*.mydomain.com) cert in the chain? Is
Java going to need to validate the Intermediate cert now too?
There is only the domain cert in the truststore so how is it
going to validate the Intermediate in this case? Is the
Intermediate ignored maybe since the domain one validates
anyway?
This behaviour makes sense for me because you are saying that you
trust in that certificate because you verified ( by other means
) it before Even if certificate is expired ( and server send it
expired too, obviously ) , I think that is validated as
trusted
Regards
To make it more clear, with a custom trusted keystore how do the
certs that are not trusted get validated? Now Java needs to
validate the whole chain so where does it look for the CAs? In its
own keystore $JAVA_HOME/jre/lib/security/cacerts or under
/etc/ssl/certs ?
Java has no notion of CAs, nor does any trust store, really.
Correct, but by loading all CAs in the trust store it kinda does,
indirectly.

A
Post by Christopher Schultz
certificate is trusted if it is present in the trust store, full stop.
It not need be a "CA". The oly thing being a CA gets you is ... in
everyone's default trust stores.
The system property javax.net.ssl.trustStore only sets the default
trust store for the JVM and any components which choose to use it. For
example, if you use HttpURLConnection without any explicit
configuration, it will use that. Same with Apache httpclient.
But both of those can be configured to use a different trust store,
which case they will *not* fall-back to the built-in trust store (the
one in JAVA_HOME/lib/security/cacerts.
Well I see couple of issues with this approach of the trsutstore being the
only source of truth. First is the obvious one, when using a custom trust
store
I have to load *all* CA certificates that already exist somewhere else on
the
server (and in multiple places) in the trust store too otherwise no
certificate will
ever get validated.
Post by Christopher Schultz
When overriding the default trust store for the JVM, the trust store
you specify should be the ONLY trust store consulted. It should not
fall back. I can confirm this is the case on Java 8 - 11, at least the
ones I happen to be using. Any other behavior would be a security proble
m.
Not sure I can agree with this reasoning too. All apps on the server use
the default
system CA store so should we consider them insecure? I see no harm of Java
looking
in the default location(s) on the server when a cert can not be validated
by looking in
the trust store. Otherwise as noted above in case of custom trust store we
need to load
all those certificates anyway ending up with same certificates stored in
multiple places,
making the size of the trust store unnecessary big.
Post by Christopher Schultz
The proper way to validate a certificate chain is to perform the
0. Start with the server's certificate (the leaf)
1. Is the certificate in the trust store?
Yes: chain is valid; stop
2. Is the certificate signed by a cert in the trust store?
Yes: chain is valid; stop
3. Is the certificate signed by the next cert in the chain?
No: chain is invalid; stop
4. Move to the next cert in the chain
5. Go to step 1
So if you use an empty trust store and try to connect to
https://www.google.com, you should find that you get an exception
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target
To conclude, the way I would expect the trust store to be used and the
whole
validation done:

1. I use custom trust store because I need to load self signed certificates
that
I need to validate when connecting to lets say partner APIs that use self
signed
certificates and I know I can trust
2. I would expect nothing else needed in this store as every other valid
certificate
under the sun is already located in default locations on the server Java is
running on
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the default) I
would
expect Java to use the system store(s) too in case a certificate can not be
validated
simply because a CA is missing in the Java store. Example, DigiCert Global
Root G2
CA is missing in the Java versions older than 8u91 causing inexplicable
PKIX exceptions
but can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates
which are (much) more frequently updated with new certs than Java versions.
This actually
applies to the case of custom trust store even more so

Thoughts?

Cheers,
Igor

- -chris
Post by Christopher Schultz
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvIo24ACgkQHPApP6U8
pFgJSBAAw/Q8vRyj5w4cB0cBx8EeNpBamr89jVvNQj5jHeuXOn6htATILjCQZpm9
FL1Ikn5lxkWvMn1Ru+x7br3F3cxTdkVvRWXH4KcksVh6KkPGcp0PEogl7sscjf4X
pSpnYocNKiCBfnlXG3bJV2u4VS4J9m9pXAh9proxnBxY1fg5NgtUp+uq4kF4q1z+
uopesloM9RdKoLREJVjXsZemBXQyF3tKGziOb0+u3Meq1eG+N+P+gSZr7AsQndCx
KQmdlKK5NyD8NhrCi8EGfGR8992JgrsvOG43gPokqfzOJ0IHXZs6vrCU8yrl7KVX
RcKtc+mBI+8biAW+41ut5cPIqM5p7h9ZMCn9KH7vLz26HuFeQYhhE+WOQ8QRt07A
QiNL2uqIbBG1wVaBzmB664wu0ifbWdUfzVq39T56Rzd9HUFDn26GaO7+IO4kfdpt
UdklRvf+Ldw/SCuXYtTI7nArrhfCFN0ALUVBkY81hUcdKNXdWrE7DEp0KLm7LD61
aFl3M/HWNSSJnGvgyT5OJ1Q1Z8wVaisccPe9vBDJ6EmE9u868uPkquY6I6jnsC1X
4Q9wGzf3OsPYQ0vzSNhN1At8kMHj6C/5RRTVAUIXW1M3FE9YiiELy25CWOdaF95f
aFNRSoiEHzRTWcAVh8M5KXb0G3rtq8b6WjWgMCPJ5Kh33uRShWM=
=g3kh
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
Mark H. Wood
2018-10-22 16:12:49 UTC
Permalink
On Fri, Oct 19, 2018 at 10:09:16AM +1100, Igor Cicimov wrote:
[snip]
Post by Igor Cicimov
To conclude, the way I would expect the trust store to be used and the
1. I use custom trust store because I need to load self signed certificates
that
I need to validate when connecting to lets say partner APIs that use self
signed
certificates and I know I can trust
2. I would expect nothing else needed in this store as every other valid
certificate
under the sun is already located in default locations on the server Java is
running on
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the default) I
would
expect Java to use the system store(s) too in case a certificate can not be
validated
simply because a CA is missing in the Java store. Example, DigiCert Global
Root G2
CA is missing in the Java versions older than 8u91 causing inexplicable
PKIX exceptions
but can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates
which are (much) more frequently updated with new certs than Java versions.
This actually
applies to the case of custom trust store even more so
Thoughts?
There are two ways that a truststore can be inadequate. (1) It can
lack a certificate that your application should trust. (2) It can
contain a certificate that your application should NOT trust.

Suppose that you had an application A which needs to distrust one of the
CAs (X) that are trusted by the OS maintainers. And suppose that you had
other applications on the same host which need to trust that same CA.

The way it currently works, you can copy the systemwide JRE truststore
once, remove from the copy the certificate for X, and tell A to use
the copy as its truststore. Every other application can just default
to the systemwide truststore. (This also works if you need to give A
an additional trusted CA not needed by other applications, and are
willing to trust all the other CAs.)

The way you suggest it should work, you must remove X from all
systemwide truststores of every type, and then configure a custom
truststore for every application except A.

Which is more error-prone?

It shouldn't be difficult to write a script that makes a copy of the
systemwide store and adjusts it to your application's specific needs.
--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
Jäkel, Guido
2018-10-23 08:49:15 UTC
Permalink
Dear Igor,
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the default) I would
expect Java to use the system store(s) too in case a certificate can not be validated
simply because a CA is missing in the Java store. Example, DigiCert Global
Root G2 CA is missing in the Java versions older than 8u91 causing inexplicable
PKIX exceptions but can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates which are (much) more frequently updated with new certs than Java versions.
This actually applies to the case of custom trust store even more so
Thoughts?
Because Java is platform-independent, it have an own store and don't use any of the underlying OS. But one told me that on some Linux distributions, there's a script tool to update the Java cert store from the CA-Certificates-Package. Using such a tool you get the update more frequently.

But actually you probably don't need. You wrote about using Java8u91, which is simply complete out of date because the latest is Java8u192. I you complain about security things like an outdated certificate store, you simply should install a Java version as recent as the CA-Certificates package of the hosting OS. Any you will catch other JAVA bugs and security issues by the way, too.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
Igor Cicimov
2018-10-23 09:42:43 UTC
Permalink
Hi Guido,
Post by Jäkel, Guido
Dear Igor,
Post by Igor Cicimov
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the
default) I would
Post by Igor Cicimov
expect Java to use the system store(s) too in case a certificate can
not be validated
Post by Igor Cicimov
simply because a CA is missing in the Java store. Example, DigiCert
Global
Post by Igor Cicimov
Root G2 CA is missing in the Java versions older than 8u91 causing
inexplicable
Post by Igor Cicimov
PKIX exceptions but can be found in the system store, both under
/etc/ssl/certs and
Post by Igor Cicimov
/usr/share/ca-certificates which are (much) more frequently updated
with new certs than Java versions.
Post by Igor Cicimov
This actually applies to the case of custom trust store even more so
Thoughts?
Because Java is platform-independent, it have an own store and don't use
any of the underlying OS. But one told me that on some Linux distributions,
there's a script tool to update the Java cert store from the
CA-Certificates-Package. Using such a tool you get the update more
frequently.
But actually you probably don't need. You wrote about using Java8u91,
which is simply complete out of date because the latest is Java8u192. I you
complain about security things like an outdated certificate store, you
simply should install a Java version as recent as the CA-Certificates
package of the hosting OS. Any you will catch other JAVA bugs and security
issues by the way, too.
Just to make it clear, when I mentioned Java8u91 as an example I meant for
the time when lets say Java8u81 was latest at that moment. In that case
when using the built in JVM store you would encounter failed connections to
servers with G2 signed certificates. Then as you said you would need to
import it into the JVM store by yourself or wait for the next Java release
that would include it by default i.e. Java8u91
Jäkel, Guido
2018-10-23 10:35:18 UTC
Permalink
Post by Igor Cicimov
Just to make it clear, when I mentioned Java8u91 as an example I meant for
the time when lets say Java8u81 was latest at that moment. In that case
when using the built in JVM store you would encounter failed connections to
servers with G2 signed certificates. Then as you said you would need to
import it into the JVM store by yourself or wait for the next Java release
that would include it by default i.e. Java8u91
I understand your meanings, but I wonder if the release interval of the "CA-Certs Distro package" is shorter than that from Java and first of all if it's likely that one will use a certificate chained with a "brand-new" CA that's not in the common trust stores :)

B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB��[��X��ܚX�KK[XZ[�\�\��][��X��ܚX�P�X�] �\X�K�ܙ�B��܈Y][ۘ[��[X[��K[
Jäkel, Guido
2018-10-23 11:06:05 UTC
Permalink
Dear Igor and others,

in this context maybe the attached Java program together with this script is useful for you ...


#!/bin/bash

if [ -z "$1" ]; then
cat >&2 <<-EOT
syntax : $0 [-d] [-a] <ssl-hostname>
purpose: check ssl certificate path. Use [#] to add to keystore"
options:
-d debug ssl handshake
-a add certificate (if not already trusted)
EOT
exit -1
fi

[ "$1" == "-d" ] && shift && DEBUG="-Djavax.net.debug=ssl:handshake" && echo "DEBUG MODE" >&2
ACTION="q"; [ "$1" == "-a" ] && shift && ACTION="1"

echo $ACTION | java $DEBUG InstallCert $*


greetings

Guido
Igor Cicimov
2018-10-23 09:05:14 UTC
Permalink
Hi Mark,
Post by Mark H. Wood
[snip]
Post by Igor Cicimov
To conclude, the way I would expect the trust store to be used and the
1. I use custom trust store because I need to load self signed
certificates
Post by Igor Cicimov
that
I need to validate when connecting to lets say partner APIs that use self
signed
certificates and I know I can trust
2. I would expect nothing else needed in this store as every other valid
certificate
under the sun is already located in default locations on the server Java
is
Post by Igor Cicimov
running on
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the
default) I
Post by Igor Cicimov
would
expect Java to use the system store(s) too in case a certificate can not
be
Post by Igor Cicimov
validated
simply because a CA is missing in the Java store. Example, DigiCert
Global
Post by Igor Cicimov
Root G2
CA is missing in the Java versions older than 8u91 causing inexplicable
PKIX exceptions
but can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates
which are (much) more frequently updated with new certs than Java
versions.
Post by Igor Cicimov
This actually
applies to the case of custom trust store even more so
Thoughts?
There are two ways that a truststore can be inadequate. (1) It can
lack a certificate that your application should trust. (2) It can
contain a certificate that your application should NOT trust.
Suppose that you had an application A which needs to distrust one of the
CAs (X) that are trusted by the OS maintainers. And suppose that you had
other applications on the same host which need to trust that same CA.
The way it currently works, you can copy the systemwide JRE truststore
once, remove from the copy the certificate for X, and tell A to use
the copy as its truststore. Every other application can just default
to the systemwide truststore. (This also works if you need to give A
an additional trusted CA not needed by other applications, and are
willing to trust all the other CAs.)
The way you suggest it should work, you must remove X from all
systemwide truststores of every type, and then configure a custom
truststore for every application except A.
I get what you are saying but I really can't imagine I would ever get into
this
kind of situation where I would need to distrust a specific CA and only for
one
single application. Most of the time I need to trust all CA's that the
world is trusting
at that moment.
Post by Mark H. Wood
Which is more error-prone?
It shouldn't be difficult to write a script that makes a copy of the
systemwide store and adjusts it to your application's specific needs.
--
Mark H. Wood
Lead Technology Analyst
University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
Christopher Schultz
2018-10-22 21:12:43 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Igor,
Post by Igor Cicimov
On Fri, Oct 19, 2018 at 2:14 AM Christopher Schultz <
does any trust store, really.
Post by Igor Cicimov
Correct, but by loading all CAs in the trust store it kinda
does, indirectly.
A
certificate is trusted if it is present in the trust store, full
stop. It not need be a "CA". The oly thing being a CA gets you is
... in everyone's default trust stores.
The system property javax.net.ssl.trustStore only sets the default
trust store for the JVM and any components which choose to use it.
For example, if you use HttpURLConnection without any explicit
configuration, it will use that. Same with Apache httpclient.
But both of those can be configured to use a different trust
store, which case they will *not* fall-back to the built-in trust
store (the one in JAVA_HOME/lib/security/cacerts.
Post by Igor Cicimov
Well I see couple of issues with this approach of the trsutstore
being the only source of truth. First is the obvious one, when
using a custom trust store I have to load *all* CA certificates
that already exist somewhere else on the server (and in multiple
places) in the trust store too otherwise no certificate will
ever get validated.
Why would you want to use a custom trust store that also includes the
whole list of trusted certs from the vendor? Either you want to
delegate everything to the vendor (e.g. Oracle) or you know who you
are connecting to, and you only need one cert (or a small selection of
them) in your trust store.

I think the problem is that people rely on "the trust store for the
JVM" as the trust store for everything, which is a Bad Idea. Use
separate trust stores for different types of connections.
Post by Igor Cicimov
When overriding the default trust store for the JVM, the trust
store you specify should be the ONLY trust store consulted. It
should not fall back. I can confirm this is the case on Java 8 -
11, at least the ones I happen to be using. Any other behavior
would be a security problem.
Post by Igor Cicimov
Not sure I can agree with this reasoning too. All apps on the
server use the default system CA store so should we consider them
insecure? I see no harm of Java looking in the default
location(s) on the server when a cert can not be validated by
looking in the trust store. Otherwise as noted above in case of
custom trust store we need to load all those certificates anyway
ending up with same certificates stored in multiple places,
making the size of the trust store unnecessary big.
You are talking about a web application connecting to an outside
service like a REST service via HTTPS, right? How many of these
services could you possibly be connecting to? Why don't you already
know their CAs?

The default trust store that ships with the JVM is really only good if
you want to connect to an arbitrary service and inherit all the certs
that e.g. Oracle trusts. That only makes sense if you don't know in
advance who you'll be connecting to.
Post by Igor Cicimov
The proper way to validate a certificate chain is to perform the
0. Start with the server's certificate (the leaf) 1. Is the
certificate in the trust store? Yes: chain is valid; stop 2. Is the
certificate signed by a cert in the trust store? Yes: chain is
valid; stop 3. Is the certificate signed by the next cert in the
chain? No: chain is invalid; stop 4. Move to the next cert in the
chain 5. Go to step 1
So if you use an empty trust store and try to connect to
https://www.google.com, you should find that you get an exception
sun.security.validator.ValidatorException: PKIX path building
unable to find valid certification path to requested target
Post by Igor Cicimov
To conclude, the way I would expect the trust store to be used
1. I use custom trust store because I need to load self signed
certificates that I need to validate when connecting to lets say
partner APIs that use self signed certificates and I know I can
trust
So... you just need to trust a self-signed certificate. That shouldn't
be a problem.
Post by Igor Cicimov
Post by Igor Cicimov
2. I would expect nothing else needed in this store as every
other valid certificate under the sun is already located in
default locations on the server Java is running on
And why do you need to be able to validate all those other
certificates if you are only connecting to one service?
Post by Igor Cicimov
Post by Igor Cicimov
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the
default) I would expect Java to use the system store(s) too in
case a certificate can not be validated simply because a CA is
missing in the Java store.
To behave otherwise would be a security problem: if you install your
own trust store, falling-back into the JVM's default trust store means
that trust is being misplaced. There would be no way to dis-trust any
certificate without physically deleting the certificate from the
system's default trust store. That may not be possible if the system
isn't writable by e.g. the application operator.
Post by Igor Cicimov
Post by Igor Cicimov
Example, DigiCert Global Root G2 CA is missing in the Java
versions older than 8u91 causing inexplicable PKIX exceptions but
can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates which are (much) more frequently
updated with new certs than Java versions.
Those errors aren't inexplicable: they occur because the certificate
is not trusted. Java doesn't use the system trust store, ever. You
can't even configure it to use the system's trust store. You'd have to
build a custom TrustManager and wire it into the system's trust store
yourself.

If you want to trust both your own custom trust store plus the
platform default, you'll have to write code to make that happen. It's
possible, but it's not "free". [1]

- -chris

[1]
https://stackoverflow.com/questions/24555890/using-a-custom-truststore-i
n-java-as-well-as-the-default-one
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvOPUoACgkQHPApP6U8
pFh1EhAAtuI+UD5R6CcVL0NOoFZYqhoDppkMLaFAFTzu5PtCyFZXEX4sDZhzd1kp
RqZCLXhliAbACnYk2AlkNKEgIaSZNoi5NxozmFQGka1o/dcQQqvJvTxYp1txj5dM
LQSXDybEJeeHT10fAThuwnRaDcVaJdtv1ILri2/FsswP1ZBnjki5kXKiRBzBZyw5
NSH/wJTm2cx7no6B2JdvckKXjExFdjO/TT8i6KOmcKjG80fjRUVbSpH3lXhV01nN
+tMQcJ/D3RVHLBR2tUKi/WhwVXgOxfx3Vkb43YAuOcrQYD+Fi0V2eQh7JdJhlvYD
EAGOs8bzOSFEDx3oQBsXSP0BwGlBCL//erxDI9iKtyzOBsxEJk+dU1fKalayj7l0
VicN2p4smHnBbeYJDFuAGvSMywuLfxXNrJJ8gY/6bP30554ydDA5lDNdXmUmaR06
gnGWX2ZaGqn6Icijdm8VpKUnwUA2gHzXPknFnOI806V6SWJzVEmnwg1913jBbq0T
FarSuVy+JuN/dkhIgBC90U9dstgnAhhjJBv920ia8wbjaCsewgpy74Yyf8UitU/Z
GJYnmKYAqk1KLjXXTXmM2KsncsbMCWwSba/OqbLtOQSQ2GtQ3qBbxQ/C+OgZacMj
Uysw7G6h8q0AeVRwN2wwDRwhfbmu2AsXN8dNSfSDU6bG6+kghE4=
=z9/7
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
Igor Cicimov
2018-10-23 09:35:20 UTC
Permalink
Hi Chris,

On Tue, Oct 23, 2018 at 8:12 AM Christopher Schultz <
Post by Christopher Schultz
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Igor,
Post by Igor Cicimov
On Fri, Oct 19, 2018 at 2:14 AM Christopher Schultz <
does any trust store, really.
Post by Igor Cicimov
Correct, but by loading all CAs in the trust store it kinda
does, indirectly.
A
certificate is trusted if it is present in the trust store, full
stop. It not need be a "CA". The oly thing being a CA gets you is
... in everyone's default trust stores.
The system property javax.net.ssl.trustStore only sets the default
trust store for the JVM and any components which choose to use it.
For example, if you use HttpURLConnection without any explicit
configuration, it will use that. Same with Apache httpclient.
But both of those can be configured to use a different trust
store, which case they will *not* fall-back to the built-in trust
store (the one in JAVA_HOME/lib/security/cacerts.
Post by Igor Cicimov
Well I see couple of issues with this approach of the trsutstore
being the only source of truth. First is the obvious one, when
using a custom trust store I have to load *all* CA certificates
that already exist somewhere else on the server (and in multiple
places) in the trust store too otherwise no certificate will
ever get validated.
Why would you want to use a custom trust store that also includes the
whole list of trusted certs from the vendor? Either you want to
delegate everything to the vendor (e.g. Oracle) or you know who you
are connecting to, and you only need one cert (or a small selection of
them) in your trust store.
I think the problem is that people rely on "the trust store for the
JVM" as the trust store for everything, which is a Bad Idea. Use
separate trust stores for different types of connections.
Post by Igor Cicimov
When overriding the default trust store for the JVM, the trust
store you specify should be the ONLY trust store consulted. It
should not fall back. I can confirm this is the case on Java 8 -
11, at least the ones I happen to be using. Any other behavior
would be a security problem.
Post by Igor Cicimov
Not sure I can agree with this reasoning too. All apps on the
server use the default system CA store so should we consider them
insecure? I see no harm of Java looking in the default
location(s) on the server when a cert can not be validated by
looking in the trust store. Otherwise as noted above in case of
custom trust store we need to load all those certificates anyway
ending up with same certificates stored in multiple places,
making the size of the trust store unnecessary big.
You are talking about a web application connecting to an outside
service like a REST service via HTTPS, right? How many of these
services could you possibly be connecting to? Why don't you already
know their CAs?
Actually many. My point is I don't have to know their CA's *except* when
they are self signed. So when I connect to an API partner with a certificate
signed by a valid CA then the cert should be obviously validated without me
putting the CA in the trusted store. Isn't this the way any other app would
expect things to happen (except java obviously)?

Another reason is containers where we try to get the smallest possible
image
size. Why would I even bother changing and bloat my image just because I
need
to add CAs in my trust store when they have already been loaded via the
installed
ca-certificates package lets say? Much easier than maintaining my own trust
store
file.
Post by Christopher Schultz
The default trust store that ships with the JVM is really only good if
you want to connect to an arbitrary service and inherit all the certs
that e.g. Oracle trusts. That only makes sense if you don't know in
advance who you'll be connecting to.
Post by Igor Cicimov
The proper way to validate a certificate chain is to perform the
0. Start with the server's certificate (the leaf) 1. Is the
certificate in the trust store? Yes: chain is valid; stop 2. Is the
certificate signed by a cert in the trust store? Yes: chain is
valid; stop 3. Is the certificate signed by the next cert in the
chain? No: chain is invalid; stop 4. Move to the next cert in the
chain 5. Go to step 1
So if you use an empty trust store and try to connect to
https://www.google.com, you should find that you get an exception
sun.security.validator.ValidatorException: PKIX path building
unable to find valid certification path to requested target
Post by Igor Cicimov
To conclude, the way I would expect the trust store to be used
1. I use custom trust store because I need to load self signed
certificates that I need to validate when connecting to lets say
partner APIs that use self signed certificates and I know I can
trust
So... you just need to trust a self-signed certificate. That shouldn't
be a problem.
Post by Igor Cicimov
Post by Igor Cicimov
2. I would expect nothing else needed in this store as every
other valid certificate under the sun is already located in
default locations on the server Java is running on
And why do you need to be able to validate all those other
certificates if you are only connecting to one service?
Post by Igor Cicimov
Post by Igor Cicimov
3. In case JAVA_HOME/lib/security/cacerts is my trust store (the
default) I would expect Java to use the system store(s) too in
case a certificate can not be validated simply because a CA is
missing in the Java store.
To behave otherwise would be a security problem: if you install your
own trust store, falling-back into the JVM's default trust store means
that trust is being misplaced. There would be no way to dis-trust any
certificate without physically deleting the certificate from the
system's default trust store. That may not be possible if the system
isn't writable by e.g. the application operator.
I get what you are saying and it is a valid point too. But as I said
previously
how many occasions do you know or heard of that someone would want to
remove a CA from the system store? It is also matter of policy though, do
you
start by trusting everyone (in this case all CAs) or no one.
Post by Christopher Schultz
Post by Igor Cicimov
Post by Igor Cicimov
Example, DigiCert Global Root G2 CA is missing in the Java
versions older than 8u91 causing inexplicable PKIX exceptions but
can be found in the system store, both under /etc/ssl/certs and
/usr/share/ca-certificates which are (much) more frequently
updated with new certs than Java versions.
Those errors aren't inexplicable: they occur because the certificate
is not trusted. Java doesn't use the system trust store, ever. You
can't even configure it to use the system's trust store. You'd have to
build a custom TrustManager and wire it into the system's trust store
yourself.
If you want to trust both your own custom trust store plus the
platform default, you'll have to write code to make that happen. It's
possible, but it's not "free". [1]
- -chris
[1]
https://stackoverflow.com/questions/24555890/using-a-custom-truststore-i
n-java-as-well-as-the-default-one
<https://stackoverflow.com/questions/24555890/using-a-custom-truststore-in-java-as-well-as-the-default-one>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvOPUoACgkQHPApP6U8
pFh1EhAAtuI+UD5R6CcVL0NOoFZYqhoDppkMLaFAFTzu5PtCyFZXEX4sDZhzd1kp
RqZCLXhliAbACnYk2AlkNKEgIaSZNoi5NxozmFQGka1o/dcQQqvJvTxYp1txj5dM
LQSXDybEJeeHT10fAThuwnRaDcVaJdtv1ILri2/FsswP1ZBnjki5kXKiRBzBZyw5
NSH/wJTm2cx7no6B2JdvckKXjExFdjO/TT8i6KOmcKjG80fjRUVbSpH3lXhV01nN
+tMQcJ/D3RVHLBR2tUKi/WhwVXgOxfx3Vkb43YAuOcrQYD+Fi0V2eQh7JdJhlvYD
EAGOs8bzOSFEDx3oQBsXSP0BwGlBCL//erxDI9iKtyzOBsxEJk+dU1fKalayj7l0
VicN2p4smHnBbeYJDFuAGvSMywuLfxXNrJJ8gY/6bP30554ydDA5lDNdXmUmaR06
gnGWX2ZaGqn6Icijdm8VpKUnwUA2gHzXPknFnOI806V6SWJzVEmnwg1913jBbq0T
FarSuVy+JuN/dkhIgBC90U9dstgnAhhjJBv920ia8wbjaCsewgpy74Yyf8UitU/Z
GJYnmKYAqk1KLjXXTXmM2KsncsbMCWwSba/OqbLtOQSQ2GtQ3qBbxQ/C+OgZacMj
Uysw7G6h8q0AeVRwN2wwDRwhfbmu2AsXN8dNSfSDU6bG6+kghE4=
=z9/7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
Christopher Schultz
2018-10-23 14:55:32 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Igor,
Post by Igor Cicimov
On Tue, Oct 23, 2018 at 8:12 AM Christopher Schultz <
application connecting to an outside service like a REST service
via HTTPS, right? How many of these services could you possibly be
connecting to? Why don't you already know their CAs?
Post by Igor Cicimov
Actually many. My point is I don't have to know their CA's
*except* when they are self signed.
So use a custom trust-store for this services with a self-signed
certificate and use the default trust store for everything else. I
don't really see a problem, here, conceptually.

The real problem is that many software libraries don't come with the
ability to easily switch to a separate trust store to connect to them.
They just assume that you'll import their certificate into a
system-wide trust store and move-on. I don't like that strategy one bit.
Post by Igor Cicimov
Post by Igor Cicimov
So when I connect to an API partner with a certificate signed by
a valid CA then the cert should be obviously validated without me
putting the CA in the trusted store. Isn't this the way any
other app would expect things to happen (except java obviously)?
All applications I know of require you to import anything it doesn't
already trust in order to trust those services (duh). The difference
is that you think it should fall-back on the system trust store and
Java does not (and I happen to agree with that strategy).

You can code around it if you absolutely want your JVM to behave that wa
y.
Post by Igor Cicimov
Post by Igor Cicimov
Another reason is containers where we try to get the smallest
possible image size. Why would I even bother changing and bloat
my image just because I need to add CAs in my trust store when
they have already been loaded via the installed ca-certificates
package lets say? Much easier than maintaining my own trust store
file.
Unfortunately, Java uses trust store bundles and e.g. Linux packages
everything as a directory of PEM-encoded DER files. Java simply can't
use those as a default trust store. Again, you can write your own
TrustManager to use those OS-provided certificates; Java doesn't
provide such code out of the box.

The cacerts file that comes with Java 8 is 113k. I don't think it's
breaking your byte allotment for containers.
Post by Igor Cicimov
Post by Igor Cicimov
I get what you are saying and it is a valid point too. But as I
said previously how many occasions do you know or heard of that
someone would want to remove a CA from the system store? It is
also matter of policy though, do you start by trusting everyone
(in this case all CAs) or no one.
Symantec? Or all the other sub-CA's who probably never should have
been in anyone's root trust store in the first place?

For my money, I always use custom trust stores for individual external
services. It protects me against all manner of Bad Things such as
stupid CAs, MitM, etc. If you trust everything without question, you
may make a connection you shouldn't have and give-up private data to a
(truly, not technically) untrusted recipient.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvPNmMACgkQHPApP6U8
pFiaahAAuL0M8hU+NzxiyKXDTTyrmwW0kMtxmhXqdmZDbIuQmVeJtX4zqH7t7bW7
dGcLomXFqnaj24jXWtg2F2yCi/xS5V9EaCypVKpgMOmr7Hd7V6FG5TTa1MNaDARL
34acLCbXi7/R934Vs6il5wQaJqT97b2sT4KfwJ8qdb3nxNK+UxpOqzrqlH7CVfKm
GcNmUXDqSaOn6+a6vib5j+QTONMxC2tS8jqSmAHASPgqaTao/MO5KypoIe3wls0w
9B/HFjUPu29T8Nfesj8CX+MpR544DUCD5QPmBzAMfgkKFXCVcQYvQ58Bu2hWybL7
T/B3dAnBUqjDtuxgC+EQeaoxC72S6AYGvWbDzgCFrOyXWai2HWkQSHi949Oh+WjY
XcaW1ykx888mFuvE/2GCC9tY/jfS+aoXwUhrnRJgZ2Qy2VQPHxPedap2xTELaD4k
ijlCHq3DhjkUSa/2m96YWF964ub2RlH2auB32Rk2CVZ88PGdjCiEKCs5HTNSdoa+
tESYbRDeiZFkmBNRn6BeZ/fChH9ReUXoAy/ggC2V5/v+QZpLqdZ3iEMpRejUY/9x
L73iNV29d+h+OXJv/1WAwajW4GuxZVkzJ2z4LWbjVSfOS3u9SQgI2kE8zuPgbXeR
5u3CpxUQAYtHLOpteefQF1peJcEiE7i9DlLE8j0zKuVwi0Q5Z+U=
=gNYA
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Loading...