Tomcat embedded with Apache Solr

[Christopher Schultz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

I've posted a question to the Solr mailing list[1] about why Jetty is
being used instead of Tomcat, and it seems that it's just "always the
way it's been" for the most part.

These seems to be some interest in moving from their existing model
(Jetty launches, then hosts Solr as a web application) to an embedded
model where a Solr-agent launches and then launches the app server
(e.g. tc) hosting itself.

Is anyone familiar enough with Solr to whip-up a quick-and-dirty POC
for Tomcat-embedded-hosting-Solr?

I'd like to make the case that Tomcat isn't some big, bad heavy
monster that should be avoided for some other product. There appears
to be soe unfounded hatred[4] going around.

The Solr team seems especially conscious of runtime memory footprint,
and they don't have any use for JSP, Websocket, or anything fancy.
They just want to pump bytes back and forth to their basic
servlet-based service.

Can anyone chime-in on that thread referenced below with some thoughts
on how Solr might be able to use Tomcat-embedded as a platform for soe
future version?

Also see [2] and [3].

Thanks,
- -chris

[1]
https://lists.apache.org/thread.html/69c3a6ef96d99aa79d5ab15b23e1eb75d8f
32b0709fa14cc7cdded91@%3Cdev.lucene.apache.org%3E
[2] https://issues.apache.org/jira/browse/SOLR-6733
[3] https://issues.apache.org/jira/browse/SOLR-6734 ??
[4]
https://lists.apache.org/thread.html/c5786fd956d6d9029ddacc5af18147d098d
5a8675ebda78db14c10f1@%3Cdev.lucene.apache.org%3E
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvCnsMACgkQHPApP6U8
pFiqwA/+PbcFqOUhXacAzRwzBGtK0WELMHrr1QCkPYWw1D6yxgNq2kKya6jWh9XK
+EFb1ZkaYDFunseLsXdLw6pN7fkxPjLRUVFLUr+57csGKR8nH+Y36uaKXErKG1TD
w1PHOIjZkNfN1i5x3aouqZVGwxbqB41HZeXYDmaknNX5yhZGoZQlgLkH8DXZRj4H
qLMIY2WV3q0tUz/9mVi4Lvtvejr01KUiALYxurwxJI9rToLdOEv3qIOk2+sNy5b7
P8G2eUgMtA7vdXQ+sajXV+KptKkzcNHwehgX0YxYsV56tkaVgWkg6d63KZj11qgm
xWzUpdiGYXXVTFfEPSEVSu3psqBGIHDvv234PMGs2A+sqM8kaynJkbQsJJ7fu9Co
oA7clrptZbyGqVB4ASAxWC+PXeuycGnMFplZRfRXagmcmkChxli+Y2MqFekdKHO6
VF6ExuzK7gg3ogZ1R+4sHYYW/KBTSaS+OVHXjJvTppATQbsnAGkZ5/93zTdlnbMh
Ua9WOICaOats7XDcDhgpSmClL5SdOWQ+THCeT5hrnI7CttefihdPp4fqrDOPJ0O+
fhcVv1J80GJIIPq2g48MzmxQoT08tKxomM9EeZbApnV9ptQxz6RgLAYYGryYUqjn
7v3Aoc6YS6WWEicZ6iR1j3AD5mE/kV0/GxZ8L52MPyVb4gvhiVU=
=/itJ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Jäkel, Guido]

Dear Christopher,

my 5ct on that: We're using Solr for years like (and together with) all other Webapps on Tomcat (and since some month on Wildfly) in a "classic" (non-cloud) setup without any the need for special tweaks.

There is no official support for an webapp artifact since some Versions, but one just have to extract the web application tree from the release.

I have no experience with embedded tomcat, but it should be also straight forward. Said that, I can't imagine the advantage of such an approach against the currently used, which just start the Web Application Server (Jetty, Tomcat or whatever) with the preinstalled Solr application.

We even have used it in the other way round: We used an embedded Solr Server as a SolrMaster instead of a SolrClient within an Ingest application to avoid the "serialization/network overhead" of big Solr documents. This Application was also hosted together with others on our Tomcat plattform


With greetings

Guido

-----Original Message-----
Sent: Sunday, October 14, 2018 3:41 AM
Subject: Tomcat embedded with Apache Solr
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
I've posted a question to the Solr mailing list[1] about why Jetty is
being used instead of Tomcat, and it seems that it's just "always the
way it's been" for the most part.
These seems to be some interest in moving from their existing model
(Jetty launches, then hosts Solr as a web application) to an embedded
model where a Solr-agent launches and then launches the app server
(e.g. tc) hosting itself.
Is anyone familiar enough with Solr to whip-up a quick-and-dirty POC
for Tomcat-embedded-hosting-Solr?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-m

[Christopher Schultz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Guido,

Post by Jäkel, Guido
Dear Christopher,
my 5ct on that: We're using Solr for years like (and together with)
all other Webapps on Tomcat (and since some month on Wildfly) in a
"classic" (non-cloud) setup without any the need for special
tweaks.
There is no official support for an webapp artifact since some
Versions, but one just have to extract the web application tree
from the release.
I have no experience with embedded tomcat, but it should be also
straight forward. Said that, I can't imagine the advantage of such
an approach against the currently used, which just start the Web
Application Server (Jetty, Tomcat or whatever) with the
preinstalled Solr application.
We even have used it in the other way round: We used an embedded
Solr Server as a SolrMaster instead of a SolrClient within an
Ingest application to avoid the "serialization/network overhead" of
big Solr documents. This Application was also hosted together with
others on our Tomcat plattform

Thanks for the reply.

Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive. I agree
that "solr.sh start" is the right model for users. I'm just working on
getting them to consider Tomcat as the hosting container instead of
Jetty. Right now, "solr.sh start" launches Jetty with Solr configured
as a web application. They are considering using "real embedding" and
so Tomcat has an opportunity to be considered as the container for
that effort.

- -chris

Post by Jäkel, Guido

-----Original Message----- From: Christopher Schultz
Subject: Tomcat embedded with Apache Solr
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
All,
I've posted a question to the Solr mailing list[1] about why
Jetty is being used instead of Tomcat, and it seems that it's
just "always the way it's been" for the most part.
These seems to be some interest in moving from their existing
model (Jetty launches, then hosts Solr as a web application) to
an embedded model where a Solr-agent launches and then launches
the app server (e.g. tc) hosting itself.
Is anyone familiar enough with Solr to whip-up a quick-and-dirty
POC for Tomcat-embedded-hosting-Solr?

---------------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvInvUACgkQHPApP6U8
pFgokQ/+IDJIEWrvtXg8WMGsnyikBwLpVafyNlje+nnNZDJ46bdL/5eAOeO0ef06
TidVZlbHtucP3gParlyDFAZE5+R5PLQw3Kb+aH5eZLShrUlF0/aBruG4sdAXt8b6
tN8eu9WmULz78AfX9wA5EvIKfKkps+ubYhbmZgMBnzHcARkMbG3FqKpIrEG36D3l
ixpPNrY/Ig7WDse20/+xbPVLpfkvYIlZuD1ks4ChRoUiMxSBUdTBcecI7b7ktzrT
OY6wxrrBPX4Ca37aA83oLw4EChPFy5bvR21P4wXFaA9yrT3GdvPNG0Rs7DiGR3PN
RSADf8dJ0Mle2azPB56+n2CPck5CLZllk8kbvyy1RV+9AQFDV0ytFPnS/34IbZf/
Q2lSgGyVLB3GAic3RzCqIAtRnUmJW6BSXtq8B8YGL5sy9HATTMj9f00xzDqTC38i
H22fnJohfTrsVj4VBklDEvE2V+9NEyj+guw/v0jKRlfiX8S98tHQuQD8kefTP4X2
vHkKaU0V5rNmO/Ix8jA9i9CVqNe5UX+nZThWe5gBHKxdFVpZ9VFWwTYgX9+eZiYc
+I1D6j1oQS+qEWw3RlPkTUei6GHVCrTsXlHxEN89k6mTOZi1x9xaq5G6th5AOa5B
VL3qxoVz/lWdSFaZLcN2kHwG/gBlSgqBa8V4m7kt83IDoUsGg2w=
=A9vT
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Shawn Heisey]

Post by Christopher Schultz
Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive.

Over on lucene-dev, I had said that I removed jetty from solr's ivy
config and found only two classes with errors in eclipse.

Turns out this was because I hadn't removed jetty from the *lucene* ivy
config, so most of the jetty jars were actually still referenced in the
eclipse build path.

When I remove jetty from ALL ivy configs, there are 335 compile errors,
across many more classes.  Some of those are on the Lucene side, where I
have less concern.  The part of Lucene that utilizes Jetty is not used
in Solr.

As I expected, a lot of the errors are in test code, but some of them
are in code that's not for tests.

If you really want to see us switch to Tomcat, we'll need help fixing
those errors ... switching the code over to generic APIs (servlet and
native Java) where possible, and to Tomcat where necessary.  I would not
expect the Tomcat community to actually do the work -- just provide
expert guidance.  Although if anyone was interested in volunteering, I
wouldn't turn away the help!

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Woonsan Ko]

Not sure if it helps, but I've spent some hours to test it out for my
curiosity. Embedding tomcat v9 with an example servlet is simple
enough, but it's not working yet with fully initialized Solr:
- https://github.com/woonsan/solr-tomcat

I'm not there yet to understand the reasons and have fixes.
Let me know if you have any suggestions.

Cheers,

Woonsan

Post by Shawn Heisey

Post by Christopher Schultz
Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive.

Over on lucene-dev, I had said that I removed jetty from solr's ivy
config and found only two classes with errors in eclipse.
Turns out this was because I hadn't removed jetty from the *lucene* ivy
config, so most of the jetty jars were actually still referenced in the
eclipse build path.
When I remove jetty from ALL ivy configs, there are 335 compile errors,
across many more classes. Some of those are on the Lucene side, where I
have less concern. The part of Lucene that utilizes Jetty is not used
in Solr.
As I expected, a lot of the errors are in test code, but some of them
are in code that's not for tests.
If you really want to see us switch to Tomcat, we'll need help fixing
those errors ... switching the code over to generic APIs (servlet and
native Java) where possible, and to Tomcat where necessary. I would not
expect the Tomcat community to actually do the work -- just provide
expert guidance. Although if anyone was interested in volunteering, I
wouldn't turn away the help!
Thanks,
Shawn
---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Igal Sapir]

Woonsan,

Post by Woonsan Ko
Not sure if it helps, but I've spent some hours to test it out for my
curiosity. Embedding tomcat v9 with an example servlet is simple
- https://github.com/woonsan/solr-tomcat

I noticed that you extend Tomcat [1] in order to avoid initializing the
default servlets. You may have already seen that, but I wanted to point
out that in the next version of Tomcat it can be done by passing false
to setAddDefaultWebXmlToWebapp()
[2].

Best,

Igal

[1]
https://github.com/woonsan/solr-tomcat/blob/master/server/src/main/java/com/github/woonsan/solr/server/tomcat/launch/Main.java#L103
[2] https://github.com/apache/tomcat/commit/0222ce9

Post by Woonsan Ko
I'm not there yet to understand the reasons and have fixes.
Let me know if you have any suggestions.
Cheers,
Woonsan

Post by Shawn Heisey

Post by Christopher Schultz
Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive.

Over on lucene-dev, I had said that I removed jetty from solr's ivy
config and found only two classes with errors in eclipse.
Turns out this was because I hadn't removed jetty from the *lucene* ivy
config, so most of the jetty jars were actually still referenced in the
eclipse build path.
When I remove jetty from ALL ivy configs, there are 335 compile errors,
across many more classes. Some of those are on the Lucene side, where I
have less concern. The part of Lucene that utilizes Jetty is not used
in Solr.
As I expected, a lot of the errors are in test code, but some of them
are in code that's not for tests.
If you really want to see us switch to Tomcat, we'll need help fixing
those errors ... switching the code over to generic APIs (servlet and
native Java) where possible, and to Tomcat where necessary. I would not
expect the Tomcat community to actually do the work -- just provide
expert guidance. Although if anyone was interested in volunteering, I
wouldn't turn away the help!
Thanks,
Shawn
---------------------------------------------------------------------

---------------------------------------------------------------------

[Woonsan Ko]

Post by Igal Sapir
Woonsan,

Post by Woonsan Ko
Not sure if it helps, but I've spent some hours to test it out for my
curiosity. Embedding tomcat v9 with an example servlet is simple
- https://github.com/woonsan/solr-tomcat

I noticed that you extend Tomcat [1] in order to avoid initializing the
default servlets. You may have already seen that, but I wanted to point
out that in the next version of Tomcat it can be done by passing false
to setAddDefaultWebXmlToWebapp()
[2].

That's a nice improvement. Thanks for the info. I added a TODO comment
above the line.

Best,

Woonsan

Post by Igal Sapir
Best,
Igal
[1]
https://github.com/woonsan/solr-tomcat/blob/master/server/src/main/java/com/github/woonsan/solr/server/tomcat/launch/Main.java#L103
[2] https://github.com/apache/tomcat/commit/0222ce9

Post by Woonsan Ko
I'm not there yet to understand the reasons and have fixes.
Let me know if you have any suggestions.
Cheers,
Woonsan

Post by Shawn Heisey

Post by Christopher Schultz
Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive.

Over on lucene-dev, I had said that I removed jetty from solr's ivy
config and found only two classes with errors in eclipse.
Turns out this was because I hadn't removed jetty from the *lucene* ivy
config, so most of the jetty jars were actually still referenced in the
eclipse build path.
When I remove jetty from ALL ivy configs, there are 335 compile errors,
across many more classes. Some of those are on the Lucene side, where I
have less concern. The part of Lucene that utilizes Jetty is not used
in Solr.
As I expected, a lot of the errors are in test code, but some of them
are in code that's not for tests.
If you really want to see us switch to Tomcat, we'll need help fixing
those errors ... switching the code over to generic APIs (servlet and
native Java) where possible, and to Tomcat where necessary. I would not
expect the Tomcat community to actually do the work -- just provide
expert guidance. Although if anyone was interested in volunteering, I
wouldn't turn away the help!
Thanks,
Shawn
---------------------------------------------------------------------

---------------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Shawn Heisey]

Post by Jäkel, Guido
I have no experience with embedded tomcat, but it should be also straight forward. Said that, I can't imagine the advantage of such an approach against the currently used, which just start the Web Application Server (Jetty, Tomcat or whatever) with the preinstalled Solr application.

Probably the top reason for embedding the network layer (currently
Jetty) is to have all of the global config (both for the
network/container layer and for the Solr node itself) live in one place.

Right now, when running in cloud mode, a node could register itself with
the cluster using completely invalid information -- wrong host/address,
wrong port, wrong context path (normally /solr -- changing it causes big
problems with 5.0 and later).  Changing to embedded would allow us to
ensure that the info used in node registration is the same info used to
initialize the node.  It would be overall much cleaner, and we would be
able to finally say that the end user is starting Solr -- not Jetty or
any other container.

Here's the info about why things are as they are currently, with a
little bit of info about where we want things to end up:

https://wiki.apache.org/solr/WhyNoWar

Christopher already shared the issue I filed for the effort in setting
Solr up as a standalone application.  It is an umbrella issue with one
child issue currently:

https://issues.apache.org/jira/browse/SOLR-6733

Another point in Jetty's favor:  We are in the process of switching from
Apache httpclient to Jetty httpclient, for http/2 support.  The new
Apache client with http/2 support is only recently into beta status.  It
was in alpha state for a REALLY long time, and we expect the beta state
to last for another several months, possibly a year or longer.  The
jetty client with http/2 support has been available in a stable version
almost as long as the http/2 RFC has been published, and has had all
that time with heavy usage for the showstopper bugs to be worked out.  I
personally think it's tragic that we're switching away from Apache for
our httpclient, but I'm in favor of the switch for the reasons
outlined.  Compatibility with the server is expected to be better if
both are from the same development team.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

[Igal Sapir]

Chris,

On Sat, Oct 13, 2018 at 6:41 PM Christopher Schultz <

Post by Christopher Schultz
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
I've posted a question to the Solr mailing list[1] about why Jetty is
being used instead of Tomcat, and it seems that it's just "always the
way it's been" for the most part.
These seems to be some interest in moving from their existing model
(Jetty launches, then hosts Solr as a web application) to an embedded
model where a Solr-agent launches and then launches the app server
(e.g. tc) hosting itself.
Is anyone familiar enough with Solr to whip-up a quick-and-dirty POC
for Tomcat-embedded-hosting-Solr?

The last time I used Solr was about 10 years ago. The Solr API seemed so
cumbersome that I opted to embed Apache Lucene directly in my application.

And ever since Elasticsearch was launched there was very little incentive
to use Solr, as it provided similar functionality (and nowadays arguably
better), with a much simpler UI/API.

Woonsan's PoC looks very close to getting the job done, possibly missing
some Solr config setting or something like that.

Post by Christopher Schultz
I'd like to make the case that Tomcat isn't some big, bad heavy
monster that should be avoided for some other product. There appears
to be soe unfounded hatred[4] going around.
The Solr team seems especially conscious of runtime memory footprint,
and they don't have any use for JSP, Websocket, or anything fancy.
They just want to pump bytes back and forth to their basic
servlet-based service.

I have seen stuff like that years ago in the Railo project (Lucee's
predecessor), where some users made similar claims and insisted on using
Jetty. I was never able to confirm those claims in my tests, and most
Lucee users run it in Tomcat nowadays.

Post by Christopher Schultz
Can anyone chime-in on that thread referenced below with some thoughts
on how Solr might be able to use Tomcat-embedded as a platform for soe
future version?

I would argue that they should have a container-agnostic configuration.
They should really let the end users the freedom to choose their servlet
container.

Best,

Igal

Post by Christopher Schultz
Also see [2] and [3].
Thanks,
- -chris
[1]
https://lists.apache.org/thread.html/69c3a6ef96d99aa79d5ab15b23e1eb75d8f
[2] https://issues.apache.org/jira/browse/SOLR-6733
[3] https://issues.apache.org/jira/browse/SOLR-6734 ??
[4]
https://lists.apache.org/thread.html/c5786fd956d6d9029ddacc5af18147d098d
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvCnsMACgkQHPApP6U8
pFiqwA/+PbcFqOUhXacAzRwzBGtK0WELMHrr1QCkPYWw1D6yxgNq2kKya6jWh9XK
+EFb1ZkaYDFunseLsXdLw6pN7fkxPjLRUVFLUr+57csGKR8nH+Y36uaKXErKG1TD
w1PHOIjZkNfN1i5x3aouqZVGwxbqB41HZeXYDmaknNX5yhZGoZQlgLkH8DXZRj4H
qLMIY2WV3q0tUz/9mVi4Lvtvejr01KUiALYxurwxJI9rToLdOEv3qIOk2+sNy5b7
P8G2eUgMtA7vdXQ+sajXV+KptKkzcNHwehgX0YxYsV56tkaVgWkg6d63KZj11qgm
xWzUpdiGYXXVTFfEPSEVSu3psqBGIHDvv234PMGs2A+sqM8kaynJkbQsJJ7fu9Co
oA7clrptZbyGqVB4ASAxWC+PXeuycGnMFplZRfRXagmcmkChxli+Y2MqFekdKHO6
VF6ExuzK7gg3ogZ1R+4sHYYW/KBTSaS+OVHXjJvTppATQbsnAGkZ5/93zTdlnbMh
Ua9WOICaOats7XDcDhgpSmClL5SdOWQ+THCeT5hrnI7CttefihdPp4fqrDOPJ0O+
fhcVv1J80GJIIPq2g48MzmxQoT08tKxomM9EeZbApnV9ptQxz6RgLAYYGryYUqjn
7v3Aoc6YS6WWEicZ6iR1j3AD5mE/kV0/GxZ8L52MPyVb4gvhiVU=
=/itJ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------