Tomcat occasionally returns a response without HTTP headers

Discussion:

Kohei Nozaki

2018-11-30 01:19:39 UTC

Hello,

I'm investigating a problem where Tomcat (7.0.92) returns a response
with no HTTP headers very occasionally. It happened during a load test.

According to the captured packets by Wireshark, after Tomcat receives a
request it just returns only a response body. It returns neither a
status line nor HTTP response headers.

This is a screenshot of Wireshark's "Follow HTTP Stream" which is
showing the problematic response: Loading Image...

This is a screen shot of "TCP Stream" of the relevant part (only
response): Loading Image...

In my opinion, neither an application nor a framework can make Tomcat
behave this way, so I'm wondering if it's a bug or something of
Tomcat. Do you think it can be a bug of Tomcat or is there something
which can make this happen? Was there a relevant change of Tomcat in
recent versions?

This question was originally posted on Stackoverflow. There are some
discussions and a bit of more information there:
https://stackoverflow.com/q/53496598/3591946

Thanks,

Kohei

Christopher Schultz

2018-11-30 16:52:04 UTC

Permalink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Kohei,

Post by Kohei Nozaki
I'm investigating a problem where Tomcat (7.0.92) returns a
response with no HTTP headers very occasionally. It happened during
a load test.
According to the captured packets by Wireshark, after Tomcat
receives a request it just returns only a response body. It returns
neither a status line nor HTTP response headers.
This is a screenshot of Wireshark's "Follow HTTP Stream" which is
https://i.stack.imgur.com/E6ttG.png

It's difficult to tell what's being shown, here. Too much has been
redacted. It looks like both request and response are being shown, and
the response appears to be part of a chunked response.

Post by Kohei Nozaki
This is a screen shot of "TCP Stream" of the relevant part (only
response): https://i.stack.imgur.com/TLKl3.png

This looks like a standard chunked response. I see no problems, there.

Post by Kohei Nozaki
In my opinion, neither an application nor a framework can make
Tomcat behave this way, so I'm wondering if it's a bug or something
of Tomcat. Do you think it can be a bug of Tomcat or is there
something which can make this happen? Was there a relevant change
of Tomcat in recent versions?

I agree that it should not be possible for an application (and a
framework counts as "the application" in this sense) to break Tomcat's
handling of the HTTP spec, but I don't see direct evidence of a bug,
yet, here.

Are your samples (shown in your images) definitely showing only a
single TCP connection, or is it possible that wireshark is merging the
requests/responses of multiple connections together in the same log
stream?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwBarQACgkQHPApP6U8
pFisaw/9G3l7InffOzzOQcTLWE7b+38dFKxrB0ejpLqoG70C1w2VgwwSGBfzrpxA
ucpA+SJeyLujI+NLyeANtB01uY/B9Ba3f0XgdQGOSl+RcT4vOQMs5qGtfS8ImitE
slCjiogr077Lp/sKMXSCkSDzEvHaJ/8jIITqGElo05q4HZqSHsHAan4crndP0lE6
9mIHfN477NmOhoHHg+HwuSzbgGbawPlNwjkFLZvQlQQIBUEc00DyGFC+avnie5nf
0z40UFULlR+rzsdXYsrfmOtzYpAV1lzksxDx7RqPbKjrOdqNRzeCkcdE2yzoi8CF
rzacG2anhxWOuuE5zKoFzcxxAv2uNDnl9sDfIcNkl00D5pBBeGAQ6fib27oDNtJ/
zZW9UsbwiiUzDOnpzNTFLNzf/TTF3NuHNpVXyVq9rBXwdqzqCQXHaI7zOSyY6NDr
6V09eP2xwsXlaDMXp7hl8hea3Ci0gIUXFm8wTq8LBjNtteoWrJ4vkx5qoVFhzORe
D0TgmZgwtablmzb76hLYMJP1t6gsnx0nbcob1YcxAx9k1pCTKEiRsdFHYO1fH0KE
8TpR4q/3xmINNhU2vb8SefvoTXIvOcI8LJpy6u9YuTAIiTlmwUQouqtdZ99U9EjN
EdG5eUVv2XRJvlKLRovIOAoJl1LL7dDWO1II6hRdBbqulmeQXHE=
=eDT+
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Kohei Nozaki

2018-11-30 22:30:14 UTC

Permalink

Hello Christopher, thank you for your help.

Post by Christopher Schultz

It's difficult to tell what's being shown, here. Too much has been
redacted. It looks like both request and response are being shown, and
the response appears to be part of a chunked response.

The problem here is that the HTTP status line and the HTTP response headers which should have been at before the beginning of the chunked response ("5d") are missing.

I'm sorry about the too much redaction but that's what I only can do due to security reasons.. If more information is needed in order to narrow down the problem, please tell me and I will do the best.

Post by Christopher Schultz

Post by Kohei Nozaki
This is a screen shot of "TCP Stream" of the relevant part (only
response): https://i.stack.imgur.com/TLKl3.png

This looks like a standard chunked response. I see no problems, there.

Actually, this screenshot shows 2 responses. There are two terminating chunk ("0" and an empty line) there. Those 2 responses were sent from a SOAP endpoint and that's why there are 2 XML documents.

And the problem here is that the HTTP status line and the HTTP response headers are missing in the latter response.

Post by Christopher Schultz

I believe my samples are showing only one single TCP connection because:

* They are filtered by a "tcp.stream eq xxx" condition as shown in the title bars in the screen shots
* Our downstream Nginx instance (The client of our Tomcat instance) recorded the error "upstream sent no valid HTTP/1.0 header while reading response header from upstream" at that time and the error makes perfect sense concerning the response which has neither HTTP status line nor HTTP headers.

Speaking of a possibility of a bug, a person commented on the Stackoverflow question and said that there might be something in the request that possibly downgrades the connection to HTTP/0.9. Do you think it's possible? The comments can be seen from the URL below:

https://stackoverflow.com/questions/53496598/tomcat-occasionally-returns-a-response-without-http-headers#comment93976313_53552752

Best regards,
Kohei

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Konstantin Kolinko

2018-12-01 10:00:53 UTC

Permalink

Post by Kohei Nozaki
Hello Christopher, thank you for your help.
* Our downstream Nginx instance (The client of our Tomcat instance) recorded the error "upstream sent no valid HTTP/1.0 header while reading response header from upstream" at that time and the error makes perfect sense concerning the response which has neither HTTP status line nor HTTP headers.
https://stackoverflow.com/questions/53496598/tomcat-occasionally-returns-a-response-without-http-headers#comment93976313_53552752

1. See the official FAQ / Troubleshoting page:
https://wiki.apache.org/tomcat/FAQ/Troubleshooting_and_Diagnostics

Especially pay attention to
1) configuring an access log
2) setting system property
org.apache.catalina.connector.RECYCLE_FACADES=true

(and Java ImageIO stream handling bug)

2. HTTP 0.9 is a valid response format (a feature, not a bug) that
does not contain status line nor headers, sends just the requested
document and closes the connection afterwards. The "Specifications"
page in the wiki has a link to the original specification, if you are
interested.

But HTTP 0.9 should never mix itself with chunked encoding. (The "5d +
CRLF" chunk size that you are seeing).

3. If somebody calls "out.flush()" (or response.flushBuffer()) before
writing a response body, the headers and the body will be sent as
separate packets and may appear separately in wireshark. It is a
valid behaviour.

(But your client should see the headers. It shouldn't report that
headers are missing).

Best regards,
Konstantin Kolinko

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org