Discussion:
Error Regarding RFC 7230 and RC 3986
M. Manna
2018-10-18 08:52:57 UTC
Permalink
Hello,

We received in error in our application after we have upgraded to 8.5.34

INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request
target. The valid characters are defined in RFC 7230 and RFC 3986


The URI we have for this problem has the following param (did work with
8.5.28)

defaultMessageType=true&locale=en_US&action=[key:label.edit]

The issue is the action parameter value. Could someone help me understand
the following?

1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if this
is CVE-2016-681 ? If not, could you please let me know which one that is?

2) Apart from refactoring code, is there any recommended corrective action?

Thanks,
Mark Thomas
2018-10-18 09:39:37 UTC
Permalink
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to 8.5.34
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me understand
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if this
is CVE-2016-681 ? If not, could you please let me know which one that is?
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.

The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
M. Manna
2018-10-18 10:55:24 UTC
Permalink
Thanks a bunch Mark.

"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If so,
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have this
option (or we might've missed it).

We are using relaxedQueryChars for now - but would like to understand the
fix you've proposed above.
Post by M. Manna
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to 8.5.34
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the
request
Post by M. Manna
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me understand
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if this
is CVE-2016-681 ? If not, could you please let me know which one that is?
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective
action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.
The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.
Mark
---------------------------------------------------------------------
Mark Thomas
2018-10-18 11:12:10 UTC
Permalink
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If so,
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have this
option (or we might've missed it).
The browsers should fix it but I doubt they will. I've seen at least one
browser vendor refuse to accept the current behaviour is a bug and claim
that the browsers are working to their own, different spec.

Investigations show that each browser has slightly different behaviour.

https://cwiki.apache.org/confluence/display/TOMCAT/Encoding+and+URIs

So even if the browsers are following a different spec, they aren't
implementing that one correctly either.

Sigh.

Mark
Post by M. Manna
We are using relaxedQueryChars for now - but would like to understand the
fix you've proposed above.
Post by M. Manna
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to 8.5.34
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the
request
Post by M. Manna
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me understand
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if this
is CVE-2016-681 ? If not, could you please let me know which one that is?
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective
action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.
The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.
Mark
---------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
M. Manna
2018-10-18 11:18:15 UTC
Permalink
I understand. We will use the connector patch for now. But thanks again for
sharing your thoughts. And the link to apache Confluence is really helpful!

Thanks,
Post by Mark Thomas
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If
so,
Post by M. Manna
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have
this
Post by M. Manna
option (or we might've missed it).
The browsers should fix it but I doubt they will. I've seen at least one
browser vendor refuse to accept the current behaviour is a bug and claim
that the browsers are working to their own, different spec.
Investigations show that each browser has slightly different behaviour.
https://cwiki.apache.org/confluence/display/TOMCAT/Encoding+and+URIs
So even if the browsers are following a different spec, they aren't
implementing that one correctly either.
Sigh.
Mark
Post by M. Manna
We are using relaxedQueryChars for now - but would like to understand the
fix you've proposed above.
Post by M. Manna
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to
8.5.34
Post by M. Manna
Post by M. Manna
Post by M. Manna
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged
at
Post by M. Manna
Post by M. Manna
Post by M. Manna
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the
request
Post by M. Manna
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me
understand
Post by M. Manna
Post by M. Manna
Post by M. Manna
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if
this
Post by M. Manna
Post by M. Manna
Post by M. Manna
is CVE-2016-681 ? If not, could you please let me know which one that
is?
Post by M. Manna
Post by M. Manna
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective
action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.
The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.
Mark
---------------------------------------------------------------------
---------------------------------------------------------------------
Johan Compagner
2018-10-18 11:17:09 UTC
Permalink
how is the browser to blame for "
defaultMessageType=true&locale=en_US&action=[key:label.edit]"

that url is not generated by a browser but by some software that uses a
browser...
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If so,
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have this
option (or we might've missed it).
We are using relaxedQueryChars for now - but would like to understand the
fix you've proposed above.
Post by M. Manna
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to
8.5.34
Post by M. Manna
Post by M. Manna
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged
at
Post by M. Manna
Post by M. Manna
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the
request
Post by M. Manna
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me
understand
Post by M. Manna
Post by M. Manna
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if
this
Post by M. Manna
Post by M. Manna
is CVE-2016-681 ? If not, could you please let me know which one that
is?
Post by M. Manna
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective
action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.
The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.
Mark
---------------------------------------------------------------------
--
Johan Compagner
Servoy
Mark Thomas
2018-10-18 11:38:44 UTC
Permalink
Post by Johan Compagner
how is the browser to blame for "
defaultMessageType=true&locale=en_US&action=[key:label.edit]"
that url is not generated by a browser but by some software that uses a
browser...
Browsers these days try to be helpful and show the user the un-encoded
URI in the address bar but send the encoded version to the server.

Personally, that sort of behaviour bugs me as I like to know exactly
what is going on - not have the technology perform 'magic' for me under
the covers to get stuff to work. But I suspect that is just me. And that
is why I do a lot of my testing in this area with telnet.

In this instance, the browser should encode the '[' and the ']' but it
does not.

See this link for how browsers should (as per the RFC) encode characters
in the path and query string compared to how they actually behave.

https://cwiki.apache.org/confluence/display/TOMCAT/Encoding+and+URIs

Mark
Post by Johan Compagner
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If so,
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have this
option (or we might've missed it).
We are using relaxedQueryChars for now - but would like to understand the
fix you've proposed above.
Post by M. Manna
Post by M. Manna
Hello,
We received in error in our application after we have upgraded to
8.5.34
Post by M. Manna
Post by M. Manna
INFO: Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged
at
Post by M. Manna
Post by M. Manna
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the
request
Post by M. Manna
target. The valid characters are defined in RFC 7230 and RFC 3986
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The issue is the action parameter value. Could someone help me
understand
Post by M. Manna
Post by M. Manna
the following?
1) Since the issue didn't happen for 8.5.28 - this means some CVE has
triggered this change to be in place. I am just trying to confirm if
this
Post by M. Manna
Post by M. Manna
is CVE-2016-681 ? If not, could you please let me know which one that
is?
Post by M. Manna
The change in request parsing was prompted by CVE-2016-6816. There
wasn't a specific attack that prompted this particular change.
CVE-2016-6816 was caused by not parsing the request line as per the
spec. Therefore, to reduce the risk of future vulnerabilities, we have
been tightening up the parsing of the request line.
Post by M. Manna
2) Apart from refactoring code, is there any recommended corrective
action?
The correct fix is to ensure that the user agents are sending
specification compliant requests.
The work-around is to use relaxedPathChars and/or relaxedQueryChars on
the Connector.
Mark
---------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org
Johan Compagner
2018-10-18 14:17:26 UTC
Permalink
Post by Mark Thomas
Post by Johan Compagner
how is the browser to blame for "
defaultMessageType=true&locale=en_US&action=[key:label.edit]"
that url is not generated by a browser but by some software that uses a
browser...
Browsers these days try to be helpful and show the user the un-encoded
URI in the address bar but send the encoded version to the server.
If this is really the case that you have a webpage where we have a href
with the right url (so the above url is encoded)
and when you click on it, the browser loads it (and i guess that should be
fine then)

But then in the browser you see the un-encoded version (as a bove) and when
you do then a refresh it goes wrong?
That would be bad...
That something the browser really needs to fix...
So yes if the browser really sets the un-encoded version in the url bar
that would be quite annoying
(bookmarking it and so on)
Mark H. Wood
2018-10-18 13:09:19 UTC
Permalink
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If so,
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have this
option (or we might've missed it).
[snip]
Post by M. Manna
Post by M. Manna
The URI we have for this problem has the following param (did work with
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The browser did not actually *compose* that parameter, did it?

If I had this problem, given only what I know from this thread, I
would suppose that the page which contained an href having such a
parameter is the source of the problem. Some link is improperly
encoded.

I would say it is debatable whether browsers should be "correcting"
hrefs which are handed to them by some site.
--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
M. Manna
2018-10-18 14:15:30 UTC
Permalink
We already know that the parameter is the issue. Having to change all the
parameter i.e. refactoring code is always the answer.
The question was more about the recommended way of handling this issue
without exposing application to any specific vulnerability. I believe Mark
T has answered this already.

Thanks,
Post by Mark H. Wood
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level ? If
so,
Post by M. Manna
is there any specific browser/update we can use? We've checked a few
browsers so far (Firefox, Edge, Chrome) and none of them seem to have
this
Post by M. Manna
option (or we might've missed it).
[snip]
Post by M. Manna
Post by M. Manna
The URI we have for this problem has the following param (did work
with
Post by M. Manna
Post by M. Manna
8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The browser did not actually *compose* that parameter, did it?
If I had this problem, given only what I know from this thread, I
would suppose that the page which contained an href having such a
parameter is the source of the problem. Some link is improperly
encoded.
I would say it is debatable whether browsers should be "correcting"
hrefs which are handed to them by some site.
--
Mark H. Wood
Lead Technology Analyst
University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu
Christopher Schultz
2018-10-18 15:46:14 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,
Post by Mark H. Wood
Post by M. Manna
Thanks a bunch Mark.
"The correct fix is to ensure that the user agents are sending
specification compliant requests." - Do you mean at browser level
? If so, is there any specific browser/update we can use? We've
checked a few browsers so far (Firefox, Edge, Chrome) and none of
them seem to have this option (or we might've missed it).
[snip]
Post by M. Manna
Post by M. Manna
The URI we have for this problem has the following param (did
work with 8.5.28)
defaultMessageType=true&locale=en_US&action=[key:label.edit]
The browser did not actually *compose* that parameter, did it?
If I had this problem, given only what I know from this thread, I
would suppose that the page which contained an href having such a
parameter is the source of the problem. Some link is improperly
encoded.
Nope. Because if the application %-encodes those bracket characters,
the browser will double-%-encode them. There is literally no way to
get a browser to send the proper RFC-compliant character string, which
is why these hacks are necessary.
Post by Mark H. Wood
I would say it is debatable whether browsers should be
"correcting" hrefs which are handed to them by some site.
They are required to do so by the HTML specification. Well... the
HTML/HTTP spec also require them to do other things which they aren't
doing, so I suppose it's all the wild west out there. Did I mention
that ignoring standards hurts everybody?

IMO, HTTP/2 should have done-away with all of this and 100% required
clients and servers to implement the spec TO THE LETTER. Basically a
lot of "servers RFC-MUST cause the client to catch fire if the
encoding is incorrect" or something similar. But we all missed an
opportunity, there.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvIqsYACgkQHPApP6U8
pFi8zA/+KtWKJrOPn85xsW59lGBvZrxRP49DoLT7Smjkolpni99zCnHbR/QYfEqP
4iAo/GS1v11OtgJythTc26Ywy70Tm4KRWyrLwb+b4NfiChBaqzcg2k1mPL3N+spp
XDC7lfjzEHRtECuO9C/P1BJZ1+yQPaqBtZlczwanCLps7rveOc2z2Tysfo8AaFAy
2aURDpn/BsUrcS08FJMHaqBBsrBbQnvQ2fOjgV1v7umE21C/oZSEZ2/DMKfw+Tx4
92TwszW+cJloWRpSYVI0GA7ydSpw8GAfBbhci9VTc4PqHTxgUVkTQcem+8D90RtR
sa8aHy1CC6VILdtunApjcJCztnKn+6+XKqNEb5gHnI7uaEcU/I/hiGxqvfGv3fDh
o+gXD3YDwyJiG3v5ldJtP5GyIlmST0ihhDS8B1PyTHaYhEabU9flZ5Ul+pibU35X
aTPfBuN3+zHqAuNCZDSekHWTOHhwCHxeMhEBSYKe/8BxSXkfDpf+n1xGQWPtI/u2
gRQeI1BsqN+PAZDPXbeSMc2ArZoqHVxQ5WCUAQfdD37PC2L4s/t72jonRXNplRLt
X7EmwuaowURf7tJMetsWLmkWxbxaHQG0Cb4BwEOl3pST2vbNg9eZ4L6yW8+VCojx
WsvwSBrbZl6k55A19ZnRnoEdCTyWZh+8cObB8HZPrCGHgTsws4E=
=2/qX
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Loading...