UTF-8 charset encoding

Discussion:

Tremal Naik

2007-11-16 13:14:12 UTC

Hello Tomcat users,
I'm developing an application in Jboss 4.0.2, which uses Tomcat 5.5.9
as web tier.

I'm trying to make Tomcat decoding the request body with the correct
encoding. I have problems with IE and Firefox as well. The html page
has the meta tag:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" >

all characters are displayed well in both browsers, the page encoding
appears correctly set to UTF-8. The problems arises when I try to
submit "strange" characters as currency symbols (euro, pound, yen,
...) in a form text box.

Debugging Tomcat source, I see the following in the
org.apache.catalina.connector.Request: the parseParameters() method
tries to get the correct encoding:

String enc = getCharacterEncoding();

the ContentType.getCharsetFromContentType() tries to infer the correct
character encoding from the request Content-Type header, but since it
is not set it returns null:

public static String getCharsetFromContentType(String type) {
if (type == null) {
return null;
}
int semi = type.indexOf(";");
if (semi == -1) {
return null;
}

Here the problem is the content type of the http request coming from
the browser is as follows (as grabbed by an http tracer):

Content-Type: application/x-www-form-urlencoded

Hence, it doesn't contain any ";charset=utf-8" appended, as expected.
The body looks to me correctly encoded in UTF-8 format.

Thus, the parseParameters() parses the parameters with its default
character encoding which is ISO-8859-1:

parameters.setEncoding
(org.apache.coyote.Constants.DEFAULT_CHARACTER_ENCODING);

Now, I know this is not a Tomcat problem, but I don't know how to
force the client to sent the correct information to the server. Any
ideas how to solve this?
It was nice also if I could set up Tomcat default encoding to UTF-8,
without recompiling the source (I have some requirements on the
Jboss/Tomcat: I can't use a modified or different version).

By the way, I already tried to update the file
$CATALINA_HOME/conf/server.xml for UTF-8 support by connectors with
the parameter URIEncoding="UTF-8".

Thanks for your help,

--
TREMALNAIK

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Mohsen Saboorian

2007-11-16 14:42:10 UTC

Permalink

Hi,

I don't know if this is the best solution. You can create a filter for
*.* in your web.xml, with the following piece of code:
response.setCharacterEncoding("UTF-8");

Mohsen.

Post by Tremal Naik
Hello Tomcat users,
I'm developing an application in Jboss 4.0.2, which uses Tomcat 5.5.9
as web tier.
I'm trying to make Tomcat decoding the request body with the correct
encoding. I have problems with IE and Firefox as well. The html page
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" >
all characters are displayed well in both browsers, the page encoding
appears correctly set to UTF-8. The problems arises when I try to
submit "strange" characters as currency symbols (euro, pound, yen,
...) in a form text box.
Debugging Tomcat source, I see the following in the
org.apache.catalina.connector.Request: the parseParameters() method
String enc = getCharacterEncoding();
the ContentType.getCharsetFromContentType() tries to infer the correct
character encoding from the request Content-Type header, but since it
public static String getCharsetFromContentType(String type) {
if (type == null) {
return null;
}
int semi = type.indexOf(";");
if (semi == -1) {
return null;
}
Here the problem is the content type of the http request coming from
Content-Type: application/x-www-form-urlencoded
Hence, it doesn't contain any ";charset=utf-8" appended, as expected.
The body looks to me correctly encoded in UTF-8 format.
Thus, the parseParameters() parses the parameters with its default
parameters.setEncoding
(org.apache.coyote.Constants.DEFAULT_CHARACTER_ENCODING);
Now, I know this is not a Tomcat problem, but I don't know how to
force the client to sent the correct information to the server. Any
ideas how to solve this?
It was nice also if I could set up Tomcat default encoding to UTF-8,
without recompiling the source (I have some requirements on the
Jboss/Tomcat: I can't use a modified or different version).
By the way, I already tried to update the file
$CATALINA_HOME/conf/server.xml for UTF-8 support by connectors with
the parameter URIEncoding="UTF-8".
Thanks for your help,
--
TREMALNAIK
---------------------------------------------------------------------

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Tremal Naik

2007-11-16 15:02:41 UTC

Permalink

Post by Mohsen Saboorian
I don't know if this is the best solution. You can create a filter for
response.setCharacterEncoding("UTF-8");

No, unfortunately the parameters are parsed before any filter is
invoked. Hence, a flag is set on the request that avoids subsequent
parameters re-evaluation:

parametersParsed = true;

When the filter sets the character encoding, it is too late.

thanks, anyway

Tremal Naik

2007-11-16 15:32:42 UTC

Permalink

Post by Tremal Naik
No, unfortunately the parameters are parsed before any filter is
invoked. Hence, a flag is set on the request that avoids subsequent

I tried with valve. It looks fine now.

But it's really annoying having to impose a default character encoding
using a valve. Isn't it possible to tell the browser to send the
encoding, somehow?

Thanks,

Ognjen Blagojevic

2007-11-16 16:03:03 UTC

Permalink

Hi Tremal,

Post by Tremal Naik
all characters are displayed well in both browsers, the page encoding
appears correctly set to UTF-8. The problems arises when I try to
submit "strange" characters as currency symbols (euro, pound, yen,
...) in a form text box.

Did you try to put acceptCharset="UTF-8" in the form tag?

Regards,
Ognjen

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Tremal Naik

2007-11-16 17:28:51 UTC

Permalink

Post by Ognjen Blagojevic
Did you try to put acceptCharset="UTF-8" in the form tag?

well, I'm using Struts and it looks the html:form tag doesn't allow
any acceptCharset attribute. I tried to set the enctype attribute, but
with no effect.

Thanks,

Ognjen Blagojevic

2007-11-19 09:57:09 UTC

Permalink

Post by Tremal Naik

Post by Ognjen Blagojevic
Did you try to put acceptCharset="UTF-8" in the form tag?

well, I'm using Struts and it looks the html:form tag doesn't allow
any acceptCharset attribute. I tried to set the enctype attribute, but
with no effect.

Which version of Struts are you using? 1.2.7 does support acceptCharset,
as you can see here:

http://struts.apache.org/1.2.7/userGuide/struts-html.html#form

Regards,
Ognjen

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Tremal Naik

2007-11-19 10:06:07 UTC

Permalink

Post by Ognjen Blagojevic
Which version of Struts are you using? 1.2.7 does support acceptCharset,

Oh, yes, you're right. I'm using version 1.1, that's why probably I
don't have that option available. Unfortunately I'm not allowed to
upgrade to a newer version...

Thanks, again

TN

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Ognjen Blagojevic

2007-11-19 10:21:59 UTC

Permalink

Post by Tremal Naik
Oh, yes, you're right. I'm using version 1.1, that's why probably I
don't have that option available. Unfortunately I'm not allowed to
upgrade to a newer version...

I suppose you are using ActionForms. Try to extend ActionForm overriding
your reset method which will set the character encoding, before the
parameters are processed. Something like this:

public class UTF8ActionForm extends ActionForm {
public void reset(ActionMapping mapping,
HttpServletRequest request) {
super.reset(mapping, request);
try {
request.setCharacterEncoding("UTF-8");
} catch (UnsupportedEncodingException ioe) {
throw new RuntimeException(ioe.getMessage());
}
}
}

And then just instead of using ActionForm use UTF8ActionForm.

And try asking on Struts mailing list, there should be more people who
solved the same problem.

Regards,
Ognjen

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Tremal Naik

2007-11-19 12:38:21 UTC

Permalink

Post by Ognjen Blagojevic
I suppose you are using ActionForms. Try to extend ActionForm overriding
your reset method which will set the character encoding, before the

well, I solved with a Valve that impose a default encoding of "UTF-8.
This is the perfect solution for me since I read request parameters in
other valves that come after in the chain, and I want them correctly
encoded. Hence, when the Struts takes control of the application, the
request parameters are parsed with the already set correct encoding.

Thanks,

TN

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Michael

2007-11-16 17:33:44 UTC

Permalink

start your JVM with -Dfile.encoding=UTF-8 and try again!

bye

--
<NO> OOXML - Say NO To Microsoft Office broken standard
http://www.noooxml.org

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Mark Thomas

2007-11-16 19:19:57 UTC

Permalink

Post by Michael
start your JVM with -Dfile.encoding=UTF-8 and try again!

That property is read-only on some JVMs and is not the way to achieve what
the OP is trying to do.

Some standard text I wrote a while ago follows. The most useful bit is
probably the URIEncoding attribute on the connector.

Character encoding summary
==========================

There are a number of situations where there may be a requirement to use non-
US ASCII characters in a URI. These include:
- Parameters in the query string
- Servlet paths

There is a standard for encoding URIs (http://www.w3.org/International/O-URL-
code.html) but this standard is not consistently followed by clients. This
causes a number of problems.

The functionality provided by Tomcat (4 and 5) to handle this less than ideal
situation is described below.

1. The Coyote HTTP/1.1 connector has a useBodyEncodingForURI attribute
which if set to true will use the request body encoding to decode the URI
query parameters.
- The default value is true for TC4 (breaks spec but gives consistent
behaviour across TC4 versions)
- The default value is false for TC5 (spec compliant but there may be
migration issues for some apps)
2. The Coyote HTTP/1.1 connector has a URIEncoding attribute which defaults
to ISO-8859-1.
3. The parameters class (o.a.t.u.http.Parameters) has a QueryStringEncoding
field which defaults to the URIEncoding. It must be set before the
parameters are parsed to have an effect.

Things to note regarding the servlet API:
1. HttpServletRequest.setCharacterEncoding() normally only applies to the
request body NOT the URI.
2. HttpServletRequest.getPathInfo() is decoded by the web container.
3. HttpServletRequest.getRequestURI() is not decoded by container.

Other tips:
1. Use POST with forms to return parameters as the parameters are then part of
the request body.

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org

Tremal Naik

2007-11-19 09:51:23 UTC

Permalink

Post by Mark Thomas
Some standard text I wrote a while ago follows. The most useful bit is
probably the URIEncoding attribute on the connector.

Thanks Mark, I think I read your paper somewhere before I decided to
write to this help request. In fact, if you read carefully the last
paragraph of my original post, I said that I already tried setting the
connector parameter URIEncoding="UTF-8". Unfortunately, this parameter
is read after the first parse occurred, as I explained there, making
the subsequent setting useless in terms of request body parsing. I
think (I didn't investigate further) this is due a Valve I wrote that
accesses the request parameters before the Connector.getURIEncoding()
is invoked the first time. I solved this with another Valve (which
code is copy/pasted from the SetCharacterEncodingFilter) that is
invoked as the first in the chain.

Thanks,

TN

---------------------------------------------------------------------
To start a new topic, e-mail: ***@tomcat.apache.org
To unsubscribe, e-mail: users-***@tomcat.apache.org
For additional commands, e-mail: users-***@tomcat.apache.org