IllegalArgumentException: Unexpected char ... in header value: ... at com.squareup.okhttp.Headers$Builder.checkNameAndValue (Headers.java:295)

Question

IllegalArgumentException: Unexpected char ... in header value: ... at com.squareup.okhttp.Headers$Builder.checkNameAndValue (Headers.java:295)

thest1 opened this issue 9 years ago · 40 comments

I get crash reports like that:

java.lang.IllegalArgumentException: Unexpected char 0x43a at 101 in header value: Mozilla/5.0 (Linux; U; Android 4.1.2; ru-ru; PMP7170B3G_DUO Build/JZO54K) AppleWebKit/534.30 (KHTML, как Gecko) Version/4.0 МобильныйSafari/534.30
    at com.squareup.okhttp.Headers$Builder.checkNameAndValue (Headers.java:295)
    at com.squareup.okhttp.Headers$Builder.set (Headers.java:275)
    at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.setRequestProperty (HttpURLConnectionImpl.java:526)
    <some 3rd library code I don't have access to>

Seems to happen due to non-ASCII chars in header value.
Similar to #1998. But #1998 is about response header and my smacktrace is about request header.

Thank you!

Answer 1 · 2015-12-01T04:13:34.000Z

Any chance you can get the 3rd party library to fix? It's difficult to fix this in OkHttp because the request header contains invalid data.

Answer 2 · 2015-12-03T13:38:39.000Z

No problem, I can ask library developers try/catch that.

Just wanted you to be aware about the issue.

Imagine some other developer. His project works fine. He adds

 URL.setURLStreamHandlerFactory(new OkUrlFactory(okHttpClient));

After that his project starts crashing. He can't add try/catch becasue it happens in some 3rd party library. He can't fix the wrong header becasue it's in some 3rd party library. He's not hapy with "your header is incorrect" answer because the same header worked just fine for a long time with HttpUrlConnection.

Probably not a big deal, I'm not sure. Just wanted issue to be created, may be helpful in further development :-) Thank you!

Answer 3 · 2015-12-03T19:45:11.000Z

Yeah, agreed. We decided to make it more strict than it was before on the expectation that most people weren't doing this deliberately.

Answer 4 · 2015-12-11T20:45:45.000Z

When someone passes you a Java String, rejecting non-ASCII (as you do now) is definitely a good idea.

However, the HTTP specification allows more than just ASCII (roughly, any octet except control characters). What happens when the response contains a header with non-ASCII? Do you just get rid of the non-ASCII parts? It might be worth having a byte[] interface to let clients receive non-ASCII header values uncorrupted.

Answer 5 · 2015-12-11T21:10:45.000Z

I had that issue a few minutes ago.
What I did was adding a Base64 encoded string as Header.

I used Android and the Base64 class with Base64.DEFAULT.

When I change it to Base64.URL_SAFE, its working. Problem is, the server
does not understand it anymore.

Patrick

https://www.streetlife.com.mx
powered by Alpha Wave Systems
On Dec 11, 2015 14:46, "Kannan Goundan" notifications@github.com wrote:

When someone passes you a Java String, rejecting non-ASCII (as you do
now) is definitely a good idea.

However, the HTTP specification allows more than just ASCII (roughly, any
octet except control characters). What happens when the response contains a
header with non-ASCII? Do you just get rid of the non-ASCII parts? It might
be worth having a byte[] interface to let clients receive non-ASCII
header values uncorrupted.

—
Reply to this email directly or view it on GitHub
#2016 (comment).

Answer 6 · 2015-12-11T21:15:55.000Z

@pbertsch: What encoding does the server say you should use? Is there public documentation for the server you're sending requests to? (Aside: OkHttp should be fine with both DEFAULT and URL_SAFE -- both use only printable ASCII characters.)

Answer 7 · 2015-12-11T21:48:34.000Z

I used only NO_WRAP and it seems to work at the moment. I will try to write
a test case with the Base64.DEFAULT flag only.

Patrick

https://www.streetlife.com.mx
powered by Alpha Wave Systems
On Dec 11, 2015 15:16, "Kannan Goundan" notifications@github.com wrote:

@pbertsch https://github.com/pbertsch: What encoding does the server
say you should use? Is there public documentation for the server you're
sending requests to? (Aside: OkHttp should be fine with both DEFAULT and
URL_SAFE -- both use only printable ASCII characters.)

—
Reply to this email directly or view it on GitHub
#2016 (comment).

Answer 8 · 2015-12-11T21:50:41.000Z

Ah, that's probably it. Newlines aren't allowed as header values.

Answer 9 · 2016-01-25T11:06:00.000Z

Hi, this restriction about just ASCII chars is very strange, we support chinese users etc... So almost all of our request fail when we try to send some chinese value using the header. This works fine in the HttpRequest apache. Why this restriction?

Answer 10 · 2016-01-25T12:42:08.000Z

@gudomau This restriction comes from official HTTP spec.

Answer 11 · 2016-01-25T20:27:36.000Z

Non-ASCII characters are allowed according to both the original and updated HTTP 1.1 specs.

The updated spec recommends against using non-ASCII characters for newly-defined headers, but clearly they're allowed:

Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.

Because of the way UTF-8 is encoded, all codepoints above 127 are composed of bytes whose values are 128 or greater, which means they should pass through untouched. The only characters you have to be careful with are ASCII control characters and, in certain situations, spaces, quotes, and backslashes.

Answer 12 · 2016-01-25T22:25:12.000Z

How can I do that since all headers are validated by the checkNameAndValue ?

Answer 13 · 2016-06-08T09:52:14.000Z

The HTTP Spec actually uses the word SHOULD, so isn't this restriction in the http client too harsh?

Answer 14 · 2016-10-18T16:36:41.000Z

so you're saying that "ç" and "é á ó" are "invalid data in headers"?

Sorry but I respectfully disagree, the RFC does not say that, is the client being too strict.

In my scenario, what's happening is that I use pagination with the Link header, but when my users are querying my API with special chars eg "ç ã é" this error is being thrown =|

Answer 15 · 2016-10-18T16:42:20.000Z

Alright, I can see the issue is fixed in new releases as mentioned #1998 sorry for my comment in an outdated thread.

Answer 16 · 2016-12-27T13:53:24.000Z

Whatever I use whether android.util.Base64.DEFAULT or URL_SAFE, as flag for Base64.encodeToString(...), it throws IllegalArgumentException.

        byte[] sigBytes = hmac.doFinal(reqStr.getBytes());
        String sig = Base64.encodeToString(sigBytes, Base64.URL_SAFE);
...

        Request.Builder builder = request.newBuilder()
                .header("x-guoer-ts", signature.ts)
                .header("x-guoer-nonce", signature.nonce)
                .header("x-guoer-sig", signature.sig);


java.lang.IllegalArgumentException: Unexpected char 0x0a at 44 in x-guoer-sig value: IkObQzVpEFRv06bEwc5-CnSUO4rqYX_9Gdqs4qEJ_vI=


	at okhttp3.Headers$Builder.checkNameAndValue(Headers.java:320)
	at okhttp3.Headers$Builder.set(Headers.java:300)
	at okhttp3.Request$Builder.header(Request.java:164)

Answer 17 · 2016-12-27T13:57:12.000Z

I regret that I used base64 which is always a trouble. I should use hex for it is stable and deterministic ascii. the code is running in production in iOS, it is hard to change now.

Answer 18 · 2016-12-27T14:05:58.000Z

Character 0x0a is newline. I'm guessing sig contains a newline at the end. The docs say you can pass in Base64.NO_WRAP to omit newlines.

For OkHttp: it might be good to quote strings in error messages so the mistake is more clear. Java string literal quoting or JSON quoting seem reasonable.

Answer 19 · 2016-12-27T14:21:22.000Z

Also consider using ByteString.base64() which might be a bit easier.

Answer 20 · 2017-06-08T10:37:59.000Z

Hi, just switched from Volley and this is the only issue we found, we understand what the standard suggests, however we have cookies we need to send in the header that contain Russian characters.

Is there a way we can bypass the checkNameAndValue method?

Answer 21 · 2017-06-08T11:04:02.000Z

@kriskast: What value are you passing to OkHttp and what error are you seeing, exactly?

Answer 22 · 2017-06-08T11:37:38.000Z

Caused by: java.lang.IllegalArgumentException: Unexpected char 0x416 at 9 in User-Agent value: M.R.User=Ж

builder.addHeader(entry.getKey(), "M.R.User=Ж");

This is only a small extract of it as the cookie string is longer and has more Russian characters.

Answer 23 · 2017-06-08T11:42:27.000Z

Thats not valid HTTP. You might need to base64 encode it client-side and decode it server-side. ByteString from Okio can help with that.

Answer 24 · 2017-06-08T12:12:25.000Z

Thanks for your response, to Clarify

builder.addHeader("Cookie", "M.R.User=Ж;"); where the value is part of a cookie that is generated by .NET and we have no access on how it is produced. Also we don't have access to the server to be able to decode.

What do you mean by not valid HTTP?

Answer 25 · 2017-06-08T12:18:07.000Z

What happens when you use this cookie in a browser like Chrome or Safari?

Answer 26 · 2017-06-08T12:19:08.000Z

Also – make sure you're using the latest OkHttp. We recently fixed things to be consistent with major web browsers.

Answer 27 · 2017-06-08T12:55:19.000Z

Just to be clear, HTTP is a byte-oriented protocol so I'm assuming you want "Ж" to be encoded as UTF-8, which would be a two-byte sequence: 0xd0, 0x96.

Loophole: use Headers.of(...), which doesn't seem to do any validation, though this is probably dangerous to rely on. (This only works because OkHttp's HTTP 1 and 2 codecs encode/decode header values as UTF-8, which seems problematic in general but oh well.)

Answer 28 · 2017-06-08T13:01:39.000Z

@swankjesse: Excluding control characters, UTF-8-encoded text is valid (in some sense) as an HTTP header value. See: #2016 (comment)

Answer 29 · 2017-06-08T13:38:49.000Z

Just checked on Chrome, looks like UserTitle=женский from the server is being encoded into UserTitle=%D0%B6%D0%B5%D0%BD%D1%81%D0%BA%D0%B8%D0%B9 on chrome, and if I send both of them back to the server both are accepted, would you know how I can convert it to the one in chrome so I send that?

Answer 30 · 2017-06-09T07:06:22.000Z

Correction, if I send UserTitle=%D0%B6%D0%B5%D0%BD%D1%81%D0%BA%D0%B8%D0%B9 to the server it seems it is not being decoded back, looks like chrome only displays it like that but actually sends UserTitle=женский in the background.

@cakoose 's method is the best I found so far of using Headers.of(...) which skips validation

Thanks everyone

Answer 31 · 2018-08-01T09:49:01.000Z

@pbertsch good job

Answer 32 · 2018-09-21T13:13:49.000Z

Headers.of() is your friend if you want non-ASCII characters in your headers... at least for know, unless someone decides to add ASCII validation there.

Answer 33 · 2018-09-22T00:35:22.000Z

@vishna yikes. #4283

Answer 34 · 2020-03-31T07:26:19.000Z

Hi,

I am trying to send "tête-à-tête.pdf" in my header and I am getting the same exception. But, I am able to send the same header from either Postman or using HttpsURLConnection. So, I don't understand why this restriction in okhttp when HTTP protocol supports this.

Answer 35 · 2020-03-31T12:25:43.000Z

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Answer 36 · 2020-03-31T13:13:05.000Z

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

Answer 37 · 2020-03-31T13:28:39.000Z

@rohitkum28 If you ask and self answer on stackoverflow, I'll bump up both. 25 points right there :)

Answer 38 · 2020-03-31T16:57:57.000Z

@rohitkum28 If you ask and self answer on stackoverflow, I'll bump up both. 25 points right there :)

I did ask the question on StackOverflow and now that I know the answer I added it on Stackoverflow.

Answer 39 · 2020-04-02T11:49:36.000Z

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

thats because it wasn't possible at the time with the current version

Answer 40 · 2020-07-11T08:44:16.000Z

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

thats because it wasn't possible at the time with the current version

is this method is renamed as addLenient() now?