clj-commons/clj-http-lite

When :body is string it is encoded using platform default encoding

lread opened this issue · 2 comments

lread commented

Symptom

Clj-http-lite was passing for me on all platforms except Windows.

Diagnosis

I captured HTTP traffic with RawCap and looked at it with WireShark.
I learned that clj-http-lite is not converting my request body to UTF-8 on Windows.

I took a peek a the code and see

(client (-> req (assoc :body (.getBytes ^String body)

This will encode the body to the platform's default character set.
I think the JDK is moving to UTF-8 for all platforms?
But I'm still seeing a default character set of windows-1252 under JDK11.

C:\Users\lee>java --version
openjdk 11.0.15 2022-04-19
OpenJDK Runtime Environment Temurin-11.0.15+10 (build 11.0.15+10)
OpenJDK 64-Bit Server VM Temurin-11.0.15+10 (build 11.0.15+10, mixed mode)

C:\Users\lee>clojure
Clojure 1.10.3
user=> (.displayName (java.nio.charset.Charset/defaultCharset))
"windows-1252"

Workaround

Clj-http-lite will pass through the body if it is not a String.

So you can do the conversion to UTF-8 yourself, I am using something similar to this:

(assoc params :body (.getBytes (json/generate-string (or payload {}))
                               "UTF-8"))

PR welcome

lread commented

Started to look at this one.

Took me a while to figure out that lein seems to be overriding the default character encoding to UTF-8 on Windows:

>lein repl
nREPL server started on port 51498 on host 127.0.0.1 - nrepl://127.0.0.1:51498
REPL-y 0.5.1, nREPL 0.9.0
Clojure 1.11.1
OpenJDK 64-Bit Client VM 11.0.15+10
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (.displayName (java.nio.charset.Charset/defaultCharset))
"UTF-8"

Clojure cli (also tested with deps.exe) does not do this on Windows:

>clojure
Clojure 1.11.0
user=> (.displayName (java.nio.charset.Charset/defaultCharset))
"windows-1252"

For this reason I'll likely flesh out the existing deps.edn and switch to running our tests from Clojure cli.