Remove charset when posting direct_post & direct_post.jwt

[babisRoutis]

[dzarras]

Our issue stems from the usage of HttpClient.submitForm() in DefaultResponseDispatcher.
Internally this method uses the class FormDataContent for the body of the form. During initialization, this class does the following:

1. Form url-encode the content of the form and serialize it to a byte array using the UTF-8 charset
2. Hardcode the content type to application/x-www-form-urlencoded; charset=UTF-8

Here's the full implementation of this class:

public class FormDataContent(
    public val formData: Parameters
) : OutgoingContent.ByteArrayContent() {
    private val content = formData.formUrlEncode().toByteArray()

    override val contentLength: Long = content.size.toLong()
    override val contentType: ContentType = ContentType.Application.FormUrlEncoded.withCharset(Charsets.UTF_8)

    override fun bytes(): ByteArray = content
}

The spec currently expects the Content-Type to be application/x-www-form-urlencoded, meaning the charset is US-ASCII by default.

A possible way forward is to replace the usage of HttpClient.submitForm() with HttpClient.post() instead and set the body to a custom OutgoingContent.ByteArrayContent that does exactly what FormDataContent is doing, but instead encode the parameters using US-ASCII charset and then set the content type accordingly.
i.e.

 val response = httpClient.post(url.toExternalForm()) {
            val bytes = parameters.formUrlEncode().toByteArray(Charsets.US_ASCII)
            body = object : OutgoingContent.ByteArrayContent() {
                override fun bytes(): ByteArray = bytes
                override val contentLength: Long = bytes.size.toLong()
                override val contentType: ContentType = ContentType.Application.FormUrlEncoded
            }
        }

This will also require updates to the tests of DefaultDispatcher because currently the MockClient expects FormDataContent to be posted.

@babisRoutis what do you think?

[dzarras]

Just found this stackoverflow question that describes the same issue, but unfortunately has no answers.

[babisRoutis]

@dzarras If you have tested it, let's go ahead.

Added I minor comment with regards to Charset param. If it is always US_ASCII let's remove it (from params)