Fetch request body as application/x-www-form-urlencoded

While upgrading ScalaJS from 1.x to 2.x, we are hitting to this issue with sending data with urlencoded type (in version 1.x we were using Ajax, now we must switch to fetch())

Example with this simple code:

import org.scalajs.dom._

val headers = new Headers()
headers.set("content-type", "application/x-www-form-urlencoded")
Fetch
      .fetch(
        "/test-api",
        new RequestInit {
          method = HttpMethod.POST
          body = "key1=value1&key2=value2"
          headers = headers
        }
      )

When executing this script in browser, the content-type header of fetch request is changed to be text/plain;charset=UTF-8. It seems ScalaJs itselves detects the body as string and decide to change the content-type to text/plain which is wrong.

If I use the same code above directly with javascript here (testing by openning the chrome console), the fetch request is sent correctly with content-type urlencoded.

window.fetch('/test-api', {
  method: 'POST',
  headers: {
  	'content-type': 'application/x-www-form-urlencoded'
  },
  body: 'key1=value1&key2=value2'
})

Could you please show me how to use this properly?

While I search some solution for this, I notice one workaround for this is to use this type URLSearchParams in the body of fetch request, this will set the content-type automatically to urlencoded. But sadly the type of request body RequestInit doesnt support this yet:

scala-js-dom/dom/src/main/scala/org/scalajs/dom/package.scala

Line 35 in e7254c8

Blob | BufferSource | FormData | String // todo: add URLSearchParams

Thanks for reporting. That is indeed strange, I can't understand why scala-js-dom would do something different than vanilla JavaScript: it is purely a facade for the JavaScript API.

What browser are you testing with?

Can you try setting the headers in Scala.js like this:

Fetch
      .fetch(
        "/test-api",
        new RequestInit {
          method = HttpMethod.POST
          body = "key1=value1&key2=value2"
          headers = js.Dictionary("content-type" -> "application/x-www-form-urlencoded")
        }
      )

This matches your JavaScript code more closely.

Similarly, can you try changing the JavaScript code to:

var headers = new Headers();
headers.set("content-type", "application/x-www-form-urlencoded");
window.fetch('/test-api', {
  method: 'POST',
  headers: headers,
  body: 'key1=value1&key2=value2'
})

This matches the original Scala.js code that has the problem.

Please try those and let me know what happens.

Regarding URLSearchParams...

But sadly the type of request body RequestInit doesnt support this yet:

It seems we have a facade for URLSearchParams already, it just needs to be added to the union there. Would you like to make a PR? :)

scala-js-dom/dom/src/main/scala/org/scalajs/dom/URLSearchParams.scala

Line 9 in e7254c8

class URLSearchParams extends js.Iterable[js.Tuple2[String, String]] {

Thanks @armanbilge for your detail reply, to make it easier testing, I setup a new project to demonstrate this issue here:

https://github.com/duchoang/scalajs-fetch-urlencoded/blob/main/client/src/main/scala/example/client.scala#L11

Button1 is using Headers type and button2 is using js.Dictionary type.

To run this example, you can run these sbt command:

$> sbt
sbt:scalajs-fetch-urlencoded> compile
sbt:scalajs-fetch-urlencoded> webserver/run

Open example web at http://localhost:8080/ and click the first button, the fetch request is sent with header content-type: text/plain

while for second button, it is sent correctly with form-urlencoded

Thanks for the suggestion with js.Dictionary, it works for our case, but why it is failing with Headers type is still a mystery, I leave this to your side to investigate more, feel free to check out the above example code and test.

In addition, I'm happy to raise a PR to let BodyInit type support this URLSearchParams, will do it soon.

Thank you for setting up the example! Very helpful. Look forward to the PR :)

Glad you have a working solution for now, we'll investigate the problem with Headers.

Ah, this is quite silly actually! This code will work.

val myHeaders = new Headers()
myHeaders.set("content-type", "application/x-www-form-urlencoded")
Fetch
      .fetch(
        "/test-api",
        new RequestInit {
          method = HttpMethod.POST
          body = "key1=value1&key2=value2"
          headers = myHeaders
        }
      )

headers was cyclically referencing itself, not the variable defined above.

Oh wow, I would never have guessed! :)

BTW, I think I did a similar thing for a different trait, but one of the scalac warnings got triggered (something like "this thing is doing nothing except referencing itself", might have been scala3)

Yes, I think also enabling warnings about unused variables (which Scala 2 has) would help catch this—not the cyclic reference of course, but the unused Headers above :)

-Xlint:unused
    Enable -Wunused:imports,privates,locals,implicits.

@armanbilge wow thanks for your debugging, didn't notice that self-referencing 😂

@duchoang btw I noticed you are using http4s in your example project, you may find this project useful: https://github.com/http4s/http4s-dom