Proposal: Support GET bodies
Closed this issue · 32 comments
More and more, applications are built as thick clients sending complex queries to a thin server (see this). Many of those queries are requests for data, and are free from side effects. A few projects that do this (ie. support arbitrarily large browser queries to retrieve data) include:
- ElasticSearch (query API uses GET with body)
- Facebook's GraphQL (query syntax is a nested JSON structure)
- Netflix's Falcor (query syntax is a JSON structure)
- Meteor's DDP (queries support EJSON params)
RFC 7231 allows GET bodies (spec), but Chromium's XMLHttpRequest implementation does not (source). This prevents us from GETting data that requires an arbitrarily large query to fetch.
There are work arounds, each with their own drawbacks:
- Use a GET, and serialize the query into the query string (subject to max-length restrictions on URIs, not human-readable)
- Use a GET, and serialize the query into a header (semantically incorrect, often subject to practical server-side limits)
- Use a POST (semantically incorrect, imposes restrictions on browser-level caching)
- Use a custom HTTP method (doesn't play well with application/network middleware)
- Use websockets with a custom subprotocol (difficult to implement in a RESTful way)
It seems to me that the best, most backward-compatible solution is to permit bodies for GET requests with the fetch API.
What does everyone think?
(This is continuing the discussions I had with @albertywu, @pathikrit, and @igrigorik)
It seems this restriction was added here for GET, and later HEAD:
- http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.src.html.diff?r1=1.165;r2=1.166;f=h
- http://dev.w3.org/cvsweb/2006/webapi/XMLHttpRequest/Overview.src.html.diff?r1=1.182;r2=1.183;f=h
Based on this thread:
Concerns are networking libraries used by user agents, unsuspecting servers, RFC conformance (now clarified), ... Convincing those skeptical in that thread that this is now a good idea would be a good start.
These examples are somewhat compelling, though. These are very established APIs.
In particular, requiring POST for use with these APIs also prevents the Request from being stored in Cache API. Maybe we should loosen the POST constrain on Cache if fetch won't change to all body on GET.
What do you think @jakearchibald?
I'm not against allowing bodied requests into the cache, although wouldn't the body be wasted space? I guess you'd need to it replay the request at a later date to update the cache.
I think it might have to take part in the matching algorithm. If you want to cache elastic search results you would probably want each query to be a separate cache entry. It seems to me the URL and headers will be identical and only the body will differ.
@bcherny How is this sort of thing handled with http caching semantics today? Do you just cache your last query or is there something different in the headers the http cache can key on?
@wanderview I agree, the pkey for a bodied GET should be its URL-headers-body combination.
In the applications I've worked on, we issued bodied POSTs and used an app-layer cache keyed on a hash of the URL-headers-body combination.
Right, but I guess my point is that you don't get the right caching semantics for this API even if you could use GET because the http cache is not going to look at the body. So I'm not sure its worth trying to support GET with a body or POST in cache. It does seem like this needs special app layer caching semantics.
That makes sense - to properly handle GET w/ body caching, you'll need to either:
- Add a checksum header, and cache the response, or
- Set
no-cachein the response, and cache in the app layer instead
For reference, the relevant spec is here: https://tools.ietf.org/html/rfc7234#section-4
To be clear, I agree that the examples are compelling and I'm saying to make progress here someone needs to convince the people in that referenced thread. This is not a high priority for me, so if you want to drive this, please do contact them.
To give some context - RFC7230 allows all request methods to have bodies, because we want generic parsers; they shouldn't have to have knowledge of a specific method's semantics to parse a message.
The relevant text in 7231:
A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.
... and indeed this is the case; most HTTP APIs (client and server side) don't provide access to GET bodies, and many firewalls, etc. will block requests with them. It's also not uncommon for GET bodies to outright cause bugs, because most software doesn't expect them.
So, HTTP doesn't "allow" GETs to have bodies in the sense that it's recommending it as a good practice; it's just an artefact of how HTTP parsing is specified.
We're having a discussion of this issue in HTTP WG now; semantic Web folks want to be able to perform complex queries as well, and the current proposal is to use a new method. See:
http://tools.ietf.org/html/draft-snell-search-method
That isn't going forward yet, because there isn't a compelling argument against just using POST.
Besides that, note that there is no defined maximum length on URIs in HTTP; 7230 says:
Various ad hoc limitations on request-line length are found in practice. It is RECOMMENDED that all HTTP senders and recipients support, at a minimum, request-line lengths of 8000 octets.
http://httpwg.github.io/specs/rfc7230.html#request.line
I.e., this could equally be addressed in fetch by requiring support for longer URLs.
Also, AFAICT, the only one of the links above actually using a body in a HTTP GET is the first one, which says:
Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well.
The upshot of a body over a URL is that you can use a wider range of a byte and therefore put more data in less space. Perhaps H2 compression takes some of this away though.
But yeah, given the complexity of introducing this now just using POST plus some custom logic for caching might make things a lot easier for everyone involved.
@bcherny the only downside seems to be about semantics. I'll happily violate some semantics in order to avoid introducing extra complexity to this system. If you're happy with SEARCH, that works for me too.
http://stackoverflow.com/a/35685945/23607 has another use case. Submitting some data with GETs for authentication in middleware.
@annevk i don't think that issue is about auth, it's about querying a store. see the 1st use case in my original post.
the only downside seems to be about semantics. I'll happily violate some semantics in order to avoid introducing extra complexity to this system. If you're happy with SEARCH, that works for me too.
@annevk semantics are ostensibly important for a specification, but more important is the ability to cache queries without resorting to app-level caches. as more of the web moves on to sending large queries from the browser, reliable, standardized, built-in caching is a necessity.
Semantics are also important for interop. If you put arguments on a GET body and send it through a proxy or gateway cache (e.g., a CDN, Varnish, etc.), you'll likely get the wrong response.
Wanting this with near-desperation.
Perhaps use the name query or search so that GET and POST variables can both be sent at the same time.
I'm going to close this. If implementer interest materializes we can reconsider. I thought about this a bit and it seems that from a conservative security perspective we could only allow this same-origin, perhaps even then requiring client-side opt-in since currently we drop the body silently. Cross-origin gets harder as even with CORS unsuspecting servers have never been exposed and tested against this so we'd have to extend CORS. That just seems like too much complexity.
I too agree that GET should support Request bodies. This used to be supported back in 2012 to 2014 via AngularJS which I think used XMLHttpRequest; but now it isn't any more. You guys do know that there is a limit on how lone the URL can be and thus we can't use query parameters? Be cause of this limitation, we need to be able to pass data for a HTTP GET via the request body. Per the HTTP Status codes, the HTTP GET does not say you can't pass data via the Request body; so I don't know why this implementation is so limiting. Lastly there are a lot of other frameworks that interpret the specification this way too. Please reconsider and allow for HTTP GET to send Request body. It kind of stupid to have to do an HTTP POST when all we really want to do is HTTP GET. I guess it also comes down to semantics; as we don't want to do a POST unless we are actually save/changing state of some server resource. We want to be a HTTP GET when we want to retrieve resource/data from the server. We need to send Request body here because the query parameter may be too long for the URL.
At very least the, we should be able to opt into sending Request body with HTTP GET!
Thank you.
Our micro-services application recently came up against this issue when the request URL blew the server's 8K limit. Having confirmed RFC 7231 does not disallow it, just that the semantics are undefined, then between private in-house client/server applications a GET-with-body should be permissible. When we went to implement a fix, we find that using a 3rd party nodejs module minipass-fetch explicitly denies it.
This issue is going to continue to return again and again while the RFC and the fetch API specification disagree. It clearly needs to be fixed.
FYI - httpwg/http-core#202
If you won't support GET bodies, at least consider ignoring a Content-Length: 0 header when there is no body. Here is an example of a library written in Ruby which includes that header and, like the Fetch devs here, are 100% convinced they are correct and close the tickets, the Ruby devs in this link are also convinced they are correct and refuse to change their behavior.
Please work it out. I just need my 3rd libraries to stop fighting and come to an agreement. Let's have some pragmatism over pedantry and remember the "Robustness Principle":
Be conservative in what you send, be liberal in what you accept.
Here's an example of a real-world issue I can't get resolved because no one can agree on what the spec is:
https://meta.discourse.org/t/link-preview-http-get-breaks-spec
Note that this is not the issue tracker for a "third-party library"; this is the issue tracker for the standard fetch() API implemented in browsers. It has nothing to do with Node.js.
@domenic I didn't write Fetch, it's just included in software I use by those developers. Therefore it's a "3rd party library" to me. And the reason I made that distinction is because while the developers here are arguing they're following the spec and the developers in the httprb issue are arguing they're following the spec, all I want is for a link preview in my Discourse software to a GitBook URL to work. Chasing down 3rd library owners to try and get them to come to an agreement is not my idea of how to spend my day. I would encourage the Fetch developers to defend their point directly to the to the httprb developers whose HTTP requests you're blocking. I linked to the thread in my previous post. I'm just here trying to facilitate discussion so my software can work.
To be clear, any library you are using is not related to this repository. This is a repository for a web standard, not a piece of software.
To be clear, this very discussion was used as supporting evidence for the Node developers of the Fetch API to reject the idea of changing their code.
node-fetch/node-fetch#795 (comment)
This is like the spiderman meme where there's a whole circle of identical spider men standing around pointing at each other. The decision made here affect real software and that software affects me.
The Fetch Standard already requires that GET requests won't have a Content-Length header. The body is null after all, not empty. HTTP does allow such GET requests however, but servers are also allowed to reject them. So I think all pieces of software are in fact doing what they are supposed/allowed to do, it just doesn't match what you want. (And it seems that per httprb/http#487 they would take a patch, if you were to write one.)
The Fetch Standard already requires that GET requests won't have a Content-Length header.
@annevk Could help me out with a link or citation for this? One of my problems here just as a middle-man user trying to get software to play nice is I've got the httprb developer adamantly claiming he's not breaking any specs (just bending them IMO) My understanding is the Fetch standard doesn't necessarily follow the HTTP RFCs-- is that correct? A link to a spec that's actually being broken would be handy here. Otherwise, it's just endless claiming that each side is correct.
they would take a patch, if you were to write one.
Yes, I'm aware, and you keep reading in that thread, you'll see the part where I explain why I'm not the person best-positioned to submit one. This isn't just about "making code work", it's also about what's right and what's wrong and agreeing on what specs we're going to follow and what those specs do or do not say. We know that https://datatracker.ietf.org/doc/html/rfc7230#section-3.3.2 does not expressly forbid an empty content-length header with a GET. If there is another, more strict spec that Fetch is following, then at least identifying that will help.
As I said, each side here is correct. Fetch imposes stricter requirements on clients for certain cases, but that doesn't mean that other clients that do something different are incorrect.
For those trying to use fetch in the browser or in nodejs or node-fetch a GET with a request body will fail with
Request with GET/HEAD method cannot have body
Workaround for Nodejs runtime (not in browser ... JS on the server)
Unfortunately the API I'm working with I cannot change at this time so I needed a work around.
You can use the axios library to make GET with request body.
// only works in node / server -- i have not tried in the browser
const res = await axios.get("/api/devices", {
data: { deviceName: 'name' }
}) Not advocating for GET with bodies, simply sharing a solution for folks writing JS on the server to get temporarily unblocked
Correct, nothing has changed since #83 (comment).