Support Ordered Headers
shrimpwtf opened this issue · 23 comments
Hello!
Is it possible to add support for ordered headers for example https://github.com/ignassew/http-ordered and https://github.com/ignassew/reqwestplus have this feature, but should be definable per request.
No plans, I have fixed the request header order for different browsers
No plans, I have fixed the request header order for different browsers
The header ordering isn't correct, and this can be a make or break on some sites with strict antibot. Imo this needs to be addressed.
that's http2 headers, im talking for example how accept-encoding is set to be at the bottom in your headermap but it appears at the top in the produced output. headers are not supposed to matter the order, however browsers produce consistent ordering of them and can be fingerprinted based upon this. a good example is datadome, one header in the wrong order on a request will result in a straight up captcha ban, even if your tls fingerprint is 1:1.
You can provide a PR to resolve this issue.
You can provide a PR to resolve this issue.
only issue is it requires modifying http to allow for this, as seen for example here https://github.com/ignassew/http-ordered/commits/master/ . let's connect and can work out a clean way for this to be implemented, and i can pr this in.
Modifying http will affect all downstream
Modifying http will affect all downstream
that's the only annoying thing. i did it before with crates patches in my own fork of the above mentioned, but it got very out of date quickly. also noticed that h2 headers do not match safari's proper ordering which is m,s,a,p not m,s,p,a
that's very interesting, my ja3 is a lot shorter and produced ja4 is different to you on 17.5 as well, this is definitely something to look into as one of our fingerprints is deviating from what it actually should be, or maybe even both of us.
I have many Apple devices and their Safari Http2 header order is the same for all of them
JA4 matches on my iPhone however h2 is m,s,a,p again. tested on my 17.4 ipad which is m,s,a,p too. this is not a coincidence.
This is very interesting
This is very interesting
it is indeed. there's a lot to be added to make these requests more 1:1, happy to work with you if you want to reach out to me on telegram @shrimpwtf
iphone 15 pro on ios 17.5, set to chinese language, same fingerprint but m,s,p,a like yours.
iphone 15 pro max, english, m,s,a,p
this can't be a coincidence, this is so strange.
iphone 14, english, 17.5, m,s,p,a
I don't use telegram, so I'll discuss it here. The problem at hand is that I don't know what details cause their order to be different.
If you have a better suggestion for the header order, please put it forward.
If you have a better suggestion for the header order, please put it forward.
The way I know of solving it is by patching http and hyper and using crate patches. It's got pros and cons, but does work.
I have a messy but working example I've done here;
https://github.com/shrimpyuk/reqwestplus
[patch.crates-io] hyper = { git = "https://github.com/shrimpyuk/hyperplus.git", branch = "0.14.28-patched" } http = { git = "https://github.com/shrimpyuk/http-ordered.git" }
If there's a different way to do this that doesnt involve modifying and patching hyperplus/http, even better.
Patching seems to be a better method, and the patch needs to be applied on the user side. If you modify http directly, you must manually convert the request header.
The HeaderMap iteration is indeed consistent with the insertion order, but I don't know why it changes the order when sending the request
