Downloads failed as "No content-length header"
udmada opened this issue · 28 comments
Hi Nelson,
I wanted to thank you for your amazing work on the [project name] extension. I'm also interested in using R2 and/or B2 as potential destinations.
I followed your tutorial using a self-hosted Cloudflare Proxy and tried to download a 50G archive from Google Takeout. However, I received an error message from the extension that said, takeout.zip - failed - No content-length header. Based on my initial assessment, I believe this issue might be related to the Cloudflare worker.
Could you please provide me with some insights on what could be causing this issue and what steps I can take to resolve it?
Thank you again for all your hard work!
Cheers!
Adam
I'm not sure to be honest. The setup is a bit jank. I would check the extension's console and see if it can give more insight. Content-length header is usually done by the extension running HEAD against the final and direct signed takeout URL.
Also, what is the URL of the direct download link that the ZIP file resolves to? I was able to whitelist EU and US in the worker but I am a bit uncertain about AUS. This message could come from the proxy blocking it. You can neuter it yourself or just give it to me after 30 minutes either by email or just posting it here.
Right now it's whitelisting these.
Thanks for your response! I tailed Cf logs and I can see it is trying to HEAD a google endpoint that starts with storage.googleapis.com/dataliberation/. Let me try whitelisting it myself :)
The HEAD is supposed to be from the extension itself. It doesn't need to go through the proxy.
Hold up, are you trying to transload from drive?
I can see Cf logs having a 403 response to a HEAD req that is sent to [my own provy]/p/storage.googleapis.com/dataliberation/[blah blah blah]. Chrome service worker console shows the same network activity.
In terms of where I am transloading from - I did a Takepout archive (of Google Photos). I understand that ultimately photos in GPhotos are stored in GDrive, but I didn't do it directly via Drive.
This is the interface I download Takeouts from. Do you use the same? I press that blue download button.
I did and the extension successfully intercepts the download.
Hmm, I can reproduce the issue. I'm not sure what's going on at the moment, but I'll see what I can find out and fix it.
Thanks!
Just a bit more information I guess:
I tried downloading another Takeout archive from another account - same 403 resp and same error message. However, I can see that the HEAD req was sent to a different endpoint: [blah]-apidata.googleusercontent.com/download/storage/v1/b/dataliberation/[blah], which is whitelisted in the code.
The format seems to have changed slightly. There's now a %2B (+) here which wasn't in this demo URL before. My guess is that Azure is unhelpful screwing this up too.
note: these are both expired of course.
Hmm, an initial hack with that assumption doesn't work. I'll have to test this out more over next weekend with https://github.com/nelsonjchen/put-block-from-url-esc-issue-demo-server and get down to how exactly azure is mangling Google's URL.
It's definitely something to do with %2B or the + that Google just added. Armoring probably did survive Azure, but now it might be something in Javascript, Cloudflare, etc. causing this unintentional replacement.
Observe the RawPath values here.
Just a +: https://put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me+You.txt
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me+You.txt"
Percent encoded, note the server sees %2B correctly: https://put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me%2BYou.txt
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"/Me%2BYou.txt", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me%2BYou.txt"
Proxy with no encoding: https://gtr-proxy.677472.xyz/p/put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me+You.txt
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me+You.txt"
Proxy with encoding: https://gtr-proxy.677472.xyz/p/put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me%2BYou.txt
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me+You.txt"
Proxy with "azure armoring": https://gtr-proxy.677472.xyz/p/put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me%252BYou.txt
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me+You.txt"
Hmm, this is a bit frustrating. I'll have to experiment a bit with CF and see what can be done to work around this CF bug.
addEventListener("fetch", event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const url = "https://put-block-from-url-esc-issue-demo-server-3vngqvvpoq-uc.a.run.app/Me%2BYou.txt"
const originalResponse = await fetch((new URL(url)).toString())
return originalResponse
}https://cloudflareworkers.com/#a4c100303434d48b75d700b7adf8e794:http://about:blank/
apparently the URL object will cause this. It preserves %2F but will clobber %2B.
Going to reopen this issue. I queued up a Takeout for myself but it's like "Start in 2 days". OK, whatever, I'll test it then and if it works, this will close for sure.
Debug: Escaped Path Recieved
&url.URL{Scheme:"", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/Me+You.txt", RawPath:"/Me%2BYou.txt", ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}
"/Me%2BYou.txt"
I think the issue is now resolved 🎉
You'll need to pull or update your GTR proxy.
I experienced this today
could it be that %3D at the end of the url?
I have my own Google Takeout in progress right now on schedule. I'm traveling but I'll check it out when I can.
gtr-proxy is returning a 403 in background.js
Alt-Svc: h3=":443"; ma=86400
Cf-Ray: 7d5b119c0acd4798-DFW
Content-Encoding: br
Content-Type: text/plain;charset=UTF-8
Date: Sun, 11 Jun 2023 16:00:45 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=bGlo90JDsy9IBJV3SD3fdtxdObN9Y4HLIMqcb2N%2BoUdciY01GxDf3KaQDveBC9djBBKHogg9pwU8cqHOQvS81Cj9YTm6BBjCwmY5btjwIQxAqvcwGVvHwsmYEGt%2B2s8bEeMmVh1j1A%3D%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Vary: Accept-Encoding
I have my own Google Takeout in progress right now on schedule. I'm traveling but I'll check it out when I can.
appreciate it, let me know if you can reproduce. If it helps this is a google workspace account, in this case for colorado.edu
I think it has to do with this function in gtr-proxy/src/handler.ts
export function validGoogleTakeoutUrl(url: URL): boolean {
return (
url.hostname.endsWith('apidata.googleusercontent.com') &&
(url.pathname.startsWith('/download/storage/v1/b/dataliberation/o/') ||
url.pathname.startsWith('/download/storage/v1/b/takeout'))
)
}perhaps you could instead use regex something like
export function validGoogleTakeoutUrl(url: URL): boolean {
return /^.*\.(googleusercontent\.com|googleapis\.com|google\.com)$/.test(url.hostname)
}or
export function validGoogleTakeoutUrl(url: URL): boolean {
return /^.*\.(googleusercontent\.com|googleapis\.com|google\.com)$/.test(url.hostname) && /^((\/download\/storage\/v1\/b\/.*)|\/takeout-eu\/.*)/.test(url.pathname)
}I did not run these expressions through regexr to validate htem, but I hope you can understand the gist
A more broad approach could simply check that the cert authority organization is Google Trust Services LLC, though i could think of a few reasons why you wouldn't want to proxy any google url
I'm handy with regex but I also am very aware of how their terseness can sometimes have holes so I reach for a larger, more obvious self-documentation version.
Yeah, the few reasons why you wouldn't want to proxy any Google URL is right and the realization that's just what one could think of.
Was this a takeout that worked in the past or is this the first time with a Workspace account?
Also, I thought and your profile says you're in colorado. "takeout-eu"?
Also, I thought and your profile says you're in colorado. "takeout-eu"?
yeah no clue why that is the case. Basically CU Boulder finally cracked down on the once 'unlimited' google drive storage, so here I am trying to download 2tb over the past 3 weeks lol
yes it worked in the past. I exceeded the retry attempts when using the extension, If you used something like preact to inject into the dom on any page when a download is intercepted (and/or fails) it would be fairly informative. I forgot I installed this extension earlier and was very confused when all of my unrelated downloads weren't working.
Here you can see takeout downloads working, except when chrome uses 40% of my ryzen 5 5600x to write files and crashes when i drag a tab out of a window.
sorry for the trouble I would have just patched the cloudflare worker myself, but I am kind of averse to using cloudflare's dev tools as I used to work for a competitor 😅
Yeah the archive downloads have a lifespan of 5 attempts. This ephemeral-ness is one of the reasons why I made GTR too. Out of danger ASAP please thanks.
OK, I'll see about adding those URLs as allowable to the proxy. It must be unique to workspaces. I'll probably make it allow anything starting with takeout on that domain.
Can you check if those are URLs have all that is needed to download the archives with a small takeout? Like, could you copy a URL and download an archive with it shortly after it is generated with an incognito window?
yes it worked in the past. I exceeded the retry attempts when using the extension, If you used something like preact to inject into the dom on any page when a download is intercepted (and/or fails) it would be fairly informative. I forgot I installed this extension earlier and was very confused when all of my unrelated downloads weren't working.
Yeah I could do more, but I explicitly elected to take out that requirement since I didn't feel like fighting a heavily post-processed page and RE-ing it. Intercepting the downloads with the downloads API in Chromium felt much more stable.
@AskAlice I've updated the gtr-proxy I host to include those domains and path prefixes. Please give it a try and let me know if that works. I can't see the full URLs from your screenshot, but if they're signed and don't need cookies, then we should be good.
Fingers crossed! 🤞
After a few tries, and changing nothing, I got a download to start! but other files are giving me 502s
takeout-20230609T011058Z-026.tgz - failed - Failed to stage block: 502
<html> <head><title>502 Bad Gateway</title></head>
<body> <center><h1>502 Bad Gateway</h1></center>
<hr><center>cloudflare</center>
</body> </html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->and others I see
takeout-20230609T011058Z-016.tgz - failed - Failed to stage block: 400
<?xml version="1.0" encoding="utf-8"?><Error>
<Code>CannotVerifyCopySource</Code>
<Message>Bad Request RequestId:81365698-301e-0032-4752-9d31c6000000 Time:2023-06-12T17:22:31.2050574Z</Message></Error>also (might be from retrying, not sure)
Failed to stage block: 500 invalid URL
and also
Failed to commit block list: 400
I hope I can just retry those and see if it works but I've got a good portion of these downloaded though which should supplement the ones that I downloaded before chrome died under the sheer load of writing files to my hard drive. So thank you for this awesome tool!
Ok, probably as good as I can get it