Fix issues with downloading files
Closed this issue · 7 comments
I was playing a bit with your plugin tonight — thanks for making it! Noticed a couple of things that could be improved with the image downloading.
First, I noticed that sometimes my image processing query failed. Looking at the bootstrap output, I noticed that files didn't finish downloading until after bootstrap was finished. Which means that you're not waiting for the Promise returned by createRemoteFileNode to resolve before telling Gatsby you're finished.
Second, fairly recently we added a new required field to createRemoteFileNode — parentNodeId https://www.gatsbyjs.org/packages/gatsby-source-filesystem/?=filesys#createremotefilenode
This is needed so that Gatsby knows what node the newly downloaded File is connected to. Without this, Gatsby will delete the File node when you restart the develop server.
Would appreciate a fix to these, thanks!
Thanks for the heads up @KyleAMathews !
@brxck is this something you'd have time to look into? :D
Yep I'll take a look!
Hey I've also been experiencing issues with the image download but for another reason. My images are hosted in a Google Cloud Platform Storage Bucket (which is public), however the request to grab those images includes an Authorization header which causes Google to send back a 401 (Unauthorized).
More details
In the gatsby-node file you create an instance of LocalFile with one of the options passed through being:
auth: { htaccess_user: secretKey }
This is then passed through to createRemoteFileNode (gatsby-source-filesystem) which is finally passed through to an npm module called got as one of its options. By passing this auth option got adds in an Authorization header to the HTTP request. This triggers google to send back a 401 because the contents of the header doesn't match anything google knows about.
Questions
Why do we add that auth option in behind the scenes? Is it necessary?
If it is could we provide an auth option to users of this plugin that allows them to remove this if necessary?
I'd be happy to create a pull request tonight that adds in that option if it sounds alright?
@Francesco-Lanciana I understand your problem. The auth is set that way because it is required to download the content of Stripe's File nodes. It's a good idea to allow the user to set (or not set) their own auth, which would be used except in the case of those File nodes.
Feel free to make a PR, or I could add this quickly myself. This might warrant a separate issue, as I just submitted a PR that should hopefully close this original one.
@brxck Ahh right that does make sense, however shouldn't it then be passed in to createRemoteFileNode as an option because grabbing the content from Stripe and then fetching the images using the URLs returned from Stripe are two seperate things? In that case the PR would just remove the htaccess_user option from LocalFile and provide an option for the user to provide their own auth? Let me know if my reasoning is still a bit off here.
I'll create a new issue now, happy for you to power out the PR if you would like, probably much quicker all in all.
@brxck Ended up doing the majority of the PR already so no need to do it yourself, I'll definitely submit it tomorrow (Y)