Uploading many files is slow
bkniffler opened this issue · 3 comments
- Do you want to request a feature or report a bug?
Feature
- What is the current behavior?
Js-Client is very slow when you need to upload >10k pages. It doesn't take advantage of parallel uploading in a single thread (say 10 files in parallel) nor does it take advantage of multiple threads (child workers). I'm pretty confident deployment could be insanely faster with these.
See: netlify/cli#318
A good example implementation of parallel and multithread work is nextjs export: https://github.com/zeit/next.js/tree/canary/packages/next/export
The client uploads 15 files at a time, which helps speed up uploading of many small files.
Netlify's API has a fairly low rate limit by default though, so you may be running into that if you are experiencing slow uploads.
I've looked into the code, it is true. Forking child processes would help hugely too. If there really is rate limits kicking in while deploying, that would be a shame. Would be cool if netlify would accept single zip file deployment.
There is a Zip upload endpoint available, but there is no de-duplicated zip upload endpoint. The zip upload requires you to upload everything you want hosted for every deploy. The file manifest endpoint enables content de-duplication which is usually faster after a slower initial deploy.
One thing you may check for in your project is if every file in your site is changing or not (e.g. if you have a timestamp in your headers or something). If this is the case you end up re-uploading every file every build. If your builds deterministically regenerate unmodified pages, you can take advantage of content deduplication, where you only upload newly changed assets.