Build optimization
limzykenneth opened this issue · 3 comments
Currently the build is taking a very long time with around 25-30+ minutes to complete. I'm looking into the causes and potential solutions. I have so far identified two large areas that are slow:
- Reference pages generation
- Asset optimization
For reference pages generation, I can track it all the way down to this pair of loops which for each execution need to loop through up to (1023 * 1023) entries for each page (including translated reference pages). This on my M1 macbook pro measures to about 250ms per page which adds up to a lot.
The thing I noticed for this function is that it seems to be fully functional with no side effects, ie. the same arguments should give the same output. This means that it can be memoize. Indeed by memoizing the function with lodash/memoize, individual page's generation time goes down to ~20ms. To that end I would propose memoizing all of the fully functional functions which should greatly speed up build overall. If there's no obvious drawback on this I can start working on this.
For asset optimization I'm still investigating.
For asset optimization mainly the problem is the number and size of assets we have for astro-compress to handle. astro-compress process everything in series which isn't very efficient while at the same time with larger images, especially animated ones, it seems to exponentially take more time to optimize.
With a couple memoize I'm able to reduce the general build (not including astro-compress) to around 125s but astro-compress will add another 600+s to that. I tried out another image optimization integration but it only reduces the time taken by several seconds which is not significant enough. The HTML minifier might benefit from a faster one if a faster one is out there.
Having some parallelism is likely to give more significant improvement but it is not implemented for astro-compress. That is to say I can't find anything that can significantly solve the asset optimization speed problem for now, I'm tempted to build one myself but I'll have to add this to a long list of side project ideas I have.
@outofambit For memoization, do you see any problem with them? The functions I plan to apply memoization to are the following:
- getCollectionInLocaleWithFallbacks
- getSketch - Also helps reduce traffic to OP on each build
- getRandomCurationSketches - Have the added benefit of having the same set of random sketches across translations
@limzykenneth thanks for looking at this!
memoizing those calls makes sense to me, i think that's a good list. getCollectionInLocaleWithFallbacks shouldn't change for a build run. there could be issues with dev mode, since you could change the file system without restarting the server. astro basically using server side rendering for dev mode, so i'm not sure exactly when your memoized cache would get busted by the server re-running, so that's worth testing. it'll be pretty obvious if it doesn't work for that, and we could always isolate memoization for production builds.
i took a quick swing at some code reorganization in #341 to remove some calls to the next loop you identified, but that didn't really speed up so i think moving deeper makes sense.
memoizing the API calls to OpenProcessing would be awesome, that was in my mental backlog of improvements that could happen someday!
Good shout on the dev server, I'll need to test that and provided there are flags accessible in the script to identify whether it is in production or development, it should be doable if it needs to be isolated for production build only.