Parallelize Gulp Tasks per file
ADCJustinH opened this issue · 3 comments
I have a lot of files in my projects and running the gulp file currently takes about half an hour all while utilizing around 8% of my cpu capacity on this entirely cpu bound work.
Currently all tasks for a file are parallelized (e.g. I can compress brotli and gzip at the same time )and while that is a 2x speedup it would be better to parallelize file processing too (e.g. files a,b,c,d are processed at the same time doing brotli and gzip = 8x speed up etc).
I tried and failed to incorporate https://github.com/mafintosh/parallel-transform to allow processing more than one file at a time but I have very limited experience with Gulp and Node so that doesn't mean much.
Node is single threaded so i have no idea how to handle more than one file at a time, sadly i don't have a project where i use this in production currently so i don't think i will have time to experiment with this.
If you only use gulp for compressing and you have some time how about creating a source generator that handels the compression?
I would gladly add it to the nuget.
Source generators can't do File IO AFAIK; but yeah I'll just write some small utility to do this since it is wasting too much time on every commit.
Didn't know that, if a source generators can't do it how about a msbuild step? https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing?view=vs-2022 (no idea if it's possible or not)