write-workspace flag can cause performance problems

[jeremybramwell]

I recently ran into a performance issue that I think it would be good to mention in the documentation somewhere. I had been working on a cloudbuild.yaml file for a couple weeks as I slowly worked through all the hurdles in getting building, testing and deployment steps working. During this time I was running the build locally using cloud-build-local. At some point I added the --write-workspace flag to save the resulting workspace to local disk for my inspection.

After many, many local runs I found that the start and end of the build runs were taking longer and longer. Eventually, it was taking 15+ minutes to start a local build. Once I realized it was getting slower each time the source of the problem was obvious. I was writing the post-build workspace to a subdirectory of the folder that I was passing in as the source for the workspace. So with each build the post-build-workspace folder was getting bigger since it would include a new copy of the full workspace which included the prior post-build-workspace folder, which included the prior post-build-workspace folder... I had created recursive workspace folders all the way down...

I think this may have also been causing the end of the build to take a long time as the post-build workspace was copied to disk. This would result in the next build indicating that the prior run had left around resources that needed to be cleaned up. I see some issues with others describing similar symptoms.

So I think it would be good to warn about this potential pitfall of using --write-workspace in your documentation. You might also consider logic that checks to see if the --write-workspace folder exists within the source folder, and either ignore that folder when copying the source or log a warning about the potential for a snowball effect with repeat local builds.

[bendory]

Close Issue as part of archive process; please see notice of archive status.