microapps-publish - OOM @ 4 GB publishing thousands of files

[huntharo]

Motivations

• Publishing 10,000+ files to S3 is causing microapps-publish to use over 6 GB of RAM and crash from OOM
• The implementation is quite inefficient and is easy to fix

To-Do

• When there are more than 200 files, use "file groups" as tasks instead of individual files
• Limit the number of file groups to 100
• Use an iterative directory traversal instead of recursive