equinor/webviz-config

Upload GitHub directory takes more time than necessary

Closed this issue · 0 comments

The current function,

def upload_directory(
github_slug: str,
source_directory: Path,
commit_message: str = "Initial commit",
branch_name: str = "main",
) -> None:
with tempfile.TemporaryDirectory() as tmp_dir:
temp_dir = Path(tmp_dir)
subprocess.run(
["git", "clone", f"git@github.com:{github_slug}"],
check=True,
cwd=temp_dir,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
clone_path = temp_dir / github_slug.split("/")[1]
copy_tree(str(source_directory), str(clone_path))
# dirs_exist_ok first available in Python 3.8.
# shutil.copytree(source_directory, clone_path, dirs_exist_ok=True)
commands = [
["git", "add", "."],
["git", "commit", "-m", commit_message],
["git", "branch", "-M", branch_name],
["git", "push", "-u", "origin", branch_name],
]
for command in commands:
subprocess.run(
command,
check=True,
cwd=clone_path,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
, use more time than necessary. It copies over resources folder (which could be huge) to the cloned path, even though this subfolder is part of .gitignore and not uploaded (instead it is uploaded to blob storage in a later step).

Ignore resources folder to potentially safe quite some runtime. shutil.copytree supports e.g. ignore_patterns (https://docs.python.org/3/library/shutil.html#shutil.ignore_patterns).