bitnami/charts-syncer

Execution lasts a long time when using a target HTTPS repo

carrodher opened this issue · 0 comments

It seems currently the tool is downloading the index.yaml per chart and version, what causes execution to be delayed.

I0803 13:28:18.365226   21482 harbor.go:49] Checking if zookeeper-5.21.1 chart exists in "https://harbor-foo.bar"
I0803 13:28:18.365238   21482 utils.go:71] Repo configures basic authentication. Downloading index.yaml...
...
I0803 13:28:33.054737   21482 harbor.go:49] Checking if zookeeper-5.21.0 chart exists in "https://harbor-foo.bar"
I0803 13:28:33.054748   21482 utils.go:71] Repo configures basic authentication. Downloading index.yaml...

in the above example, first is looking for zookeeper-5.21.1 where the index.yaml is downloaded, once checked this version, the tool is going to check zookeeper-5.21.0 and the index.yaml is downloaded again.

When using HTTPS and basic authentication the download of the index.yaml is not instantaneous, taking between 15-20 seconds per chart and version. That means when synchronizing a repository with a huge number of charts/versions the execution is going to take a lot of time:

I0803 13:28:33.054737   21482 harbor.go:49] Checking if zookeeper-5.21.0 chart exists in "https://harbor-foo.bar"
I0803 13:28:33.054748   21482 utils.go:71] Repo configures basic authentication. Downloading index.yaml...
I0803 13:28:51.806207   21482 utils.go:41] Chart "zookeeper" exists in index.yaml file. Searching "5.21.0" version
I0803 13:28:51.806226   21482 utils.go:44] Version "5.21.0" found for chart "zookeeper" in index.yaml file

18 seconds in the above example, from 13:28:33 to 13:28:51 according to the timestamp.

In my current use case, just synchronizing a catalog of 39 charts with versions from the last 20 days, it is taking around 15 minutes without any push as all the charts are up-to-date.