consider using quick-fedora-mirror for fedora syncing
Closed this issue · 3 comments
https://pagure.io/quick-fedora-mirror
It's using zsh, but aside that it should be pretty portable. It's really a lot nicer/faster than bare rsync (for both us and you!)
I don't see any upside to using quick-fedora-mirror. Our upstream mirror.fcix.net is easily able to handle the 20 simultaneous rsync metadata walks that are triggered when we push-mirror out to the Micro Mirror fleet for the EPEL and fedora modules.
quick mirror would also require us to install an additional script plus supporting config files and give up flexibility just for the one project.
What behavior of the current rsync commands that we're using should we be trying to improve?
Ah, I missed that you were pulling from fcix.net. I thought you were pulling from our master mirrors.
Then yes, no advantage unless you hit problems on mirror.fcix.net with metadata walking. :)
Yeah, we definitely couldn't be adding 20 mirrors to some other projects given the state of their tier0 / tier1 mirrors, so currently our big iron in Fremont pulls from all the projects, then immediately sends out MQTT messages to push the Micro Mirror fleet to all pull the updates at the same time, since that reduces the IOPS load on our main mirror since everyone is asking for the same diff at the same time.