integrate firerequests for faster downloads

[mratanusarkar]

I stumbled upon firerequests library by Rishiraj since its inception and have been using it in projects to download files and data. It makes the download speed super fast!

just thought of an idea to integrate the same with huggingface_hub, which might help increase the model download speed and reduce wait time!

any thoughts?

[Wauplin]

Hi @mratanusarkar, thanks for the suggestion. Though firerequests seems promising, it doesn't seem mature enough to be integrated in a library as widely used as huggingface_hub. Also, we already have a binding with hf_transfer which can be used to speed-up file transfer (both upload and download) on machine with large bandwidth. See https://github.com/huggingface/hf_transfer for more details.