Hugging Face accelerates distribution of models and datasets based on Dragonfly
gaius-qi opened this issue · 3 comments
Distribute Hugging Face's LFS download file request through Dragonfly P2P, refer to https://d7y.io/docs/next/setup/integration/hugging-face.
During the downloading of datasets or models, the file size is large and there are many services downloading the files at the same time. The bandwidth of the storage will reach the limit and the download will be slow.
Dragonfly can be used to eliminate the bandwidth limit of the storage through P2P technology, thereby accelerating file downloading.
Related comment:
huggingface/huggingface_hub#1780 (comment)
huggingface/huggingface_hub#1780 (comment)
huggingface/huggingface_hub#1780 (comment)
TODO List:
- Publish document to Dragonfly website, refer to https://d7y.io/docs/next/setup/integration/hugging-face.
- Finish technical article《Hugging Face accelerates distribution of models and datasets based on Dragonfly》.
- Publish technical article to CNCF public account in Wechat, refer to https://mp.weixin.qq.com/s/WnI6cIs2LTlaOB2MKwcrqA.
- Publish technical article to CNCF blog on November 16, refer to https://www.cncf.io/blog/2023/11/16/hugging-face-accelerates-distribution-of-models-and-datasets-based-on-dragonfly/.
- Publish technical article to dragonfly_oss twitter.
- Publish technical article to blog-explorers, refer to https://huggingface.co/blog/gaius-qi/hugging-face-distribution-based-on-dragonfly.
- Publish technical article to Hugging Face public account in Wechat, refer to https://mp.weixin.qq.com/s/NM1WB65KiVH6iimxBltn9Q.
@Wauplin Can you help us push this article to Hugging Face’s Blog or Hugging Face's Twitter or Hugging Face's Wechat public account? Or provide users with an acceleration reference plan in the Hugging Face website?
Hey @gaius-qi what I would suggest you is to write a community blog article. You can find all the instructions on this page: https://huggingface.co/blog-explorers. Once done, your article will be showcased in the Community posts section on https://huggingface.co/blog. Please let me know if you have any questions or if you want a quick review of the article. I think the content from this comment of you would be especially beneficial to explain when this solution is useful (and when it's not).
Hey @gaius-qi what I would suggest you is to write a community blog article. You can find all the instructions on this page: https://huggingface.co/blog-explorers. Once done, your article will be showcased in the Community posts section on https://huggingface.co/blog. Please let me know if you have any questions or if you want a quick review of the article. I think the content from this comment of you would be especially beneficial to explain when this solution is useful (and when it's not).
@Wauplin Thanks! 😊😊😊
I will do this as soon as possible so that more people can understand this efficient data distribution solution of Hugging Face & Dragonfly. 🚀🚀🚀