huggingface/tokenizers

Support `pip install` directly from GitHub

jamesbraza opened this issue · 3 comments

jamesbraza commented

With Python 3.12.7 and pip==24.3.1, the below does not work:

> pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers
Collecting git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers
  Cloning https://github.com/huggingface/tokenizers.git to /private/var/folders/5c/20jqnfqx4sv1_6_bdkf765cr0000gn/T/pip-req-build-b1vkwjey
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/tokenizers.git /private/var/folders/5c/20jqnfqx4sv1_6_bdkf765cr0000gn/T/pip-req-build-b1vkwjey
  Resolved https://github.com/huggingface/tokenizers.git to commit 6c15458868bc79dce134dc821c1c0359f3180f2f
ERROR: git+https://github.com/huggingface/tokenizers.git#subdirectory=tokenizers does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

It would be cool if tokenizers supported pip installing from GitHub to get the latest source code, e.g. for Python 3.13 support.

ArthurZucker commented

Yeah sounds interesting, will look into this!
Will do the release today!

gau-nernst commented

@jamesbraza I think you point to the wrong subdirectory. This should work

pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
  • 👍1
jamesbraza commented

Oh wow you are right, I did. Closing this as resolved, thanks @gau-nernst