huggingface/tokenizers

Discrepancy Between GitHub Release and NPM Package Version & Missing Dependencies

superBertBerg opened this issue · 2 comments

superBertBerg commented

Hello,

I've encountered two issues with the https://www.npmjs.com/package/tokenizers package that I believe need attention:

  1. Version Discrepancy: The latest release on GitHub is v0.15.2 (Feb 12), while the NPM package is still at 0.13.3. This gap means that users of the NPM package are missing out on the latest features and fixes. That also applies to the binaries which exist.

  2. Missing Dependencies: The older NPM version lacks several dependencies, such as tokenizers-darwin-arm64 (link), critical for ARM-based systems.

I was able to clone the repository, build, and use the project without any issues, which suggests the current GitHub version is stable. Could you provide insight into the following:

  • Is the NPM package version lag intentional, or is an update planned?
  • Are the missing dependencies like tokenizers-darwin-arm64 intended to be excluded from the NPM release?

Your guidance on these points would be greatly appreciated as it impacts project integration.

Thank you.

superBertBerg commented

#1480 (comment) that is 100% the same issue
#1403 that as well i guess