chongdashu/Tokenizers-UE5

Use Huggingface tokenizers in Unreal Engine 5

Tokenizers-UE5

Use Hugging Face Tokenizers in Unreal Engine 5

Credits and Licenses

This project integrates components from the following projects:

• tokenizers-cpp:
  • Source: https://github.com/mlc-ai/tokenizers-cpp
  • License: Apache License 2.0
  • Files: tokenizers_c.h