kayces/gpt_tokenizer

BPE tokenizer used for Dart/Flutter applications when calling ChatGPT APIs

• cargo build only re-run tokenizer/build.rs when the symbols are changed

• cocopods 1.12.0 supports slice_type: .dylib. see github

• build-apple.sh

  • if LIBNAME=libgpt_tokenizer.a, everything works well