Issue with tokenizers
PatiB87 opened this issue · 2 comments
Hey, on Apple Silicon M2 Max Mac with Sonoms 14.4 I get this error:
Running rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=a838c54d75003fe3 -C extra-filename=-a838c54d75003fe3 --out-dir /private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps -L dependency=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps --extern aho_corasick=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libaho_corasick-cf7f722ec57c198c.rmeta --extern cached_path=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libcached_path-a143f921f9453d11.rmeta --extern clap=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libclap-d33a0ce1bf9623e1.rmeta --extern derive_builder=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libderive_builder-2b414fe86f3f7dae.rmeta --extern dirs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libdirs-40dac3a47aae9a63.rmeta --extern esaxx_rs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libesaxx_rs-34939aa2deba81ff.rmeta --extern getrandom=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libgetrandom-d27a1b9442658553.rmeta --extern indicatif=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libindicatif-c0504f481b9fa0be.rmeta --extern itertools=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libitertools-90031d8d651a7dea.rmeta --extern lazy_static=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblazy_static-d31bb14af56b19a1.rmeta --extern log=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblog-9190404a24feee53.rmeta --extern macro_rules_attribute=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmacro_rules_attribute-04b2257e6a22bdf7.rmeta --extern monostate=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmonostate-b60e123f4c54ab51.rmeta --extern onig=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libonig-4308de740a029f74.rmeta --extern paste=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libpaste-0d5ab52f5414cf39.dylib --extern rand=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librand-f8add6882e4a6275.rmeta --extern rayon=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon-cf05149d401ef4ba.rmeta --extern rayon_cond=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon_cond-c33184e37c11d202.rmeta --extern regex=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex-afa2ad7903fb991a.rmeta --extern regex_syntax=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex_syntax-5547d7a028bf4216.rmeta --extern reqwest=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libreqwest-c185a395a99ee36b.rmeta --extern serde=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde-773632c5ddd7bea9.rmeta --extern serde_json=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde_json-1dd05001fee61fc2.rmeta --extern spm_precompiled=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libspm_precompiled-d9e9ebfcc69aabea.rmeta --extern thiserror=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libthiserror-bb5c7bf2954fac7d.rmeta --extern unicode_normalization_alignments=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_normalization_alignments-b092663a307c74ed.rmeta --extern unicode_segmentation=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_segmentation-d1383c684a17ba8d.rmeta --extern unicode_categories=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_categories-4cf6f47b5f7bd286.rmeta -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/bzip2-sys-afef6ee6e2a804f2/out/lib -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/zstd-sys-5955d8c7b8dca691/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/esaxx-rs-f21eeb3477fea3ce/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/onig_sys-32c35aba3220b1d3/outwarning: variable does not need to be mutable --> tokenizers-lib/src/models/unigram/model.rs:265:21 | 265 | let mut target_node = &mut best_path_ends_at[key_pos]; | ----^^^^^^^^^^^ | | | help: remove thismut| = note:#[warn(unused_mut)]` on by default
warning: variable does not need to be mutable
--> tokenizers-lib/src/models/unigram/model.rs:282:21
|
282 | let mut target_node = &mut best_path_ends_at[starts_at + mblen];
| ----^^^^^^^^^^^
| |
| help: remove this `mut`
warning: variable does not need to be mutable
--> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
|
200 | encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
| ----^^^^^^^
| |
| help: remove this `mut`
error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
--> tokenizers-lib/src/models/bpe/trainer.rs:526:47
|
522 | let w = &words[*i] as *const _ as *mut _;
| -------------------------------- casting happend here
...
526 | let word: &mut Word = &mut (*w);
| ^^^^^^^^^
|
= note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
= note: `#[deny(invalid_reference_casting)]` on by default
warning: `tokenizers` (lib) generated 3 warnings
error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted
Caused by:
process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=a838c54d75003fe3 -C extra-filename=-a838c54d75003fe3 --out-dir /private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps -L dependency=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps --extern aho_corasick=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libaho_corasick-cf7f722ec57c198c.rmeta --extern cached_path=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libcached_path-a143f921f9453d11.rmeta --extern clap=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libclap-d33a0ce1bf9623e1.rmeta --extern derive_builder=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libderive_builder-2b414fe86f3f7dae.rmeta --extern dirs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libdirs-40dac3a47aae9a63.rmeta --extern esaxx_rs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libesaxx_rs-34939aa2deba81ff.rmeta --extern getrandom=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libgetrandom-d27a1b9442658553.rmeta --extern indicatif=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libindicatif-c0504f481b9fa0be.rmeta --extern itertools=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libitertools-90031d8d651a7dea.rmeta --extern lazy_static=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblazy_static-d31bb14af56b19a1.rmeta --extern log=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblog-9190404a24feee53.rmeta --extern macro_rules_attribute=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmacro_rules_attribute-04b2257e6a22bdf7.rmeta --extern monostate=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmonostate-b60e123f4c54ab51.rmeta --extern onig=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libonig-4308de740a029f74.rmeta --extern paste=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libpaste-0d5ab52f5414cf39.dylib --extern rand=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librand-f8add6882e4a6275.rmeta --extern rayon=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon-cf05149d401ef4ba.rmeta --extern rayon_cond=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon_cond-c33184e37c11d202.rmeta --extern regex=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex-afa2ad7903fb991a.rmeta --extern regex_syntax=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex_syntax-5547d7a028bf4216.rmeta --extern reqwest=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libreqwest-c185a395a99ee36b.rmeta --extern serde=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde-773632c5ddd7bea9.rmeta --extern serde_json=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde_json-1dd05001fee61fc2.rmeta --extern spm_precompiled=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libspm_precompiled-d9e9ebfcc69aabea.rmeta --extern thiserror=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libthiserror-bb5c7bf2954fac7d.rmeta --extern unicode_normalization_alignments=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_normalization_alignments-b092663a307c74ed.rmeta --extern unicode_segmentation=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_segmentation-d1383c684a17ba8d.rmeta --extern unicode_categories=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_categories-4cf6f47b5f7bd286.rmeta -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/bzip2-sys-afef6ee6e2a804f2/out/lib -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/zstd-sys-5955d8c7b8dca691/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/esaxx-rs-f21eeb3477fea3ce/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/onig_sys-32c35aba3220b1d3/out` (exit status: 1)
error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101`
=> I cannot seem to fix this.
Any help?
I found the solution mentioned here, same issue:
#728 (comment)
The steps to install:
python3 -m venv .venv
source .venv/bin/activate
pip install numba inflect psutil
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
pip install transformers
git clone https://github.com/neonbjb/tortoise-tts.git
cd tortoise-tts
export RUSTFLAGS="-A invalid_reference_casting"
pip install .
I had a similar problem under Windows 11. I had the latest versions of Rust and cargo installed, latest transformers and tokenizers but no luck. I tried the above instructions but they did not work. I found that using set in a virtual environment wasn't working. I had to use:
$env:RUSTFLAGS="-A invalid_reference_casting"