huggingface/tokenizers

Failing to build bindings with 0.19.1

Opened this issue · 4 comments

With the 0.19.1 release I get the following error:

-> % python -m build --wheel                                                                                                                                                                                                                  
* Creating isolated environment: venv+pip...                                                                                                                                                                                                  
* Installing packages in isolated environment:                                                                                                                                                                                                
  - maturin>=1.0,<2.0                                                                                                                                                                                                                         
* Getting build dependencies for wheel...                                                                                                                                                                                                     
* Building wheel...                                                                                                                                                                                                                           
Running `maturin pep517 build-wheel -i /tmp/build-env-2cqz6gzk/bin/python --compatibility off`                                                                                                                                                
error: failed to select a version for `env_logger`.                                                                                                                                                                                           
    ... required by package `tokenizers-python v0.19.1 (/home/wdouglas/clearlinux/packages/pypi-tokenizers/tokenizers-0.19.1/bindings/python)`                                                                                                
versions that meet the requirements `^0.11` (locked to 0.11.3) are: 0.11.3                                                                                                                                                                    
                                                                                                                                                                                                                                              
the package `tokenizers-python` depends on `env_logger`, with features: `anstream` but `env_logger` does not have these features.                                                                                                             
 It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.                                                                      
                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                              
failed to select a version for `env_logger` which could resolve this conflict                                                                                                                                                                 
💥 maturin failed                                                                                                                                                                                                                             
  Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?                                                                                                                                                               
  Caused by: `cargo metadata` exited with an error:                                                                    
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/build-env-2cqz6gzk/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
                                                                                                                       
ERROR Backend subprocess exited when trying to invoke build_wheel

And it looks like the issue is the Cargo.toml enforcing the use of env_logger 0.11 as there is some incompatability with anstream. If I update the config to use any version of env_logger it select 0.10 and the bindings are able to build.

It’s weird since we used this for the recent release. Will check

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Did you see anything? I am seeing the same error though I did update maturin.

Oh I think I understand the issue. I was using a single cargo vendor but this has two rust repos in it which don't share the same dependencies.

In order to build offline I try to keep a single vendor per module and this breaks that assumption.