huggingface/transformers

Unable to set up developer environment on Mac M1

KMFODA opened this issue · 7 comments

KMFODA commented

System Info

As MacBooks with the M1 chip need tensorflow-macos installed instead of tensorflow>=2.3 (as listed in the setup.py) file trying to set up a developer environment on a M1 MacBook produces the following error:

ERROR: Could not find a version that satisfies the requirement tensorflow>=2.3; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for tensorflow>=2.3; extra == "dev"

Is there any way around this? I tried replacing with tensorflow-macos but that creates a myriad of other issues when trying to set up the developer environment.

transformers 4.21.0
python 3.9.12

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

pip install -e ".[dev]"

Expected behavior

Developer environment should be installed without errors.

ArthurZucker commented

Hey! Thanks for the issue. I managed to get it work doing the following :

  1. Replace tensorflow with tensorflow-macos in the setup.py file.
  2. Install particular dependencies manually
brew install llvm
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
pip install fugashi==1.1.2a6
pip install numba
brew install cmake 
brew install rust 
conda install -c conda-forge onnxruntime
  1. Just run pip install -e ".[dev]" and it should work :).
  • 👍1
KMFODA commented

Thanks so much for the prompt response @ArthurZucker. This is great although when I run the final pip install -e ".[dev]" command I get the following error:

ERROR: Could not find a version that satisfies the requirement tensorflow-text; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for tensorflow-text; extra == "dev"

when I remove tensor flow-text from setup.py the pip install -e ".[dev]" runs fine. Did you experience a similar issue?

ArthurZucker commented

At the time (~2 month ago), I did not. But it seems like it is a pretty known issue mentioned here. You should apparently build tensorflow-text from source or use the python wheel made available 👍🏻 (installing with pip install tensorflow-text does not work either)

KMFODA commented

Amazing that seems to work for now. Thanks for the help!

  • ❤️1
ArEnSc commented

sadly this guide no longer works as
using conda 3.9.11
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal also fails

ArEnSc commented

The issue is you want to use miniforge (community version of conda) and not the conda...

LWprogramming commented

Following up here with what I've tried so far with my issue, in case it's useful for anyone in the future:

  • tensorflow-text doesn't have a Python 3.11 prebuilt wheel, so I used 3.10 for everything.
  • Install miniforge as mentioned here instead of conda, because it has tensorflow-deps and conda doesn't.
  • follow instructions in here, in an active conda environment.
  • Remove decord manually from setup.py because it's not actively maintained anymore according to this, and my use case shouldn't need decord.
  • While conda environment is active, create virtual environment and try installing:
    python3.10 -m venv venv && source venv/bin/activate && pip install --upgrade pip && pip uninstall transformers && pip install -e ".[dev]"
  • I then got a resolution error:
ERROR: Cannot install transformers and transformers[dev]==4.30.0.dev0 because these package versions have conflicting dependencies.

ERROR: Cannot install transformers and transformers[dev]==4.30.0.dev0 because these package versions have conflicting dependencies.

The conflict is caused by:
    transformers[dev] 4.30.0.dev0 depends on jax!=0.3.2, <=0.3.6 and >=0.2.8; extra == "dev"
    flax 0.6.9 depends on jax>=0.4.2
    transformers[dev] 4.30.0.dev0 depends on jax!=0.3.2, <=0.3.6 and >=0.2.8; extra == "dev"
    flax 0.6.8 depends on jax>=0.4.2
# ... many similar lines of text
  • Installing with pip install -e ".[quality]" instead of dev worked, which is fine for my use case because I'm not modifying anything with Jax but not a complete solution unfortunately.
  • 👍1