Inference code for the Mistral's "mixtral" 8x7B mixture of experts model. Largely based on the Mistral 7B inference repository. Requires ~100GB of VRAM.
PyTorch, SentencePiece, and xformers.
pip install -r requirements.txt
Assumes you have 8 CUDA devices. You can modify this near the bottom of main.py.
python main.py