Support for MoE models (see Switch Tranformer, NLLB)

[fiqas]

Hi, have you guys considered adding a support for Mixture-of-Experts models?
They're usually quite hefty in terms of size and would be a great opportunity to have them offload parameters to CPU.

Examples:
Switch Transformers (https://huggingface.co/google/switch-base-256)
NLLB (https://github.com/facebookresearch/fairseq/tree/nllb/)