Add basic Mamba block
Quentin-Anthony opened this issue · 0 comments
Quentin-Anthony commented
We want to add Mamba to gpt-neox:
- Add basic mamba block, without kernels, from https://github.com/state-spaces/mamba/tree/main/mamba_ssm/modules to https://github.com/EleutherAI/gpt-neox/tree/main/megatron/model
- Add mamba kernels from https://github.com/state-spaces/mamba/tree/main/mamba_ssm/ops
- Add config options for mamba
- Add assertions to gpt-neox so that parallelism schemes and other architectures are disabled when mamba is enabled in config