Entropy-xcy/bitnet158

(Re)-Implementation of BitNet 1.58

Original Paper

• BitNet: Scaling 1-bit Transformers for Large Language Models
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Part of Code from

• BitNet-Transformers
• BitNet

Tested Environment:

• Python 3.9
• PyTorch 2.2.1

Installation

pip install -r requirements.txt
pip install -e .

Training with DeepSpeed

<TODO>