This repos contains a basic implementation for the model merging procedure in Fisher Mask Nodes for Language Model Merging. The code is based on the original implementations of the papers Merging Models with Fisher-Weighted Averaging, A Fast Post-Training Pruning Framework for Transformers, and Editing Models with Task Arithmetic.
To install the required packages, run:
pip install -r requirements.txtexample.ipynb contains an example of how to use the code to merge two models. To recreate our evaluations, run eval_compare.py with the appropriate config.json file. An example of the config.json file is provided in this repo.