125G of memory, executing merge-weights.py on 30B will oom

Question

125G of memory, executing merge-weights.py on 30B will oom

Chting opened this issue 2 years ago · 4 comments

Chting commented 2 years ago

Answer 1 · 2023-03-10T07:41:57.000Z

Set up at least a 40G swap, it need about 130G of memory for merging 30B

Answer 2 · 2023-03-10T07:57:47.000Z

Yes, I set 100G SWP, I don't know if 65B is enough

Answer 3 · 2023-03-10T08:03:36.000Z

I was successful. My system has 3 separate swap files. 1 ZRAM (ram compression), 1 on an NVME SSD (32GB), and 1 on a PCI-e SSD (4GB). Set the Zram and NVME priority 1 and PCI-e Priority -1. Ubuntu 22.04, Ryzen 5900x, 64GB RAM.

Answer 4 · 2023-03-10T08:52:18.000Z

Mine swap is set to 256 Gb in addition to a 128 Gb of RAM. Max swap usage was about 180 Gb while merging 65B model.