Models fail to load
peter-ciccolo opened this issue · 0 comments
peter-ciccolo commented
I'm getting the following error in the final step of the quickstart:
unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
Input line:
./main -m models/bigcode/gpt_bigcode-santacoder-ggml.bin -p "def fibonnaci(" --top_k 0 --top_p 0.95 --temp 0.2
Output:
main: seed = 1687068338
starcoder_model_load: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49280
starcoder_model_load: n_ctx = 2048
starcoder_model_load: n_embd = 2048
starcoder_model_load: n_head = 16
starcoder_model_load: n_layer = 24
starcoder_model_load: ftype = 1003
starcoder_model_load: qntvr = 1
starcoder_model_load: ggml ctx size = 1794.97 MB
starcoder_model_load: memory size = 768.00 MB, n_mem = 49152
starcoder_model_load: unknown tensor 'transformer.h.0.attn.q_attn.weight' in model file
main: failed to load model from 'models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin'
Notable differences from the sample output:
starcoder_model_load: ftype = 1in my output vsstarcoder_model_load: ftype = 3
(quanitzed models were produced with./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3; non-quanitzed model fails with a similar error)starcoder_model_load: qntvr = 1in my output vs. no info onqntvrin the sample output
Other notes:
- this is running on a 2019 Intel MBP, not an M1
conda listis reproduced below in case I'm somehow missing a dependency
# Name Version Build Channel
accelerate 0.20.3 pyhd8ed1ab_0 conda-forge
blas 1.0 mkl
brotlipy 0.7.0 py310hca72f7f_1002
bzip2 1.0.8 h1de35cc_0
ca-certificates 2023.5.7 h8857fd0_0 conda-forge
certifi 2023.5.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h6c40b1e_3
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.0.4 py310hecd8cb5_0
cryptography 39.0.1 py310hf6deb26_2
dataclasses 0.8 pyh6d0b6a4_7
filelock 3.9.0 py310hecd8cb5_0
future 0.18.3 py310hecd8cb5_0
huggingface_hub 0.15.1 py_0 huggingface
idna 3.4 py310hecd8cb5_0
importlib-metadata 6.0.0 py310hecd8cb5_0
importlib_metadata 6.0.0 hd3eb1b0_0
intel-openmp 2023.1.0 ha357a0b_43547
joblib 1.2.0 py310hecd8cb5_0
libcxx 14.0.6 h9765a3e_0
libffi 3.4.4 hecd8cb5_0
libgfortran 5.0.0 11_3_0_hecd8cb5_28
libgfortran5 11.3.0 h9dfd629_28
libopenblas 0.3.21 h54e7dc3_0
libprotobuf 3.20.3 hfff2838_0
libuv 1.44.2 h6c40b1e_0
llvm-openmp 14.0.6 h0dcd299_0
mkl 2023.1.0 h59209a4_43558
mkl-service 2.4.0 py310h6c40b1e_1
mkl_fft 1.3.6 py310h3ea8b11_1
mkl_random 1.2.2 py310h3ea8b11_1
ncurses 6.4 hcec6c5f_0
ninja 1.10.2 hecd8cb5_5
ninja-base 1.10.2 haf03e11_5
numpy 1.24.3 py310h827a554_1
numpy-base 1.24.3 py310ha186be2_1
openssl 3.1.1 h8a1eda9_1 conda-forge
packaging 23.0 py310hecd8cb5_0
pip 23.1.2 py310hecd8cb5_0
protobuf 3.20.3 py310hcec6c5f_0
psutil 5.9.5 py310h90acd4f_0 conda-forge
pycparser 2.21 pyhd3eb1b0_0
pyopenssl 23.0.0 py310hecd8cb5_0
pysocks 1.7.1 py310hecd8cb5_0
python 3.10.11 h5ee71fb_3
python_abi 3.10 2_cp310 conda-forge
pytorch 1.13.1 cpu_py310h9e40b02_0
pyyaml 6.0 py310h6c40b1e_1
readline 8.2 hca72f7f_0
regex 2022.7.9 py310hca72f7f_0
requests 2.29.0 py310hecd8cb5_0
sacremoses master py_0 huggingface
setuptools 67.8.0 py310hecd8cb5_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h6c40b1e_0
tbb 2021.8.0 ha357a0b_0
tk 8.6.12 h5d9f67b_0
tokenizers 0.11.4 py310h8776b5c_1
tqdm 4.65.0 py310h20db666_0
transformers 4.28.1 py_0 huggingface
typing-extensions 4.6.3 py310hecd8cb5_0
typing_extensions 4.6.3 py310hecd8cb5_0
tzdata 2023c h04d1e81_0
urllib3 1.26.16 py310hecd8cb5_0
wheel 0.38.4 py310hecd8cb5_0
xz 5.4.2 h6c40b1e_0
yaml 0.2.5 haf1e3a3_0
zipp 3.11.0 py310hecd8cb5_0
zlib 1.2.13 h4dc903c_0