GPU build issue: failed to load model

Question

GPU build issue: failed to load model

mymagicpower opened this issue a year ago · 2 comments

GPU build script:

mkdir build
cd build
cmake -DLLAMA_CUBLAS=ON -DBUILD_SHARED_LIBS=ON ..
cmake --build . --config Release

Error log:
(Same source code, it works for CPU build.)

Extracted 'llama.dll' to 'C:\Users\admin\AppData\Local\Temp\13d0dd2e-1917-452f-a21f-aa3660df1dba-llama.dll'
Extracted 'jllama.dll' to 'C:\Users\admin\AppData\Local\Temp\f659dc4f-1852-4ef5-a333-ffb1aa37f5af-jllama.dll'
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6
llama_init_from_gpt_params: error: failed to load model 'unknown'
Exception in thread "main" de.kherud.llama.LlamaException: could not load model from given file path
at de.kherud.llama.LlamaModel.loadModel(Native Method)
at de.kherud.llama.LlamaModel.(LlamaModel.java:54)
at examples.MainExample.main(MainExample.java:32)
error loading model: failed to open unknown: No such file or directory
llama_load_model_from_file: failed to load model

Process finished with exit code 1

Answer 1 · 2023-09-21T09:30:01.000Z

I make a new CPU build again.
I forgot to copy new dll files to "D:\Program Files\Java\jdk-17\bin".
But it works on GPU.

Confused ......

"D:\Program Files\Java\jdk-17\bin\java.exe" Extracted 'llama.dll' to Extracted 'jllama.dll' to ggml_init_cublas: found 1 Device 0: NVIDIA GeForce llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - type f32: llama_model_loader: - type q4_0: llama_model_loader: - type q6_K: llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_ctx llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: n_ff llm_load_print_meta: freq_base llm_load_print_meta: freq_scale llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: using CUDA llm_load_tensors: mem required = llm_load_tensors: offloading llm_load_tensors: offloading llm_load_tensors: offloading llm_load_tensors: offloading llm_load_tensors: offloaded llm_load_tensors: VRAM used: .................................. llama_new_context_with_model: kv self size llama_new_context_with_model: llama_new_context_with_model: This is a conversation between Llama is helpful, kind, honest, "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2023.1\lib\idea_rt.jar=52979:C:\Program Files\JetBrains\IntelliJ IDEA 2023.1\bin" -Dfile.encoding=UTF-8 -classpath F:\LLM\java-llama.cpp\target\test-classes;F:\LLM\java-llama.cpp\target\classes;C:\Users\admin.m2\repository\junit\junit\4.13.1\junit-4.13.1.jar;C:\Users\admin.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Users\admin.m2\repository\org\jetbrains\annotations\24.0.1\annotations-24.0.1.jar examples.MainExample
'C:\Users\admin\AppData\Local\Temp\aefd639b-1ac4-4e2a-a535-001eedb79c60-llama.dll'
'C:\Users\admin\AppData\Local\Temp\c86ab26c-14f9-4581-b3b7-0370331a1d92-jllama.dll'
CUDA devices:
RTX 3060, compute capability 8.6
meta data with 19 key-value pairs and 291 tensors from F:\LLM\java-llama.cpp\models\llama-2-7b-chat.Q4_0.gguf (version GGUF V2 (latest))
token_embd.weight q4_0 [ 4096, 32000, 1, 1 ]
blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.0.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.0.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.0.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.0.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.0.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.1.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.1.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.1.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.10.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.10.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.10.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.11.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.11.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.11.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.12.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.12.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.12.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.13.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.13.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.13.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.14.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.14.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.14.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.15.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.15.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.15.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.16.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.16.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.16.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.17.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.17.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.17.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.18.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.18.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.18.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.19.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.19.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.19.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.2.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.2.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.2.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.20.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.20.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.20.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.21.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.21.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.21.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.22.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.22.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.22.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.23.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.23.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.23.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.3.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.3.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.3.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.4.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.4.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.4.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.5.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.5.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.5.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.6.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.6.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.6.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.7.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.7.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.7.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.8.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.8.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.8.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.9.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.9.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.9.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
output.weight q6_K [ 4096, 32000, 1, 1 ]
blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.24.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.24.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.24.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.24.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.24.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.25.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.25.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.25.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.26.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.26.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.26.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.27.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.27.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.27.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.28.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.28.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.28.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.29.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.29.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.29.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.30.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.30.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.30.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.ffn_down.weight q4_0 [ 11008, 4096, 1, 1 ]
blk.31.ffn_gate.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.31.ffn_up.weight q4_0 [ 4096, 11008, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.attn_k.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_output.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_q.weight q4_0 [ 4096, 4096, 1, 1 ]
blk.31.attn_v.weight q4_0 [ 4096, 4096, 1, 1 ]
output_norm.weight f32 [ 4096, 1, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
tokenizer.ggml.unknown_token_id u32
general.quantization_version u32
65 tensors
225 tensors
1 tensors
= GGUF V2 (latest)
= llama
= SPM
= 32000
= 0
= 4096
= 512
= 4096
= 32
= 32
= 32
= 128
= 1
= 0.0e+00
= 1.0e-06
= 11008
= 10000.0
= 1
= 7B
= mostly Q4_0
= 6.74 B
= 3.56 GiB (4.54 BPW)
= LLaMA v2
token = 1 ''
token = 2 ''
token = 0 ''
= 13 '<0x0A>'
0.09 MB
for GPU acceleration
70.41 MB (+ 256.00 MB per state)
32 repeating layers to GPU
non-repeating layers to GPU
v cache to GPU
k cache to GPU
35/35 layers to GPU
3834 MB
................................................................
= 256.00 MB
compute buffer total size = 71.97 MB
VRAM scratch buffer: 70.50 MB
User and Llama, a friendly chatbot.
good at writing, and never fails to answer any requests immediately and with precision.

User: hi
Llama: 🐮 Hey there! How can I help you today? Please let me know if you have any questions or need assistance with something.

Answer 2 · 2023-09-25T06:32:29.000Z

duplicate issue:
#11

GPU build script:

Error log: (Same source code, it works for CPU build.)

Confused ......

Error log:
(Same source code, it works for CPU build.)