docker/genai-stack

genai-stack-llm-gpu-1: ggml-cuda.cu is out of memory, raises SIGABRT

clvgt12 opened this issue · 0 comments

When running a query through the pdf_bot container, the Ollama container throws a SIGABRT if it is built with Nvidia GPU support. The error indicates an out of memory condition.

From the streamlit web UI:

image

From the console:

genai-stack-pdf_bot-1     | 2024-01-06 20:15:25.882 Embedding: Using SentenceTransformer
genai-stack-pdf_bot-1     | 2024-01-06 20:15:25.882 LLM: Using Ollama: llama2
genai-stack-pdf_bot-1     | /usr/local/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:189: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
genai-stack-pdf_bot-1     |   warn_deprecated(
genai-stack-pdf_bot-1     | /usr/local/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:189: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
genai-stack-pdf_bot-1     |   warn_deprecated(
genai-stack-pdf_bot-1     | /usr/local/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:189: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
genai-stack-pdf_bot-1     |   warn_deprecated(
genai-stack-pdf_bot-1     | /usr/local/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:189: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
genai-stack-pdf_bot-1     |   warn_deprecated(
genai-stack-pdf_bot-1     | /usr/local/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:189: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
genai-stack-pdf_bot-1     |   warn_deprecated(
genai-stack-api-1         | INFO:     127.0.0.1:60418 - "GET / HTTP/1.1" 200 OK
genai-stack-llm-gpu-1     | 2024/01/06 20:15:28 shim_ext_server_linux.go:24: Updating PATH to /usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tmp/ollama1468013993/cuda
genai-stack-llm-gpu-1     | 2024/01/06 20:15:28 shim_ext_server.go:92: Loading Dynamic Shim llm server: /tmp/ollama1468013993/cuda/libext_server.so
genai-stack-llm-gpu-1     | 2024/01/06 20:15:28 gpu.go:146: 4031 MB VRAM available, loading up to 26 cuda GPU layers out of 32
genai-stack-llm-gpu-1     | 2024/01/06 20:15:28 ext_server_common.go:143: Initializing internal llama server
genai-stack-llm-gpu-1     | ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
genai-stack-llm-gpu-1     | ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
genai-stack-llm-gpu-1     | ggml_init_cublas: found 1 CUDA devices:
genai-stack-llm-gpu-1     |   Device 0: NVIDIA GeForce GTX 1050 Ti, compute capability 6.1
genai-stack-llm-gpu-1     | llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256:8934d96d3f08982e95922b2b7a2c626a1fe873d7c3b06e8e56d7bc0a1fef9246 (version GGUF V3 (latest))
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    0:                token_embd.weight q4_0     [  4096, 32000,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    1:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    2:            blk.0.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    3:            blk.0.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    4:              blk.0.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    5:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    6:              blk.0.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    7:         blk.0.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    8:              blk.0.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor    9:              blk.0.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   10:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   11:            blk.1.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   12:            blk.1.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   13:              blk.1.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   14:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   15:              blk.1.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   16:         blk.1.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   17:              blk.1.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   18:              blk.1.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   19:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   20:           blk.10.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   21:           blk.10.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   22:             blk.10.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   23:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   24:             blk.10.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   25:        blk.10.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   26:             blk.10.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   27:             blk.10.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   28:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   29:           blk.11.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   30:           blk.11.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   31:             blk.11.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   32:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   33:             blk.11.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   34:        blk.11.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   35:             blk.11.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   36:             blk.11.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   37:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   38:           blk.12.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   39:           blk.12.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   40:             blk.12.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   41:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   42:             blk.12.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   43:        blk.12.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   44:             blk.12.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   45:             blk.12.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   46:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   47:           blk.13.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   48:           blk.13.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   49:             blk.13.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   50:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   51:             blk.13.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   52:        blk.13.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   53:             blk.13.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   54:             blk.13.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   55:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   56:           blk.14.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   57:           blk.14.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   58:             blk.14.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   59:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   60:             blk.14.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   61:        blk.14.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   62:             blk.14.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   63:             blk.14.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   64:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   65:           blk.15.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   66:           blk.15.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   67:             blk.15.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   68:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   69:             blk.15.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   70:        blk.15.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   71:             blk.15.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   72:             blk.15.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   73:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   74:           blk.16.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   75:           blk.16.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   76:             blk.16.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   77:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   78:             blk.16.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   79:        blk.16.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   80:             blk.16.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   81:             blk.16.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   82:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   83:           blk.17.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   84:           blk.17.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   85:             blk.17.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   86:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   87:             blk.17.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   88:        blk.17.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   89:             blk.17.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   90:             blk.17.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   91:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   92:           blk.18.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   93:           blk.18.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   94:             blk.18.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   95:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   96:             blk.18.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   97:        blk.18.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   98:             blk.18.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor   99:             blk.18.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  100:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  101:           blk.19.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  102:           blk.19.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  103:             blk.19.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  104:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  105:             blk.19.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  106:        blk.19.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  107:             blk.19.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  108:             blk.19.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  109:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  110:            blk.2.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  111:            blk.2.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  112:              blk.2.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  113:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  114:              blk.2.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  115:         blk.2.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  116:              blk.2.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  117:              blk.2.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  118:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  119:           blk.20.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  120:           blk.20.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  121:             blk.20.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  122:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  123:             blk.20.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  124:        blk.20.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  125:             blk.20.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  126:             blk.20.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  127:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  128:           blk.21.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  129:           blk.21.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  130:             blk.21.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  131:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  132:             blk.21.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  133:        blk.21.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  134:             blk.21.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  135:             blk.21.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  136:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  137:           blk.22.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  138:           blk.22.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  139:             blk.22.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  140:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  141:             blk.22.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  142:        blk.22.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  143:             blk.22.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  144:             blk.22.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  145:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  146:           blk.23.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  147:           blk.23.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  148:             blk.23.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  149:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  150:             blk.23.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  151:        blk.23.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  152:             blk.23.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  153:             blk.23.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  154:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  155:            blk.3.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  156:            blk.3.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  157:              blk.3.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  158:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  159:              blk.3.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  160:         blk.3.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  161:              blk.3.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  162:              blk.3.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  163:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  164:            blk.4.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  165:            blk.4.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  166:              blk.4.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  167:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  168:              blk.4.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  169:         blk.4.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  170:              blk.4.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  171:              blk.4.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  172:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  173:            blk.5.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  174:            blk.5.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  175:              blk.5.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  176:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  177:              blk.5.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  178:         blk.5.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  179:              blk.5.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  180:              blk.5.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  181:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  182:            blk.6.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  183:            blk.6.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  184:              blk.6.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  185:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  186:              blk.6.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  187:         blk.6.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  188:              blk.6.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  189:              blk.6.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  190:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  191:            blk.7.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  192:            blk.7.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  193:              blk.7.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  194:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  195:              blk.7.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  196:         blk.7.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  197:              blk.7.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  198:              blk.7.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  199:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  200:            blk.8.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  201:            blk.8.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  202:              blk.8.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  203:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  204:              blk.8.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  205:         blk.8.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  206:              blk.8.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  207:              blk.8.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  208:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  209:            blk.9.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  210:            blk.9.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  211:              blk.9.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  212:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  213:              blk.9.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  214:         blk.9.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  215:              blk.9.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  216:              blk.9.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  217:                    output.weight q6_K     [  4096, 32000,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  218:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  219:           blk.24.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  220:           blk.24.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  221:             blk.24.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  222:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  223:             blk.24.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  224:        blk.24.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  225:             blk.24.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  226:             blk.24.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  227:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  228:           blk.25.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  229:           blk.25.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  230:             blk.25.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  231:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  232:             blk.25.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  233:        blk.25.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  234:             blk.25.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  235:             blk.25.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  236:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  237:           blk.26.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  238:           blk.26.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  239:             blk.26.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  240:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  241:             blk.26.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  242:        blk.26.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  243:             blk.26.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  244:             blk.26.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  245:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  246:           blk.27.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  247:           blk.27.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  248:             blk.27.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  249:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  250:             blk.27.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  251:        blk.27.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  252:             blk.27.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  253:             blk.27.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  254:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  255:           blk.28.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  256:           blk.28.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  257:             blk.28.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  258:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  259:             blk.28.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  260:        blk.28.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  261:             blk.28.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  262:             blk.28.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  263:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  264:           blk.29.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  265:           blk.29.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  266:             blk.29.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  267:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  268:             blk.29.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  269:        blk.29.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  270:             blk.29.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  271:             blk.29.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  272:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  273:           blk.30.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  274:           blk.30.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  275:             blk.30.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  276:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  277:             blk.30.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  278:        blk.30.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  279:             blk.30.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  280:             blk.30.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  281:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  282:           blk.31.ffn_down.weight q4_0     [ 11008,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  283:           blk.31.ffn_gate.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  284:             blk.31.ffn_up.weight q4_0     [  4096, 11008,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  285:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  286:             blk.31.attn_k.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  287:        blk.31.attn_output.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  288:             blk.31.attn_q.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  289:             blk.31.attn_v.weight q4_0     [  4096,  4096,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: - tensor  290:               output_norm.weight f32      [  4096,     1,     1,     1 ]
genai-stack-llm-gpu-1     | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
genai-stack-llm-gpu-1     | llama_model_loader: - kv   0:                       general.architecture str              = llama
genai-stack-llm-gpu-1     | llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
genai-stack-llm-gpu-1     | llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
genai-stack-llm-gpu-1     | llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
genai-stack-llm-gpu-1     | llama_model_loader: - kv   4:                          llama.block_count u32              = 32
genai-stack-llm-gpu-1     | llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
genai-stack-llm-gpu-1     | llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
genai-stack-llm-gpu-1     | llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
genai-stack-llm-gpu-1     | llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
genai-stack-llm-gpu-1     | llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
genai-stack-llm-gpu-1     | llama_model_loader: - kv  10:                          general.file_type u32              = 2
genai-stack-llm-gpu-1     | llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
genai-stack-llm-gpu-1     | llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
genai-stack-llm-gpu-1     | llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
genai-stack-llm-gpu-1     | llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
genai-stack-llm-gpu-1     | llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
genai-stack-llm-gpu-1     | llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
genai-stack-llm-gpu-1     | llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
genai-stack-llm-gpu-1     | llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
genai-stack-llm-gpu-1     | llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
genai-stack-llm-gpu-1     | llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
genai-stack-llm-gpu-1     | llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% if messages[0]['role'] == 'system'...
genai-stack-llm-gpu-1     | llama_model_loader: - kv  22:               general.quantization_version u32              = 2
genai-stack-llm-gpu-1     | llama_model_loader: - type  f32:   65 tensors
genai-stack-llm-gpu-1     | llama_model_loader: - type q4_0:  225 tensors
genai-stack-llm-gpu-1     | llama_model_loader: - type q6_K:    1 tensors
genai-stack-llm-gpu-1     | llm_load_vocab: special tokens definition check successful ( 259/32000 ).
genai-stack-llm-gpu-1     | llm_load_print_meta: format           = GGUF V3 (latest)
genai-stack-llm-gpu-1     | llm_load_print_meta: arch             = llama
genai-stack-llm-gpu-1     | llm_load_print_meta: vocab type       = SPM
genai-stack-llm-gpu-1     | llm_load_print_meta: n_vocab          = 32000
genai-stack-llm-gpu-1     | llm_load_print_meta: n_merges         = 0
genai-stack-llm-gpu-1     | llm_load_print_meta: n_ctx_train      = 4096
genai-stack-llm-gpu-1     | llm_load_print_meta: n_embd           = 4096
genai-stack-llm-gpu-1     | llm_load_print_meta: n_head           = 32
genai-stack-llm-gpu-1     | llm_load_print_meta: n_head_kv        = 32
genai-stack-llm-gpu-1     | llm_load_print_meta: n_layer          = 32
genai-stack-llm-gpu-1     | llm_load_print_meta: n_rot            = 128
genai-stack-llm-gpu-1     | llm_load_print_meta: n_gqa            = 1
genai-stack-llm-gpu-1     | llm_load_print_meta: f_norm_eps       = 0.0e+00
genai-stack-llm-gpu-1     | llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
genai-stack-llm-gpu-1     | llm_load_print_meta: f_clamp_kqv      = 0.0e+00
genai-stack-llm-gpu-1     | llm_load_print_meta: f_max_alibi_bias = 0.0e+00
genai-stack-llm-gpu-1     | llm_load_print_meta: n_ff             = 11008
genai-stack-llm-gpu-1     | llm_load_print_meta: n_expert         = 0
genai-stack-llm-gpu-1     | llm_load_print_meta: n_expert_used    = 0
genai-stack-llm-gpu-1     | llm_load_print_meta: rope scaling     = linear
genai-stack-llm-gpu-1     | llm_load_print_meta: freq_base_train  = 10000.0
genai-stack-llm-gpu-1     | llm_load_print_meta: freq_scale_train = 1
genai-stack-llm-gpu-1     | llm_load_print_meta: n_yarn_orig_ctx  = 4096
genai-stack-llm-gpu-1     | llm_load_print_meta: rope_finetuned   = unknown
genai-stack-llm-gpu-1     | llm_load_print_meta: model type       = 7B
genai-stack-llm-gpu-1     | llm_load_print_meta: model ftype      = Q4_0
genai-stack-llm-gpu-1     | llm_load_print_meta: model params     = 6.74 B
genai-stack-llm-gpu-1     | llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) 
genai-stack-llm-gpu-1     | llm_load_print_meta: general.name     = LLaMA v2
genai-stack-llm-gpu-1     | llm_load_print_meta: BOS token        = 1 '<s>'
genai-stack-llm-gpu-1     | llm_load_print_meta: EOS token        = 2 '</s>'
genai-stack-llm-gpu-1     | llm_load_print_meta: UNK token        = 0 '<unk>'
genai-stack-llm-gpu-1     | llm_load_print_meta: LF token         = 13 '<0x0A>'
genai-stack-llm-gpu-1     | llm_load_tensors: ggml ctx size =    0.11 MiB
genai-stack-llm-gpu-1     | llm_load_tensors: using CUDA for GPU acceleration
genai-stack-llm-gpu-1     | llm_load_tensors: mem required  =  824.54 MiB
genai-stack-llm-gpu-1     | llm_load_tensors: offloading 26 repeating layers to GPU
genai-stack-llm-gpu-1     | llm_load_tensors: offloaded 26/33 layers to GPU
genai-stack-llm-gpu-1     | llm_load_tensors: VRAM used: 2823.44 MiB
genai-stack-llm-gpu-1     | ..................................................................................................
genai-stack-llm-gpu-1     | llama_new_context_with_model: n_ctx      = 3072
genai-stack-llm-gpu-1     | llama_new_context_with_model: freq_base  = 10000.0
genai-stack-llm-gpu-1     | llama_new_context_with_model: freq_scale = 1
genai-stack-llm-gpu-1     | 
genai-stack-llm-gpu-1     | CUDA error 2 at /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:9132: out of memory
genai-stack-llm-gpu-1     | current device: 0
genai-stack-llm-gpu-1     | GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:9132: !"CUDA error"
genai-stack-llm-gpu-1     | Lazy loading /tmp/ollama1468013993/cuda/libext_server.so library
genai-stack-llm-gpu-1     | SIGABRT: abort
genai-stack-llm-gpu-1     | PC=0x7f725cd779fc m=23 sigcode=18446744073709551610
genai-stack-llm-gpu-1     | signal arrived during cgo execution