keldenl/gpt-llama.cpp

Error after connecting to chat UI and sending message (Windows)

InfernalDread opened this issue · 51 comments

Hello again,

I went through the instructions to connect to the Chat UI and it worked! However, after sending a test "Hello" message, I got the following error code on my cmd:

(gptllamaapi) C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\gpt_llama_cpp\gpt-llama.cpp>npm start

gpt-llama.cpp@0.1.8 start
node index.js

Server is listening on:

  • localhost:443
  • 192.168.0.34:443 (for other devices on the same network)

--LLAMA.CPP SPAWNED--
C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin/llama.cpp/main -m C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin --temp 0 --n_predict 1000 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942 --reverse-prompt user: --reverse-prompt
user --reverse-prompt system: --reverse-prompt
system --reverse-prompt ## --reverse-prompt

--reverse-prompt ### -i -p ### Instructions

Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.

Inputs

system: You are a helpful assistant.
user: How are you?
assistant: Hi, how may I help you today?
system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.

Response

user: Hello
assistant:

--REQUEST--
user: Hello
node:events:515
throw er; // Unhandled 'error' event
^

Error: spawn C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin/llama.cpp/main ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -4058,
code: 'ENOENT',
syscall: 'spawn C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin/llama.cpp/main',
path: 'C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin/llama.cpp/main',
spawnargs: [
'-m',
'C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\llama_cpp\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin',
'--temp',
0,
'--n_predict',
1000,
'--top_p',
'0.1',
'--top_k',
'40',
'-b',
'512',
'-c',
'2048',
'--repeat_penalty',
'1.1764705882352942',
'--reverse-prompt',
'user:',
'--reverse-prompt',
'\nuser',
'--reverse-prompt',
'system:',
'--reverse-prompt',
'\nsystem',
'--reverse-prompt',
'##',
'--reverse-prompt',
'\n##',
'--reverse-prompt',
'###',
'-i',
'-p',
'### Instructions\n' +
'Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.\n' +
'\n' +
'### Inputs\n' +
'system: You are a helpful assistant.\n' +
'user: How are you?\n' +
'assistant: Hi, how may I help you today?\n' +
"system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.\n" +
'\n' +
'### Response\n' +
'user: Hello\n' +
'assistant:'
]
}

Node.js v18.4.0

(gptllamaapi) C:\Users\gamet\Documents\Transfer_To_External_Hard_Drive\gpt_llama_cpp\gpt-llama.cpp>

o crap it must be the front and back slashes i have hardcoded.. let me make a change..

Something that was interesting was that the code would add "/llama.cpp/main" after my path for specifying the .bin file in the llama.cpp folder, I am not sure if it has anything to do with that though

oh I see, lol, similar thought processes

I think the hardcoded values are in the "utils.js" file, near the end of the code. I am assuming that you would have to remove them and include a variable to take the path that the user input?

sorry i'll push out a fix tonight that supports both windows and mac (for front and backslashes). for now, u can try replacing all the front slashes with backslashes and lmk if that works?

oh ok, I can try that, hopefully it works lol

Stumbled upon this issue as well. Will also need to change scriptPath on line 80 in chatRoutes.js

yup, made note of that just now, I will try to make the changes and see if it works

hey @brandonvessel @InfernalDread , i just pushed a change that should "fix" \ and / in paths for windows and mac in 3055f5e

do a fresh pull and lmk if it works! (i haven't published it to npm yet pending this test)

Not sure if this is on my end. Getting the following error:

  const readable = new ReadableStream({
                   ^

ReferenceError: ReadableStream is not defined
    at file:///E:/Projects/_AI/gpt-llama.cpp/routes/chatRoutes.js:164:20
    at Layer.handle [as handle_request] (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\layer.js:95:5)
    at next (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\route.js:144:13)
    at Route.dispatch (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\route.js:114:3)
    at Layer.handle [as handle_request] (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\layer.js:95:5)
    at E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\index.js:284:15
    at Function.process_params (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\index.js:346:12)
    at next (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\index.js:280:10)
    at Function.handle (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\index.js:175:3)
    at router (E:\Projects\_AI\gpt-llama.cpp\node_modules\express\lib\router\index.js:47:12)```

hmm could you run npm i again?

No updates needed. npm version 9.6.5 if that helps. It did print the --LLAMA.CPP SPAWNED-- log message and the raw query information, but errors out on trying to read the response.

this is actually probably related to readablestream only being supported on node 16+. can you double check your node version? you can check this by doing node -v. i'm on v19.8.1

v16.14.0

Going to upgrade node and see if that fixes it

Screenshot 2023-04-19 at 5 54 41 PM

actually i lied @brandonvessel , it requires v18+. upgrading should do the trick

source

I still am getting this error (also, yes, I changed the port to 8080):

(venv) C:\Users\Mike's PC\Documents\transfer_to_external_storage\gpt_llama_cpp\gpt-llama.cpp>npm start

gpt-llama.cpp@0.1.8 start
node index.js

Server is listening on:

  • localhost:8080
  • 192.168.0.33:8080 (for other devices on the same network)

--LLAMA.CPP SPAWNED--
C:\Users\Mike's\llama.cpp\main -m C:\Users\Mike's --temp 0 --n_predict 1000 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942 --reverse-prompt user: --reverse-prompt
user --reverse-prompt system: --reverse-prompt
system --reverse-prompt ## --reverse-prompt

--reverse-prompt ### -i -p ### Instructions

Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.

Inputs

system: You are a helpful assistant.
user: How are you?
assistant: Hi, how may I help you today?
system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.

Response

user: hello
assistant:

--REQUEST--
user: hello
node:events:515
throw er; // Unhandled 'error' event
^

Error: spawn C:\Users\Mike's\llama.cpp\main ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -4058,
code: 'ENOENT',
syscall: "spawn C:\Users\Mike's\llama.cpp\main",
path: "C:\Users\Mike's\llama.cpp\main",
spawnargs: [
'-m',
"C:\Users\Mike's",
'--temp',
0,
'--n_predict',
1000,
'--top_p',
'0.1',
'--top_k',
'40',
'-b',
'512',
'-c',
'2048',
'--repeat_penalty',
'1.1764705882352942',
'--reverse-prompt',
'user:',
'--reverse-prompt',
'\nuser',
'--reverse-prompt',
'system:',
'--reverse-prompt',
'\nsystem',
'--reverse-prompt',
'##',
'--reverse-prompt',
'\n##',
'--reverse-prompt',
'###',
'-i',
'-p',
'### Instructions\n' +
'Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.\n' +
'\n' +
'### Inputs\n' +
'system: You are a helpful assistant.\n' +
'user: How are you?\n' +
'assistant: Hi, how may I help you today?\n' +
"system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.\n" +
'\n' +
'### Response\n' +
'user: hello\n' +
'assistant:'
]
}

Node.js v18.4.0

(venv) C:\Users\Mike's PC\Documents\transfer_to_external_storage\gpt_llama_cpp\gpt-llama.cpp>

i'm going to add a minimum node version of 18 if that fixes it. unless something else comes up that forces backwards compatibility with the experimental 16.5-17.5 version i'm going to stick with that

@keldenl That fixed it! Looks like my instance is working as expected now

@InfernalDread does running

C:\Users\Mike's\llama.cpp\main -m C:\Users\Mike's --temp 0 --n_predict 1000 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942 -p "the sky is"

work for you?

update: nvm, i don't see your model path at all, C:\Users\Mike's\llama.cpp\main -m C:\Users\Mike's isn't right. it should show your model path

it says path cannot be specified. I believe I correctly authorized the model. Let me check again

mine looks like this: ../llama.cpp/main -m ../llama.cpp/models/vicuna/13B/ggml-vicuna-unfiltered-13b-4bit.bin --temp 1 --n_predict 1000 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942

can you paste the api key you provided for your model?

C:\Users\Mike's PC\Documents\transfer_to_external_storage\llama_cpp\3\llama-master-f7d0509-bin-win-avx-x64\ggml-model-q4_0.bin

EDIT: do I have to create a specific directory to match something like yours?

ahhhh.. your llama.cpp path looks vastly different from what i expected

i believe the standard setup should be

\llama.cpp\models\ggml-model-q4_0.bin

the problematic parts of your path is the fact that i rely on

  • llama.cpp being the project folder name
  • models being stored in llama.cpp\models\<HERE> helps, but not required. it relies on llama.cpp being in the path tho

any chance you could tweak your path and folder naming to fix that?

oh ya, of course I can, let me get on that right now!

all the other ".exe" files like "main.exe" should also be there too right?

did you have to include .exe when getting it working with llama.cpp? what's an example command you run in cmd to get llama.cpp working (not gpt-llama.cpp)

I actually do not remember if I had to, I haven't used llama.cpp in a while, but I know that it worked

I am now using this as the model path:

C:\Users\Mike's PC\Documents\llama.cpp\models\ggml-model-q4_0.bin

will this work better?

EDIT: I forgot to include the ".bin" in a folder in the models folder, does that matter?

I think it would be best for me to try and reinstall everything over again

interesting, even after completely installing it fresh, and using this path:

C:\Users\Mike's PC\Documents\llama.cpp\models\vicuna_13b\ggml-model-q4_0.bin

I still get this error:

(venv) C:\Users\Mike's PC\Documents\transfer_to_external_storage\gpt_llama_cpp\gpt-llama.cpp>npm start

gpt-llama.cpp@0.1.8 start
node index.js

Server is listening on:

  • localhost:8080
  • 192.168.0.33:8080 (for other devices on the same network)

--LLAMA.CPP SPAWNED--
C:\Users\Mike's\llama.cpp\main -m C:\Users\Mike's --temp 0 --n_predict 1000 --top_p 0.1 --top_k 40 -b 512 -c 2048 --repeat_penalty 1.1764705882352942 --reverse-prompt user: --reverse-prompt
user --reverse-prompt system: --reverse-prompt
system --reverse-prompt ## --reverse-prompt

--reverse-prompt ### -i -p ### Instructions

Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.

Inputs

system: You are a helpful assistant.
user: How are you?
assistant: Hi, how may I help you today?
system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.

Response

user: hello
assistant:

--REQUEST--
user: hello
node:events:515
throw er; // Unhandled 'error' event
^

Error: spawn C:\Users\Mike's\llama.cpp\main ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -4058,
code: 'ENOENT',
syscall: "spawn C:\Users\Mike's\llama.cpp\main",
path: "C:\Users\Mike's\llama.cpp\main",
spawnargs: [
'-m',
"C:\Users\Mike's",
'--temp',
0,
'--n_predict',
1000,
'--top_p',
'0.1',
'--top_k',
'40',
'-b',
'512',
'-c',
'2048',
'--repeat_penalty',
'1.1764705882352942',
'--reverse-prompt',
'user:',
'--reverse-prompt',
'\nuser',
'--reverse-prompt',
'system:',
'--reverse-prompt',
'\nsystem',
'--reverse-prompt',
'##',
'--reverse-prompt',
'\n##',
'--reverse-prompt',
'###',
'-i',
'-p',
'### Instructions\n' +
'Complete the following chat conversation between the user and the assistant. System messages should be strictly followed as additional instructions.\n' +
'\n' +
'### Inputs\n' +
'system: You are a helpful assistant.\n' +
'user: How are you?\n' +
'assistant: Hi, how may I help you today?\n' +
"system: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.\n" +
'\n' +
'### Response\n' +
'user: hello\n' +
'assistant:'
]
}

Node.js v18.4.0

(venv) C:\Users\Mike's PC\Documents\transfer_to_external_storage\gpt_llama_cpp\gpt-llama.cpp>

FINALLY, so apparently, this program does NOT like spaces in the path. I had to put this in a path without a single space. I know that this can happen at times with coding.

This is the final path:

C:\Users\Mike's\llama.cpp\models\vicuna_13b\ggml-model-q4_0.bin

so, you probably want to include in the README file, to not have ANY spaces in your path

Also, one last thing, I got a timeout error from the chat UI cmd, but the response still showed up on the gpt-llama.cpp cmd, so this issue is resolved, but it is weird that I got a timeout error from chat UI

EDIT: And now Chat UI is working fine.......I am confused, but happy confused.

Thank you for all of your help!

glad you figured it out! hopefully this thread will be useful for other folks if they run into similar issues – sorry i had to step away to make some dinner :/

regarding the spaces.. let me see if there's a way around it, there may be a bug perhaps

glad you figured it out! hopefully this thread will be useful for other folks if they run into similar issues – sorry i had to step away to make some dinner :/

regarding the spaces.. let me see if there's a way around it, there may be a bug perhaps

oh no, don't worry about it, we all gotta eat lol. As for the spaces, it's not the worst thing to deal with, but I am glad that this issue was resolved. Hopefully it is something that can be fixed easily, if not, no big deal!

I think I may know why this happened, maybe you have it in your code to always use the %username% of the host machine in the path variable. Hence why, no matter what previous path I chose, it always defaulted to my username.

hey @InfernalDread weird question but how did you get llama.cpp set up? do you have a main file? somebody else is trying to get it working and seem to be stuck on not having the main file

Oh, no problem. So, I went to the main llama.cpp repo and clicked on the releases button. Then, I downloaded the avx zip file that doesn't have a number attached to it (should be the first file of the assets). I unzipped the file and clicked into it and bam, all the .exe files that you need (including of course the "main.exe" file. Then I moved them to a separate folder named "llama.cpp" and created the "models" where I put my folder with the ".bin" file in it. If you need more clarification and or a visual representation, fill free to ask!

are you running on cmd, powershell or git bash or something else?

are you running on cmd, powershell or git bash or something else?

I am running fully on CMD using "venv" python virtual environment

got it! thank you :) also do you think if i made a discord channel it'd be helpful?

got it! thank you :) also do you think if i made a discord channel it'd be helpful?

I'd think so, this way more people can either contribute or ask questions without bloating the repo, but it's up to you of course! I'm just glad I can be of assistance!

Hello guys, how are you doing? Firstly, great job here keldenl! Your work is amazing! And thanks for all the help you gave him InfernalDread.

I'm stuck in the same error as you InfernalDread:

node:events:489
      throw er; // Unhandled 'error' event
      ^

Error: spawn C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:285:19)
    at onErrorNT (node:internal/child_process:483:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:291:12)
    at onErrorNT (node:internal/child_process:483:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main',
  path: 'C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main',
  spawnargs: [
    '-m',
    'C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/models/ggml-vicuna-13b-4bit-rev1.bin',
    '--temp',
    1,
    '--n_predict',
    1000,
    '--top_p',
    '0.1',
    '--top_k',
    '40',
    '-b',
    '512',
    '-c',
    '2048',
    '--repeat_penalty',
    '1.1764705882352942',
    '--reverse-prompt',
    'user:',
    '--reverse-prompt',
    '\nuser',
    '--reverse-prompt',
    'system:',
    '--reverse-prompt',
    '\nsystem',
    '--reverse-prompt',
    '##',
    '--reverse-prompt',
    '\n##',
    '--reverse-prompt',
    '###',
    '-i',
    '-p'

You have mentioned the spaces between the folders in the path, I've checked that already and it's everything ok, any toughts?

Hello guys, how are you doing? Firstly, great job here keldenl! Your work is amazing! And thanks for all the help you gave him InfernalDread.

I'm stuck in the same error as you InfernalDread:

node:events:489
      throw er; // Unhandled 'error' event
      ^

Error: spawn C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:285:19)
    at onErrorNT (node:internal/child_process:483:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:291:12)
    at onErrorNT (node:internal/child_process:483:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main',
  path: 'C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/main',
  spawnargs: [
    '-m',
    'C:/Users/Vitor/Desktop/Hax_Prog/llama.cpp/models/ggml-vicuna-13b-4bit-rev1.bin',
    '--temp',
    1,
    '--n_predict',
    1000,
    '--top_p',
    '0.1',
    '--top_k',
    '40',
    '-b',
    '512',
    '-c',
    '2048',
    '--repeat_penalty',
    '1.1764705882352942',
    '--reverse-prompt',
    'user:',
    '--reverse-prompt',
    '\nuser',
    '--reverse-prompt',
    'system:',
    '--reverse-prompt',
    '\nsystem',
    '--reverse-prompt',
    '##',
    '--reverse-prompt',
    '\n##',
    '--reverse-prompt',
    '###',
    '-i',
    '-p'

You have mentioned the spaces between the folders in the path, I've checked that already and it's everything ok, any toughts?

What is your node version? It needs to be 18+ to work. Do "node -v" and tell me what the output is.

@Vitorhsantos i think you're missing a / in your path, it should start with C:// right?

also @InfernalDread and @Vitorhsantos , i just made a discord channel that hopefully avoids a 50 reply long thread on GitHub (but hopefully we'll still have actual issues posted here too): https://discord.gg/aWHBQnJaFC come through!

@InfernalDread It says Node.js v20.0.0

@keldenl I've tried both options, same error, any other toughts?

@Vitorhsantos wanna join the discord we could try to figure it out there faster

@InfernalDread It says Node.js v20.0.0

@keldenl I've tried both options, same error, any other toughts?

You made sure to authenticate and use the same model structure and path as in the README?

Actually I got lost in that part, how can I authenticate properly? I would really appreciate it if you could explain it to me

Actually I got lost in that part, how can I authenticate properly? I would really appreciate it if you could explain it to me

Join the discord, it'll be easier for us to help you there!