pooyahrtn/RunpodOllama

(Error) Command not found: run_proxy

Closed this issue · 4 comments

ZoccoCss commented

I've set up the Runpod endpoint and it's working properly with runpod_serverless_llm.py. However when I run the start-proxy.sh I get a Command not found: run_proxy error.

In the shell script there's a line with poetry run run_proxy that calls a file that doesn't exist in the repository. I think you are referring to ollama_runpod_proxy.py but when I make the necessary changes to point to that file, I still get some errors (Now anOpenAIException).

Should I continue to consider ollama_runpod_proxy.py as the missing run_proxy file?

pooyahrtn commented

@ZoccoCss Thanks for sharing. Indeed ollama_runpod_proxy.py is the intended file.

ZoccoCss commented

Thank you @pooyahrtn for your quick response. I'm going to close this issue now.

I'm still getting an anOpenAIException error. Probably I have an issue with the config.yml. If I can't make it work I'll open a new issue.

pooyahrtn commented

@ZoccoCss I just fixed the script and added an example. Please let me know if it fixed the issue.

ZoccoCss commented

Yes, the issues where fixed. Thank you @pooyahrtn.

I would only suggest that you could add the following in the Readme to make it more clear.

  • In the config.yml file only add and pod id, leave the model and the api_key as they are.
  • set ´RUNPOD_API_TOKEN as an environmental variable.

Also, there is a minor typo in the example and python is complaining. I don't think it deserves to open a separate issue.

Cannot access member "message" for type "CompletionChoice"
  Member "message" is unknownPylance[reportGeneralTypeIssues](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportGeneralTypeIssues)
```
  • 👍1