Add o1 support

Question

Add o1 support

Closed this issue 14 days ago · 8 comments

I believe it needs to be added to default_models.py and also add some documentation.

I just tried a local patch and the API reported that o1-preview does not exist or I don't have access to it, so not sure if API access has not been enabled yet or if our company doesn't have it yet.

Answer 1 · 2024-09-12T21:09:55.000Z

you can add locally with this config https://llm.datasette.io/en/stable/openai-models.html#adding-more-openai-models

Answer 2 · 2024-09-12T21:23:45.000Z

the docs there are not quite right... you need to specify both model_name and model_id, without model_name I get errors.

Answer 3 · 2024-09-12T21:24:37.000Z

OK, it's there for me now, however I'm getting this - "Error: Error code: 400 - {'error': {'message': "Unsupported value: 'stream' does not support true with this model. Only the default (false) value is supported.", 'type': 'invalid_request_error', 'param': 'stream', 'code': 'unsupported_value'}}"

So seems like you need to use --no-stream in order to get any output, wonder if there is a way to make that the default

Answer 4 · 2024-09-12T21:27:17.000Z

the docs there are not quite right... you need to specify both model_name and model_id, without model_name I get errors.

Good catch! I submitted a PR to fix the doc.

Answer 5 · 2024-09-12T22:40:50.000Z

So seems like you need to use --no-stream in order to get any output, wonder if there is a way to make that the default

I think we need a new model option for setting can_stream = False.

Answer 6 · 2024-09-12T22:52:26.000Z

I've published a branch with this implemented, but I can't test it myself yet as I don't have a tier 5 API key.

Could someone else test this for me? You can install the branch like this:

pip install https://github.com/simonw/llm/archive/refs/heads/openai-o1.zip
# Or maybe even this:
llm install https://github.com/simonw/llm/archive/refs/heads/openai-o1.zip

Then:

llm -m o1-preview 'hello to o1-preview'
llm -m o1-mini 'hello to o1-preview'

If someone reports those running without incident I'll ship a release with them.

Answer 7 · 2024-09-12T23:04:29.000Z

$ llm -m o1-mini 'hello to o1-preview'
Hello! It sounds like you're referring to "o1-preview." Could you provide a bit more context or let me know how I can assist you with it?

$ llm -m o1-preview 'hello to o1-preview'
Hello! How can I assist you today?

@simonw your branch works!

Answer 8 · 2024-09-12T23:08:18.000Z

Merged that, about to release it.