openai/evals

Getting started example doesn't work - oieval attempts to update a None type object

jswang opened this issue · 1 comments

Describe the bug

When running the example provided in https://github.com/openai/evals/blob/main/docs/run-evals.md: oaieval gpt-3.5-turbo test-match, the following error is encountered out of the box:

 File "/Users/juliewang/Documents/evals/evals/cli/oaieval.py", line 167, in run
    eval_spec.args.update(extra_eval_params)
    ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'update'

This is due to this line: https://github.com/openai/evals/blob/main/evals/cli/oaieval.py#L158. The args field of EvalSpec is optional:

args: Optional[Dict[str, Any]] = None
, and in the test-match case, eval_spec.args is None.

To Reproduce

  1. Run oaieval gpt-3.5-turbo test-match

Code snippets

No response

OS

macOS

Python version

Python 3.12.2

Library version

openai-evals commit b5853eb

Thx for raising this! I've opened a PR that should address it.