openai/evals

`Failed to open: ../registry/data/social_iqa/few_shot.jsonl` with custom registry

LoryPack opened this issue · 0 comments

Describe the bug

I have created a new eval within a custom registry and installed evals from pypi (using a recent checkout from github). Calling oaievals works fine if I do not use few_show samples, but it does not if you include few shot examples.

As can be seen in the error thread below, the issue is caused by calling self._prefix_registry_path(self.few_shot_jsonl) (see _prefix_registry_path) which transforms the path string into a Path object. Then, open_by_file_pattern (here) still treats it as a string, unaware of the fact that it is actually a Path

To Reproduce

  1. Create a new eval with few-shot samples.
  2. Call oaieval, for instance:
oaieval text-ada-001 social_iqa_few_shot --registry_path=../registry
  1. You'll get an error thread as follows:
[2023-11-03 14:40:52,931] [registry.py:254] Loading registry from /home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/registry/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from /home/lorenzo/.evals/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from ../registry/evals
[2023-11-03 14:40:55,007] [oaieval.py:189] �[1;35mRun started: 2311031440552KUYNQGH�[0m
[2023-11-03 14:40:55,010] [data.py:90] Fetching ../registry/data/social_iqa/few_shot.jsonl
Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 54, in open_by_file_pattern
    if filename.endswith(".gz"):
AttributeError: 'PosixPath' object has no attribute 'endswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lorenzo/venv/recog-LLM_capabilities/bin/oaieval", line 8, in <module>
    sys.exit(main())
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 274, in main
    run(args)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 215, in run
    eval: Eval = eval_class(
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/elsuite/basic/match.py", line 28, in __init__
    self.few_shot = evals.get_jsonl(self._prefix_registry_path(self.few_shot_jsonl))
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 129, in get_jsonl
    return _get_jsonl_file(path)
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 91, in _get_jsonl_file
    with open_by_file_pattern(path, mode="r") as f:
  File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 75, in open_by_file_pattern
    raise RuntimeError(f"Failed to open: {filename}") from e
RuntimeError: Failed to open: ../registry/data/social_iqa/few_shot.jsonl

Code snippets

No response

OS

Ubuntu 20.04

Python version

python 3.9

Library version

git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8