`Failed to open: ../registry/data/social_iqa/few_shot.jsonl` with custom registry
LoryPack opened this issue · 0 comments
LoryPack commented
Describe the bug
I have created a new eval within a custom registry and installed evals from pypi (using a recent checkout from github). Calling oaievals
works fine if I do not use few_show samples, but it does not if you include few shot examples.
As can be seen in the error thread below, the issue is caused by calling self._prefix_registry_path(self.few_shot_jsonl)
(see _prefix_registry_path) which transforms the path string into a Path
object. Then, open_by_file_pattern
(here) still treats it as a string, unaware of the fact that it is actually a Path
To Reproduce
- Create a new eval with few-shot samples.
- Call
oaieval
, for instance:
oaieval text-ada-001 social_iqa_few_shot --registry_path=../registry
- You'll get an error thread as follows:
[2023-11-03 14:40:52,931] [registry.py:254] Loading registry from /home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/registry/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from /home/lorenzo/.evals/evals
[2023-11-03 14:40:54,382] [registry.py:254] Loading registry from ../registry/evals
[2023-11-03 14:40:55,007] [oaieval.py:189] �[1;35mRun started: 2311031440552KUYNQGH�[0m
[2023-11-03 14:40:55,010] [data.py:90] Fetching ../registry/data/social_iqa/few_shot.jsonl
Traceback (most recent call last):
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 54, in open_by_file_pattern
if filename.endswith(".gz"):
AttributeError: 'PosixPath' object has no attribute 'endswith'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/lorenzo/venv/recog-LLM_capabilities/bin/oaieval", line 8, in <module>
sys.exit(main())
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 274, in main
run(args)
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/cli/oaieval.py", line 215, in run
eval: Eval = eval_class(
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/elsuite/basic/match.py", line 28, in __init__
self.few_shot = evals.get_jsonl(self._prefix_registry_path(self.few_shot_jsonl))
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 129, in get_jsonl
return _get_jsonl_file(path)
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 91, in _get_jsonl_file
with open_by_file_pattern(path, mode="r") as f:
File "/home/lorenzo/venv/recog-LLM_capabilities/lib/python3.9/site-packages/evals/data.py", line 75, in open_by_file_pattern
raise RuntimeError(f"Failed to open: {filename}") from e
RuntimeError: Failed to open: ../registry/data/social_iqa/few_shot.jsonl
Code snippets
No response
OS
Ubuntu 20.04
Python version
python 3.9
Library version
git+https://github.com/openai/evals.git@dd96814dd96bd64f3098afca8dc873aa8d8ce4c8