multi-needle-eval-pizza-3 dataset not found
gkamradt opened this issue · 1 comments
Hey @rlancemartin , I'm running the command on the readme right now
needlehaystack.run_test --evaluator langsmith --context_lengths_num_intervals 3 --document_depth_percent_intervals 3 --provider openai --model_name "gpt-4-0125-preview" --multi_needle True --eval_set multi-needle-eval-pizza --needles '["Figs are one of the three most delicious pizza toppings.", "Prosciutto is one of the three most delicious pizza toppings.", "Goat cheese is one of the three most delicious pizza toppings."]'
The eval kicks off find but then errors out on the first tests with
langsmith.utils.LangSmithNotFoundError: Dataset multi-needle-eval-pizza-3 not found
I switched --eval_set multi-needle-eval-pizza to --eval_set multi-needle-eval-pizza-3 (per the blog post), but that didn't fix the issue.
Have an idea of what's going on?
Fixed this issue after debugging offline with Lance.
The setup instructions on the readme - I just needed to follow them ;)