stanford-crfm/helm

HEIMHumanEvalScenario requires permissions to download data from codalab

slymane opened this issue · 1 comments

slymane commented

Hi folks, thanks for the work here! I'm trying to use the HEIMHumanEvalScenario class but run into an error after get_instances() is called. Specifically, https://worksheets.codalab.org/rest/bundles/0x502d646c366c4f1d8c4a2ccf163b958f/contents/blob/ returns ERROR 403: Forbidden when using wget to download data during the HEIMHumanEvalScenario.get_instances() call:

helm/src/helm/benchmark/scenarios/vision_language/heim_human_eval_scenario.py

Line 62 in 60a5865

def get_instances(self, output_path: str) -> List[Instance]:

Visiting the URL manually shows a forbidden error specifying that read permissions are needed. Is this intended behavior?

yifanmai commented

@teetone could you answer regarding HEIM?