stanford-crfm/helm

run_specs returns null when using expand

xvanQ opened this issue · 4 comments

Why does the RunExpander class in run_dexpander. py not have a specific implementation of expand, and returns empty when running “run_specs = [child_run_spec for parent_run_spec in run_specs for child_run_spec in expander.expand(parent_run_spec)] ”

when debugging i found:
class RunExpander(ABC):
"""
A RunExpander takes a RunSpec and returns a list of RunSpecs.
For example, it might fill out the model field with a variety of different models.
"""

name: str

@abstractmethod
def expand(self, run_spec: RunSpec) -> List[RunSpec]:
    pass

Hi @xvanQ, RunExpander is an Abstract Base Class, which means it an abstract interface that is intended to be implemented by subclasses and not intended to be instantiated directly. For more information, refer to the Python documentation on Abstract Base Classes and PEP 3119.

Hi @xvanQ, RunExpander is an Abstract Base Class, which means it an abstract interface that is intended to be implemented by subclasses and not intended to be instantiated directly. For more information, refer to the Python documentation on Abstract Base Classes and PEP 3119.

Thank you for your answer. I found that the problem is that if I use model='text ', the value in the corresponding values_dict is empty, which causes run_specs to be empty after expand, resulting in an index error when calling run_specs [0]

My command line parameters are as follows. Could you please take a look and see if there are any issues
--description wikifact:model=text,k=5,subject=plaintiff --model facebook/opt-iml-30b

#2142 (comment)
I think I have identified the problem, as not registering resulted in ALL_MODEL_METADATA being empty.
add:
register_builtin_configs_from_helm_package()
register_configs_from_directory(args.local_path)