Proposal: User-provided custom adapters

[yifanmai]

Background

There are a number of scenarios that require user-provided custom adapters. Currently, HELM only supports built-in adapters.

Example use cases:

• MedAlign in #3038 needs to truncate the EHR context in a special way before it is interpolated into the prompt.
• MTBench, which is not yet implemented, needs to generate multi-turn requests in which the model's previous response is incorporated into the message history.
• Minimal-pairs scenarios like BLIMP and EWOK may want to send two requests per instance and take the instance with lower logprobs.
• Model-as-examiner scenarios like AutoBencher may want to query a model to generate instances dynamically, rather than using predefined instances from a static dataset.
• Stand-alone projects (like HEIM and VHELM) and external users of HELM may want to store their project-specific adapters in a separate repository outside of HELM, especially if the adapters contain private or proprietary information.

The custom adapters subclass might also require additional arguments to be passed in, besides existing arguments in AdapterSpec.

Proposal

Currently, adapters are instantiated inside AdapterFactory.get_adapter() (code) which expects adapter_spec.method to be one of several canned enum values. It instantiates an adapter from one of several built-in classes based on this enum value.

We could allow users to set method to either an enum value or a class name. If method is an enum, we revert to the default behavior, otherwise, we instantiate an instance of the provided class and pass in adapter_spec and tokenizer_service.

We could also add an args: Dict[str, Any] field to AdapterSpec which can be used to passing additional arguments to AdapterSpec.

Alternatives Considered

We could make class_name: str a separate field from method. A valid AdapterSpec will have exactly one of method and class_name set. The semantics are otherwise the same as the main proposal.

We could make AdapterSpec an ObjectSpec. This would mirror ScenarioSpec and MetricSpec, which are also subclasses ObjectSpec. This would also allow users to pass arbitrary extra parameters to the Adapter. However, this doesn't work because there are a massive number of dependencies on specific fields in AdapterSpec - various RunExpanders modify specific fields in ObjectSpec, and Runner directly accesses several fields including max_eval_instances and eval_splits.

We could put in an custom_adapter_spec: Object field inside AdapterSpec. If set, we instantiate the adapter using the usual ObjectSpec semantics, additionally passing in adapter_spec and tokenizer_service as additional parameters.