EleutherAI/delphi

Unified autoencoder loader

Closed this issue · 1 comments

Have a unified way to load autoencoders into the model for caching. Should allow for some flexibility for people to add new autoencoders. Completed when:

  • Function returns a hooked model for some choice of autoencoder
  • Support for SAE lens
  • Tests that verify that an autoencoder is correctly "hooked"

Closing because the LatentCache now takes a hookpoint_to_sparse_encode dictionary that can be built for various autoencoders and other models as follows:

hookpoint_to_sparse_encode = {
   hookpoint: autoencoders[hookpoint].forward
   for hookpoint in hookpoints
}