"The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable" Replication
Minimal replication of the core result of Millidge & Black's "The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable". This replicates OV circuit interpretability via SVD for GPT-2-medium. See the linked paper for context and the full results.