/anthology-of-modern-ml

Collection of important articles to be treated as a textbook

Primary LanguageJupyter NotebookMIT LicenseMIT

Anthology of Modern Machine Learning

A curated collection of significant/impactful articles to be treated as a textbook, because sometimes it's just best to go straight to the source. My hope is to provide a reference for understanding important developments in the historical context that motivated them, e.g. the problems the authors were attempting to solve, what particular features of the discovery were considered especially novel or impressive when it was first published, what the competing theories or techniques at the time were, etc.

Someday this will be organized better.

"Classic" ML

Network Graphs / combinatorial optimization

Geometric Deep Learning and ML applications of group theory/representation theory

Misc optimization and numerical methods

Neural optimizers

Neural activations

Neural initializations

Neural layers

RL

Good list here: https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#citations-below

Hyperparameter tuning / Architecture Search

Implicit Representation

Specific architectures/achievements, and other misc milestones

Computer Vision / representation learning

NLP

Representation Learning

Misc

Learning theory / Deep learning theory / model compression / interpretability / Information Geometry

Surprisingly Relevant Group Theory

Information theory

Causal Modeling / experimentation

  • Double machine learning
  • Doubly robust inference
  • Pearl's do calculus and graphical modeling / structural equation modeling
  • Rubin's potential outcomes model
  • model identification
  • d-separation
  • propensity scoring/matching
  • item-response model and adaptive testing
  • bandit learning for on-line experimentation
  • belief propagation

Time series forecasting

Misc Generative Art milestones and techniques

Ethics in ML

  • Data Privacy
    • See Netflix Prize
  • Differential Privacy
  • k-anonymity
  • Dataset bias - gendered words, differential treatment of skin color, race and zipcode in legal applications
  • YOLO author's resignation (blog post + reddit thread)
  • CV techniques used to subjugate minorities in SE Asia and China
  • Ethical issues surrounding classification of behavioral health and interventions
  • Metadata deanonymization and leaks of US domestic data collection programs with corporate participation
  • "fairness" algorithms
  • gerrymandering and algorithmic redistricting
  • Facebook's influence on elections and live-testing to influence people's emotions and behaviors w/o consent

Analytic Process

Misc important papers for generative models/art, misc modern era

LLMs, in-context learning, prompt engineering

Development of attention mechanisms

largely via https://twitter.com/karpathy/status/1668302116576976906