Sora-mmh/Optimization-for-ML

implement from scratch and explain the fundamentals in optimization for ML through experimentations

Optimization-for-ML

This project is about different optimization algorithms for machine learning in different settings (constrained, unconstrained, automatic differentiation...).

A quick overview which is more detailed in this experimentations and observations notebook :

• Part 1: Exploratory Data Analysis about Diabetes dataset
  • 1.1 Import Dataset
  • 1.2 Exploration
• Part 2: Gradient Descent
  • 2.1 Logistic Regression GD for classification
  • 2.2 Step Size
  • 2.3 Convergence Rate
  • 2.4 Performance Test Set (change the RIDGE penalty)
• Part 3: Automatic Differentiation
• Part 4: Stochtastic Gradient Descent
  • 4.1 SGD and Comparison with GD
  • 4.2 Optimal Batch Size
  • 4.3 SVRG
• Part 5: Convexity and Constrained Optimization
  • 5.1 Conditional Gradient Algorithm
  • 5.2 Projected Gradient Optimization
• Part 6: Regularization
  • 6.1 L-1 Regularization : LASSO
  • 6.2 L-2 Regularization
• Part 7: Large-Scale and Distributed Optimization
  • 7.1 Randomized Block Coordinate Descent
  • 7.2 Randomized Block Coordinate Descent + SGD
• Part 8: Advanced Topics On Gradient Descent
  • 8.1 Heavy Ball
  • 8.2 Add Non-Convex Penalty to the loss (L-0.5 penalty) + GD