This repository contains implementations of various numerical optimization algorithms commonly used in machine learning and optimization problems.
-
Batch Gradient Descent: A classic optimization algorithm that computes the gradient of the loss function w.r.t. all training examples to update model parameters.
-
Stochastic Gradient Descent (SGD): An optimization algorithm that computes the gradient of the loss function w.r.t. a single training example at each iteration to update model parameters.
-
Mini-Batch Gradient Descent: A variation of SGD that computes the gradient using a small subset (mini-batch) of training examples to update model parameters.
-
Momentum-based Gradient Descent: An optimization algorithm that adds momentum to the gradient update to accelerate convergence and escape local minima.
-
Nesterov Accelerated Gradient (NAG): A modification of momentum-based gradient descent that reduces the oscillations and overshooting by updating the parameters with a look-ahead gradient.
-
Adagrad: An adaptive learning rate optimization algorithm that adapts the learning rate for each parameter based on the historical gradients.
-
RMSProp: Another adaptive learning rate optimization algorithm that uses a moving average of squared gradients to adjust the learning rate.
-
ADAM (Adaptive Moment Estimation): A popular optimization algorithm that combines the benefits of both momentum-based methods and adaptive learning rate methods.
-
Second Order Optimization: Optimization algorithms that utilize second-order information such as Hessian matrix for more accurate updates. Examples include Newton's Method and Quasi-Newton methods like BFGS and L-BFGS.
Each optimization algorithm is implemented as a separate module. You can find the implementation and usage instructions in the respective directories.
To use any of the optimization algorithms, simply import the corresponding module and call the optimization function with appropriate parameters.