/Deep-Learning-Optimization-Algorithms

Visualization of various deep learning optimization algorithms using PyTorch automatic differentiation and optimizers.

Primary LanguageJupyter Notebook

Deep-Learning-Optimization-Algorithms

This repository contains the code for visualization of various deep learning optimization algorithms. PyTorch is used for automatic differentiation and implementing the various optimizers.

Stochastic Gradient Descent (SGD)

2D Convex Surface 2D Non-Convex Surface 3D Surface with Saddle Point

SGD with Momentum

2D Convex Surface 2D Non-Convex Surface 3D Surface with Saddle Point

Ada-delta

2D Convex Surface 2D Non-Convex Surface 3D Surface with Saddle Point

Adaptive Moment Estimation (ADAM)

2D Convex Surface 2D Non-Convex Surface 3D Surface with Saddle Point