In this work we show that the inherent noise of SGD is sufficient for escaping from saddles points in polynomial time. This results builds upon the empirical observation that noise from subsampling finite sum objectives is highly anisotropic and somewhat aligned with the leftmost eigenvectors.
