/AIMS

AIMS South Africa Short Course on Optimization

Primary LanguageHTMLMIT LicenseMIT

AIMS

AIMS South Africa Short Course on Optimization

1 week short-course taught for AIMS South Africa, September 2019

Taught by Stephen Becker (University of Colorado, Applied Mathematics)

Details at the AIMS Spring School on Mathematics of Data Science website

See the Lipschitz/strong-convexity inequalities cheat-sheet for the course

Convex Optimization References

There are many great books to suggest. The ones below are all excellent, and these are the books I used most for preparing these notes.

  • Convex Optimization by S. Boyd and L. Vandenberghe, (Cambridge U. Press, 2004; available online at Boyd's website)
    • A popular (and rightly so) introduction to convex optimization, with many examples. This is a great source to get ideas on how to prove that your function is convex (follow the examples and exercises)
  • Numerical Optimization by J. Nocedal and S. Wright (Springer, 2005; available online via SpringerLink)
    • This is a comprehensive review of numerical methods; very practical, and most useful when you are implementing a method
  • First-Order Methods in Optimization by Amir Beck, (SIAM, 2017; available online via SIAM eBooks)
    • A recent book that is algorithmic in focus, but includes many results that are not in standard algorithmic books (e.g., the results were previously only in Bauschke and Combettes)
    • Do not confuse this with Amir Beck's other recent book, Intro to Nonlinear Optimization (SIAM, 2014)
  • Convex Analysis and Monotone Operator Theory in Hilbert Spaces by H. Bauschke and P. Combettes (Springer, 2nd ed 2017; available online via SpringerLink)
    • A nearly comprehensive set of results on convex analysis, superceding Rockafellar's 1970 Convex Analysis, and overlapping some with Rockafellar and Wets' 1997/2009 Variational Analysis

For analysis

We covered gradient descent and variants (sub-gradient, stochastic gradient, accelerated gradient, proximal gradient). If you're interested in the analysis, here are some references:

For a history of stochastic approximation, see

For more on degeneracy, focused on SDP, see