Journal Club – 101 Topics in ML, Control & Optimization

Introduction

Optimization has been playing a critical role in modern machine learning, leading to its significantly successful applications in many different fields. Researchers and practitioners typically spend less time investigating optimizers when they do the model training, mostly following some empirical rules or recommendations from the community. In this event, we will go relatively deeper in the optimizer behind, particularly the popular first-order stochastic optimizers (e.g., SGD, Adam, etc.). 

A big picture of what and how optimization for machine learning has been done will first be given, even including typical venues and well-known domain experts. Some fundamentals and basic knowledge regarding the objective loss, accuracy metrics (asymptotic and non-asymptotic) and learning rate will be covered, as well as the critical inequalities and basic proof techniques. We will also show how to reach some simple convergence rates from scratch. We are not going very deep in terms of math, while most of contents is on top of calculus and linear algebra.