Skip to main content

Learn Skills Tools Games Exams Newsletter Community

Learn Skills Tools Games Exams Newsletter Community Search

AI From Scratch/Phase 01/Lesson 08/~75 minutes

Optimization

BuildNo prerequisites

Training a neural network is nothing more than finding the bottom of a valley.

Back to phaseMath Foundations Next lessonInformation Theory

Phase 01

Math Foundations

01Linear Algebra Intuition 02Vectors, Matrices & Operations 03Matrix Transformations 04Calculus for Machine Learning 05Chain Rule & Automatic Differentiation 06Probability and Distributions 07Bayes' Theorem 08Optimization 09Information Theory 10Dimensionality Reduction 11Singular Value Decomposition 12Tensor Operations 13Numerical Stability 14Norms and Distances 15Statistics for Machine Learning 16Sampling Methods 17Linear Systems 18Convex Optimization 19Complex Numbers for AI 20The Fourier Transform 21Graph Theory for Machine Learning 22Stochastic Processes

Loading lesson page...

Progress

0 / 22 phase lessons0%

Continue target

Phase 01

Math Foundations

01Linear Algebra Intuition 02Vectors, Matrices & Operations 03Matrix Transformations 04Calculus for Machine Learning 05Chain Rule & Automatic Differentiation 06Probability and Distributions 07Bayes' Theorem 08Optimization 09Information Theory 10Dimensionality Reduction 11Singular Value Decomposition 12Tensor Operations 13Numerical Stability 14Norms and Distances 15Statistics for Machine Learning 16Sampling Methods 17Linear Systems 18Convex Optimization 19Complex Numbers for AI 20The Fourier Transform 21Graph Theory for Machine Learning 22Stochastic Processes

On this page

Resources

Sebastian Ruder: An overview of gradient descent optimization algorithmsReading Why Momentum Really Works (Distill)Reading Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014)Reading Visualizing the Loss Landscape of Neural Nets (Li et al., 2018)Reading Optimizer GuidePrompt main.jlCode