Skip to main content

Learn Tools Games Exams Newsletter Community

Learn Tools Games Exams Newsletter Community Search

Phase 01: Math Foundations

AI From Scratch/Lesson 09/~60 minutes

Information Theory

Information theory measures surprise. Loss functions are built on it.

Learn

Back to phaseMath Foundations Next lessonDimensionality Reduction

Phase 01

Math Foundations

01Linear Algebra Intuition 02Vectors, Matrices & Operations 03Matrix Transformations 04Calculus for Machine Learning 05Chain Rule & Automatic Differentiation 06Probability and Distributions 07Bayes' Theorem 08Optimization 09Information Theory 10Dimensionality Reduction 11Singular Value Decomposition 12Tensor Operations 13Numerical Stability 14Norms and Distances 15Statistics for Machine Learning 16Sampling Methods 17Linear Systems 18Convex Optimization 19Complex Numbers for AI 20The Fourier Transform 21Graph Theory for Machine Learning 22Stochastic Processes

Loading lesson page...

Progress

0 / 22 phase lessons0%

Continue target

Phase 01

Math Foundations

01Linear Algebra Intuition 02Vectors, Matrices & Operations 03Matrix Transformations 04Calculus for Machine Learning 05Chain Rule & Automatic Differentiation 06Probability and Distributions 07Bayes' Theorem 08Optimization 09Information Theory 10Dimensionality Reduction 11Singular Value Decomposition 12Tensor Operations 13Numerical Stability 14Norms and Distances 15Statistics for Machine Learning 16Sampling Methods 17Linear Systems 18Convex Optimization 19Complex Numbers for AI 20The Fourier Transform 21Graph Theory for Machine Learning 22Stochastic Processes

On this page

Resources

Shannon 1948: A Mathematical Theory of CommunicationReading Visual Information Theory (Chris Olah)Reading PyTorch CrossEntropyLoss docsReading Information TheorySkill information_theory.pyCode Source lessonSource