Learning theory
Aperçu des semaines
-
This master class on learning theory covers the classical PAC framework for learning, stochastic gradient descent, tensor methods. We also touch upon topics from the recent literature on mean field methods for NN and the double descent phenomenon.
Teacher: Nicolas Macris: nicolas.macris@epfl.ch - with also some lectures by Rodrigo Veiga: rodrigo.veiga@epfl.ch
Teaching Assitant: Anand Jerry Georges - anand.george@epfl.ch
Courses: Mondays 8h15-10h in presence Room INM202; Exercises: Tuesdays 17h15-19h in presence Room INR219.
We will use this moodle page to distribute homeworks, solutions and also collect graded ones. As well as use the discussion and questions forum. Dont hesitate to actively use this forum.
Lectures are in presence. If you miss a lecture an old recorded version is accessible here https://mediaspace.epfl.ch/channel/CS-526+Learning+theory/29761 however the material, instructors and order of lectures might be slightly different this year
EXAM: its open book. You can bring your notes, printed material, the UML book. If you dont want to print you can upload your material on your laptop beforehand, and have wifi switched of (and we may check that you switch of the wifi during the exam).
Textbooks and notes:
- Understanding Machine Learning (UML) by Shalev-Shwartz and Ben David
- Bayesian Reasoning and Machine Learning by David Barber(Cambridge)
- Pattern recognition and Machine Learning by Christopher Bishop (Springer)
- Introduction to Tensor Decompositions and their Applications in Machine Learning (Ranbaser, Shchur, Gunneman)
- One lecture on two-layer neural networks
-
If you have a question or want to start a discussion on a topic, post here
- Understanding Machine Learning (UML) by Shalev-Shwartz and Ben David
-
PAC learning framework. Finite classes. Uniform convergence.
See chapters 3 and 4 in UML
Homework 1: exercises 1, 3, 7, 8 of Chapter 3.
-
No free lunch theorem.
See chapter 5 in UML
Homework 2: exercises 1 and 2 of chapter 4
-
Learning infinite classes I
Chapter 6 in UML
Graded hmw 3 due date Monday 18 March 23h59
-
Learning infinite classes II (VC dimension)
Chapter 6 continued
graded hmw 3 continued due date monday 18 March 23h59
-
Bias variance tradeoff and the double descent phenomenon
We will study the double descent of generalization error based on the paper "Two models of double descent for weak features" by Belkin, Hsu, Xu
-
Double descent phenomenon: continuation and derivation for weak features model
The derivations use the notion of Moore-Penrose inverse which is fully reviewed as a problem and solution in the two files attached below.
-
Easter week break
-
Gradient descent (convexity, Lipshitzness, Approach to optimal solution)
Stochastic gradient descent, application to learning
Chapter 14 in UML
Graded hmw 6 due date monday 21th April at 23h59
-
Mean field approach for two layer neural networks
based on the paper "One lecture on two layer neural networks" by A. Montanari
graded hmw 6 extended: due tuesday 23nd April at 23h59
-
I will finish discussing the main idea of the mean analysis of two layer neural networks.
graded hmw 6 extended deadline: due tuesday 23nd April at 23h59 -
Tensors 1. Motivations and examples, multi-dimensional arrays, tensor product, tensor rank.
Tensors 2. Tensor rank and decompositions, Jennrich's theorem
Graded hmw 7 due date 13 may 23h59 ---> Extended 15th may 23h59
-
Tensors 2bis. Tensor decomposizion and Jennrich's algorithm
Continuation of graded homework 7 --> due date extended 15th may 23h59
Extra hmw 8 is on Jenrich's theorem
-
Tensors 3. Alternating least square algorithm
Tensors 4. Multilinear rank Tucker higher order singular value decomposition
Graded hmw 7 has extended due date 15th may 23h59. We still answer questions and help this tuesady.
Hmw 8 above for those who finsished the no 7.
-
Monday 20 holiday
Tuesday 21: Homework 10 - this is the 4th graded homework due date 31 May
(Problem 3 of this hmw reviews the whitening process of a tensor which will be discussed in the last lecture. However part of this problem 3 you can already do - and leave the power method question for next time).
-
Tensors 5. Power method and Applications: Gaussian Mixture Models, Topic models of documents
Homework: continuation of graded hmw 10 (deadline 31 May) -
Here are old exams with solutions which will allow you to train yourself. Note that most some problems (but not all) are already included in this year's material. In exam 2019 ignore problems on graphical models that we do not treat this year.
-
ignore problems on Markov Random Fields that we did no treat this year (problem 3 and question 3 in MCQ)
-
Ignore question 5(a) on NTK not treated this year.
-
-