We never Had Truly Understood the Bias-Variance Trade-off!!!
In this interview with Prof. Mikhail Belkin, we will discuss his amazing paper: “Reconciling modern machine learning practice and the bias-variance trade-off” which introduces the other half of the bias-variance trade-off that we have been missing all these years. Behold the Double-Descent behavior!
In this exciting interview with Prof. Mikhail Belkin, we go over one of his publications entitled: “Reconciling modern machine-learning practice and the classical bias–variance trade-off”.
00:51 Prof. Belkin introduces himself
01:27 The research interests of Prof. Belkin
02:35 What is the story behind the paper and how did they even think about challenging the traditional single-descent U-shape plot of bias-variance trade-off?
05:40 The double-descent behavior of the interpolation models.
06:56 What if over-parametrized models did not have any regularization mechanism? Would they still generalize well (according to the double-descent behavior found in this paper)?
10:34 What is this idea of Interpolation Threshold and can it be computed mathematically?
14:32 What is Inductive Bias and how can we control it? (related to the ‘smoothness’ which is chosen as the inductive bias in the paper)
17:31 How did you determine that ‘smoothness’ is the right inductive-bias for the problems you have worked on in the paper? What if it was not? Would we still see the double-descent behavior?
25:57 The bottom of the second descent (for an over-parametrized model) does NOT necessarily correspond to less Test Error (better generalizability), compared to the bottom of the traditional U-shape model (for an under-parametrized model).
29:60 If we were not able to create highly rich models, maybe we would not have never questioned the U-shape model. What other well-founded theoretical foundation is out there (similar to the old regime and the U-shape behavior) that you think are ONLY half the picture and need us to investigate them more?
36:46 A lot of biological models out there are over-parametrized!