Weight Decay Gains Weight in Machine Learning
Weight decay used to be mostly ignored parameter that you leave it to default. Now it is becoming most significant parameter. The age of diet machine learning is upon us. https://x.com/ykilcher/status/1446241396072099849