Splitting Exponents: The Magic of 3x10^-4 Weight Decay
·34 words·1 min
Why do we see a weight decay values like 3x10^-4? Why not 5 or 6 or some other digit? Because this is exponent bisection of 10^-3 and 10^-4, i.e., 10^-3.5 which is 3x10^-4.