Deep Learning Unplugged: SOTA with Just Convs and ReLU

Some of the papers I enjoy are the ones which lays out simple architectures achieving SOTA or close through theory insight. This paper uses no dropout, no batchnorm, no pooling. Just convs + ReLU + weight delay.

Identity Matters in Deep Learning: https://arxiv.org/abs/1611.04231

Discussion