Small Data, Big Gains: Pre-training Revolution
·43 words·1 min
Surprising and important paper:
TLDR; All the gains we get by first pre-training on large dataset and then fine tuning on small dataset could be obtained by just small dataset but with pre-training objective!! Big hole in our understanding of SSL!