ImageNet Models Can't Even ImageNet

Do ImageNet Classifiers Generalize to ImageNet? Authors in this paper create new test set of 10,000 images using same methodology as original ImageNet and find that accuracy drops for all models by at least 10%.

Relative ranking of models still remains more or less same and newer models have surprisingly less gap than older. NasNet large version by Zoph etc al (2017) remains the state of the art at 72.2% top-1 accuracy. https://arxiv.org/abs/1902.10811

Discussion