BERT's Secret Sauce for Multiple Choice

How do you answer multi-choice question using BERT/GPT? This paper has cool trick: concatenate quation, answers with delimiters, convert hidden state for each answer to logits by linear xform. You get 56% accuracy on CommonsenseQA with this (humans 89%)! https://arxiv.org/abs/1811.00937

Discussion