Code First, Ask Questions Later: The Understudied Secret Behind GPT-3.5's Success

Code-first training is absolutely one of the the most understudied phenomenon in current LLM training landscape. Many recent models have been trained without any regards to this phenomenon and they consistently under perform compared to GPT 3.5. https://x.com/Mascobot/status/1651022921056555008

Discussion