Perimeter Institute for Theoretical Physics · Improving LLMs: The Bitter Lesson

1. 1. The primary driver of improvement in LLMs is scale, not clever algorithms.
2. 2. Rich Sutton coined the term 'bitter lesson' to describe how scaling outperforms human-designed cleverness.
3. 3. Human-crafted clever tricks become obsolete when the model is scaled up and learns them on its own.
4. 4. Each new generation of general-purpose model surpasses the specialized models of the previous generation.