As models improve, the data that helps them improve changes. Large volumes of generic examples give way to smaller sets of difficult, carefully reviewed material written by people who know the domain.
Difficulty that matches the frontier
Useful examples sit just beyond what the model can already do. Experts are better at finding that edge because they know where real problems get hard.
Consistency through calibration
Two experts can disagree on style while agreeing on substance. Calibration rounds, clear guidelines and agreement tracking keep data consistent enough to learn from.
Verifiability
Where possible, tasks should have answers that can be checked automatically, such as tests for code or numerical answers for maths. Verifiable data makes training signals more reliable and evaluation more honest.
- Prefer fewer, harder examples over many easy ones
- Invest in reviewer calibration before scaling
- Design tasks with verifiable outcomes wherever possible
- Hold out evaluation sets that the data team never sees
