Model checks

Small, repeatable tests of language models on real scientific tasks, with dated results.

Benchmarks rarely look like the work scientists actually do. A model check is a short, fixed task taken from real research (a unit conversion with a trap, a derivation with a known sign error, a FITS header to interpret) run on several models, with the exact prompt, the expected answer and the date.

Re-running the same check on new models shows what has really improved.

First checks in preparation

The format and the first checks are being defined. Propose a check.