Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
An article on Hacker News examines shortcomings in current software engineering benchmarks, focusing on the Senior SWE‑Bench suite. It argues that many evaluations rely on oversimplified calculations, likened to ‘napkin math,’ and fail to reflect real‑world performance. The author uses the winter‑tire analogy to illustrate how inadequate metrics can lead to unsafe or inefficient outcomes, urging more rigorous testing standards.