用于自我改进的 Harness 工程
表现最优的AI智能体在2小时预算下得分比人类高4倍；但人类在更长预算下收益更高，并在8小时和32小时设置下反超AI智能体。
原始摘录
Best AI agents scored 4× higher than humans at a 2-hour budget, but humans had better returns to longer budgets and exceeded agents at 8-hour and 32-hour settings.
上下文
RE-Bench：在真实ML研究-工程环境中，以前沿AI智能体为对象、以人类专家为基准开展评测。共设7个具挑战性、开放式的ML研究-工程环境。每个环境 =（评分函数，初始解，参考解）；每个环境均可在8块或更少H100 GPU上运行。示例任务包括：优化一个核函数、运行缩放律实验、修复嵌入向量、对GPT-2进行问答任务微调等。数据涵盖61位不同人类专家所完成的71次八小时尝试。人类专家在82%的八小时尝试中取得了非零得分；其中24%的尝试达到或超过了强参考解水平。
原始上下文
RE-Bench : evaluate frontier AI agents on realistic ML research-engineering envs against human experts. 7 challenging, open-ended ML research-engineering environments. Each environment = (scoring function, starting solution, reference solution); each can be run with 8 or fewer H100 GPUs. Examples: optimize a kernel, run a scaling-law experiment, fix an embedding, fine-tune GPT-2 for QA, etc. Includes data from 71 eight-hour attempts by 61 distinct human experts. Human experts achieved non-zero score in 82% of 8-hour attempts; 24% matched or exceeded strong reference solutions.