Automated discovery is only as good as the benchmark that tells you a model works. BioMate evaluates discovery models with the controls that separate real, transferable learning from the artifacts a weak method can hide behind.
Results in brief
Why a good score is not enough
A model that ranks members of a familiar chemical series can look effective without learning anything that transfers to new chemistry or a new laboratory. Apparent gains can come from the initial predictor, from a nearby measured analogue, or from a validity check whose threshold is miscalibrated. A benchmark that doesn't control for these rewards the wrong thing.
A framework built around controls
BioMate's benchmark framework separates three questions with coordinated controls: does updating the model on new measurements actually improve selection; do learned rankings transfer across experimental sources; and does the benchmark's own validity gate detect shortcuts without falsely flagging sound data. Each is tested with frozen-model, one-shot, shuffled-feedback and planted-shortcut controls, with inference over both seeds and sources.
Calibration that holds its false-alarm rate
A validity gate is only useful if it rarely cries wolf. A null-calibrated gate holds its false-alarm rate where an uncalibrated one flags clean benchmarks — while still detecting planted shortcuts. Trust in a verdict comes from controls that are themselves measured, not from a single headline metric.
The point
BioMate treats benchmark interpretation as an empirical object of study. The result is evaluation you can act on: when BioMate reports that a model learned something, the controls are there to show it is real.
The complete study — methods, controls and every result — is archived open-access under CC-BY at Zenodo (doi:10.5281/zenodo.24112627).
Read the full paper on Zenodo →