Automated discovery is only as good as the benchmark that tells you a model works. BioMate evaluates discovery models with the controls that separate real, transferable learning from the artifacts a weak method can hide behind.

A benchmark framework to guide automated cyclic-peptide discovery — study overview
Study overview — a pre-registered benchmark with permutation, ablation and placebo controls separates transferable learning from test-set luck, and a calibrated gate holds false alarms to 3–8%.

Results in brief

3–8%
gate false-alarm rate, calibrated (vs 24.5–32.5% uncalibrated)
+0.075
held-out-source ranking gain — the honest transfer signal
pre-registered
margins, controls and verdicts fixed before the run

Why a good score is not enough

A model that ranks members of a familiar chemical series can look effective without learning anything that transfers to new chemistry or a new laboratory. Apparent gains can come from the initial predictor, from a nearby measured analogue, or from a validity check whose threshold is miscalibrated. A benchmark that doesn't control for these rewards the wrong thing.

A framework built around controls

BioMate's benchmark framework separates three questions with coordinated controls: does updating the model on new measurements actually improve selection; do learned rankings transfer across experimental sources; and does the benchmark's own validity gate detect shortcuts without falsely flagging sound data. Each is tested with frozen-model, one-shot, shuffled-feedback and planted-shortcut controls, with inference over both seeds and sources.

Calibration that holds its false-alarm rate

A validity gate is only useful if it rarely cries wolf. A null-calibrated gate holds its false-alarm rate where an uncalibrated one flags clean benchmarks — while still detecting planted shortcuts. Trust in a verdict comes from controls that are themselves measured, not from a single headline metric.

The point

BioMate treats benchmark interpretation as an empirical object of study. The result is evaluation you can act on: when BioMate reports that a model learned something, the controls are there to show it is real.

Read the full paper A benchmark framework to guide automated cyclic-peptide discovery

The complete study — methods, controls and every result — is archived open-access under CC-BY at Zenodo (doi:10.5281/zenodo.24112627).

Read the full paper on Zenodo →