Modern drug discovery and genomics run on multi-step computational pipelines — pick the right analysis, configure it, quality-control the data, model the drug, choose a safe trial dose. Teams increasingly want an AI agent to drive them. RealBio asks the blunt question: can an agent get the objectively-checkable answer right? 505 fixed public items across 5 real drug-development / genomics tasks, each with published ground truth, graded by one shared open scorer (score.py — deterministic, no AI judge). The finding: frontier LLMs know biology but fail at execution — worst on the clinical-pharmacology tasks (predicting drug exposure, choosing a trial dose) — while a purpose-built platform (BioMate) leads every task.
Data + scorer on GitHub → Dataset on HuggingFace See the leaderboard
Tasks are real bioinformatics automation — not multiple-choice trivia. Every number traces to committed score.py output on committed predictions; every leaderboard cell re-scores from the released files.
Taking a therapy from a molecule or a dataset toward the clinic is a computational relay. RealBio's five tasks are five real hand-offs in it — grouped into Orchestration (set the analysis up) and Execution (run the science, including the clinical-pharmacology computations). Quality control isn't a separate stage — it's embedded in every task.
| Band | Task | In plain English | Why an error is costly | Coverage |
|---|---|---|---|---|
| Orchestration | Pipeline routing | Pick the single correct analysis pipeline for a plain-English request (RNA-seq vs. variant-calling vs. single-cell …). | The wrong pipeline wastes days of compute and real sample money. | 200 items · 63 pipelines, all omics domains |
| Orchestration | Parameter pre-fill | Fill the pipeline's settings from the request — and only the ones actually specified, never invented. | Over-filling silently corrupts a run; both failures look like success. | 170 items · 11 categories, 143 parameters |
| Execution | QC gating | Decide PASS/FAIL on a real sequencing sample; a universal cutoff false-alarms on specialized assays (single-cell, small-RNA, cryo-EM). | Discarding good data — or keeping bad data — corrupts every downstream result. | 100 items · 99 assays · GEO/PRIDE/EMDB |
| Execution | Trial dose-finding (BOIN) | BOIN = Bayesian Optimal INterval, a Phase I clinical-trial design. From toxicity data, recommend the Maximum Tolerated Dose (MTD). | A first-in-human patient-safety call: too high harms patients, too low fails the drug. | 20 items · 20 trial scenarios (oncology, ADCs, pediatric) |
| Execution | Drug-exposure prediction (PBPK) | PBPK = Physiologically-Based PharmacoKinetics: predict blood concentration (Cmax, AUC, half-life) from a molecule's chemistry, dose and route. | Sets the first-in-human dose and underpins FDA submissions. | 15 items · 15 marketed drugs across classes |
Every abbreviation (BOIN, PBPK, MTD, Cmax, AUC, GEO/PRIDE/EMDB) is a standard drug-development or bioinformatics term. Items are original tasks authored from public sources — not drawn from any system's training set. The pattern: LLMs used directly do fine on knowledge-heavy orchestration but collapse on the execution tasks — PBPK simulation (mean 0.08 within-2-fold) and BOIN dose-finding (mean 0.24 exact-MTD) — exactly where a wrong answer is a safety error.
On the same fixed items and the same open scorer, BioMate is first on all five tasks — routing 0.965, parameter-F1 0.848, protocol-QC 0.925, PBPK 1.00, BOIN 0.80. The runner-up differs by task (Biomni ties routing; Claude Opus 5 is second on parameters; Gemini 3.1 Pro second on protocol), so the lead is broad, not a one-task artifact. The clearest pattern among the LLMs: used the way they are actually deployed, they fail execution tasks — PBPK simulation (mean 0.08 within-2-fold) and deterministic BOIN dose-finding (mean 0.24 exact-MTD) — while doing well on knowledge-driven extraction.
| System | Routing | Param (F1) | Protocol QC | PBPK (2×) | BOIN (MTD) |
|---|---|---|---|---|---|
| BioMate (product) | 0.965 | 0.848 | 0.925 | 1.00 | 0.80 |
| Claude Opus 5 | 0.615 | 0.757 | 0.330 | 0.133 | 0.444 |
| Gemini 3.1 Pro | 0.580 | 0.693 | 0.630 | 0.000 | 0.278 |
| GPT-5.6 | 0.530 | 0.671 | 0.460 | 0.200 | 0.500 |
| Kimi K3 | 0.450 | 0.628 | 0.360 | 0.000 | 0.167 |
| DeepSeek V4 | 0.415 | 0.621 | 0.420 | 0.067 | 0.167 |
| GLM-5.2 | 0.400 | 0.641 | 0.440 | 0.000 | 0.111 |
| Qwen3.8-Max | 0.345 | 0.692 | 0.460 | 0.000 | 0.111 |
| Biomni (Stanford A1) | 0.950 | 0.708 | 0.500 | 0.000 | 0.500 |
Every cell is a same-item, same-scorer result that re-scores from the committed prediction files under results/.
Latency below is the routing task (same 200 items) — the one metric measured identically for every system, so the only clean head-to-head. Cost is reported separately, because no single task logged tokens for all systems.
| System | Routing latency (median) | Engine |
|---|---|---|
| BioMate (product) | 2.3 s | Mixture of LLMs — primary Claude Sonnet 4.5; secondary Claude Haiku 4.5, Gemini 3.5 Flash, Gemini 3.1 Pro, GPT-5.6-luna |
| Claude Opus 5 | 2.7 s | Anthropic Claude Opus 5 |
| Gemini 3.1 Pro | 3.6 s | Google Gemini 3.1 Pro |
| GPT-5.6 | — (subset only) | OpenAI GPT-5.6 |
| GPT-5.6-luna | 1.6 s | OpenAI GPT-5.6-luna |
| Kimi K3 | 3.2 s | Moonshot Kimi K3 |
| DeepSeek V4 | 3.8 s | DeepSeek V4 |
| GLM-5.2 | 0.8 s | Z.ai GLM-5.2 |
| Qwen3.8-Max | 4.0 s | Alibaba Qwen3.8-Max |
| Biomni (A1) | 3.9 s | Agent scaffold — Claude Opus 5 (routing/param/protocol), Claude Sonnet 4.5 (PBPK/BOIN) |
Per-call cost — being revised. The token/cost analysis is temporarily withheld while we make it a clean, same-task comparison across all systems; it will be added back once corrected. Latency above is the interim same-task efficiency metric.
Every system was run on the same fixed items and scored by the same score.py. LLMs were run the way they are actually deployed (model + task); BioMate as the product; Biomni as its published agent scaffold.
| System | Organization | Link |
|---|---|---|
| BioMate (product) | BioMate AI | biomate.ai |
| Claude Opus 5 | Anthropic | anthropic.com/claude |
| Gemini 3.1 Pro | Google DeepMind | deepmind.google/models/gemini |
| GPT-5.6 · GPT-5.6-luna | OpenAI | openai.com |
| Kimi K3 | Moonshot AI | moonshot.ai |
| DeepSeek V4 | DeepSeek | deepseek.com |
| GLM-5.2 | Z.ai (Zhipu AI) | z.ai |
| Qwen3.8-Max | Alibaba Qwen | qwen.ai |
| Biomni (A1) | Stanford (Zou Lab) | biomni.stanford.edu · Science 2025 |
The items, ground truth, and scorer are public. Run your system on the fixed items, score with the shared score.py, and your result is directly comparable to the leaderboard.