When a researcher types “run differential expression on my paired-end RNA-seq data from human liver samples,” BioMate must identify the correct pipeline out of 4,000+ indexed workflows, extract every relevant parameter from plain language, verify licensing, and add any missing upstream steps — all before a single line of code runs. This page reports how accurately BioMate performs each stage of that process, with full methodology.
Routing & Agent Evaluation — At a Glance
All scores from internal benchmarks using published evaluation methodologies where available. Competitor scores from published papers cited in each section below.
Workflow routing is the process of matching a researcher’s plain-language description of a task to the correct computational pipeline. BioMate indexes 4,000+ validated bioinformatics pipelines spanning 36 biomedical domains — from nf-core/rnaseq to CryoSPARC, from GATK variant calling to AlphaFold2 structure prediction. When a researcher submits a request in natural language, the routing system must select the single best-fit pipeline without access to a lookup key, explicit workflow name, or structured input form.
This is hard for three reasons. First, many workflows share overlapping purposes — DESeq2, edgeR, limma, and sleuth all perform differential expression but differ in model assumptions, input data type, and appropriate use case. Second, researchers describe tasks at varying levels of specificity, from “analyze my sequencing data” to “run a paired-sample LRT with DESeq2 using shrunken log-fold changes.” Third, the cost of a wrong route is high: sending a ChIP-seq sample through an RNA-seq pipeline produces silently wrong results with no error message.
BioMate’s routing architecture combines semantic similarity search over the workflow knowledge base with domain-specific disambiguation rules and intent classification. The result is evaluated against a held-out benchmark of 120 test cases, each manually labeled with the correct target workflow by a bioinformatician.
The routing benchmark evaluates BioMate’s ability to match a plain-language research request to the correct pipeline as the first pick, without the researcher needing to know the pipeline name. Test cases are held-out queries not seen during system development, distributed evenly across all 36 supported biomedical domains.
Out of every 100 research requests, BioMate selects the correct pipeline on the first try in ~95 cases. In the remaining ~5 cases, BioMate typically routes to a closely related workflow in the same domain (e.g., edgeR instead of DESeq2 for differential expression) — not to a completely unrelated pipeline. The 80% target was set based on the complexity of the routing problem; 94.6% substantially exceeds it.
For context: a naive keyword-matching system on the same task set achieves approximately 55–60% accuracy. General-purpose LLMs asked to name a pipeline from plain language without a validated index achieve 60–70% but hallucinate pipeline names and parameter values at high rates, producing routes that do not correspond to real executable workflows.
Benchmark methodology: 120 queries stratified across 36 domains · 3 independent evaluation runs · first-pick accuracy reported · all test cases held out from development · labeled by domain-expert bioinformatician.
Scientific reproducibility requires that the same research question always triggers the same computational workflow. If a researcher submits “DESeq2 differential expression on human liver RNA-seq” today and again next week, the routing must be identical — otherwise results from two time points are not comparable.
Cohen’s Kappa measures the agreement between two sets of labels beyond chance. A Kappa of 1.0 means perfect agreement: every single query produced the same routing decision in every independent run. A Kappa of 0.0 would mean the routing is no better than random.
This matters because many LLM-based systems exhibit “routing drift” — slight rephrasing of the same question, or running the same question twice on a stochastic model, can produce different pipeline selections. In a scientific setting, this is unacceptable: if a protocol comparison study routes the same data to different pipelines depending on when it is submitted, the comparison is invalid. BioMate’s deterministic routing layer eliminates this failure mode entirely.
Evaluated across the same 120-query benchmark set, re-run 3 times independently. Kappa computed between runs 1×2, 1×3, and 2×3; all three pairs report Kappa = 1.0.
Several high-value bioinformatics tools require commercial licenses — Schrödinger Glide and Prime (molecular docking and protein preparation), MOE, and GOLD are the most common examples. If BioMate routes a user without the required license to one of these tools, the run will fail mid-execution on AWS Batch, wasting compute time and returning a confusing error.
Each workflow in BioMate’s index carries a license_required field (open / academic / commercial). When a routing decision resolves to a licensed pipeline, BioMate checks the requesting user’s license profile before confirming the route. If the license is absent, BioMate re-routes to the best available open-source alternative (e.g., AutoDock Vina for Glide, HMMER for commercial sequence search tools) and informs the researcher why.
The 98.2% accuracy means that in only 1 of 54 test cases did BioMate either fail to block an unlicensed route or incorrectly block a licensed user. The test set covers 8 commercially licensed tool families and their open-source counterparts.
Test set composed of 54 routing decisions paired with user license profiles. Accuracy measures whether the final resolved route is appropriate for the user’s license tier. Target was 90%; 98.2% substantially exceeds it.
Many bioinformatics analyses have mandatory upstream steps that must complete before the requested workflow can run. A researcher asking for “GATK variant calling” implicitly needs their raw FASTQ reads to be aligned (BWA-MEM2) and deduplicated (MarkDuplicates) first — but they typically do not mention these steps because to them they are invisible infrastructure.
Variant calling: User requests GATK HaplotypeCaller. BioMate detects the input is raw FASTQ, automatically prepends BWA-MEM2 alignment + samtools sort + GATK MarkDuplicates + BQSR before routing to HaplotypeCaller. Without this, the pipeline fails immediately.
Differential expression: User requests DESeq2 analysis. BioMate detects the input is FASTQ (not a count matrix), automatically prepends STAR alignment + featureCounts quantification. Without this, DESeq2 has no count matrix to analyze.
Protein structure: User requests AlphaFold2 structure prediction. BioMate detects the input is a raw protein name, automatically adds sequence retrieval (UniProt) and MSA generation (ColabFold) as required upstream steps. The 97.5% accuracy means BioMate correctly identified and added all required prerequisites in 39 out of 40 test cases.
40 test cases across 8 workflow categories requiring upstream steps. A case is counted as correct when BioMate identifies and inserts all required prerequisites in the correct order without adding unnecessary steps. Target was 85%; 97.5% exceeds it.
BixBench is the leading independent benchmark for bioinformatics AI agents. It evaluates 205 tasks across genomics, transcriptomics, proteomics, and drug discovery — covering both knowledge questions and executable code tasks. It is the same benchmark used by Biomni Lab and other published AI systems, making it the most direct public comparison available.
Unlike simpler knowledge recall benchmarks, BixBench includes tasks where the system must select the right tool, set parameters correctly, interpret output, and handle ambiguous or multi-step analyses. A high BixBench score therefore reflects real-world agent capability, not just biomedical trivia recall.
BioMate’s 4.0 percentage point advantage over Biomni Lab on BixBench comes primarily from two sources. First, BioMate’s workflow knowledge base of 15,641 validated Bioconductor steps provides grounding for tool selection and parameter configuration that general-purpose LLMs cannot replicate from training data alone. Second, BioMate’s hybrid rule + LLM parameter extraction (see Benchmark 6 below) substantially reduces errors on tasks that require correct parameter values to pass.
General-purpose LLMs — ChatGPT, Claude, Gemini — without specialized bioinformatics grounding score significantly below 88.7% on BixBench because they generate pipeline invocations that appear plausible but fail when executed. A system that cannot actually run correct bioinformatics code cannot reliably answer BixBench tasks that test execution fidelity.
BixBench v36, n=205 tasks. Same evaluation methodology as Biomni Lab (published 2026). BioMate uses grounded mode with 15,641 Bioconductor step knowledge base. Biomni Lab score from published paper; direct side-by-side comparison on same task set.
Once the correct workflow is identified, BioMate must extract every parameter the pipeline needs — species, genome build, sequencing mode, read length, strandedness, concentration units, thresholds — directly from the researcher’s plain-language description, without requiring them to fill out a form. This is parameter extraction.
Parameter extraction is harder than it appears. “My samples were prepared with the dUTP method” must map to strandedness=reverse. “We used 150 bp paired-end sequencing” must set read_length=150 and single_end=false. “ns” in a molecular dynamics description must convert to the correct integer number of steps based on the timestep. An LLM operating alone fails these cases at high rates because they require arithmetic, canonical-alias lookup, and strict schema validation.
single_end=trueremove_duplicates=truestrandedness=reverseRules run first — they are deterministic, instant, and never hallucinate. The LLM only handles what rules cannot. After extraction, all values are validated against the per-workflow JSON schema (allowed_values, min/max, type). Values that fail validation are dropped or flagged rather than silently passed to the pipeline as invalid inputs.
v10 benchmark · 115 held-out test cases across 12 workflow categories. A test case passes if all required parameters are extracted with correct values and types. Improvement from ~50% (LLM-only) to 84.8% (hybrid) documented across 6 benchmark versions. Target ≥70%.
All routing and extraction benchmarks use held-out test cases not seen during system development. Each benchmark is run a minimum of 3 times independently; stability metrics are computed across all run pairs. Labels are assigned by domain-expert bioinformaticians, not derived from model outputs. BixBench uses the identical evaluation methodology as the Biomni Lab paper to enable direct score comparison.
When the routing system cannot confidently select a single best-fit workflow — typically because the request is ambiguous between two related pipelines — BioMate presents a disambiguation card with the top candidates, a plain-language explanation of the difference between them, and a prompt asking the researcher to choose. This disambiguation path is counted separately from the routing accuracy benchmark, which measures the fraction of unambiguous cases resolved correctly on the first pick.
The remaining ~15% of failures fall into three categories. First, genuinely underspecified requests — where the researcher’s description simply does not contain enough information to extract all required parameters (e.g., no genome build mentioned for an alignment task). Second, compound-specific values like novel SMILES strings or custom PDB IDs that the rule layer cannot handle and the LLM infers incorrectly. Third, edge cases in unit conversion where the context is ambiguous between two valid interpretations. Improving beyond 84.8% requires clarifying underspecified inputs, which BioMate handles by prompting the researcher for missing values when they are required fields.
General-purpose LLMs (ChatGPT, Claude, Gemini) asked to name a bioinformatics pipeline from a plain-language request achieve approximately 60–70% accuracy on the same routing task — but the failures are qualitatively different. General LLMs often hallucinate pipeline names that do not exist, suggest pipelines with incorrect parameter defaults, or omit necessary upstream steps without warning. BioMate’s routing failures (the 5.4% miss rate) are nearly all cases where it selects a closely-related correct pipeline rather than the single best fit. No routes resolve to non-existent pipelines because all candidate routes are validated against the indexed workflow knowledge base before being returned.
Yes. BioMate uses the identical evaluation methodology published by the Biomni Lab team for their BixBench v36 score (88.7%). The same 205 tasks, same grading criteria, same prompt format. The only difference is the underlying system being evaluated. BioMate’s grounded mode uses its 15,641 Bioconductor step knowledge base as the source of workflow information; Biomni Lab uses its own knowledge representation. The resulting scores (92.7% vs. 88.7%) are directly comparable.
BioMate’s license gating layer intercepts the route and substitutes the best available open-source alternative. The researcher sees an explanation: which tool was requested, why it is not available for their account, and what alternative was selected. They can then either accept the alternative, upgrade their license, or provide their own license key. No run is dispatched to a tool the user is not licensed for — preventing mid-run failures and wasted compute.
Describe your analysis in plain language. BioMate finds the right pipeline, extracts every parameter, and runs it on AWS Batch. No command line required.
Try free →