{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2503-00096","title":"BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology","arxiv_id":"2503.00096","date":"2025-02-28","proceeding":null,"authors":["Ludovico Mitchener","Jon M Laurent","Benjamin Tenmann","Siddharth Narayanan","Geemi P Wellawatte","Andrew White","Lorenzo Sani","Samuel G Rodriques"],"abstract":"Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and rote knowledge tasks, towards more practical work such as literature review and experimental planning. Bioinformatics is a domain where fully autonomous AI-driven discovery may be near, but no extensive benchmarks for measuring progress have been introduced to date. We therefore present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis with nearly 300 associated open-answer questions designed to measure the ability of LLM-based agents to explore biological datasets, perform long, multi-step analytical trajectories, and interpret the nuanced results of those analyses. We evaluate the performance of two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework we open source. We find that even the latest frontier models only achieve 17% accuracy in the open-answer regime, and no better than random in a multiple-choice setting. By exposing the current limitations of frontier models, we hope BixBench can spur the development of agents capable of conducting rigorous bioinformatic analysis and accelerate scientific discovery.","url_abs":"https://arxiv.org/abs/2503.00096v2","url_pdf":"https://arxiv.org/pdf/2503.00096v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2503-00096","repo_url":"https://github.com/Future-House/BixBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"2503-00096","repo_url":"https://github.com/Future-House/data-analysis-crow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"scientific-discovery","task_name":"scientific discovery"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.00096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.00096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Future-House/BixBench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Future-House/data-analysis-crow","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6205d4627a1c839a","entry":"compute_metrics","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6205d4627a1c839a"}},{"code_sha256_prefix":"8a7353d569b0249e","entry":"flatten_list","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/postprocessing_utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/postprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a7353d569b0249e"}},{"code_sha256_prefix":"e2b63330a93e9f10","entry":"get_config_help_string","repo":"Future-House/data-analysis-crow","repo_kind":"official","path":"src/fhda/models.py","file_url":"https://github.com/Future-House/data-analysis-crow/blob/HEAD/src/fhda/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2b63330a93e9f10"}},{"code_sha256_prefix":"68f3bbb6fc1812dd","entry":"load_arg_dict","repo":"Future-House/data-analysis-crow","repo_kind":"official","path":"src/fhda/models.py","file_url":"https://github.com/Future-House/data-analysis-crow/blob/HEAD/src/fhda/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68f3bbb6fc1812dd"}},{"code_sha256_prefix":"e9b3fd78985fb2aa","entry":"load_config","repo":"Future-House/data-analysis-crow","repo_kind":"official","path":"src/fhda/models.py","file_url":"https://github.com/Future-House/data-analysis-crow/blob/HEAD/src/fhda/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9b3fd78985fb2aa"}},{"code_sha256_prefix":"07f6784a25c3b03b","entry":"load_dataframe_from_json_directory","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/postprocessing_utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/postprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07f6784a25c3b03b"}},{"code_sha256_prefix":"a6c564d5b5c99437","entry":"parse_response","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6c564d5b5c99437"}},{"code_sha256_prefix":"aa2c6e922c733527","entry":"randomize_choices","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa2c6e922c733527"}},{"code_sha256_prefix":"2167072e3d4ff590","entry":"send_message_to_llm","repo":"Future-House/BixBench","repo_kind":"official","path":"bixbench/postprocessing_utils.py","file_url":"https://github.com/Future-House/BixBench/blob/HEAD/bixbench/postprocessing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2167072e3d4ff590"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}