{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/molcap-arena-a-comprehensive-captioning","title":"MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction","arxiv_id":"2411.00737","date":"2024-11-01","proceeding":null,"authors":["Carl Edwards","Ziqing Lu","Ehsan Hajiramezanali","Tommaso Biancalani","Heng Ji","Gabriele Scalia"],"abstract":"Bridging biomolecular modeling with natural language information, particularly through large language models (LLMs), has recently emerged as a promising interdisciplinary research area. LLMs, having been trained on large corpora of scientific documents, demonstrate significant potential in understanding and reasoning about biomolecules by providing enriched contextual and domain knowledge. However, the extent to which LLM-driven insights can improve performance on complex predictive tasks (e.g., toxicity) remains unclear. Further, the extent to which relevant knowledge can be extracted from LLMs also remains unknown. In this study, we present Molecule Caption Arena: the first comprehensive benchmark of LLM-augmented molecular property prediction. We evaluate over twenty LLMs, including both general-purpose and domain-specific molecule captioners, across diverse prediction tasks. To this goal, we introduce a novel, battle-based rating system. Our findings confirm the ability of LLM-extracted knowledge to enhance state-of-the-art molecular representations, with notable model-, prompt-, and dataset-specific variations. Code, resources, and data are available at github.com/Genentech/molcap-arena.","url_abs":"https://arxiv.org/abs/2411.00737v1","url_pdf":"https://arxiv.org/pdf/2411.00737v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"molcap-arena-a-comprehensive-captioning","repo_url":"https://github.com/genentech/molcap-arena","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"},{"task_slug":"property-prediction","task_name":"Property Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.00737","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.00737"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/genentech/molcap-arena","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"dd19815ada629f82","entry":"create_captions","repo":"genentech/molcap-arena","repo_kind":"official","path":"captioners/BioT5_captioner.py","file_url":"https://github.com/genentech/molcap-arena/blob/HEAD/captioners/BioT5_captioner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dd19815ada629f82"}},{"code_sha256_prefix":"44139e924e222def","entry":"create_captions","repo":"genentech/molcap-arena","repo_kind":"official","path":"captioners/BioT5_plus_captioner.py","file_url":"https://github.com/genentech/molcap-arena/blob/HEAD/captioners/BioT5_plus_captioner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"44139e924e222def"}},{"code_sha256_prefix":"5bdad06d966e5cb1","entry":"create_captions","repo":"genentech/molcap-arena","repo_kind":"official","path":"captioners/Blank_captioner.py","file_url":"https://github.com/genentech/molcap-arena/blob/HEAD/captioners/Blank_captioner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5bdad06d966e5cb1"}},{"code_sha256_prefix":"7c696edce66383b1","entry":"create_captions","repo":"genentech/molcap-arena","repo_kind":"official","path":"captioners/ChemDFM_captioner.py","file_url":"https://github.com/genentech/molcap-arena/blob/HEAD/captioners/ChemDFM_captioner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7c696edce66383b1"}},{"code_sha256_prefix":"0d020a16cec6bb3c","entry":"predict_ensemble","repo":"genentech/molcap-arena","repo_kind":"official","path":"MolCapArena/model/components.py","file_url":"https://github.com/genentech/molcap-arena/blob/HEAD/MolCapArena/model/components.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d020a16cec6bb3c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}