{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-molecular-multimodal-foundation-model","title":"A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language","arxiv_id":"2209.05481","date":"2022-09-12","proceeding":null,"authors":["Bing Su","Dazhao Du","Zhao Yang","Yujie Zhou","Jiangmeng Li","Anyi Rao","Hao Sun","Zhiwu Lu","Ji-Rong Wen"],"abstract":"Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality. Since the hierarchy of molecular knowledge is profound, even humans learn from different modalities including both intuitive diagrams and professional texts to assist their understanding. Inspired by this, we propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data (crawled from published Scientific Citation Index papers) via contrastive learning. This AI model represents a critical attempt that directly bridges molecular graphs and natural language. Importantly, through capturing the specific and complementary information of the two modalities, our proposed model can better grasp molecular expertise. Experimental results show that our model not only exhibits promising performance in cross-modal tasks such as cross-modal retrieval and molecule caption, but also enhances molecular property prediction and possesses capability to generate meaningful molecular graphs from natural language descriptions. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine, among others.","url_abs":"https://arxiv.org/abs/2209.05481v1","url_pdf":"https://arxiv.org/pdf/2209.05481v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-molecular-multimodal-foundation-model","repo_url":"https://github.com/bingsu12/momu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-molecular-multimodal-foundation-model","repo_url":"https://github.com/ai-hpc-research-team/git-mol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-molecular-multimodal-foundation-model","repo_url":"https://github.com/ai-hpc-research-team/slm4mol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-molecular-multimodal-foundation-model","repo_url":"https://github.com/yangzhao1230/graphtextretrieval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"},{"task_slug":"molecule-captioning","task_name":"Molecule Captioning"},{"task_slug":"property-prediction","task_name":"Property Prediction"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/molecule-captioning-on-chebi-20","task":"Molecule Captioning","dataset":"ChEBI-20","model":"MoMu+MolT5-Large","rank_in_archive_order":13,"of":33,"metrics":{"BLEU-2":"59.9","BLEU-4":"51.5","METEOR":"59.7","Text2Mol":"58.2"},"uses_additional_data":false},{"leaderboard":"/sota/molecule-captioning-on-chebi-20","task":"Molecule Captioning","dataset":"ChEBI-20","model":"MoMu+MolT5-Base","rank_in_archive_order":25,"of":33,"metrics":{"BLEU-2":"54.9","BLEU-4":"46.2","METEOR":"57.6","Text2Mol":"55.8"},"uses_additional_data":false},{"leaderboard":"/sota/molecule-captioning-on-chebi-20","task":"Molecule Captioning","dataset":"ChEBI-20","model":"MoMu+MolT5-Small","rank_in_archive_order":30,"of":33,"metrics":{"BLEU-2":"53.2","BLEU-4":"44.5","METEOR":"55.7","Text2Mol":"55.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.05481","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.05481"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai-hpc-research-team/git-mol","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bingsu12/momu","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangzhao1230/graphtextretrieval","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai-hpc-research-team/slm4mol","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":6},"by_repo_kind":{"listed":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"138b87fa967c256f","entry":"get_latest_checkpoint","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"138b87fa967c256f"}},{"code_sha256_prefix":"556ccfd77bd3c202","entry":"safe_index","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/feature/graph_featurizer.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/feature/graph_featurizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"556ccfd77bd3c202"}},{"code_sha256_prefix":"7091cac2247e9821","entry":"ToDevice","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/utils/xutils.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/utils/xutils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7091cac2247e9821"}},{"code_sha256_prefix":"b55b14ce1d8befcc","entry":"custom_collate_fn","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/utils/xutils.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/utils/xutils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b55b14ce1d8befcc"}},{"code_sha256_prefix":"54e7398c07a02f5c","entry":"generate_mols","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"54e7398c07a02f5c"}},{"code_sha256_prefix":"0365b968d90ed84e","entry":"generate_scaffold","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/evaluations/mol_translation_metrics.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/evaluations/mol_translation_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0365b968d90ed84e"}},{"code_sha256_prefix":"96dcc6542582fbce","entry":"pad_tensors_to_max_length","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/models/model_manager.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/models/model_manager.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96dcc6542582fbce"}},{"code_sha256_prefix":"8ca70db968a79f18","entry":"text_evaluate","repo":"ai-hpc-research-team/slm4mol","repo_kind":"listed","path":"src/evaluations/text_translation_metrics.py","file_url":"https://github.com/ai-hpc-research-team/slm4mol/blob/HEAD/src/evaluations/text_translation_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ca70db968a79f18"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}