{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uni-mol2-exploring-molecular-pretraining","title":"Uni-Mol2: Exploring Molecular Pretraining Model at Scale","arxiv_id":"2406.14969","date":"2024-06-21","proceeding":null,"authors":["Xiaohong Ji","Zhen Wang","Zhifeng Gao","Hang Zheng","Linfeng Zhang","Guolin Ke","Weinan E"],"abstract":"In recent years, pretraining models have made significant advancements in the fields of natural language processing (NLP), computer vision (CV), and life sciences. The significant advancements in NLP and CV are predominantly driven by the expansion of model parameters and data size, a phenomenon now recognized as the scaling laws. However, research exploring scaling law in molecular pretraining models remains unexplored. In this work, we present Uni-Mol2 , an innovative molecular pretraining model that leverages a two-track transformer to effectively integrate features at the atomic level, graph level, and geometry structure level. Along with this, we systematically investigate the scaling law within molecular pretraining models, characterizing the power-law correlations between validation loss and model size, dataset size, and computational resources. Consequently, we successfully scale Uni-Mol2 to 1.1 billion parameters through pretraining on 800 million conformations, making it the largest molecular pretraining model to date. Extensive experiments show consistent improvement in the downstream tasks as the model size grows. The Uni-Mol2 with 1.1B parameters also outperforms existing methods, achieving an average 27% improvement on the QM9 and 14% on COMPAS-1D dataset.","url_abs":"https://arxiv.org/abs/2406.14969v2","url_pdf":"https://arxiv.org/pdf/2406.14969v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uni-mol2-exploring-molecular-pretraining","repo_url":"https://github.com/deepmodeling/Uni-Mol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"uni-mol2-exploring-molecular-pretraining","repo_url":"https://github.com/dptech-corp/Uni-Mol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.14969","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.14969"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dptech-corp/Uni-Mol","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmodeling/Uni-Mol","reach":null}],"summary":{"ran":2,"ran_violates":1,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"32f631a2eb5edfee","entry":"calc_mask","repo":"dptech-corp/Uni-Mol","repo_kind":"listed","path":"unimol/unimol/losses/conf_gen.py","file_url":"https://github.com/dptech-corp/Uni-Mol/blob/HEAD/unimol/unimol/losses/conf_gen.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32f631a2eb5edfee"}},{"code_sha256_prefix":"294ba20b052bfbc7","entry":"gaussian","repo":"dptech-corp/Uni-Mol","repo_kind":"listed","path":"unimol/unimol/models/unimol.py","file_url":"https://github.com/dptech-corp/Uni-Mol/blob/HEAD/unimol/unimol/models/unimol.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"294ba20b052bfbc7"}},{"code_sha256_prefix":"698e12d1755a5461","entry":"masked_mean","repo":"dptech-corp/Uni-Mol","repo_kind":"listed","path":"unimol_plus/inference.py","file_url":"https://github.com/dptech-corp/Uni-Mol/blob/HEAD/unimol_plus/inference.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"698e12d1755a5461"}},{"code_sha256_prefix":"aa538ebbc52c74ed","entry":"realign_coord","repo":"dptech-corp/Uni-Mol","repo_kind":"listed","path":"unimol/unimol/losses/conf_gen.py","file_url":"https://github.com/dptech-corp/Uni-Mol/blob/HEAD/unimol/unimol/losses/conf_gen.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa538ebbc52c74ed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}