{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/molxpt-wrapping-molecules-with-text-for","title":"MolXPT: Wrapping Molecules with Text for Generative Pre-training","arxiv_id":"2305.10688","date":"2023-05-18","proceeding":null,"authors":["Zequn Liu","Wei zhang","Yingce Xia","Lijun Wu","Shufang Xie","Tao Qin","Ming Zhang","Tie-Yan Liu"],"abstract":"Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.","url_abs":"https://arxiv.org/abs/2305.10688v2","url_pdf":"https://arxiv.org/pdf/2305.10688v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"molxpt-wrapping-molecules-with-text-for","repo_url":"https://huggingface.co/zequnl/molxpt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"},{"task_slug":"molecule-captioning","task_name":"Molecule Captioning"},{"task_slug":"property-prediction","task_name":"Property Prediction"},{"task_slug":"text-based-de-novo-molecule-generation","task_name":"Text-based de novo Molecule Generation"},{"task_slug":"scientific-discovery","task_name":"scientific discovery"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/molecular-property-prediction-on-bace-1","task":"Molecular Property Prediction","dataset":"BACE","model":"MolXPT","rank_in_archive_order":1,"of":20,"metrics":{"ROC-AUC":"88.4"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-bbbp-1","task":"Molecular Property Prediction","dataset":"BBBP","model":"MolXPT","rank_in_archive_order":11,"of":29,"metrics":{"ROC-AUC":"80.5 ± 0.5"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-hiv-dataset","task":"Molecular Property Prediction","dataset":"HIV dataset","model":"MolXPT","rank_in_archive_order":6,"of":11,"metrics":{"AUC":"0.781"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-sider-1","task":"Molecular Property Prediction","dataset":"SIDER","model":"MolXPT","rank_in_archive_order":3,"of":19,"metrics":{"ROC-AUC":"71.7"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-tox21-1","task":"Molecular Property Prediction","dataset":"Tox21","model":"MolXPT","rank_in_archive_order":10,"of":20,"metrics":{"ROC-AUC":"77.1"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-clintox-1","task":"Molecular Property Prediction","dataset":"clintox","model":"MolXPT","rank_in_archive_order":3,"of":20,"metrics":{"ROC-AUC":"95.3±0.2"},"uses_additional_data":false},{"leaderboard":"/sota/molecule-captioning-on-chebi-20","task":"Molecule Captioning","dataset":"ChEBI-20","model":"MolXPT","rank_in_archive_order":16,"of":33,"metrics":{"BLEU-2":"59.4","BLEU-4":"50.5","METEOR":"62.6","ROUGE-1":"66","ROUGE-2":"51.1","ROUGE-L":"59.7","Text2Mol":"59.4"},"uses_additional_data":false},{"leaderboard":"/sota/text-based-de-novo-molecule-generation-on","task":"Text-based de novo Molecule Generation","dataset":"ChEBI-20","model":"MolXPT","rank_in_archive_order":20,"of":20,"metrics":{"Exact Match":"21.5","Frechet ChemNet Distance (FCD)":"0.45","MACCS FTS":"85.9","Morgan FTS":"66.7","Parameter Count":"350000000","RDK FTS":"75.7","Text2Mol":"57.8","Validity":"98.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.10688","atlas_url":"https://app.syntology.ai/?focus=2305.10688","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}