{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/foundational-large-language-models-for","title":"Foundational Large Language Models for Materials Research","arxiv_id":"2412.09560","date":"2024-12-12","proceeding":null,"authors":["Vaibhav Mishra","Somaditya Singh","Dhruv Ahlawat","Mohd Zaki","Vaibhav Bihani","Hargun Singh Grover","Biswajit Mishra","Santiago Miret","Mausam","N. M. Anoop Krishnan"],"abstract":"Materials discovery and development are critical for addressing global challenges. Yet, the exponential growth in materials science literature comprising vast amounts of textual data has created significant bottlenecks in knowledge extraction, synthesis, and scientific reasoning. Large Language Models (LLMs) offer unprecedented opportunities to accelerate materials research through automated analysis and prediction. Still, their effective deployment requires domain-specific adaptation for understanding and solving domain-relevant tasks. Here, we present LLaMat, a family of foundational models for materials science developed through continued pretraining of LLaMA models on an extensive corpus of materials literature and crystallographic data. Through systematic evaluation, we demonstrate that LLaMat excels in materials-specific NLP and structured information extraction while maintaining general linguistic capabilities. The specialized LLaMat-CIF variant demonstrates unprecedented capabilities in crystal structure generation, predicting stable crystals with high coverage across the periodic table. Intriguingly, despite LLaMA-3's superior performance in comparison to LLaMA-2, we observe that LLaMat-2 demonstrates unexpectedly enhanced domain-specific performance across diverse materials science tasks, including structured information extraction from text and tables, more particularly in crystal structure generation, a potential adaptation rigidity in overtrained LLMs. Altogether, the present work demonstrates the effectiveness of domain adaptation towards developing practically deployable LLM copilots for materials research. Beyond materials science, our findings reveal important considerations for domain adaptation of LLMs, such as model selection, training methodology, and domain-specific performance, which may influence the development of specialized scientific AI systems.","url_abs":"https://arxiv.org/abs/2412.09560v2","url_pdf":"https://arxiv.org/pdf/2412.09560v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"foundational-large-language-models-for","repo_url":"https://github.com/M3RG-IITD/llamat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.09560","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.09560"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/M3RG-IITD/llamat","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9026d9da5d085bd4","entry":"compute_exact","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"src/Kshot-val.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/src/Kshot-val.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9026d9da5d085bd4"}},{"code_sha256_prefix":"aa6bc32d9c17a2f1","entry":"get_tokens","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"src/Kshot-val.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/src/Kshot-val.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa6bc32d9c17a2f1"}},{"code_sha256_prefix":"6a96435eba311b08","entry":"normalize_answer","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"src/Kshot-val.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/src/Kshot-val.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96435eba311b08"}},{"code_sha256_prefix":"d47e68a6fa4c4bfe","entry":"beautify_leg","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"new_plot.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/new_plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d47e68a6fa4c4bfe"}},{"code_sha256_prefix":"61457250a2af1e2f","entry":"get_attention_mask_and_position_ids","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"Megatron-LLM/finetune.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/Megatron-LLM/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61457250a2af1e2f"}},{"code_sha256_prefix":"6b72184dd788b134","entry":"my_formatter","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"new_plot.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/new_plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b72184dd788b134"}},{"code_sha256_prefix":"50dcab670604971a","entry":"panel","repo":"M3RG-IITD/llamat","repo_kind":"official","path":"new_plot.py","file_url":"https://github.com/M3RG-IITD/llamat/blob/HEAD/new_plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50dcab670604971a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}