{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sciglm-training-scientific-language-models","title":"SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Models","arxiv_id":"2401.07950","date":"2024-01-15","proceeding":null,"authors":["Dan Zhang","Ziniu Hu","Sining Zhoubian","Zhengxiao Du","Kaiyu Yang","Zihan Wang","Yisong Yue","Yuxiao Dong","Jie Tang"],"abstract":"Large Language Models (LLMs) have shown promise in assisting scientific discovery. However, such applications are currently limited by LLMs' deficiencies in understanding intricate scientific concepts, deriving symbolic equations, and solving advanced numerical calculations. To bridge these gaps, we introduce SciInstruct, a suite of scientific instructions for training scientific language models capable of college-level scientific reasoning. Central to our approach is a novel self-reflective instruction annotation framework to address the data scarcity challenge in the science domain. This framework leverages existing LLMs to generate step-by-step reasoning for unlabelled scientific questions, followed by a process of self-reflective critic-and-revise. Applying this framework, we curated a diverse and high-quality dataset encompassing physics, chemistry, math, and formal proofs. We analyze the curated SciInstruct from multiple interesting perspectives (e.g., domain, scale, source, question type, answer length, etc.). To verify the effectiveness of SciInstruct, we fine-tuned different language models with SciInstruct, i.e., ChatGLM3 (6B and 32B), Llama3-8B-Instruct, and Mistral-7B: MetaMath, enhancing their scientific and mathematical reasoning capabilities, without sacrificing the language understanding capabilities of the base model. We release all codes and SciInstruct at https://github.com/THUDM/SciGLM.","url_abs":"https://arxiv.org/abs/2401.07950v3","url_pdf":"https://arxiv.org/pdf/2401.07950v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sciglm-training-scientific-language-models","repo_url":"https://github.com/thudm/sciglm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"scientific-discovery","task_name":"scientific discovery"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.07950","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.07950"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/THUDM/SciGLM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/sciglm","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"938a3cf8db709378","entry":"build_prompt","repo":"THUDM/SciGLM","repo_kind":"official","path":"inference/cli_demo.py","file_url":"https://github.com/THUDM/SciGLM/blob/HEAD/inference/cli_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"938a3cf8db709378"}},{"code_sha256_prefix":"b50f9d8043b40d41","entry":"eval","repo":"thudm/sciglm","repo_kind":"official","path":"evaluation/SciEval/evaluate_SciEval.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/evaluation/SciEval/evaluate_SciEval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b50f9d8043b40d41"}},{"code_sha256_prefix":"6c4c3157688289fd","entry":"get_label","repo":"thudm/sciglm","repo_kind":"official","path":"self_reflective_annotation/gpt_labeling/gpt_judge.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/self_reflective_annotation/gpt_labeling/gpt_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c4c3157688289fd"}},{"code_sha256_prefix":"8ccd0ed4e4557aa8","entry":"gpt_usage","repo":"thudm/sciglm","repo_kind":"official","path":"evaluation/SciEval/models.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/evaluation/SciEval/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ccd0ed4e4557aa8"}},{"code_sha256_prefix":"eb4df61bb116b17c","entry":"is_choice","repo":"thudm/sciglm","repo_kind":"official","path":"self_reflective_annotation/reflective_generation/api_gen_gpt_revise.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/self_reflective_annotation/reflective_generation/api_gen_gpt_revise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb4df61bb116b17c"}},{"code_sha256_prefix":"dae309af08c4bf0a","entry":"read_json","repo":"THUDM/SciGLM","repo_kind":"official","path":"self_reflective_annotation/reflective_generation/api_gen_gpt.py","file_url":"https://github.com/THUDM/SciGLM/blob/HEAD/self_reflective_annotation/reflective_generation/api_gen_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dae309af08c4bf0a"}},{"code_sha256_prefix":"fb4b23e8ddc8bc80","entry":"read_json","repo":"thudm/sciglm","repo_kind":"official","path":"self_reflective_annotation/reflective_generation/api_gen_gpt_revise.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/self_reflective_annotation/reflective_generation/api_gen_gpt_revise.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb4b23e8ddc8bc80"}},{"code_sha256_prefix":"8b94a0e33c0124ec","entry":"read_json","repo":"thudm/sciglm","repo_kind":"official","path":"self_reflective_annotation/data_classifier/rm_eval.py","file_url":"https://github.com/thudm/sciglm/blob/HEAD/self_reflective_annotation/data_classifier/rm_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b94a0e33c0124ec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}