{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unleashing-reasoning-capability-of-llms-via","title":"Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch","arxiv_id":"2410.18693","date":"2024-10-24","proceeding":null,"authors":["Yuyang Ding","Xinyu Shi","Xiaobo Liang","Juntao Li","Qiaoming Zhu","Min Zhang"],"abstract":"The availability of high-quality data is one of the most important factors in improving the reasoning capability of LLMs. Existing works have demonstrated the effectiveness of creating more instruction data from seed questions or knowledge bases. Recent research indicates that continually scaling up data synthesis from strong models (e.g., GPT-4) can further elicit reasoning performance. Though promising, the open-sourced community still lacks high-quality data at scale and scalable data synthesis methods with affordable costs. To address this, we introduce ScaleQuest, a scalable and novel data synthesis method that utilizes \"small-size\" (e.g., 7B) open-source models to generate questions from scratch without the need for seed data with complex augmentation constraints. With the efficient ScaleQuest, we automatically constructed a mathematical reasoning dataset consisting of 1 million problem-solution pairs, which are more effective than existing open-sourced datasets. It can universally increase the performance of mainstream open-source models (i.e., Mistral, Llama3, DeepSeekMath, and Qwen2-Math) by achieving 29.2% to 46.4% gains on MATH. Notably, simply fine-tuning the Qwen2-Math-7B-Base model with our dataset can even surpass Qwen2-Math-7B-Instruct, a strong and well-aligned model on closed-source data, and proprietary models such as GPT-4-Turbo and Claude-3.5 Sonnet.","url_abs":"https://arxiv.org/abs/2410.18693v1","url_pdf":"https://arxiv.org/pdf/2410.18693v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unleashing-reasoning-capability-of-llms-via","repo_url":"https://github.com/yyding1/scalequest","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.18693","atlas_url":"https://app.syntology.ai/?focus=2410.18693","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.18693"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yyding1/scalequest","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b44e74560a2f7e1","entry":"createDifficultyPrompt","repo":"yyding1/scalequest","repo_kind":"official","path":"src/train_question_generator/question_optim/prompts.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/train_question_generator/question_optim/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b44e74560a2f7e1"}},{"code_sha256_prefix":"e35afc303e481325","entry":"createSolvabilityPrompt","repo":"yyding1/scalequest","repo_kind":"official","path":"src/train_question_generator/question_optim/prompts.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/train_question_generator/question_optim/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e35afc303e481325"}},{"code_sha256_prefix":"b65ce61293fa5c19","entry":"filter_answer","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/question_filtering/language_solvability_check.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/question_filtering/language_solvability_check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b65ce61293fa5c19"}},{"code_sha256_prefix":"7d6fdd2ac2e21fdd","entry":"filter_query","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/question_filtering/language_solvability_check.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/question_filtering/language_solvability_check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d6fdd2ac2e21fdd"}},{"code_sha256_prefix":"686a5f1d92047af3","entry":"generate_score","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/question_filtering/generate_difficulty_score.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/question_filtering/generate_difficulty_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"686a5f1d92047af3"}},{"code_sha256_prefix":"10e724226473950f","entry":"generate_score_parallel","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/reward_filtering/rm_score.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/reward_filtering/rm_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10e724226473950f"}},{"code_sha256_prefix":"5086e0bcf250b990","entry":"process_data_on_device","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/question_filtering/generate_difficulty_score.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/question_filtering/generate_difficulty_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5086e0bcf250b990"}},{"code_sha256_prefix":"747dcdb948f1521d","entry":"process_data_on_device","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/reward_filtering/rm_score.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/reward_filtering/rm_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"747dcdb948f1521d"}},{"code_sha256_prefix":"fb330c46d149f244","entry":"process_wrapper","repo":"yyding1/scalequest","repo_kind":"official","path":"src/data_generation/question_filtering/generate_difficulty_score.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/data_generation/question_filtering/generate_difficulty_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb330c46d149f244"}},{"code_sha256_prefix":"c7214b71033ab87a","entry":"run_openai_inference","repo":"yyding1/scalequest","repo_kind":"official","path":"src/train_question_generator/question_optim/openai_gen.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/train_question_generator/question_optim/openai_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7214b71033ab87a"}},{"code_sha256_prefix":"d99b1fd214fb4ca1","entry":"run_openai_inference_async","repo":"yyding1/scalequest","repo_kind":"official","path":"src/train_question_generator/question_optim/openai_gen.py","file_url":"https://github.com/yyding1/scalequest/blob/HEAD/src/train_question_generator/question_optim/openai_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d99b1fd214fb4ca1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}