{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-fine-tuning-when-scaling-test-time","title":"Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning","arxiv_id":"2502.07154","date":"2025-02-11","proceeding":null,"authors":["Feng Chen","Allan Raventos","Nan Cheng","Surya Ganguli","Shaul Druckmann"],"abstract":"Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a subsequent test-time compute strategy and budget? To explore this, we focus on pass@N, a simple test-time strategy that searches for a correct answer in $N$ independent samples. We show, surprisingly, that training with cross-entropy (CE) loss can be ${\\it misaligned}$ with pass@N in that pass@N accuracy ${\\it decreases}$ with longer training. We explain the origins of this misalignment in terms of model overconfidence induced by CE, and experimentally verify our prediction of overconfidence as an impediment to scaling test-time compute via pass@N. Furthermore we suggest a principled, modified training loss that is better aligned to pass@N by limiting model confidence and rescuing pass@N test performance. Our algorithm demonstrates improved mathematical reasoning on MATH and MiniF2F benchmarks under several scenarios: (1) providing answers to math questions; and (2) proving theorems by searching over proof trees of varying shapes. Overall our work underscores the importance of co-designing two traditionally separate phases of LLM development: training-time protocols and test-time search and reasoning strategies.","url_abs":"https://arxiv.org/abs/2502.07154v1","url_pdf":"https://arxiv.org/pdf/2502.07154v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-fine-tuning-when-scaling-test-time","repo_url":"https://github.com/allanraventos/refine","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.07154","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.07154"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allanraventos/refine","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54c2a2d5a5c4d295","entry":"compute_mean_loss_per_example","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/loss_functions.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/loss_functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54c2a2d5a5c4d295"}},{"code_sha256_prefix":"bece4e38f793fbc2","entry":"create_wandb_config","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/arg_parser.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/arg_parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bece4e38f793fbc2"}},{"code_sha256_prefix":"671c3b01c68f169e","entry":"custom_loss_function2","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/loss_functions.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/loss_functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"671c3b01c68f169e"}},{"code_sha256_prefix":"40a1820babef89cf","entry":"f","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/loss_functions.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/loss_functions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40a1820babef89cf"}},{"code_sha256_prefix":"f771bfcb5cd13900","entry":"f","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/NCO.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/NCO.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f771bfcb5cd13900"}},{"code_sha256_prefix":"83a493ca7eadac63","entry":"factor","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/NCO.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/NCO.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83a493ca7eadac63"}},{"code_sha256_prefix":"6a7299e5e2b3a131","entry":"init_process_group","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/ray_dist_utils.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/ray_dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a7299e5e2b3a131"}},{"code_sha256_prefix":"b6819f7fc99f31be","entry":"log_prob","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/NCO.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/NCO.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6819f7fc99f31be"}},{"code_sha256_prefix":"8464320e407bc6cd","entry":"normalize_final_answer","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/filter_samples.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/filter_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8464320e407bc6cd"}},{"code_sha256_prefix":"30b50c454a1ff5a6","entry":"process_results","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/filter_samples.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/filter_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30b50c454a1ff5a6"}},{"code_sha256_prefix":"e43277bef67f5b81","entry":"ray_noset_visible_devices","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/run_vllm.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/run_vllm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e43277bef67f5b81"}},{"code_sha256_prefix":"45f1d7f7f2d4a1e5","entry":"remove_boxed","repo":"allanraventos/refine","repo_kind":"official","path":"refine/training/filter_samples.py","file_url":"https://github.com/allanraventos/refine/blob/HEAD/refine/training/filter_samples.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45f1d7f7f2d4a1e5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}