{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cognitive-behaviors-that-enable-self","title":"Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs","arxiv_id":"2503.01307","date":"2025-03-03","proceeding":null,"authors":["Kanishk Gandhi","Ayush Chakravarthy","Anikait Singh","Nathan Lile","Noah D. Goodman"],"abstract":"Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors -- verification, backtracking, subgoal setting, and backward chaining -- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor -- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.","url_abs":"https://arxiv.org/abs/2503.01307v1","url_pdf":"https://arxiv.org/pdf/2503.01307v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cognitive-behaviors-that-enable-self","repo_url":"https://github.com/kanishkg/cognitive-behaviors","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.01307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.01307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kanishkg/cognitive-behaviors","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"71d5fbb13555643e","entry":"compute_completions_metrics","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"behavioral_evals/process_data.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/behavioral_evals/process_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71d5fbb13555643e"}},{"code_sha256_prefix":"236383c3f64bc48e","entry":"get_completion_files","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"behavioral_evals/api_eval.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/behavioral_evals/api_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"236383c3f64bc48e"}},{"code_sha256_prefix":"9c83e3e6e06264ea","entry":"load_all_results","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"behavioral_evals/process_data.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/behavioral_evals/process_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c83e3e6e06264ea"}},{"code_sha256_prefix":"8c5d18a9e7a574a1","entry":"read_completions","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"behavioral_evals/process_data.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/behavioral_evals/process_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5d18a9e7a574a1"}},{"code_sha256_prefix":"b72bb46f62bd47cf","entry":"remove_think_content","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"generate_cot_datasets/generate_empty_cot.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/generate_cot_datasets/generate_empty_cot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b72bb46f62bd47cf"}},{"code_sha256_prefix":"ea1fe462b951a92d","entry":"replace_think_tags","repo":"kanishkg/cognitive-behaviors","repo_kind":"official","path":"generate_cot_datasets/generate_dots_cot.py","file_url":"https://github.com/kanishkg/cognitive-behaviors/blob/HEAD/generate_cot_datasets/generate_dots_cot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea1fe462b951a92d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}