{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tinyv-reducing-false-negatives-in","title":"TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning","arxiv_id":"2505.14625","date":"2025-05-20","proceeding":null,"authors":["Zhangchen Xu","Yuetai Li","Fengqing Jiang","Bhaskar Ramasubramanian","Luyao Niu","Bill Yuchen Lin","Radha Poovendran"],"abstract":"Reinforcement Learning (RL) has become a powerful tool for enhancing the reasoning abilities of large language models (LLMs) by optimizing their policies with reward signals. Yet, RL's success relies on the reliability of rewards, which are provided by verifiers. In this paper, we expose and analyze a widespread problem--false negatives--where verifiers wrongly reject correct model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals that over 38% of model-generated responses suffer from false negatives, where the verifier fails to recognize correct answers. We show, both empirically and theoretically, that these false negatives severely impair RL training by depriving the model of informative gradient signals and slowing convergence. To mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments existing rule-based methods, which dynamically identifies potential false negatives and recovers valid responses to produce more accurate reward estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts pass rates by up to 10% and accelerates convergence relative to the baseline. Our findings highlight the critical importance of addressing verifier false negatives and offer a practical approach to improve RL-based fine-tuning of LLMs. Our code is available at https://github.com/uw-nsl/TinyV.","url_abs":"https://arxiv.org/abs/2505.14625v1","url_pdf":"https://arxiv.org/pdf/2505.14625v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tinyv-reducing-false-negatives-in","repo_url":"https://github.com/uw-nsl/tinyv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.14625","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.14625"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uw-nsl/tinyv","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":18},"by_repo_kind":{"official":{"samples":19,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e88118a107b326d1","entry":"is_equiv","repo":"uw-nsl/tinyv","repo_kind":"official","path":"analysis_tool/verl_reward_score/math.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/analysis_tool/verl_reward_score/math.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e88118a107b326d1"}},{"code_sha256_prefix":"054091f2e78d9eec","entry":"compute_score","repo":"uw-nsl/tinyv","repo_kind":"official","path":"analysis_tool/verl_reward_score/math.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/analysis_tool/verl_reward_score/math.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"054091f2e78d9eec"}},{"code_sha256_prefix":"e3290e962238260a","entry":"compute_score","repo":"uw-nsl/tinyv","repo_kind":"official","path":"analysis_tool/verl_reward_score/math_verify.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/analysis_tool/verl_reward_score/math_verify.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3290e962238260a"}},{"code_sha256_prefix":"262d6673bdfffe7a","entry":"format_score","repo":"uw-nsl/tinyv","repo_kind":"official","path":"verl/verl/utils/reward_score/tinyv.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/verl/verl/utils/reward_score/tinyv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"262d6673bdfffe7a"}},{"code_sha256_prefix":"7450ca31ebcfaeab","entry":"get_tinyv_config","repo":"uw-nsl/tinyv","repo_kind":"official","path":"verl/verl/utils/reward_score/tinyv.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/verl/verl/utils/reward_score/tinyv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7450ca31ebcfaeab"}},{"code_sha256_prefix":"93948694e210f54c","entry":"handle_one","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/classification_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/classification_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93948694e210f54c"}},{"code_sha256_prefix":"b6ed7a24ff6124d2","entry":"handle_one","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6ed7a24ff6124d2"}},{"code_sha256_prefix":"b6bc0ef47c33095e","entry":"last_boxed_only_string","repo":"uw-nsl/tinyv","repo_kind":"official","path":"verl/verl/utils/reward_score/tinyv.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/verl/verl/utils/reward_score/tinyv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6bc0ef47c33095e"}},{"code_sha256_prefix":"e2f7c5004770cf05","entry":"load_dataset_from_file","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/utils.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2f7c5004770cf05"}},{"code_sha256_prefix":"e7ea11fb04de2a2f","entry":"load_jsonl_to_list","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/utils.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7ea11fb04de2a2f"}},{"code_sha256_prefix":"ec2e2e4d69080db8","entry":"make_api_request_with_retry","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/utils.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec2e2e4d69080db8"}},{"code_sha256_prefix":"6eabad50936c956f","entry":"process_engine_responses","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/classification_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/classification_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6eabad50936c956f"}},{"code_sha256_prefix":"a412a24eeee524e7","entry":"process_engine_responses","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a412a24eeee524e7"}},{"code_sha256_prefix":"d653b0c2e4d84a40","entry":"process_engine_responses","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_rollouts.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_rollouts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d653b0c2e4d84a40"}},{"code_sha256_prefix":"a66a8238b37b114f","entry":"process_engine_responses","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_vllm.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_vllm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a66a8238b37b114f"}},{"code_sha256_prefix":"53a8a2e9d14cedfd","entry":"remove_boxed","repo":"uw-nsl/tinyv","repo_kind":"official","path":"analysis_tool/verl_reward_score/math.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/analysis_tool/verl_reward_score/math.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53a8a2e9d14cedfd"}},{"code_sha256_prefix":"3518669a2b95257a","entry":"template_generator","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/classification_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/classification_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3518669a2b95257a"}},{"code_sha256_prefix":"4386904e9b413e5e","entry":"template_generator","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_grok.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_grok.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4386904e9b413e5e"}},{"code_sha256_prefix":"e283f10667270773","entry":"template_generator","repo":"uw-nsl/tinyv","repo_kind":"official","path":"tagging/unitag_vllm.py","file_url":"https://github.com/uw-nsl/tinyv/blob/HEAD/tagging/unitag_vllm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e283f10667270773"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}