{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-21327","title":"Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning","arxiv_id":"2604.21327","date":"2026-04-23","proceeding":null,"authors":["Yongcan Yu","Lingxiao He","Jian Liang","Kuangpu Guo","Meng Wang","Qianlong Xie","Xingxing Wang","Ran He"],"abstract":"Test-time reinforcement learning (TTRL) always adapts models at inference time via pseudo-labeling, leaving it vulnerable to spurious optimization signals from label noise. Through an empirical study, we observe that responses with medium consistency form an ambiguity region and constitute the primary source of reward noise. Crucially, we find that such spurious signals can be even amplified through group-relative advantage estimation. Motivated by these findings, we propose a unified framework, Debiased and Denoised test-time Reinforcement Learning (DDRL), to mitigate spurious signals. Concretely, DDRL first applies a frequency-based sampling strategy to exclude ambiguous samples while maintaining a balanced set of positive and negative examples. It then adopts a debiased advantage estimation with fixed advantages, removing the bias introduced by group-relative policy optimization. Finally, DDRL incorporates a consensus-based off-policy refinement stage, which leverages the rejection-sampled dataset to enable efficient and stable model updates. Experiments on three large language models across multiple mathematical reasoning benchmarks demonstrate that DDRL consistently outperforms existing TTRL baselines. The code will soon be released at https://github.com/yuyongcan/DDRL.","url_abs":"https://arxiv.org/abs/2604.21327","url_pdf":"https://arxiv.org/pdf/2604.21327","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2604.21327","atlas_url":"https://app.syntology.ai/?focus=2604.21327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.21327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/yuyongcan/DDRL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9c496dae615ba2a8","entry":"generate_config_from_args","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/model_merger/base_model_merger.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/model_merger/base_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c496dae615ba2a8"}},{"code_sha256_prefix":"735e93af30b15fa9","entry":"is_ipv4","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/utils/net_utils.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735e93af30b15fa9"}},{"code_sha256_prefix":"21331a58f93375e4","entry":"union_tensor_dict","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/protocol.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21331a58f93375e4"}},{"code_sha256_prefix":"83b1686d7ac03c21","entry":"unpad_dataproto","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/protocol.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83b1686d7ac03c21"}},{"code_sha256_prefix":"c9d58036da0286f2","entry":"get_generation_config","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/utils/model.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9d58036da0286f2"}},{"code_sha256_prefix":"59904a8955b34252","entry":"get_huggingface_actor_config","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/utils/model.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59904a8955b34252"}},{"code_sha256_prefix":"acf240079d434746","entry":"get_weight_loader","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/models/weight_loader_registry.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acf240079d434746"}},{"code_sha256_prefix":"cd6b1e0d6afb2762","entry":"get_weight_saver","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/models/weight_loader_registry.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd6b1e0d6afb2762"}},{"code_sha256_prefix":"705afe175ad8e404","entry":"is_ipv6","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/utils/net_utils.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/utils/net_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"705afe175ad8e404"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"yuyongcan/DDRL","repo_kind":"found_in_text","path":"verl/verl/utils/model.py","file_url":"https://github.com/yuyongcan/DDRL/blob/HEAD/verl/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}