{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/easy-to-hard-generalization-scalable","title":"Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision","arxiv_id":"2403.09472","date":"2024-03-14","proceeding":null,"authors":["Zhiqing Sun","Longhui Yu","Yikang Shen","Weiyang Liu","Yiming Yang","Sean Welleck","Chuang Gan"],"abstract":"Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision.","url_abs":"https://arxiv.org/abs/2403.09472v2","url_pdf":"https://arxiv.org/pdf/2403.09472v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"easy-to-hard-generalization-scalable","repo_url":"https://github.com/edward-sun/easy-to-hard","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"re-ranking","task_name":"Re-Ranking"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reranking","task_name":"Reranking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.09472","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.09472"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/edward-sun/easy-to-hard","reach":null}],"summary":{"ran_draft_wrong":4,"ran_honours":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2154e44a6b59e432","entry":"batch_select","repo":"edward-sun/easy-to-hard","repo_kind":"official","path":"models/reward_model.py","file_url":"https://github.com/edward-sun/easy-to-hard/blob/HEAD/models/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2154e44a6b59e432"}},{"code_sha256_prefix":"a6989412aca721c1","entry":"unpack_dict","repo":"edward-sun/easy-to-hard","repo_kind":"official","path":"models/reward_model.py","file_url":"https://github.com/edward-sun/easy-to-hard/blob/HEAD/models/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a6989412aca721c1"}},{"code_sha256_prefix":"36772c05893440aa","entry":"compute_pairwise_reward_modeling_loss","repo":"edward-sun/easy-to-hard","repo_kind":"official","path":"models/reward_model.py","file_url":"https://github.com/edward-sun/easy-to-hard/blob/HEAD/models/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"36772c05893440aa"}},{"code_sha256_prefix":"d4969006e20577f6","entry":"compute_reward_modeling_metrics","repo":"edward-sun/easy-to-hard","repo_kind":"official","path":"train_rl_dpo.py","file_url":"https://github.com/edward-sun/easy-to-hard/blob/HEAD/train_rl_dpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d4969006e20577f6"}},{"code_sha256_prefix":"63f447f7130d312a","entry":"model_forward","repo":"edward-sun/easy-to-hard","repo_kind":"official","path":"train_rl_dpo.py","file_url":"https://github.com/edward-sun/easy-to-hard/blob/HEAD/train_rl_dpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"63f447f7130d312a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}