{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-rating-based-reinforcement-learning","title":"Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models","arxiv_id":"2506.12822","date":"2025-06-15","proceeding":null,"authors":["Tung Minh Luu","Younghwan Lee","Donghoon Lee","Sunho Kim","Min Jun Kim","Chang D. Yoo"],"abstract":"Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feedback is costly and labor-intensive, limiting its scalability. Recent advancements in foundation models present a promising alternative--leveraging AI-generated feedback to reduce reliance on human supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an enhanced rating-based RL method that effectively learns reward functions from AI feedback. Unlike prior methods that rely on pairwise comparisons, ERL-VLM queries large vision-language models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, we propose key enhancements to rating-based RL, addressing instability issues caused by data imbalance and noisy labels. Through extensive experiments across both low-level and high-level control tasks, we demonstrate that ERL-VLM significantly outperforms existing VLM-based reward generation methods. Our results demonstrate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient reward learning.","url_abs":"https://arxiv.org/abs/2506.12822v1","url_pdf":"https://arxiv.org/pdf/2506.12822v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enhancing-rating-based-reinforcement-learning","repo_url":"https://github.com/tunglm2203/erlvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.12822","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.12822"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tunglm2203/erlvlm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3f01f9fd0d0c237e","entry":"fanin_init","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"conv_net.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/conv_net.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f01f9fd0d0c237e"}},{"code_sha256_prefix":"a17d99b8eda680ff","entry":"gen_net","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"reward_model_vlm_rating.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/reward_model_vlm_rating.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a17d99b8eda680ff"}},{"code_sha256_prefix":"9910e2fc297f8665","entry":"identity","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"conv_net.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/conv_net.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9910e2fc297f8665"}},{"code_sha256_prefix":"115ab9ed9d356045","entry":"compute_state_entropy","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"agent/sac.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/agent/sac.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"115ab9ed9d356045"}},{"code_sha256_prefix":"7a9d9d3896b47d5b","entry":"extract_query_index","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"reward_model_vlm_rating.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/reward_model_vlm_rating.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a9d9d3896b47d5b"}},{"code_sha256_prefix":"0df23daf0145741e","entry":"setup_wandb","repo":"tunglm2203/erlvlm","repo_kind":"official","path":"logger.py","file_url":"https://github.com/tunglm2203/erlvlm/blob/HEAD/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0df23daf0145741e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}