{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automated-multi-level-preference-for-mllms","title":"Automated Multi-level Preference for MLLMs","arxiv_id":"2405.11165","date":"2024-05-18","proceeding":null,"authors":["Mengxi Zhang","Wenhao Wu","Yu Lu","Yuxin Song","Kang Rong","Huanjin Yao","Jianbo Zhao","Fanglong Liu","Yifan Sun","Haocheng Feng","Jingdong Wang"],"abstract":"Current multimodal Large Language Models (MLLMs) suffer from ``hallucination'', occasionally generating responses that are not grounded in the input images. To tackle this challenge, one promising path is to utilize reinforcement learning from human feedback (RLHF), which steers MLLMs towards learning superior responses while avoiding inferior ones. We rethink the common practice of using binary preferences (i.e., superior, inferior), and find that adopting multi-level preferences (e.g., superior, medium, inferior) is better for two benefits: 1) It narrows the gap between adjacent levels, thereby encouraging MLLMs to discern subtle differences. 2) It further integrates cross-level comparisons (beyond adjacent-level comparisons), thus providing a broader range of comparisons with hallucination examples. To verify our viewpoint, we present the Automated Multi-level Preference (AMP) framework for MLLMs. To facilitate this framework, we first develop an automated dataset generation pipeline that provides high-quality multi-level preference datasets without any human annotators. Furthermore, we design the Multi-level Direct Preference Optimization (MDPO) algorithm to robustly conduct complex multi-level preference learning. Additionally, we propose a new hallucination benchmark, MRHal-Bench. Extensive experiments across public hallucination and general benchmarks, as well as our MRHal-Bench, demonstrate the effectiveness of our proposed method. Code is available at https://github.com/takomc/amp.","url_abs":"https://arxiv.org/abs/2405.11165v4","url_pdf":"https://arxiv.org/pdf/2405.11165v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automated-multi-level-preference-for-mllms","repo_url":"https://github.com/takomc/amp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dataset-generation","task_name":"Dataset Generation"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11165"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/takomc/amp","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"5aa59c85970b28a7","entry":"all_gather_and_cat","repo":"takomc/amp","repo_kind":"official","path":"models/distributed_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/models/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5aa59c85970b28a7"}},{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"takomc/amp","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"89ce33341af66037","entry":"get_last_checkpoint","repo":"takomc/amp","repo_kind":"official","path":"lora_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/lora_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89ce33341af66037"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"takomc/amp","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"feb20052f9ff6506","entry":"pad_to_length","repo":"takomc/amp","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"feb20052f9ff6506"}},{"code_sha256_prefix":"4098e9c2afde4a5d","entry":"setup","repo":"takomc/amp","repo_kind":"official","path":"models/distributed_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/models/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4098e9c2afde4a5d"}},{"code_sha256_prefix":"344dff4791fd1381","entry":"process_images","repo":"takomc/amp","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"344dff4791fd1381"}},{"code_sha256_prefix":"3b88619418072c5f","entry":"trl_sanitze_kwargs_for_tagging","repo":"takomc/amp","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/takomc/amp/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b88619418072c5f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}