{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-and-analyzing-relationship","title":"Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models","arxiv_id":"2406.16449","date":"2024-06-24","proceeding":null,"authors":["Mingrui Wu","Jiayi Ji","Oucheng Huang","Jiale Li","Yuhang Wu","Xiaoshuai Sun","Rongrong Ji"],"abstract":"The issue of hallucinations is a prevalent concern in existing Large Vision-Language Models (LVLMs). Previous efforts have primarily focused on investigating object hallucinations, which can be easily alleviated by introducing object detectors. However, these efforts neglect hallucinations in inter-object relationships, which is essential for visual comprehension. In this work, we introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object. The visual instruction tuning dataset's long-tail distribution significantly impacts LVLMs' understanding of visual relationships. Furthermore, our analysis reveals that current LVLMs tend to disregard visual content and overly rely on the common sense knowledge of Large Language Models. They also struggle with reasoning about spatial relationships based on contextual information.","url_abs":"https://arxiv.org/abs/2406.16449v4","url_pdf":"https://arxiv.org/pdf/2406.16449v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-and-analyzing-relationship","repo_url":"https://github.com/mrwu-mac/R-Bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"object","task_name":"Object"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16449","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16449"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrwu-mac/R-Bench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"81ab869982b6c33d","entry":"draw_box","repo":"mrwu-mac/R-Bench","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mrwu-mac/R-Bench/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81ab869982b6c33d"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"mrwu-mac/R-Bench","repo_kind":"official","path":"demos/eval_rbench.py","file_url":"https://github.com/mrwu-mac/R-Bench/blob/HEAD/demos/eval_rbench.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"mrwu-mac/R-Bench","repo_kind":"official","path":"demos/eval_rbench.py","file_url":"https://github.com/mrwu-mac/R-Bench/blob/HEAD/demos/eval_rbench.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"1a3b4176337c42b0","entry":"draw_mask","repo":"mrwu-mac/R-Bench","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mrwu-mac/R-Bench/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a3b4176337c42b0"}},{"code_sha256_prefix":"52eb8934678c8a69","entry":"show_mask_pil","repo":"mrwu-mac/R-Bench","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mrwu-mac/R-Bench/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52eb8934678c8a69"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}