{"url":"/dataset/hallusionbench","name":"HallusionBench","full_name":null,"description_markdown":"Large language models (LLMs), after being aligned with vision models and integrated into vision-language models (VLMs), can bring impressive improvement in image reasoning tasks. This was shown by the recently released GPT-4V(ison), LLaVA-1.5, etc. However, the strong language prior in these SOTA LVLMs can be a double-edged sword: they may ignore the image context and solely rely on the (even contradictory) language prior for reasoning. In contrast, the vision modules in VLMs are weaker than LLMs and may result in misleading visual representations, which are then translated to confident mistakes by LLMs. \r\n\r\nTo study these two types of VLM mistakes, i.e., language hallucination and visual illusion, we curated HallusionBench, an image-context reasoning benchmark that is still challenging to even GPT-4V and LLaVA-1.5. We provide a detailed analysis of examples in HallusionBench, which sheds novel insights on the illusion or hallucination of VLMs and how to improve them in the future.","description_withheld":null,"homepage":"https://github.com/tianyi-lab/HallusionBench","introduced_date":"2023-10-23","introduced_date_note":null,"introduced_by":{"paper":"/paper/hallusionbench-you-see-what-you-think-or-you","title":"HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models","first_author":"Tianrui Guan","url":null},"license":{"name":"BSD 3-Clause License","url":null},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Videos","url":"/datasets/modality/videos"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Visual Question Answering (VQA)","url":"/task/visual-question-answering","datasets_with_task":"/datasets/task/visual-question-answering"}],"languages":[{"name":"English","url":"/datasets/language/english"},{"name":"Chinese","url":"/datasets/language/chinese"}],"variants":["HallusionBench"],"data_loaders":[{"repo":"https://github.com/tianyi-lab/hallusionbench","url":"https://github.com/tianyi-lab/hallusionbench","frameworks":["pytorch"]}],"num_papers_in_archive":53,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/visual-question-answering-vqa-on-3","task":"Visual Question Answering (VQA)","dataset_variant":"HallusionBench","rows":4,"metrics":["Question Pair Acc\t","Question Pair Acc"],"first_row_in_archive_order":{"model":"GPT-4V","paper":"/paper/hallusionbench-you-see-what-you-think-or-you","metrics":{"Question Pair Acc\t":"12.2047"},"code_links":[{"title":"codelion/adaptive-classifier","url":"https://github.com/codelion/adaptive-classifier"},{"title":"tianyi-lab/hallusionbench","url":"https://github.com/tianyi-lab/hallusionbench"},{"title":"FuxiaoLiu/LRV-Instruction","url":"https://github.com/FuxiaoLiu/LRV-Instruction"},{"title":"fuxiaoliu/mmc","url":"https://github.com/fuxiaoliu/mmc"},{"title":"FuxiaoLiu/VisualNews-Repository","url":"https://github.com/FuxiaoLiu/VisualNews-Repository"},{"title":"dongping-chen/mllm-as-a-judge","url":"https://github.com/dongping-chen/mllm-as-a-judge"},{"title":"zli12321/qa_metrics","url":"https://github.com/zli12321/qa_metrics"},{"title":"wuxiyang1996/AutoHallusion","url":"https://github.com/wuxiyang1996/AutoHallusion"},{"title":"zli12321/videohallu","url":"https://github.com/zli12321/videohallu"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/hallusionbench-you-see-what-you-think-or-you","title":"HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models","date":"2023-10-23","rows_on_this_dataset":1,"code_links":9,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":8,"samples_ran":2,"samples_unverified":6,"pointer_only_for_licence":1,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/aligning-large-multi-modal-model-with-robust","title":"Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning","date":"2023-06-26","rows_on_this_dataset":1,"code_links":4,"syntology":null},{"paper":"/paper/mplug-owl-modularization-empowers-large","title":"mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality","date":"2023-04-27","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":1,"samples_harvested":8,"samples_ran":2,"samples_unverified":6,"pointer_only_for_licence":1,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}