{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/faithscore-evaluating-hallucinations-in-large","title":"FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models","arxiv_id":"2311.01477","date":"2023-11-02","proceeding":null,"authors":["Liqiang Jing","Ruosen Li","Yunmo Chen","Xinya Du"],"abstract":"We introduce FaithScore (Faithfulness to Atomic Image Facts Score), a reference-free and fine-grained evaluation metric that measures the faithfulness of the generated free-form answers from large vision-language models (LVLMs). The FaithScore evaluation first identifies sub-sentences containing descriptive statements that need to be verified, then extracts a comprehensive list of atomic facts from these sub-sentences, and finally conducts consistency verification between fine-grained atomic facts and the input image. Meta-evaluation demonstrates that our metric highly correlates with human judgments of faithfulness. We collect two benchmark datasets (i.e. LLaVA-1k and MSCOCO-Cap) for evaluating LVLMs instruction-following hallucinations. We measure hallucinations in state-of-the-art LVLMs with FaithScore on the datasets. Results reveal that current systems are prone to generate hallucinated content unfaithful to the image, which leaves room for future improvements. We hope our metric FaithScore can help evaluate future LVLMs in terms of faithfulness and provide insightful advice for enhancing LVLMs' faithfulness.","url_abs":"https://arxiv.org/abs/2311.01477v2","url_pdf":"https://arxiv.org/pdf/2311.01477v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"faithscore-evaluating-hallucinations-in-large","repo_url":"https://github.com/bcdnlp/faithscore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.01477","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.01477"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bcdnlp/faithscore","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/llava15.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/llava15.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"0bd26486b7be82c7","entry":"llava15","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/utils.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bd26486b7be82c7"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/llava15.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/llava15.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"26df052b7d003ed0","entry":"stage2_llama","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/llama_pre.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/llama_pre.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26df052b7d003ed0"}},{"code_sha256_prefix":"4db6df4a33e512a2","entry":"blip_2","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/utils.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4db6df4a33e512a2"}},{"code_sha256_prefix":"81d9bfef9f4682ec","entry":"load_llama","repo":"bcdnlp/faithscore","repo_kind":"official","path":"src/faithscore/llama_pre.py","file_url":"https://github.com/bcdnlp/faithscore/blob/HEAD/src/faithscore/llama_pre.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81d9bfef9f4682ec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}