{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/image-tell-me-your-story-predicting-the","title":"\"Image, Tell me your story!\" Predicting the original meta-context of visual misinformation","arxiv_id":"2408.09939","date":"2024-08-19","proceeding":null,"authors":["Jonathan Tonglet","Marie-Francine Moens","Iryna Gurevych"],"abstract":"To assist human fact-checkers, researchers have developed automated approaches for visual misinformation detection. These methods assign veracity scores by identifying inconsistencies between the image and its caption, or by detecting forgeries in the image. However, they neglect a crucial point of the human fact-checking process: identifying the original meta-context of the image. By explaining what is actually true about the image, fact-checkers can better detect misinformation, focus their efforts on check-worthy visual content, engage in counter-messaging before misinformation spreads widely, and make their explanation more convincing. Here, we fill this gap by introducing the task of automated image contextualization. We create 5Pils, a dataset of 1,676 fact-checked images with question-answer pairs about their original meta-context. Annotations are based on the 5 Pillars fact-checking framework. We implement a first baseline that grounds the image in its original meta-context using the content of the image and textual evidence retrieved from the open web. Our experiments show promising results while highlighting several open challenges in retrieval and reasoning. We make our code and data publicly available.","url_abs":"https://arxiv.org/abs/2408.09939v2","url_pdf":"https://arxiv.org/pdf/2408.09939v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"image-tell-me-your-story-predicting-the","repo_url":"https://github.com/ukplab/5pils","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"misinformation","task_name":"Misinformation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.09939","atlas_url":"https://app.syntology.ai/?focus=2408.09939","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.09939"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ukplab/5pils","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/UKPLab/5pils","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":15,"unverified":2},"by_repo_kind":{"official":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ad9e047fb83b6d3","entry":"concatenate_entry","repo":"UKPLab/5pils","repo_kind":"official","path":"utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ad9e047fb83b6d3"}},{"code_sha256_prefix":"bbfbef96b946a60d","entry":"convert_to_date","repo":"UKPLab/5pils","repo_kind":"official","path":"evaluation/evaluation_metrics.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/evaluation/evaluation_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbfbef96b946a60d"}},{"code_sha256_prefix":"fc2f37b8acf5766b","entry":"cosine_similarity","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/generation_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc2f37b8acf5766b"}},{"code_sha256_prefix":"dd767a28b135bd80","entry":"date_distance","repo":"UKPLab/5pils","repo_kind":"official","path":"evaluation/evaluation_metrics.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/evaluation/evaluation_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd767a28b135bd80"}},{"code_sha256_prefix":"04823be297d284be","entry":"entry_exists","repo":"UKPLab/5pils","repo_kind":"official","path":"utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"04823be297d284be"}},{"code_sha256_prefix":"2a8e9af6866ca933","entry":"get_geonames_hierarchy","repo":"UKPLab/5pils","repo_kind":"official","path":"evaluation/geonames_collection.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/evaluation/geonames_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a8e9af6866ca933"}},{"code_sha256_prefix":"11b07f7f8addccd3","entry":"get_organization","repo":"UKPLab/5pils","repo_kind":"official","path":"dataset_collection/preprocessing_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/dataset_collection/preprocessing_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11b07f7f8addccd3"}},{"code_sha256_prefix":"408a24dade610ca2","entry":"gpt4_prompting","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/llm_prompting.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/llm_prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"408a24dade610ca2"}},{"code_sha256_prefix":"bab48b3fe0c88b5f","entry":"gpt4_vision_prompting","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/llm_prompting.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/llm_prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bab48b3fe0c88b5f"}},{"code_sha256_prefix":"c8af83dd089eb7c5","entry":"is_english_article","repo":"UKPLab/5pils","repo_kind":"official","path":"dataset_collection/scrape_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/dataset_collection/scrape_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8af83dd089eb7c5"}},{"code_sha256_prefix":"eb82c897d04b22b2","entry":"load_json","repo":"UKPLab/5pils","repo_kind":"official","path":"utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb82c897d04b22b2"}},{"code_sha256_prefix":"eaa4649b3cda1658","entry":"load_urls","repo":"UKPLab/5pils","repo_kind":"official","path":"dataset_collection/scrape_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/dataset_collection/scrape_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eaa4649b3cda1658"}},{"code_sha256_prefix":"b73e80aac59eccfa","entry":"search_location","repo":"UKPLab/5pils","repo_kind":"official","path":"evaluation/geonames_collection.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/evaluation/geonames_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b73e80aac59eccfa"}},{"code_sha256_prefix":"12ea3e061c3f8091","entry":"sort_with_clip_score","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/generation_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12ea3e061c3f8091"}},{"code_sha256_prefix":"d78f28d04bd9687a","entry":"sort_with_image_similarity","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/generation_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d78f28d04bd9687a"}},{"code_sha256_prefix":"ba1226b6627f813f","entry":"assemble_prompt_gpt4","repo":"UKPLab/5pils","repo_kind":"official","path":"baseline/llm_prompting.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/baseline/llm_prompting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1226b6627f813f"}},{"code_sha256_prefix":"16506989e60a009d","entry":"scrape_image","repo":"UKPLab/5pils","repo_kind":"official","path":"dataset_collection/scrape_utils.py","file_url":"https://github.com/UKPLab/5pils/blob/HEAD/dataset_collection/scrape_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16506989e60a009d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}