{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-abduction-of-sherlock-holmes-a-dataset","title":"The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning","arxiv_id":"2202.04800","date":"2022-02-10","proceeding":null,"authors":["Jack Hessel","Jena D. Hwang","Jae Sung Park","Rowan Zellers","Chandra Bhagavatula","Anna Rohrbach","Kate Saenko","Yejin Choi"],"abstract":"Humans have remarkable capacity to reason abductively and hypothesize about what lies beyond the literal content of an image. By identifying concrete visual clues scattered throughout a scene, we almost can't help but draw probable inferences beyond the literal scene based on our everyday experience and knowledge about the world. For example, if we see a \"20 mph\" sign alongside a road, we might assume the street sits in a residential area (rather than on a highway), even if no houses are pictured. Can machines perform similar visual reasoning? We present Sherlock, an annotated corpus of 103K images for testing machine capacity for abductive reasoning beyond literal image contents. We adopt a free-viewing paradigm: participants first observe and identify salient clues within images (e.g., objects, actions) and then provide a plausible inference about the scene, given the clue. In total, we collect 363K (clue, inference) pairs, which form a first-of-its-kind abductive visual reasoning dataset. Using our corpus, we test three complementary axes of abductive reasoning. We evaluate the capacity of models to: i) retrieve relevant inferences from a large candidate corpus; ii) localize evidence for inferences via bounding boxes, and iii) compare plausible inferences to match human judgments on a newly-collected diagnostic corpus of 19K Likert-scale judgments. While we find that fine-tuning CLIP-RN50x64 with a multitask objective outperforms strong baselines, significant headroom exists between model performance and human agreement. Data, models, and leaderboard available at http://visualabduction.com/","url_abs":"https://arxiv.org/abs/2202.04800v2","url_pdf":"https://arxiv.org/pdf/2202.04800v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-abduction-of-sherlock-holmes-a-dataset","repo_url":"https://github.com/lunaproject22/rpa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"visual-abductive-reasoning","task_name":"Visual Abductive Reasoning"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"sherlock","name":"SHERLOCK","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2202.04800","atlas_url":"https://app.syntology.ai/?focus=2202.04800","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.04800"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lunaproject22/rpa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7},"by_repo_kind":{"listed":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"480e30b5df38e0fc","entry":"autocontrast_func","repo":"lunaproject22/rpa","repo_kind":"listed","path":"images/randaugment.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/images/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"480e30b5df38e0fc"}},{"code_sha256_prefix":"bd5fff3dc2dc17b2","entry":"clip_forward","repo":"lunaproject22/rpa","repo_kind":"listed","path":"leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd5fff3dc2dc17b2"}},{"code_sha256_prefix":"a456fba9a4b4a6b2","entry":"clip_forward_image","repo":"lunaproject22/rpa","repo_kind":"listed","path":"leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a456fba9a4b4a6b2"}},{"code_sha256_prefix":"5ab00467725c0903","entry":"clip_forward_text","repo":"lunaproject22/rpa","repo_kind":"listed","path":"leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/leaderboard_eval/clip_leaderboard/clip_inference_iterator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ab00467725c0903"}},{"code_sha256_prefix":"6686455a295a4e15","entry":"equalize_func","repo":"lunaproject22/rpa","repo_kind":"listed","path":"images/randaugment.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/images/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6686455a295a4e15"}},{"code_sha256_prefix":"79b8769b61433979","entry":"identity_func","repo":"lunaproject22/rpa","repo_kind":"listed","path":"images/randaugment.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/images/randaugment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79b8769b61433979"}},{"code_sha256_prefix":"a1f80d0c1ee2b1e6","entry":"pairwise_acc","repo":"lunaproject22/rpa","repo_kind":"listed","path":"leaderboard_eval/score_comparison.py","file_url":"https://github.com/lunaproject22/rpa/blob/HEAD/leaderboard_eval/score_comparison.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1f80d0c1ee2b1e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}