{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bongard-in-wonderland-visual-puzzles-that","title":"Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?","arxiv_id":"2410.19546","date":"2024-10-25","proceeding":null,"authors":["Antonia Wüst","Tim Tobiasch","Lukas Helff","Devendra S. Dhami","Constantin A. Rothkopf","Kristian Kersting"],"abstract":"Recently, newly developed Vision-Language Models (VLMs), such as OpenAI's GPT-4o, have emerged, seemingly demonstrating advanced reasoning capabilities across text and image modalities. Yet, the depth of these advances in language-guided perception and abstract reasoning remains underexplored, and it is unclear whether these models can truly live up to their ambitious promises. To assess the progress and identify shortcomings, we enter the wonderland of Bongard problems, a set of classical visual reasoning puzzles that require human-like abilities of pattern recognition and abstract reasoning. While VLMs occasionally succeed in identifying discriminative concepts and solving some of the problems, they frequently falter, failing to understand and reason about visual concepts. Surprisingly, even elementary concepts that may seem trivial to humans, such as simple spirals, pose significant challenges. Moreover, even when asked to explicitly focus on and analyze these concepts, they continue to falter, suggesting not only a lack of understanding of these elementary visual concepts but also an inability to generalize to unseen concepts. These observations underscore the current limitations of VLMs, emphasize that a significant gap remains between human-like visual reasoning and machine cognition, and highlight the ongoing need for innovation in this area.","url_abs":"https://arxiv.org/abs/2410.19546v1","url_pdf":"https://arxiv.org/pdf/2410.19546v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bongard-in-wonderland-visual-puzzles-that","repo_url":"https://github.com/ml-research/bongard-in-wonderland","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.19546","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19546"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/bongard-in-wonderland","reach":{"status":"ok"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"a114d362506700c0","entry":"get_categpry","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/collect_results_q1_q2.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/collect_results_q1_q2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a114d362506700c0"}},{"code_sha256_prefix":"ba2971e84747285b","entry":"get_current_judge_prompt","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/bp_llm_judge.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/bp_llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba2971e84747285b"}},{"code_sha256_prefix":"d690247952eb20f2","entry":"map_model_names","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/model_names.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/model_names.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d690247952eb20f2"}},{"code_sha256_prefix":"4e1b6d4d2b1f83ce","entry":"parse_answer","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/bp_llm_judge.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/bp_llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e1b6d4d2b1f83ce"}},{"code_sha256_prefix":"e59329369ae20f74","entry":"parse_answer","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/bp_llm_judge_human.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/bp_llm_judge_human.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e59329369ae20f74"}},{"code_sha256_prefix":"f3d7664734bd8f19","entry":"parse_dict","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/bp_llm_judge.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/bp_llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3d7664734bd8f19"}},{"code_sha256_prefix":"24725c4738f9931e","entry":"parse_dict","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/bp_llm_judge_human.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/bp_llm_judge_human.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24725c4738f9931e"}},{"code_sha256_prefix":"3580d02883c8fc63","entry":"parse_response","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"experiments/evaluate/eval_bp_with_solutions.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/experiments/evaluate/eval_bp_with_solutions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3580d02883c8fc63"}},{"code_sha256_prefix":"04742c844992deee","entry":"retrieve_images","repo":"ml-research/bongard-in-wonderland","repo_kind":"official","path":"utils/crop_images.py","file_url":"https://github.com/ml-research/bongard-in-wonderland/blob/HEAD/utils/crop_images.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04742c844992deee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}