{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/iconary-a-pictionary-based-game-for-testing-1","title":"Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text","arxiv_id":"2112.00800","date":"2021-12-01","proceeding":"EMNLP 2021 11","authors":["Christopher Clark","Jordi Salvador","Dustin Schwenk","Derrick Bonafilia","Mark Yatskar","Eric Kolve","Alvaro Herrasti","Jonghyun Choi","Sachin Mehta","Sam Skjonsberg","Carissa Schoenick","Aaron Sarnat","Hannaneh Hajishirzi","Aniruddha Kembhavi","Oren Etzioni","Ali Farhadi"],"abstract":"Communicating with humans is challenging for AIs because it requires a shared understanding of the world, complex semantics (e.g., metaphors or analogies), and at times multi-modal gestures (e.g., pointing with a finger, or an arrow in a diagram). We investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community. In Iconary, a Guesser tries to identify a phrase that a Drawer is drawing by composing icons, and the Drawer iteratively revises the drawing to help the Guesser in response. This back-and-forth often uses canonical scenes, visual metaphor, or icon compositions to express challenging words, making it an ideal test for mixing language and visual/symbolic communication in AI. We propose models to play Iconary and train them on over 55,000 games between human players. Our models are skillful players and are able to employ world knowledge in language models to play with words unseen during training. Elite human players outperform our models, particularly at the drawing task, leaving an important gap for future research to address. We release our dataset, code, and evaluation setup as a challenge to the community at http://www.github.com/allenai/iconary.","url_abs":"https://arxiv.org/abs/2112.00800v1","url_pdf":"https://arxiv.org/pdf/2112.00800v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"iconary-a-pictionary-based-game-for-testing-1","repo_url":"https://github.com/allenai/iconary","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[{"slug":"iconary","name":"Iconary","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2112.00800","atlas_url":"https://app.syntology.ai/?focus=2112.00800","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.00800"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/iconary","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6aeb9b6b8d1117e0","entry":"get_icon_tokens","repo":"allenai/iconary","repo_kind":"official","path":"iconary/models/drawing_to_text.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/models/drawing_to_text.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6aeb9b6b8d1117e0"}},{"code_sha256_prefix":"b625e8b10728c9d5","entry":"get_soft_win_leeway","repo":"allenai/iconary","repo_kind":"official","path":"iconary/eval/eval_guess.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/eval/eval_guess.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b625e8b10728c9d5"}},{"code_sha256_prefix":"0f0e3c476a76aaa4","entry":"get_tokens_with_space","repo":"allenai/iconary","repo_kind":"official","path":"iconary/models/t5_decoding.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/models/t5_decoding.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f0e3c476a76aaa4"}},{"code_sha256_prefix":"6c85723b1c1d9f86","entry":"t5_construct_labels","repo":"allenai/iconary","repo_kind":"official","path":"iconary/models/t5_models.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/models/t5_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c85723b1c1d9f86"}},{"code_sha256_prefix":"e460214e2611bf7a","entry":"get_starts_word","repo":"allenai/iconary","repo_kind":"official","path":"iconary/models/t5_decoding.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/models/t5_decoding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e460214e2611bf7a"}},{"code_sha256_prefix":"e31c68a263cbb61e","entry":"guesses_to_dictionary","repo":"allenai/iconary","repo_kind":"official","path":"iconary/models/generation.py","file_url":"https://github.com/allenai/iconary/blob/HEAD/iconary/models/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e31c68a263cbb61e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}