{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/grit-general-robust-image-task-benchmark","title":"GRIT: General Robust Image Task Benchmark","arxiv_id":"2204.13653","date":"2022-04-28","proceeding":null,"authors":["Tanmay Gupta","Ryan Marten","Aniruddha Kembhavi","Derek Hoiem"],"abstract":"Computer vision models excel at making predictions when the test distribution closely resembles the training distribution. Such models have yet to match the ability of biological vision to learn from multiple sources and generalize to new data sources and tasks. To facilitate the development and evaluation of more general vision systems, we introduce the General Robust Image Task (GRIT) benchmark. GRIT evaluates the performance, robustness, and calibration of a vision system across a variety of image prediction tasks, concepts, and data sources. The seven tasks in GRIT are selected to cover a range of visual skills: object categorization, object localization, referring expression grounding, visual question answering, segmentation, human keypoint detection, and surface normal estimation. GRIT is carefully designed to enable the evaluation of robustness under image perturbations, image source distribution shift, and concept distribution shift. By providing a unified platform for thorough assessment of skills and concepts learned by a vision model, we hope GRIT catalyzes the development of performant and robust general purpose vision systems.","url_abs":"https://arxiv.org/abs/2204.13653v2","url_pdf":"https://arxiv.org/pdf/2204.13653v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"grit-general-robust-image-task-benchmark","repo_url":"https://github.com/allenai/grit_official","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"keypoint-detection","task_name":"Keypoint Detection"},{"task_slug":"object-categorization","task_name":"Object Categorization"},{"task_slug":"object-localization","task_name":"Object Localization"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"surface-normal-estimation","task_name":"Surface Normal Estimation"},{"task_slug":"surface-normals-estimation","task_name":"Surface Normals Estimation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[{"slug":"grit","name":"GRIT","full_name":"General Robust Image Task Benchmark"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.13653","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.13653"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/grit_official","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0721c0dabf3ece54","entry":"get_metric","repo":"allenai/grit_official","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/allenai/grit_official/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0721c0dabf3ece54"}},{"code_sha256_prefix":"8df16a0784c57da1","entry":"get_parallel_undist_example_ids","repo":"allenai/grit_official","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/allenai/grit_official/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8df16a0784c57da1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}