{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-systemic-error-detection-methods","title":"Towards a More Rigorous Science of Blindspot Discovery in Image Classification Models","arxiv_id":"2207.04104","date":"2022-07-08","proceeding":null,"authors":["Gregory Plumb","Nari Johnson","Ángel Alexander Cabrera","Ameet Talwalkar"],"abstract":"A growing body of work studies Blindspot Discovery Methods (\"BDM\"s): methods that use an image embedding to find semantically meaningful (i.e., united by a human-understandable concept) subsets of the data where an image classifier performs significantly worse. Motivated by observed gaps in prior work, we introduce a new framework for evaluating BDMs, SpotCheck, that uses synthetic image datasets to train models with known blindspots and a new BDM, PlaneSpot, that uses a 2D image representation. We use SpotCheck to run controlled experiments that identify factors that influence BDM performance (e.g., the number of blindspots in a model, or features used to define the blindspot) and show that PlaneSpot is competitive with and in many cases outperforms existing BDMs. Importantly, we validate these findings by designing additional experiments that use real image data from MS-COCO, a large image benchmark dataset. Our findings suggest several promising directions for future work on BDM design and evaluation. Overall, we hope that the methodology and analyses presented in this work will help facilitate a more rigorous science of blindspot discovery.","url_abs":"https://arxiv.org/abs/2207.04104v3","url_pdf":"https://arxiv.org/pdf/2207.04104v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-systemic-error-detection-methods","repo_url":"https://github.com/njohnson99/spotcheck","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evaluating-systemic-error-detection-methods","repo_url":"https://github.com/hazyresearch/domino","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.04104","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.04104"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/domino","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/njohnson99/spotcheck","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"630d3d44485d9ee2","entry":"binary_roll","repo":"njohnson99/spotcheck","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/njohnson99/spotcheck/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"630d3d44485d9ee2"}},{"code_sha256_prefix":"8034828ee260512e","entry":"generate_shape_mask","repo":"njohnson99/spotcheck","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/njohnson99/spotcheck/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8034828ee260512e"}},{"code_sha256_prefix":"7cdefdd18f9ea351","entry":"generate_text_mask","repo":"njohnson99/spotcheck","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/njohnson99/spotcheck/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cdefdd18f9ea351"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}