{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distilling-model-failures-as-directions-in","title":"Distilling Model Failures as Directions in Latent Space","arxiv_id":"2206.14754","date":"2022-06-29","proceeding":null,"authors":["Saachi Jain","Hannah Lawrence","Ankur Moitra","Aleksander Madry"],"abstract":"Existing methods for isolating hard subpopulations and spurious correlations in datasets often require human intervention. This can make these methods labor-intensive and dataset-specific. To address these shortcomings, we present a scalable method for automatically distilling a model's failure modes. Specifically, we harness linear classifiers to identify consistent error patterns, and, in turn, induce a natural representation of these failure modes as directions within the feature space. We demonstrate that this framework allows us to discover and automatically caption challenging subpopulations within the training dataset. Moreover, by combining our framework with off-the-shelf diffusion models, we can generate images that are especially challenging for the analyzed model, and thus can be used to perform synthetic data augmentation that helps remedy the model's failure modes. Code available at https://github.com/MadryLab/failure-directions","url_abs":"https://arxiv.org/abs/2206.14754v2","url_pdf":"https://arxiv.org/pdf/2206.14754v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distilling-model-failures-as-directions-in","repo_url":"https://github.com/madrylab/failure-directions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.14754","atlas_url":"https://app.syntology.ai/?focus=2206.14754","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.14754"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/madrylab/failure-directions","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0800a7a994ffeaf5","entry":"ffcv_read_check_override_config","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/config_parsing.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/config_parsing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0800a7a994ffeaf5"}},{"code_sha256_prefix":"f9c7d0f4c66bf055","entry":"get_pairwise_squared_dists","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/dist_shift_utils.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/dist_shift_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9c7d0f4c66bf055"}},{"code_sha256_prefix":"e6ee1c111aeef119","entry":"inv_norm","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/ffcv_utils.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/ffcv_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6ee1c111aeef119"}},{"code_sha256_prefix":"b994e6794b2055c4","entry":"pairwise_exp_kernel","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/dist_shift_utils.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/dist_shift_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b994e6794b2055c4"}},{"code_sha256_prefix":"74b176ef329f93cb","entry":"perm_2d_tensor","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/dist_shift_utils.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/dist_shift_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74b176ef329f93cb"}},{"code_sha256_prefix":"2ad398a336dccf5d","entry":"svm_read_check_override_config","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/config_parsing.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/config_parsing.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ad398a336dccf5d"}},{"code_sha256_prefix":"ce1c8775425db385","entry":"create_dataset_split","repo":"madrylab/failure-directions","repo_kind":"official","path":"failure_directions/src/ds_utils.py","file_url":"https://github.com/madrylab/failure-directions/blob/HEAD/failure_directions/src/ds_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce1c8775425db385"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}