{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-driven-slice-discovery-and-error","title":"LADDER: Language Driven Slice Discovery and Error Rectification","arxiv_id":"2408.07832","date":"2024-07-31","proceeding":null,"authors":["Shantanu Ghosh","Rayan Syed","Chenyu Wang","Clare B. Poynton","Shyam Visweswaran","Kayhan Batmanghelich"],"abstract":"Error slice discovery is crucial to diagnose and mitigate model errors. Current clustering or discrete attribute-based slice discovery methods face key limitations: 1) clustering results in incoherent slices, while assigning discrete attributes to slices leads to incomplete coverage of error patterns due to missing or insufficient attributes; 2) these methods lack complex reasoning, preventing them from fully explaining model biases; 3) they fail to integrate \\textit{domain knowledge}, limiting their usage in specialized fields \\eg radiology. We propose\\ladder (\\underline{La}nguage-\\underline{D}riven \\underline{D}iscovery and \\underline{E}rror \\underline{R}ectification), to address the limitations by: (1) leveraging the flexibility of natural language to address incompleteness, (2) employing LLM's latent \\textit{domain knowledge} and advanced reasoning to analyze sentences and derive testable hypotheses directly, identifying biased attributes, and form coherent error slices without clustering. Existing mitigation methods typically address only the worst-performing group, often amplifying errors in other subgroups. In contrast,\\ladder generates pseudo attributes from the discovered hypotheses to mitigate errors across all biases without explicit attribute annotations or prior knowledge of bias. Rigorous evaluations on 6 datasets spanning natural and medical images -- comparing 200+ classifiers with diverse architectures, pretraining strategies, and LLMs -- show that\\ladder consistently outperforms existing baselines in discovering and mitigating biases.","url_abs":"https://arxiv.org/abs/2408.07832v8","url_pdf":"https://arxiv.org/pdf/2408.07832v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-driven-slice-discovery-and-error","repo_url":"https://github.com/batmanlab/Ladder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"slice-discovery","task_name":"Slice Discovery"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.07832","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.07832"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batmanlab/Ladder","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"summary":{"ran":15,"unverified":2},"by_repo_kind":{"listed":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"688ce138721d837e","entry":"compute_AUC","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/metrics.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"688ce138721d837e"}},{"code_sha256_prefix":"8f49eb5b19d3ac4e","entry":"compute_accuracy","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/metrics.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8f49eb5b19d3ac4e"}},{"code_sha256_prefix":"896063eb2a6b5fdd","entry":"compute_auprc","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/metrics.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"896063eb2a6b5fdd"}},{"code_sha256_prefix":"0dbfc40c7885054d","entry":"custom_collate","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/caption_images.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/caption_images.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0dbfc40c7885054d"}},{"code_sha256_prefix":"8c7e7d4a40a6ed72","entry":"generate_ds_last_layer_retrain","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/mitigate_error_slices.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/mitigate_error_slices.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8c7e7d4a40a6ed72"}},{"code_sha256_prefix":"3b0e08bad449b6ab","entry":"get_Paths","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/utils.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3b0e08bad449b6ab"}},{"code_sha256_prefix":"45216ce5a16d2a2e","entry":"get_device","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/utils.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45216ce5a16d2a2e"}},{"code_sha256_prefix":"6af797ee849743d4","entry":"get_hypothesis_from_llama","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/validate_error_slices_w_LLM.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/validate_error_slices_w_LLM.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6af797ee849743d4"}},{"code_sha256_prefix":"2058541939bc9d91","entry":"init_additional_info","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/save_img_reps.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/save_img_reps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2058541939bc9d91"}},{"code_sha256_prefix":"559c536421e60ded","entry":"init_path","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/learn_aligner.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/learn_aligner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"559c536421e60ded"}},{"code_sha256_prefix":"9d091c247f43d1f2","entry":"load_model_with_retries","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/caption_images.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/caption_images.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9d091c247f43d1f2"}},{"code_sha256_prefix":"2da591b7c5b0cc23","entry":"process_batch","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/save_img_reps.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/save_img_reps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2da591b7c5b0cc23"}},{"code_sha256_prefix":"3a2653bc82bd52d6","entry":"resize_and_encode_image","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/caption_images_gpt_4.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/caption_images_gpt_4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3a2653bc82bd52d6"}},{"code_sha256_prefix":"b29a01fd0e697f3d","entry":"stratified_sample","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/utils.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b29a01fd0e697f3d"}},{"code_sha256_prefix":"54000d2c28a0531d","entry":"update_additional_info","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/save_img_reps.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/save_img_reps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"54000d2c28a0531d"}},{"code_sha256_prefix":"4ffff391e5439ad1","entry":"caption_image","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/caption_images_gpt_4.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/caption_images_gpt_4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4ffff391e5439ad1"}},{"code_sha256_prefix":"8c7dfc479e1f9844","entry":"get_hypothesis_from_GPT","repo":"batmanlab/Ladder","repo_kind":"listed","path":"src/codebase/validate_error_slices_w_LLM.py","file_url":"https://github.com/batmanlab/Ladder/blob/HEAD/src/codebase/validate_error_slices_w_LLM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8c7dfc479e1f9844"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}