{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/label-descriptive-patterns-and-their","title":"Label-Descriptive Patterns and Their Application to Characterizing Classification Errors","arxiv_id":"2110.09599","date":"2021-10-18","proceeding":null,"authors":["Michael Hedderich","Jonas Fischer","Dietrich Klakow","Jilles Vreeken"],"abstract":"State-of-the-art deep learning methods achieve human-like performance on many tasks, but make errors nevertheless. Characterizing these errors in easily interpretable terms gives insight into whether a classifier is prone to making systematic errors, but also gives a way to act and improve the classifier. We propose to discover those feature-value combinations (i.e., patterns) that strongly correlate with correct resp. erroneous predictions to obtain a global and interpretable description for arbitrary classifiers. We show this is an instance of the more general label description problem, which we formulate in terms of the Minimum Description Length principle. To discover a good pattern set, we develop the efficient Premise algorithm. Through an extensive set of experiments we show it performs very well in practice on both synthetic and real-world data. Unlike existing solutions, it ably recovers ground truth patterns, even on highly imbalanced data over many features. Through two case studies on Visual Question Answering and Named Entity Recognition, we confirm that Premise gives clear and actionable insight into the systematic errors made by modern NLP classifiers.","url_abs":"https://arxiv.org/abs/2110.09599v3","url_pdf":"https://arxiv.org/pdf/2110.09599v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"label-descriptive-patterns-and-their","repo_url":"https://github.com/uds-lsv/premise","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"label-descriptive-patterns-and-their","repo_url":"https://github.com/uds-lsv/pypremise","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.09599","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.09599"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/uds-lsv/premise","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uds-lsv/pypremise","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bf04d577bbf2ab0a","entry":"Pattern","repo":"uds-lsv/pypremise","repo_kind":"listed","path":"src/pypremise/core.py","file_url":"https://github.com/uds-lsv/pypremise/blob/HEAD/src/pypremise/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf04d577bbf2ab0a"}},{"code_sha256_prefix":"cc1fe08bf9887a8a","entry":"Premise","repo":"uds-lsv/pypremise","repo_kind":"listed","path":"src/pypremise/core.py","file_url":"https://github.com/uds-lsv/pypremise/blob/HEAD/src/pypremise/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc1fe08bf9887a8a"}},{"code_sha256_prefix":"f68c9d8148d1c0c4","entry":"PremiseResult","repo":"uds-lsv/pypremise","repo_kind":"listed","path":"src/pypremise/core.py","file_url":"https://github.com/uds-lsv/pypremise/blob/HEAD/src/pypremise/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f68c9d8148d1c0c4"}},{"code_sha256_prefix":"c610d635a6ef042e","entry":"Direction","repo":"uds-lsv/pypremise","repo_kind":"listed","path":"src/pypremise/core.py","file_url":"https://github.com/uds-lsv/pypremise/blob/HEAD/src/pypremise/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c610d635a6ef042e"}},{"code_sha256_prefix":"6ab5be9a22760f72","entry":"PremiseInstance","repo":"uds-lsv/pypremise","repo_kind":"listed","path":"src/pypremise/core.py","file_url":"https://github.com/uds-lsv/pypremise/blob/HEAD/src/pypremise/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ab5be9a22760f72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}