{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-empirical-study-on-large-scale-multi-label","title":"An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels","arxiv_id":"2010.01653","date":"2020-10-04","proceeding":"EMNLP 2020 11","authors":["Ilias Chalkidis","Manos Fergadiotis","Sotiris Kotitsas","Prodromos Malakasiotis","Nikolaos Aletras","Ion Androutsopoulos"],"abstract":"Large-scale Multi-label Text Classification (LMTC) has a wide range of Natural Language Processing (NLP) applications and presents interesting challenges. First, not all labels are well represented in the training set, due to the very large label set and the skewed label distributions of LMTC datasets. Also, label hierarchies and differences in human labelling guidelines may affect graph-aware annotation proximity. Finally, the label hierarchies are periodically updated, requiring LMTC models capable of zero-shot generalization. Current state-of-the-art LMTC models employ Label-Wise Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label classification; (2) may use the label hierarchy to improve zero-shot learning, although this practice is vastly understudied; and (3) have not been combined with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art results in several NLP benchmarks. Here, for the first time, we empirically evaluate a battery of LMTC methods from vanilla LWANs to hierarchical classification approaches and transfer learning, on frequent, few, and zero-shot learning on three datasets from different domains. We show that hierarchical methods based on Probabilistic Label Trees (PLTs) outperform LWANs. Furthermore, we show that Transformer-based approaches outperform the state-of-the-art in two of the datasets, and we propose a new state-of-the-art method which combines BERT with LWANs. Finally, we propose new models that leverage the label hierarchy to improve few and zero-shot learning, considering on each dataset a graph-aware annotation proximity measure that we introduce.","url_abs":"https://arxiv.org/abs/2010.01653v1","url_pdf":"https://arxiv.org/pdf/2010.01653v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-empirical-study-on-large-scale-multi-label","repo_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-text-classification-1","task_name":"Multi Label Text Classification"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"multi-label-text-classification","task_name":"Multi-Label Text Classification"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.01653","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.01653"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iliaschalkidis/lmtc-eurlex57k","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1923fc05163d207d","entry":"convert_to_unicode","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"neural_networks/layers/bert_tokenization.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/neural_networks/layers/bert_tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1923fc05163d207d"}},{"code_sha256_prefix":"bcf6da6d6b37e4d1","entry":"dot_product","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"neural_networks/layers/attention.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/neural_networks/layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bcf6da6d6b37e4d1"}},{"code_sha256_prefix":"ff83ccc8b0b6462d","entry":"load_vocab","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"neural_networks/layers/bert_tokenization.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/neural_networks/layers/bert_tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff83ccc8b0b6462d"}},{"code_sha256_prefix":"e38d16c8dc98d937","entry":"mean_ndcg_score","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e38d16c8dc98d937"}},{"code_sha256_prefix":"3cc7db202885c138","entry":"mean_precision_k","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cc7db202885c138"}},{"code_sha256_prefix":"c098f170b5b2019d","entry":"mean_recall_k","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c098f170b5b2019d"}},{"code_sha256_prefix":"0e5615f8994003cf","entry":"printable_text","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"neural_networks/layers/bert_tokenization.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/neural_networks/layers/bert_tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e5615f8994003cf"}},{"code_sha256_prefix":"e08f3bf74e750d72","entry":"tokenize_doc_en","repo":"iliaschalkidis/lmtc-eurlex57k","repo_kind":"official","path":"spacy_tagger.py","file_url":"https://github.com/iliaschalkidis/lmtc-eurlex57k/blob/HEAD/spacy_tagger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e08f3bf74e750d72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}