{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taming-the-sigmoid-bottleneck-provably","title":"Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification","arxiv_id":"2310.10443","date":"2023-10-16","proceeding":null,"authors":["Andreas Grivas","Antonio Vergari","Adam Lopez"],"abstract":"Sigmoid output layers are widely used in multi-label classification (MLC) tasks, in which multiple labels can be assigned to any input. In many practical MLC tasks, the number of possible labels is in the thousands, often exceeding the number of input features and resulting in a low-rank output layer. In multi-class classification, it is known that such a low-rank output layer is a bottleneck that can result in unargmaxable classes: classes which cannot be predicted for any input. In this paper, we show that for MLC tasks, the analogous sigmoid bottleneck results in exponentially many unargmaxable label combinations. We explain how to detect these unargmaxable outputs and demonstrate their presence in three widely used MLC datasets. We then show that they can be prevented in practice by introducing a Discrete Fourier Transform (DFT) output layer, which guarantees that all sparse label combinations with up to $k$ active labels are argmaxable. Our DFT layer trains faster and is more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters. Our code is publicly available at https://github.com/andreasgrv/sigmoid-bottleneck.","url_abs":"https://arxiv.org/abs/2310.10443v2","url_pdf":"https://arxiv.org/pdf/2310.10443v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taming-the-sigmoid-bottleneck-provably","repo_url":"https://github.com/andreasgrv/sigmoid-bottleneck","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"},{"task_slug":"multi-class-classification","task_name":"Multi-class Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.10443","atlas_url":"https://app.syntology.ai/?focus=2310.10443","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.10443"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andreasgrv/sigmoid-bottleneck","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":13,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa97babdb6f4b212","entry":"affinise","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/modules.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa97babdb6f4b212"}},{"code_sha256_prefix":"467640b6cac88182","entry":"count_alternating_bits","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/verify.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/verify.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"467640b6cac88182"}},{"code_sha256_prefix":"95a1b13f612e6900","entry":"cyclic_polytope_trig","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/modules.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95a1b13f612e6900"}},{"code_sha256_prefix":"f1080c6e6756f465","entry":"encode_batch_labels","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/data.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1080c6e6756f465"}},{"code_sha256_prefix":"83ded52943d23cf3","entry":"encode_label_list","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/data.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83ded52943d23cf3"}},{"code_sha256_prefix":"77bf2eea0dac3863","entry":"f1","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/metrics.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77bf2eea0dac3863"}},{"code_sha256_prefix":"09b29a8958f3dcca","entry":"f1_counts","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/metrics.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09b29a8958f3dcca"}},{"code_sha256_prefix":"f072f1c9d0e306c2","entry":"generate_card_samples","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"plots/argmaxability_slack/check_infeasible_cyclic.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/plots/argmaxability_slack/check_infeasible_cyclic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f072f1c9d0e306c2"}},{"code_sha256_prefix":"9da5519e0206563b","entry":"k_hot_to_dense","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/verify.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/verify.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9da5519e0206563b"}},{"code_sha256_prefix":"50a7ff0312b87b82","entry":"linear_dependence","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/math.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/math.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50a7ff0312b87b82"}},{"code_sha256_prefix":"43e11696974c4c73","entry":"orthogonal_complement","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/math.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/math.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43e11696974c4c73"}},{"code_sha256_prefix":"a51b6a4a498e44a8","entry":"pattern_is_feasible_cyclic","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/verify.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/verify.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a51b6a4a498e44a8"}},{"code_sha256_prefix":"e29b1f502ab0b31f","entry":"torch_interleave_columns","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/modules.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e29b1f502ab0b31f"}},{"code_sha256_prefix":"b1095adb35aeb739","entry":"encode_bpe_label_list","repo":"andreasgrv/sigmoid-bottleneck","repo_kind":"official","path":"spmlbl/data.py","file_url":"https://github.com/andreasgrv/sigmoid-bottleneck/blob/HEAD/spmlbl/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1095adb35aeb739"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}