{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-11168","title":"ELMO: Efficiency via Low-precision and Peak Memory Optimization in Large Output Spaces","arxiv_id":"2510.11168","date":"2025-10-13","proceeding":"ICML","authors":["Jinbin Zhang","Nasib Ullah","Erik Schultheis","Rohit Babbar"],"abstract":"Large output spaces, also referred to as Extreme multilabel classification (XMC), is a setting that arises, e.g., in large-scale tagging and product-to-product recommendation, and is characterized by the number of labels ranging from hundreds of thousands to millions. This means that the linear classification head, usually only a tiny fraction of the overall model, turns into the main driver for compute and memory demand. Current state-of-the-art XMC methods predominantly rely on FP16-FP32 mixed-precision training, which we show can be unstable, and inefficient in terms of memory usage and computational overhead. Meanwhile, existing low-precision methods typically retain higher precision for the classification layer. In this work, we propose ELMO, a pure low-precision training framework for XMC models using BFloat16 and Float8 data types. By leveraging Kahan summation and stochastic rounding, we demonstrate that XMC models can be effectively trained entirely in Float8, without relying on single-precision master weights or tensor scaling. Low-precision training, combined with our proposed memory optimizations -- gradient fusion and chunking -- enables significant reductions in GPU memory usage. For example, we train a 3-million-label XMC model with only 6.6 GiB of GPU memory, compared to the 39.7 GiB required by the optimized SOTA method, Renee without compromising accuracy.","url_abs":"https://arxiv.org/abs/2510.11168","url_pdf":"https://arxiv.org/pdf/2510.11168","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.11168","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.11168"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/xmc-aalto/elmo","reach":{"status":"ok","spdx":"MIT"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/warner-benjamin/optimi","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"found_in_text":{"samples":10,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"90b6172fa5dbab0b","entry":"clean_str","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90b6172fa5dbab0b"}},{"code_sha256_prefix":"8468686478a08bfd","entry":"collate","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/data.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8468686478a08bfd"}},{"code_sha256_prefix":"a42b7da783263c1b","entry":"convert","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/preprocess.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a42b7da783263c1b"}},{"code_sha256_prefix":"7f03739c008199a5","entry":"debias","repo":"warner-benjamin/optimi","repo_kind":"found_in_text","path":"optimi/utils.py","file_url":"https://github.com/warner-benjamin/optimi/blob/HEAD/optimi/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f03739c008199a5"}},{"code_sha256_prefix":"902b134b00ed8ce7","entry":"debias_beta","repo":"warner-benjamin/optimi","repo_kind":"found_in_text","path":"optimi/utils.py","file_url":"https://github.com/warner-benjamin/optimi/blob/HEAD/optimi/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"902b134b00ed8ce7"}},{"code_sha256_prefix":"61ff1f7c9fe41c1d","entry":"param_groups_weight_decay","repo":"warner-benjamin/optimi","repo_kind":"found_in_text","path":"optimi/utils.py","file_url":"https://github.com/warner-benjamin/optimi/blob/HEAD/optimi/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61ff1f7c9fe41c1d"}},{"code_sha256_prefix":"63eb98e4f891edda","entry":"print_trainable_parameters","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63eb98e4f891edda"}},{"code_sha256_prefix":"edd92d3fba08fcc2","entry":"sparsification","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/evaluate.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edd92d3fba08fcc2"}},{"code_sha256_prefix":"655433dc7bc2388f","entry":"timeit","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/preprocess.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"655433dc7bc2388f"}},{"code_sha256_prefix":"975ff6d8c2816c5a","entry":"to_wandb_dict","repo":"xmc-aalto/elmo","repo_kind":"found_in_text","path":"src/log.py","file_url":"https://github.com/xmc-aalto/elmo/blob/HEAD/src/log.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"975ff6d8c2816c5a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}