{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/polar-sparsity-high-throughput-batched-llm","title":"Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity","arxiv_id":"2505.14884","date":"2025-05-20","proceeding":null,"authors":["Susav Shrestha","Brad Settlemyer","Nikoli Dryden","Narasimha Reddy"],"abstract":"Accelerating large language model (LLM) inference is critical for real-world deployments requiring high throughput and low latency. Contextual sparsity, where each token dynamically activates only a small subset of the model parameters, shows promise but does not scale to large batch sizes due to union of active neurons quickly approaching dense computation. We introduce Polar Sparsity, highlighting a key shift in sparsity importance from MLP to Attention layers as we scale batch size and sequence length. While MLP layers become more compute-efficient under batching, their sparsity vanishes. In contrast, attention becomes increasingly more expensive at scale, while their head sparsity remains stable and batch-invariant. We develop hardware-efficient, sparsity-aware GPU kernels for selective MLP and Attention computations, delivering up to \\(2.2\\times\\) end-to-end speedups for models like OPT, LLaMA-2 \\& 3, across various batch sizes and sequence lengths without compromising accuracy. To our knowledge, this is the first work to demonstrate that contextual sparsity can scale effectively to large batch sizes, delivering substantial inference acceleration with minimal changes, making Polar Sparsity practical for large-scale, high-throughput LLM deployment systems. Our code is available at: https://github.com/susavlsh10/Polar-Sparsity.","url_abs":"https://arxiv.org/abs/2505.14884v1","url_pdf":"https://arxiv.org/pdf/2505.14884v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"polar-sparsity-high-throughput-batched-llm","repo_url":"https://github.com/susavlsh10/polar-sparsity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.14884","atlas_url":"https://app.syntology.ai/?focus=2505.14884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.14884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/susavlsh10/Polar-Sparsity","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/susavlsh10/polar-sparsity","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"af8781316172eaf2","entry":"initialize_data_structures","repo":"susavlsh10/polar-sparsity","repo_kind":"official","path":"HybridTensor/routers/datacollection/data_collection.py","file_url":"https://github.com/susavlsh10/polar-sparsity/blob/HEAD/HybridTensor/routers/datacollection/data_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af8781316172eaf2"}},{"code_sha256_prefix":"738f12bdf709b633","entry":"load_layer_data","repo":"susavlsh10/polar-sparsity","repo_kind":"official","path":"HybridTensor/routers/datacollection/data_collection.py","file_url":"https://github.com/susavlsh10/polar-sparsity/blob/HEAD/HybridTensor/routers/datacollection/data_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"738f12bdf709b633"}},{"code_sha256_prefix":"314abce3a7becf22","entry":"process_batch","repo":"susavlsh10/polar-sparsity","repo_kind":"official","path":"HybridTensor/routers/datacollection/data_collection.py","file_url":"https://github.com/susavlsh10/polar-sparsity/blob/HEAD/HybridTensor/routers/datacollection/data_collection.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"314abce3a7becf22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}