{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/liger-kernel-efficient-triton-kernels-for-llm","title":"Liger Kernel: Efficient Triton Kernels for LLM Training","arxiv_id":"2410.10989","date":"2024-10-14","proceeding":null,"authors":["Pin-Lun Hsu","Yun Dai","Vignesh Kothapalli","Qingquan Song","Shao Tang","Siyu Zhu","Steven Shimizu","Shivam Sahni","Haowen Ning","Yanning Chen"],"abstract":"Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.","url_abs":"https://arxiv.org/abs/2410.10989v3","url_pdf":"https://arxiv.org/pdf/2410.10989v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"liger-kernel-efficient-triton-kernels-for-llm","repo_url":"https://github.com/linkedin/liger-kernel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"chunking","task_name":"Chunking"},{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.10989","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10989"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linkedin/liger-kernel","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1b0aca09b120c2b8","entry":"infer_device_arch","repo":"linkedin/liger-kernel","repo_kind":"official","path":"src/liger_kernel/utils.py","file_url":"https://github.com/linkedin/liger-kernel/blob/HEAD/src/liger_kernel/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1b0aca09b120c2b8"}},{"code_sha256_prefix":"5713983eec4222cc","entry":"transformers_version_dispatch","repo":"linkedin/liger-kernel","repo_kind":"official","path":"src/liger_kernel/utils.py","file_url":"https://github.com/linkedin/liger-kernel/blob/HEAD/src/liger_kernel/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5713983eec4222cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}