{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimized-inference-for-1-58-bit-llms-a-time","title":"An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks","arxiv_id":"2411.06360","date":"2024-11-10","proceeding":null,"authors":["Mohsen Dehghankar","Mahdi Erfanian","Abolfazl Asudeh"],"abstract":"Despite their tremendous success and versatility, Deep Neural Networks (DNNs) such as Large Language Models (LLMs) suffer from inference inefficiency and rely on advanced computational infrastructure. To address these challenges and make these models more accessible and cost-effective, in this paper, we propose algorithms to improve the inference time and memory efficiency of DNNs with binary and ternary weight matrices. Particularly focusing on matrix multiplication as the bottleneck operation of inference, we observe that, once trained, the weight matrices of a model no longer change. This allows us to preprocess these matrices and create indices that help reduce the storage requirements by a logarithmic factor while enabling our efficient inference algorithms. Specifically, for a $n\\times n$ weight matrix, our efficient algorithm guarantees a time complexity of $O(\\frac{n^2}{\\log n})$, a logarithmic factor improvement over the standard vector-matrix multiplication. Besides theoretical analysis, we conduct extensive experiments to evaluate the practical efficiency of our algorithms. Our results confirm the superiority of our approach both with respect to time and memory, as we observed a reduction in the multiplication time up to 29x and memory usage up to 6x. When applied to LLMs, our experiments show up to a 5.24x speedup in the inference time.","url_abs":"https://arxiv.org/abs/2411.06360v3","url_pdf":"https://arxiv.org/pdf/2411.06360v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimized-inference-for-1-58-bit-llms-a-time","repo_url":"https://github.com/uic-indexlab/rsr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.06360","atlas_url":"https://app.syntology.ai/?focus=2411.06360","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.06360"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uic-indexlab/rsr","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b649e4517a3d9889","entry":"activation_quant","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/archive/old_inference.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/archive/old_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b649e4517a3d9889"}},{"code_sha256_prefix":"1e91e1006c2eaabf","entry":"comparing_forward","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/gpu_impl/patch_gpu.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/gpu_impl/patch_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e91e1006c2eaabf"}},{"code_sha256_prefix":"ee3e6fed4ba5b8af","entry":"get_space_usage","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/cpu_impl/patch_cpu.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/cpu_impl/patch_cpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee3e6fed4ba5b8af"}},{"code_sha256_prefix":"b1a34b2406f0beb6","entry":"preprocess_patch","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/cpu_impl/patch_cpu.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/cpu_impl/patch_cpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a34b2406f0beb6"}},{"code_sha256_prefix":"fc065371833a4888","entry":"preprocess_patch","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/gpu_impl/patch_gpu.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/gpu_impl/patch_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc065371833a4888"}},{"code_sha256_prefix":"9852263fdc043649","entry":"rsr_forward","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/cpu_impl/patch_cpu.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/cpu_impl/patch_cpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9852263fdc043649"}},{"code_sha256_prefix":"f317f62e45257a87","entry":"weight_quant","repo":"uic-indexlab/rsr","repo_kind":"official","path":"llm/archive/old_inference.py","file_url":"https://github.com/uic-indexlab/rsr/blob/HEAD/llm/archive/old_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f317f62e45257a87"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}