{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/matrix-compression-via-randomized-low-rank","title":"Matrix Compression via Randomized Low Rank and Low Precision Factorization","arxiv_id":"2310.11028","date":"2023-10-17","proceeding":"NeurIPS 2023 11","authors":["Rajarshi Saha","Varun Srivastava","Mert Pilanci"],"abstract":"Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix $\\mathbf{A}$ as $\\mathbf{A} \\approx \\mathbf{L}\\mathbf{R}$, where $\\mathbf{L}$ and $\\mathbf{R}$ are the low rank factors. The total number of elements in $\\mathbf{L}$ and $\\mathbf{R}$ can be significantly less than that in $\\mathbf{A}$. Furthermore, the entries of $\\mathbf{L}$ and $\\mathbf{R}$ are quantized to low precision formats $--$ compressing $\\mathbf{A}$ by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of $\\mathbf{A}$ by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of $\\mathbf{A}$ onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-$7$b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.","url_abs":"https://arxiv.org/abs/2310.11028v1","url_pdf":"https://arxiv.org/pdf/2310.11028v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"matrix-compression-via-randomized-low-rank","repo_url":"https://github.com/pilancilab/matrix-compressor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-compression","task_name":"Image Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.11028","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.11028"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pilancilab/matrix-compressor","reach":{"status":"ok"}}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"6547238fd017af95","entry":"batch_cutmix","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6547238fd017af95"}},{"code_sha256_prefix":"80750b7d0796de5a","entry":"batch_normalize_images","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80750b7d0796de5a"}},{"code_sha256_prefix":"246fd19433ca457a","entry":"extract_embeddings","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/mnet/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/mnet/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"246fd19433ca457a"}},{"code_sha256_prefix":"d7186cdef9b0b0a5","entry":"get_patches","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/nn/builder.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/nn/builder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7186cdef9b0b0a5"}},{"code_sha256_prefix":"1dd822fea50fd772","entry":"get_whitening_parameters","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/nn/builder.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/nn/builder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1dd822fea50fd772"}},{"code_sha256_prefix":"5d8137fb7e543a54","entry":"init_whitening_conv","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/nn/builder.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/nn/builder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d8137fb7e543a54"}},{"code_sha256_prefix":"b82ee0719ccaa6a5","entry":"make_random_square_masks","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/hlb/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/hlb/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b82ee0719ccaa6a5"}},{"code_sha256_prefix":"6b7a27a3ef3f576d","entry":"maximum_output_rank","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/lplr/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/lplr/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b7a27a3ef3f576d"}},{"code_sha256_prefix":"a81fb3527c31a385","entry":"normalize_and_shift_wrt_inner_prod","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/lplr/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/lplr/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a81fb3527c31a385"}},{"code_sha256_prefix":"bab310a37b20a439","entry":"relative_tensor_error","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/shared/error.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/shared/error.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bab310a37b20a439"}},{"code_sha256_prefix":"70c7d56564a6351e","entry":"sparse_jl_transform","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/lplr/utils.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/lplr/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70c7d56564a6351e"}},{"code_sha256_prefix":"4e097d543e13c385","entry":"uncompressed","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/lplr/compressors.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/lplr/compressors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e097d543e13c385"}},{"code_sha256_prefix":"3fcac282a5058eac","entry":"load_mobilenet_without_head","repo":"pilancilab/matrix-compressor","repo_kind":"official","path":"src/mnet/loader.py","file_url":"https://github.com/pilancilab/matrix-compressor/blob/HEAD/src/mnet/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3fcac282a5058eac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}