{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-26389","title":"LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression","arxiv_id":"2608.26389","date":"2026-08-26","proceeding":null,"authors":["Zishan Shao","Lixun Zhang","Kangning Cui","Wenhao Wu","Jinhee Kim","Yixiao Wang","Ting Jiang","Hancheng Ye","Qinsi Wang","Fan Yang","Danyang Zhuo","Yiran Chen","Hai Li"],"abstract":"SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.","url_abs":"https://arxiv.org/abs/2608.26389","url_pdf":"https://arxiv.org/pdf/2608.26389","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.26389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Zishan-Shao/lowrankarena","reach":null}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e797677814828841","entry":"bytes_to_gib","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/memory_runner.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/memory_runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e797677814828841"}},{"code_sha256_prefix":"575a3bbe6e2420cf","entry":"choose_task_metric","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/scoring.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"575a3bbe6e2420cf"}},{"code_sha256_prefix":"a1586acd6e287b0f","entry":"collect_tracked_task_metrics","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/scoring.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1586acd6e287b0f"}},{"code_sha256_prefix":"7f22d86655d615ff","entry":"flatten_result","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/report.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/report.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f22d86655d615ff"}},{"code_sha256_prefix":"015b8f97d6c3e9f2","entry":"iter_numeric_task_metrics","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/scoring.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"015b8f97d6c3e9f2"}},{"code_sha256_prefix":"424f24bf67d2470e","entry":"tensor_bytes","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/memory_runner.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/memory_runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424f24bf67d2470e"}},{"code_sha256_prefix":"e56bc9d9fe612aa4","entry":"resolve_dtype","repo":"Zishan-Shao/lowrankarena","repo_kind":"found_in_text","path":"src/memory_runner.py","file_url":"https://github.com/Zishan-Shao/lowrankarena/blob/HEAD/src/memory_runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e56bc9d9fe612aa4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}