{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/maestro-uncovering-low-rank-structures-via","title":"Maestro: Uncovering Low-Rank Structures via Trainable Decomposition","arxiv_id":"2308.14929","date":"2023-08-28","proceeding":null,"authors":["Samuel Horvath","Stefanos Laskaridis","Shashank Rajput","Hongyi Wang"],"abstract":"Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming and typically yields a single model to fit all targets. Various techniques have been proposed in the literature to mitigate this, including pruning, sparsification, or quantization of model weights and updates. While achieving high compression rates, they often incur significant computational overheads at training or lead to non-negligible accuracy penalty. Alternatively, factorization methods have been leveraged for low-rank compression of DNNs. Similarly, such techniques (e.g., SVD) frequently rely on heavy iterative decompositions of layers and are potentially sub-optimal for non-linear models, such as DNNs. We take a further step in designing efficient low-rank models and propose Maestro, a framework for trainable low-rank layers. Instead of iteratively applying a priori decompositions, the low-rank structure is baked into the training process through LoD, a low-rank ordered decomposition. Not only is this the first time importance ordering via sampling is applied on the decomposed DNN structure, but it also allows selecting ranks at a layer granularity. Our theoretical analysis demonstrates that in special cases LoD recovers the SVD decomposition and PCA. Applied to DNNs, Maestro enables the extraction of lower footprint models that preserve performance. Simultaneously, it enables the graceful trade-off between accuracy-latency for deployment to even more constrained devices without retraining.","url_abs":"https://arxiv.org/abs/2308.14929v2","url_pdf":"https://arxiv.org/pdf/2308.14929v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"maestro-uncovering-low-rank-structures-via","repo_url":"https://github.com/samuelhorvath/maestro-lod","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"low-rank-compression","task_name":"Low-rank compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"pca","method_name":"PCA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.14929","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.14929"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SamuelHorvath/Maestro-LoD","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/samuelhorvath/maestro-lod","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_violates":2,"ran_honours":2,"ran_draft_wrong":3,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":18,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"838b3ced47c739f8","entry":"MaestroConv2d","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"maestro/layers/conv.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/maestro/layers/conv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"838b3ced47c739f8"}},{"code_sha256_prefix":"396d2b49a5391613","entry":"bool_string","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro_opts.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro_opts.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"396d2b49a5391613"}},{"code_sha256_prefix":"a80d13a84b21f1cf","entry":"check_layer","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"maestro/layers/conv.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/maestro/layers/conv.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a80d13a84b21f1cf"}},{"code_sha256_prefix":"18a414ece7a6c3c2","entry":"create_conv2d_layer","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/models/lenet.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/models/lenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18a414ece7a6c3c2"}},{"code_sha256_prefix":"33eb693ad533396c","entry":"create_linear_layer","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/models/utils.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33eb693ad533396c"}},{"code_sha256_prefix":"31759f14336427d4","entry":"create_norm_layer","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"cifar_mnist_main.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/cifar_mnist_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31759f14336427d4"}},{"code_sha256_prefix":"11a9508cc4b6b3fa","entry":"decompose_conv2d","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/layers/conv.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/layers/conv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11a9508cc4b6b3fa"}},{"code_sha256_prefix":"db892158fa6646a4","entry":"decompose_linear","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"maestro/layers/linear.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/maestro/layers/linear.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db892158fa6646a4"}},{"code_sha256_prefix":"db6320dbc007c7bd","entry":"do_not_decompose_linear","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"maestro/layers/linear.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/maestro/layers/linear.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db6320dbc007c7bd"}},{"code_sha256_prefix":"54c0134104fb7718","entry":"get_submodel_and_name","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/layers/decomposition.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/layers/decomposition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54c0134104fb7718"}},{"code_sha256_prefix":"88bb5d3e4db2e20f","entry":"group_lasso_criterion","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/layers/utils.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/layers/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88bb5d3e4db2e20f"}},{"code_sha256_prefix":"fb70500cc298c8d1","entry":"initialise_model","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"cifar_mnist_main.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/cifar_mnist_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb70500cc298c8d1"}},{"code_sha256_prefix":"c4f3e05b55394eae","entry":"maestro_to_full_linear","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"maestro/layers/linear.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/maestro/layers/linear.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c4f3e05b55394eae"}},{"code_sha256_prefix":"4998b554a5d3b133","entry":"parse_maestro_opts","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro_opts.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro_opts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4998b554a5d3b133"}},{"code_sha256_prefix":"c63928dac29e7a1d","entry":"validate","repo":"samuelhorvath/maestro-lod","repo_kind":"official","path":"imagenet_main.py","file_url":"https://github.com/samuelhorvath/maestro-lod/blob/HEAD/imagenet_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c63928dac29e7a1d"}},{"code_sha256_prefix":"752dd8c48fb5d47b","entry":"create_linear_layer","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/models/lenet.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/models/lenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"752dd8c48fb5d47b"}},{"code_sha256_prefix":"a02b80587a98bf46","entry":"do_not_decompose_conv2d","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/layers/conv.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/layers/conv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a02b80587a98bf46"}},{"code_sha256_prefix":"991b739d35882910","entry":"maestro_to_full_conv2d","repo":"SamuelHorvath/Maestro-LoD","repo_kind":"official","path":"maestro/layers/conv.py","file_url":"https://github.com/SamuelHorvath/Maestro-LoD/blob/HEAD/maestro/layers/conv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"991b739d35882910"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}