{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-private-learning-via-low-rank","title":"Large Scale Private Learning via Low-rank Reparametrization","arxiv_id":"2106.09352","date":"2021-06-17","proceeding":null,"authors":["Da Yu","Huishuai Zhang","Wei Chen","Jian Yin","Tie-Yan Liu"],"abstract":"We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks, which are 1) the huge memory cost of storing individual gradients, 2) the added noise suffering notorious dimensional dependence. Specifically, we reparametrize each weight matrix with two \\emph{gradient-carrier} matrices of small dimension and a \\emph{residual weight} matrix. We argue that such reparametrization keeps the forward/backward process unchanged while enabling us to compute the projected gradient without computing the gradient itself. To learn with differential privacy, we design \\emph{reparametrized gradient perturbation (RGP)} that perturbs the gradients on gradient-carrier matrices and reconstructs an update for the original weight from the noisy gradients. Importantly, we use historical updates to find the gradient-carrier matrices, whose optimality is rigorously justified under linear regression and empirically verified with deep learning tasks. RGP significantly reduces the memory cost and improves the utility. For example, we are the first able to apply differential privacy on the BERT model and achieve an average accuracy of $83.9\\%$ on four downstream tasks with $\\epsilon=8$, which is within $5\\%$ loss compared to the non-private baseline but enjoys much lower privacy leakage risk.","url_abs":"https://arxiv.org/abs/2106.09352v4","url_pdf":"https://arxiv.org/pdf/2106.09352v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-scale-private-learning-via-low-rank","repo_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-regression","method_name":"Linear Regression"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.09352","atlas_url":"https://app.syntology.ai/?focus=2106.09352","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.09352"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dayu11/Differentially-Private-Deep-Learning","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ca59b90ba970fcda","entry":"LrkConv2d","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca59b90ba970fcda"}},{"code_sha256_prefix":"d075692ad1036499","entry":"conv1x1","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d075692ad1036499"}},{"code_sha256_prefix":"6e6cd72f51ebc400","entry":"conv3x3","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e6cd72f51ebc400"}},{"code_sha256_prefix":"b6b42d33ba802e1c","entry":"weight_decomposition","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6b42d33ba802e1c"}},{"code_sha256_prefix":"f99fafb1402a6e8c","entry":"ConvBackwardHook","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f99fafb1402a6e8c"}},{"code_sha256_prefix":"44f4381827c7be33","entry":"ConvFowardHook","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44f4381827c7be33"}},{"code_sha256_prefix":"ef5092f0e5bd3563","entry":"_compute_conv_grad_sample","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef5092f0e5bd3563"}},{"code_sha256_prefix":"18baa6e835949a4e","entry":"orthogonalize","repo":"dayu11/Differentially-Private-Deep-Learning","repo_kind":"official","path":"vision/RGP/models/resnet_cifar.py","file_url":"https://github.com/dayu11/Differentially-Private-Deep-Learning/blob/HEAD/vision/RGP/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18baa6e835949a4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}