{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/differentially-private-adaptive-optimization","title":"Differentially Private Adaptive Optimization with Delayed Preconditioners","arxiv_id":"2212.00309","date":"2022-12-01","proceeding":null,"authors":["Tian Li","Manzil Zaheer","Ken Ziyu Liu","Sashank J. Reddi","H. Brendan McMahan","Virginia Smith"],"abstract":"Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training. Prior works typically address this issue by using auxiliary information (e.g., public data) to boost the effectiveness of adaptive optimization. In this work, we explore techniques to estimate and efficiently adapt to gradient geometry in private adaptive optimization without auxiliary data. Motivated by the observation that adaptive methods can tolerate stale preconditioners, we propose differentially private adaptive training with delayed preconditioners (DP^2), a simple method that constructs delayed but less noisy preconditioners to better realize the benefits of adaptivity. Theoretically, we provide convergence guarantees for our method for both convex and non-convex problems, and analyze trade-offs between delay and privacy noise reduction. Empirically, we explore DP^2 across several real-world datasets, demonstrating that it can improve convergence speed by as much as 4x relative to non-adaptive baselines and match the performance of state-of-the-art optimization methods that require auxiliary data.","url_abs":"https://arxiv.org/abs/2212.00309v2","url_pdf":"https://arxiv.org/pdf/2212.00309v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"differentially-private-adaptive-optimization","repo_url":"https://github.com/kenziyuliu/dp2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.00309","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.00309"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/kenziyuliu/DP2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kenziyuliu/dp2","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":17,"ran_fixture":2,"ran_honours":4,"unverified":8},"by_repo_kind":{"official":{"samples":31,"ran":23,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57022752f9995f59","entry":"_config_tf_loader","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57022752f9995f59"}},{"code_sha256_prefix":"5d8c2870379d3384","entry":"_create_tf_loader","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d8c2870379d3384"}},{"code_sha256_prefix":"09495c898e5d8e94","entry":"_load_movielens_100k","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09495c898e5d8e94"}},{"code_sha256_prefix":"b3407cf1906789ea","entry":"bce_loss","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3407cf1906789ea"}},{"code_sha256_prefix":"735e56344d7303e1","entry":"global_l2_clip","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"735e56344d7303e1"}},{"code_sha256_prefix":"bbd8d1efcf5829de","entry":"global_l2_norm_sq","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbd8d1efcf5829de"}},{"code_sha256_prefix":"26f3490ca9bbe47e","entry":"logreg_classify","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26f3490ca9bbe47e"}},{"code_sha256_prefix":"c3b75484cf237418","entry":"movielens_loader","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3b75484cf237418"}},{"code_sha256_prefix":"f320dea1d5541ada","entry":"mse_loss","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f320dea1d5541ada"}},{"code_sha256_prefix":"af4f5b33931d44c0","entry":"num_params","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af4f5b33931d44c0"}},{"code_sha256_prefix":"591c9ea59777949b","entry":"privatize_grad","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"591c9ea59777949b"}},{"code_sha256_prefix":"19770acf0fa0aa44","entry":"stackoverflow_tag_loader","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19770acf0fa0aa44"}},{"code_sha256_prefix":"919aea51fdf26b6d","entry":"stackoverflow_tag_loader_cached","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"919aea51fdf26b6d"}},{"code_sha256_prefix":"cca3cf0b80500ec6","entry":"struct_add","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cca3cf0b80500ec6"}},{"code_sha256_prefix":"34bf4ad4230cf55d","entry":"struct_add_scalar","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34bf4ad4230cf55d"}},{"code_sha256_prefix":"7d340e00912389f7","entry":"struct_div","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d340e00912389f7"}},{"code_sha256_prefix":"e488dedf8e381c96","entry":"struct_div_scalar","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e488dedf8e381c96"}},{"code_sha256_prefix":"155ccee594f8679d","entry":"struct_flatten","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"155ccee594f8679d"}},{"code_sha256_prefix":"6050b70841d335c1","entry":"struct_mul_scalar","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6050b70841d335c1"}},{"code_sha256_prefix":"145eda8014bfd82a","entry":"struct_sq","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"145eda8014bfd82a"}},{"code_sha256_prefix":"ebb3869ec1266c1e","entry":"struct_sqrt","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebb3869ec1266c1e"}},{"code_sha256_prefix":"0bf286afd514640c","entry":"struct_sub","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bf286afd514640c"}},{"code_sha256_prefix":"06cd2aedfcb73bb6","entry":"struct_zeros_like","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06cd2aedfcb73bb6"}},{"code_sha256_prefix":"dca6afaa2e23a3be","entry":"BaseTrainer","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dca6afaa2e23a3be"}},{"code_sha256_prefix":"458f345611469a88","entry":"Trainer","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"458f345611469a88"}},{"code_sha256_prefix":"f4a878bc9479f4bc","entry":"imdb_loader","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4a878bc9479f4bc"}},{"code_sha256_prefix":"29b863abc869ce7a","entry":"linear_model_fn","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29b863abc869ce7a"}},{"code_sha256_prefix":"32b7bb707740780e","entry":"matfac_model_fn","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32b7bb707740780e"}},{"code_sha256_prefix":"aeca8b544abd9191","entry":"multiclass_classify","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aeca8b544abd9191"}},{"code_sha256_prefix":"defe6270f77abbbd","entry":"print_log","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"defe6270f77abbbd"}},{"code_sha256_prefix":"1d3f9fc33b06bb7f","entry":"sce_loss","repo":"kenziyuliu/dp2","repo_kind":"official","path":"trainers/dp2_adagrad.py","file_url":"https://github.com/kenziyuliu/dp2/blob/HEAD/trainers/dp2_adagrad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d3f9fc33b06bb7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}