{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mast-model-agnostic-sparsified-training","title":"MAST: Model-Agnostic Sparsified Training","arxiv_id":"2311.16086","date":"2023-11-27","proceeding":null,"authors":["Yury Demidovich","Grigory Malinovsky","Egor Shulgin","Peter Richtárik"],"abstract":"We introduce a novel optimization problem formulation that departs from the conventional way of minimizing machine learning model loss as a black-box function. Unlike traditional formulations, the proposed approach explicitly incorporates an initially pre-trained model and random sketch operators, allowing for sparsification of both the model and gradient during training. We establish insightful properties of the proposed objective function and highlight its connections to the standard formulation. Furthermore, we present several variants of the Stochastic Gradient Descent (SGD) method adapted to the new problem formulation, including SGD with general sampling, a distributed version, and SGD with variance reduction techniques. We achieve tighter convergence rates and relax assumptions, bridging the gap between theoretical principles and practical applications, covering several important techniques such as Dropout and Sparse training. This work presents promising opportunities to enhance the theoretical understanding of model training through a sparsification-aware optimization approach.","url_abs":"https://arxiv.org/abs/2311.16086v1","url_pdf":"https://arxiv.org/pdf/2311.16086v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mast-model-agnostic-sparsified-training","repo_url":"https://github.com/konstmish/opt_methods","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.16086","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.16086"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/konstmish/opt_methods","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fccd6875c293069d","entry":"relative_round","repo":"konstmish/opt_methods","repo_kind":"official","path":"optmethods/utils.py","file_url":"https://github.com/konstmish/opt_methods/blob/HEAD/optmethods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fccd6875c293069d"}},{"code_sha256_prefix":"5cd5ec08dd0ca1a0","entry":"get_dataset","repo":"konstmish/opt_methods","repo_kind":"official","path":"optmethods/datasets/utils.py","file_url":"https://github.com/konstmish/opt_methods/blob/HEAD/optmethods/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cd5ec08dd0ca1a0"}},{"code_sha256_prefix":"45ea328b237ba674","entry":"get_trace","repo":"konstmish/opt_methods","repo_kind":"official","path":"optmethods/utils.py","file_url":"https://github.com/konstmish/opt_methods/blob/HEAD/optmethods/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45ea328b237ba674"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}