{"url":"/method/nt-asgd","slug":"nt-asgd","name":"NT-ASGD","full_name":"Non-monotonically Triggered ASGD","full_name_withheld":false,"description_markdown":"**NT-ASGD**, or **Non-monotonically Triggered ASGD**, is an averaged stochastic gradient descent technique. \r\n\r\nIn regular ASGD, we take steps identical to [regular SGD](https://paperswithcode.com/method/sgd) but instead of returning the last iterate as the solution, we return $\\frac{1}{\\left(K-T+1\\right)}\\sum^{T}\\_{i=T}w\\_{i}$, where $K$ is the total number of iterations and $T < K$ is a user-specified averaging trigger.\r\n\r\nNT-ASGD has a non-monotonic criterion that conservatively triggers the averaging when the validation metric fails to improve for multiple cycles. Given that the choice of triggering is irreversible, this conservatism ensures that the randomness of training does not play a major role in the decision.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Regularizing and Optimizing LSTM Language Models","paper":"/paper/regularizing-and-optimizing-lstm-language","first_author":"Stephen Merity","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/regularizing-and-optimizing-lstm-language"},"source":{"url":"http://arxiv.org/abs/1708.02182v1","title":"Regularizing and Optimizing LSTM Language Models","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/salesforce/awd-lstm-lm/blob/32fcb42562aeb5c7e6c9dec3f2a3baaaf68a5cb5/main.py#L275","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":6,"archive_num_papers":6,"papers_newest_first":[{"paper":null,"title":"Probing for Referential Information in Language Models","date":"2020-07-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/maxup-a-simple-way-to-improve-generalization","title":"MaxUp: A Simple Way to Improve Generalization of Neural Network Training","date":"2020-02-20","arxiv_id":"2002.09024","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":"/paper/define-deep-factorized-input-word-embeddings-1","title":"DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling","date":"2019-11-27","arxiv_id":"1911.12385","n_code_links":1,"syntology":null},{"paper":null,"title":"A Subword Level Language Model for Bangla Language","date":"2019-11-15","arxiv_id":"1911.07613","n_code_links":0,"syntology":null},{"paper":null,"title":"Language Informed Modeling of Code-Switched Text","date":"2018-07-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/regularizing-and-optimizing-lstm-language","title":"Regularizing and Optimizing LSTM Language Models","date":"2017-08-07","arxiv_id":"1708.02182","n_code_links":45,"syntology":{"ran":6,"of":7,"unverified":1,"pointer_only":7}}],"papers_shown":6,"tasks":[{"task":"/task/language-modeling","name":"Language Modeling","papers":4},{"task":"/task/language-modelling","name":"Language Modelling","papers":4},{"task":"/task/translation","name":"Translation","papers":3},{"task":"/task/image-classification","name":"Image Classification","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/articles","name":"Articles","papers":1},{"task":"/task/few-shot-image-classification","name":"Few-Shot Image Classification","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":1},{"task":"/task/text-classification","name":"Text Classification","papers":1},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1},{"task":"/task/model","name":"model","papers":1},{"task":"/task/text-classification-1","name":"text-classification","papers":1}],"tasks_shown":15,"n_tasks":15,"usage_by_year":[{"year":"2017","papers":1},{"year":"2018","papers":1},{"year":"2019","papers":2},{"year":"2020","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/nt-asgd"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}