{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/memory-efficient-stochastic-methods-for","title":"Memory-efficient Stochastic methods for Memory-based Transformers","arxiv_id":"2311.08123","date":"2023-11-14","proceeding":null,"authors":["Vishwajit Kumar Vishnu","C. Chandra Sekhar"],"abstract":"Training Memory-based transformers can require a large amount of memory and can be quite inefficient. We propose a novel two-phase training mechanism and a novel regularization technique to improve the training efficiency of memory-based transformers, which are often used for long-range context problems. For our experiments, we consider transformer-XL as our baseline model which is one of memorybased transformer models. We show that our resultant model, Skip Cross-head TransformerXL, outperforms the baseline on character level language modeling task with similar parameters and outperforms the baseline on word level language modelling task with almost 20% fewer parameters. Our proposed methods do not require any additional memory. We also demonstrate the effectiveness of our regularization mechanism on BERT which shows similar performance with reduction in standard deviation of scores of around 30% on multiple GLUE tasks.","url_abs":"https://arxiv.org/abs/2311.08123v1","url_pdf":"https://arxiv.org/pdf/2311.08123v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memory-efficient-stochastic-methods-for","repo_url":"https://github.com/vishwajit-vishnu/memory-efficient-stochastic-methods-for-memory-based-transformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"paraphrase-identification","task_name":"Paraphrase Identification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adaptive-input-representations","method_name":"Adaptive Input Representations"},{"method_slug":"adaptive-softmax","method_name":"Adaptive Softmax"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer-xl","method_name":"Transformer-XL"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Skip Cross-Head Transformer-XL","rank_in_archive_order":49,"of":89,"metrics":{"Number of params":"122M","Test perplexity":"22.91","Validation perplexity":"21.87"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"Skip Cross-Head Transformer-XL","rank_in_archive_order":24,"of":42,"metrics":{"Bit per Character (BPC)":"1.033","Number of params":"41M"},"uses_additional_data":false},{"leaderboard":"/sota/paraphrase-identification-on-quora-question-1","task":"Paraphrase Identification","dataset":"Quora Question Pairs Dev","model":"BERT + SCH attm","rank_in_archive_order":1,"of":2,"metrics":{"Val Accuracy":"91.422"},"uses_additional_data":false},{"leaderboard":"/sota/paraphrase-identification-on-quora-question-1","task":"Paraphrase Identification","dataset":"Quora Question Pairs Dev","model":"BERT + SCH attn","rank_in_archive_order":2,"of":2,"metrics":{"Val F1 Score":"88.436"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}