{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alexatm-20b-few-shot-learning-using-a-large","title":"AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model","arxiv_id":"2208.01448","date":"2022-08-02","proceeding":null,"authors":["Saleh Soltan","Shankar Ananthakrishnan","Jack FitzGerald","Rahul Gupta","Wael Hamza","Haidar Khan","Charith Peris","Stephen Rawls","Andy Rosenbaum","Anna Rumshisky","Chandana Satya Prakash","Mukund Sridhar","Fabian Triefenbach","Apurv Verma","Gokhan Tur","Prem Natarajan"],"abstract":"In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.","url_abs":"https://arxiv.org/abs/2208.01448v2","url_pdf":"https://arxiv.org/pdf/2208.01448v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alexatm-20b-few-shot-learning-using-a-large","repo_url":"https://github.com/amazon-science/alexa-teacher-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"Causal Language Modeling"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"palm","method_name":"PaLM"},{"method_slug":"seq2seq","method_name":"Seq2Seq"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"AlexaTM 20B","rank_in_archive_order":41,"of":45,"metrics":{"F1":"88.4"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"AlexaTM 20B","rank_in_archive_order":38,"of":82,"metrics":{"Accuracy":"68.3"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"AlexaTM 20B","rank_in_archive_order":15,"of":20,"metrics":{"Accuracy":"67.9"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"AlexaTM 20B","rank_in_archive_order":60,"of":90,"metrics":{"Accuracy":"68.6%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"AlexaTM 20B","rank_in_archive_order":43,"of":65,"metrics":{"Accuracy":"69.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"AlexaTM 20B","rank_in_archive_order":40,"of":60,"metrics":{"Accuracy":"78.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"AlexaTM 20B","rank_in_archive_order":21,"of":30,"metrics":{"F1":"59.6"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"AlexaTM 20B","rank_in_archive_order":23,"of":37,"metrics":{"Accuracy":"53.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.01448","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.01448"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/alexa-teacher-models","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a315b8fc32a18efe","entry":"shift_tokens_right","repo":"amazon-science/alexa-teacher-models","repo_kind":"listed","path":"alexa_teacher_models/modeling_atm.py","file_url":"https://github.com/amazon-science/alexa-teacher-models/blob/HEAD/alexa_teacher_models/modeling_atm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a315b8fc32a18efe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}