{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/palm-2-technical-report-1","title":"PaLM 2 Technical Report","arxiv_id":"2305.10403","date":"2023-05-17","proceeding":null,"authors":["Rohan Anil","Andrew M. Dai","Orhan Firat","Melvin Johnson","Dmitry Lepikhin","Alexandre Passos","Siamak Shakeri","Emanuel Taropa","Paige Bailey","Zhifeng Chen","Eric Chu","Jonathan H. Clark","Laurent El Shafey","Yanping Huang","Kathy Meier-Hellstern","Gaurav Mishra","Erica Moreira","Mark Omernick","Kevin Robinson","Sebastian Ruder","Yi Tay","Kefan Xiao","Yuanzhong Xu","Yujing Zhang","Gustavo Hernandez Abrego","Junwhan Ahn","Jacob Austin","Paul Barham","Jan Botha","James Bradbury","Siddhartha Brahma","Kevin Brooks","Michele Catasta","Yong Cheng","Colin Cherry","Christopher A. Choquette-Choo","Aakanksha Chowdhery","Clément Crepy","Shachi Dave","Mostafa Dehghani","Sunipa Dev","Jacob Devlin","Mark Díaz","Nan Du","Ethan Dyer","Vlad Feinberg","Fangxiaoyu Feng","Vlad Fienber","Markus Freitag","Xavier Garcia","Sebastian Gehrmann","Lucas Gonzalez","Guy Gur-Ari","Steven Hand","Hadi Hashemi","Le Hou","Joshua Howland","Andrea Hu","Jeffrey Hui","Jeremy Hurwitz","Michael Isard","Abe Ittycheriah","Matthew Jagielski","Wenhao Jia","Kathleen Kenealy","Maxim Krikun","Sneha Kudugunta","Chang Lan","Katherine Lee","Benjamin Lee","Eric Li","Music Li","Wei Li","Yaguang Li","Jian Li","Hyeontaek Lim","Hanzhao Lin","Zhongtao Liu","Frederick Liu","Marcello Maggioni","Aroma Mahendru","Joshua Maynez","Vedant Misra","Maysam Moussalem","Zachary Nado","John Nham","Eric Ni","Andrew Nystrom","Alicia Parrish","Marie Pellat","Martin Polacek","Alex Polozov","Reiner Pope","Siyuan Qiao","Emily Reif","Bryan Richter","Parker Riley","Alex Castro Ros","Aurko Roy","Brennan Saeta","Rajkumar Samuel","Renee Shelby","Ambrose Slone","Daniel Smilkov","David R. So","Daniel Sohn","Simon Tokumine","Dasha Valter","Vijay Vasudevan","Kiran Vodrahalli","Xuezhi Wang","Pidong Wang","ZiRui Wang","Tao Wang","John Wieting","Yuhuai Wu","Kelvin Xu","Yunhan Xu","Linting Xue","Pengcheng Yin","Jiahui Yu","Qiao Zhang","Steven Zheng","Ce Zheng","Weikang Zhou","Denny Zhou","Slav Petrov","Yonghui Wu"],"abstract":"We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.","url_abs":"https://arxiv.org/abs/2305.10403v3","url_pdf":"https://arxiv.org/pdf/2305.10403v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"palm-2-technical-report-1","repo_url":"https://github.com/eternityyw/tram-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"}],"methods":[{"method_slug":"palm","method_name":"PaLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"PaLM 2 (few-shot, k=8, SC)","rank_in_archive_order":18,"of":164,"metrics":{"Accuracy":"91.0"},"uses_additional_data":false},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"PaLM 2 (few-shot, k=8, CoT)","rank_in_archive_order":71,"of":164,"metrics":{"Accuracy":"80.7"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"PaLM 2-S* (few-shot)","rank_in_archive_order":59,"of":99,"metrics":{"Accuracy":"50"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"PaLM 2 (few-shot, CoT, SC)","rank_in_archive_order":2,"of":54,"metrics":{"Accuracy":"95.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"PaLM 2-L (1-shot)","rank_in_archive_order":19,"of":54,"metrics":{"Accuracy":"69.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"PaLM 2-M (1-shot)","rank_in_archive_order":22,"of":54,"metrics":{"Accuracy":"64.9"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"PaLM 2-S (1-shot)","rank_in_archive_order":25,"of":54,"metrics":{"Accuracy":"59.6"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"PaLM 2-L (1-shot)","rank_in_archive_order":3,"of":47,"metrics":{"Accuracy":"89.7"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"PaLM 2-M (1-shot)","rank_in_archive_order":4,"of":47,"metrics":{"Accuracy":"88.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"PaLM 2-S (1-shot)","rank_in_archive_order":7,"of":47,"metrics":{"Accuracy":"85.6"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-causal","task":"Common Sense Reasoning","dataset":"BIG-bench (Causal Judgment)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"62.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-causal","task":"Common Sense Reasoning","dataset":"BIG-bench (Causal Judgment)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":3,"of":9,"metrics":{"Accuracy":"58.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-date","task":"Common Sense Reasoning","dataset":"BIG-bench (Date Understanding)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"91.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-date","task":"Common Sense Reasoning","dataset":"BIG-bench (Date Understanding)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"74.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench","task":"Common Sense Reasoning","dataset":"BIG-bench (Disambiguation QA)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"78.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench","task":"Common Sense Reasoning","dataset":"BIG-bench (Disambiguation QA)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"77.6"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-sports","task":"Common Sense Reasoning","dataset":"BIG-bench (Sports Understanding)","model":"PaLM 2(few-shot, k=3, CoT)","rank_in_archive_order":1,"of":8,"metrics":{"Accuracy":"98"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-sports","task":"Common Sense Reasoning","dataset":"BIG-bench (Sports Understanding)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":8,"metrics":{"Accuracy":"90.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-commonsenseqa","task":"Common Sense Reasoning","dataset":"CommonsenseQA","model":"PaLM 2 (few‑shot, CoT, SC)","rank_in_archive_order":3,"of":38,"metrics":{"Accuracy":"90.4"},"uses_additional_data":true},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"PaLM 2-L (one-shot)","rank_in_archive_order":37,"of":45,"metrics":{"F1":"93.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"PaLM 2-M (one-shot)","rank_in_archive_order":38,"of":45,"metrics":{"F1":"92.4"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"PaLM 2-S (one-shot)","rank_in_archive_order":39,"of":45,"metrics":{"F1":"92.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM 2-L (1-shot)","rank_in_archive_order":11,"of":77,"metrics":{"Accuracy":"83.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM 2-M (1-shot)","rank_in_archive_order":17,"of":77,"metrics":{"Accuracy":"79.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM 2-S (1-shot)","rank_in_archive_order":19,"of":77,"metrics":{"Accuracy":"77.9"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 2-M (1-shot)","rank_in_archive_order":13,"of":82,"metrics":{"Accuracy":"88.1"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 2-L (1-shot)","rank_in_archive_order":14,"of":82,"metrics":{"Accuracy":"86.9"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 2-S (1-shot)","rank_in_archive_order":18,"of":82,"metrics":{"Accuracy":"84.6"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"PaLM 2-L (one-shot)","rank_in_archive_order":9,"of":11,"metrics":{"F1":"73.6"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"PaLM 2-S (one-shot)","rank_in_archive_order":10,"of":11,"metrics":{"F1":"73.3"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"PaLM 2-M (one-shot)","rank_in_archive_order":11,"of":11,"metrics":{"F1":"73.3"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-transfer-on-xcopa","task":"Cross-Lingual Transfer","dataset":"XCOPA","model":"PaLM 2 (few-shot)","rank_in_archive_order":1,"of":6,"metrics":{"Accuracy":"94.4"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM 2-L (one-shot)","rank_in_archive_order":2,"of":37,"metrics":{"Accuracy":"86.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM 2-M (one-shot)","rank_in_archive_order":6,"of":37,"metrics":{"Accuracy":"83.7"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM 2-S (one-shot)","rank_in_archive_order":11,"of":37,"metrics":{"Accuracy":"80.7"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-formal","task":"Logical Reasoning","dataset":"BIG-bench (Formal Fallacies Syllogisms Negation)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"64.8"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-formal","task":"Logical Reasoning","dataset":"BIG-bench (Formal Fallacies Syllogisms Negation)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"57.2"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-logic-grid","task":"Logical Reasoning","dataset":"BIG-bench (Logic Grid Puzzle)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"42.4"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-logic-grid","task":"Logical Reasoning","dataset":"BIG-bench (Logic Grid Puzzle)","model":"PaLM-62B (few-shot, k=5)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"36.5"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-penguins-in-a","task":"Logical Reasoning","dataset":"BIG-bench (Penguins In A Table)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"84.9"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-penguins-in-a","task":"Logical Reasoning","dataset":"BIG-bench (Penguins In A Table)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"65.8"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-reasoning","task":"Logical Reasoning","dataset":"BIG-bench (Reasoning About Colored Objects)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"91.2"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-reasoning","task":"Logical Reasoning","dataset":"BIG-bench (Reasoning About Colored Objects)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"61.2"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-temporal","task":"Logical Reasoning","dataset":"BIG-bench (Temporal Sequences)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"100"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-temporal","task":"Logical Reasoning","dataset":"BIG-bench (Temporal Sequences)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"96.4"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-mainland","task":"Machine Translation","dataset":"FRMT (Chinese - Mainland)","model":"PaLM 2","rank_in_archive_order":1,"of":3,"metrics":{"BLEURT":"74.4"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-mainland","task":"Machine Translation","dataset":"FRMT (Chinese - Mainland)","model":"Google Translate","rank_in_archive_order":2,"of":3,"metrics":{"BLEURT":"72.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-mainland","task":"Machine Translation","dataset":"FRMT (Chinese - Mainland)","model":"PaLM","rank_in_archive_order":3,"of":3,"metrics":{"BLEURT":"70.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-taiwan","task":"Machine Translation","dataset":"FRMT (Chinese - Taiwan)","model":"PaLM 2","rank_in_archive_order":1,"of":3,"metrics":{"BLEURT":"72.0"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-taiwan","task":"Machine Translation","dataset":"FRMT (Chinese - Taiwan)","model":"PaLM","rank_in_archive_order":2,"of":3,"metrics":{"BLEURT":"68.6"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-chinese-taiwan","task":"Machine Translation","dataset":"FRMT (Chinese - Taiwan)","model":"Google Translate","rank_in_archive_order":3,"of":3,"metrics":{"BLEURT":"68.5"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese-brazil","task":"Machine Translation","dataset":"FRMT (Portuguese - Brazil)","model":"PaLM 2","rank_in_archive_order":1,"of":3,"metrics":{"BLEURT":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese-brazil","task":"Machine Translation","dataset":"FRMT (Portuguese - Brazil)","model":"Google Translate","rank_in_archive_order":2,"of":3,"metrics":{"BLEURT":"80.2"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese-brazil","task":"Machine Translation","dataset":"FRMT (Portuguese - Brazil)","model":"PaLM","rank_in_archive_order":3,"of":3,"metrics":{"BLEURT":"78.5"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese","task":"Machine Translation","dataset":"FRMT (Portuguese - Portugal)","model":"PaLM 2","rank_in_archive_order":1,"of":3,"metrics":{"BLEURT":"78.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese","task":"Machine Translation","dataset":"FRMT (Portuguese - Portugal)","model":"PaLM","rank_in_archive_order":2,"of":3,"metrics":{"BLEURT":"76.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-frmt-portuguese","task":"Machine Translation","dataset":"FRMT (Portuguese - Portugal)","model":"Google Translate","rank_in_archive_order":3,"of":3,"metrics":{"BLEURT":"75.3"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"PaLM 2 (few-shot, k=4, SC)","rank_in_archive_order":52,"of":135,"metrics":{"Accuracy":"48.8"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"PaLM 2 (few-shot, k=4, CoT)","rank_in_archive_order":81,"of":135,"metrics":{"Accuracy":"34.3"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mgsm","task":"Multi-task Language Understanding","dataset":"MGSM","model":"PaLM 2 (few-shot, k=8, SC)","rank_in_archive_order":1,"of":12,"metrics":{"Average (%)":"87.0"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mgsm","task":"Multi-task Language Understanding","dataset":"MGSM","model":"PaLM 2 (8-shot, CoT)","rank_in_archive_order":2,"of":12,"metrics":{"Average (%)":"72.2"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-27","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Hyperbaton)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":5,"of":9,"metrics":{"Accuracy":"84.8"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-27","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Hyperbaton)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":6,"of":9,"metrics":{"Accuracy":"82.4"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-28","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Movie Recommendation)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"94.4"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-28","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Movie Recommendation)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"93.6"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-29","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Navigate)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"91.2"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-29","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Navigate)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"68.8"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-30","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Ruin Names)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy":"90"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-30","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Ruin Names)","model":"PaLM 2 (few-shot, k=3, CoT)","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"83.6"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"PaLM 2-L (one-shot)","rank_in_archive_order":4,"of":25,"metrics":{"A1":"73.1","A2":"63.4","A3":"67.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"PaLM 2-M (one-shot)","rank_in_archive_order":9,"of":25,"metrics":{"A1":"58.1","A2":"49.5","A3":"54.5"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"PaLM 2-S (one-shot)","rank_in_archive_order":10,"of":25,"metrics":{"A1":"53.1","A2":"48.8","A3":"53.2"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"PaLM 2-L (one-shot)","rank_in_archive_order":10,"of":20,"metrics":{"Accuracy":"87.5"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"PaLM 2-S (one-shot)","rank_in_archive_order":11,"of":20,"metrics":{"Accuracy":"82.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"PaLM 2-M (one-shot)","rank_in_archive_order":12,"of":20,"metrics":{"Accuracy":"80.4"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 2-M (1-shot)","rank_in_archive_order":33,"of":90,"metrics":{"Accuracy":"81.9%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 2-L (1-shot)","rank_in_archive_order":39,"of":90,"metrics":{"Accuracy":"79.3%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 2-S (1-shot)","rank_in_archive_order":41,"of":90,"metrics":{"Accuracy":"78.7%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"PaLM 2-L (1-shot)","rank_in_archive_order":6,"of":65,"metrics":{"Accuracy":"90.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"PaLM 2-M (1-shot)","rank_in_archive_order":10,"of":65,"metrics":{"Accuracy":"88.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"PaLM 2-S (1-shot)","rank_in_archive_order":12,"of":65,"metrics":{"Accuracy":"88.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"PaLM 2-L (1-shot)","rank_in_archive_order":8,"of":60,"metrics":{"Accuracy":"96.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"PaLM 2-M (1-shot)","rank_in_archive_order":18,"of":60,"metrics":{"Accuracy":"90.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"PaLM 2-S (1-shot)","rank_in_archive_order":20,"of":60,"metrics":{"Accuracy":"89.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-drop-test","task":"Question Answering","dataset":"DROP Test","model":"PaLM 2 (few-shot)","rank_in_archive_order":3,"of":16,"metrics":{"F1":"85.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"PaLM 2-L (one-shot)","rank_in_archive_order":6,"of":30,"metrics":{"F1":"88.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"PaLM 2-M (one-shot)","rank_in_archive_order":9,"of":30,"metrics":{"F1":"84.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"PaLM 2-S (one-shot)","rank_in_archive_order":10,"of":30,"metrics":{"F1":"84.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM 2-L (one-shot)","rank_in_archive_order":27,"of":47,"metrics":{"EM":"37.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM 2-M (one-shot)","rank_in_archive_order":33,"of":47,"metrics":{"EM":"32.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM 2-S (one-shot)","rank_in_archive_order":41,"of":47,"metrics":{"EM":"25.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"PaLM 2-L (1-shot)","rank_in_archive_order":30,"of":45,"metrics":{"Accuracy":"58.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"PaLM 2-S (1-shot)","rank_in_archive_order":32,"of":45,"metrics":{"Accuracy":"57.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"PaLM 2-M (1-shot)","rank_in_archive_order":34,"of":45,"metrics":{"Accuracy":"56.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"PaLM 2-L (1-shot)","rank_in_archive_order":12,"of":67,"metrics":{"Accuracy":"85.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"PaLM 2-M (1-shot)","rank_in_archive_order":14,"of":67,"metrics":{"Accuracy":"83.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"PaLM 2-S (1-shot)","rank_in_archive_order":21,"of":67,"metrics":{"Accuracy":"82.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-story-cloze","task":"Question Answering","dataset":"Story Cloze","model":"PaLM 2-L (one-shot)","rank_in_archive_order":3,"of":7,"metrics":{"Accuracy":"87.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-story-cloze","task":"Question Answering","dataset":"Story Cloze","model":"PaLM 2-M (one-shot)","rank_in_archive_order":4,"of":7,"metrics":{"Accuracy":"86.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-story-cloze","task":"Question Answering","dataset":"Story Cloze","model":"PaLM 2-S (one-shot)","rank_in_archive_order":5,"of":7,"metrics":{"Accuracy":"85.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-strategyqa","task":"Question Answering","dataset":"StrategyQA","model":"PaLM 2 (few-shot, CoT, SC)","rank_in_archive_order":1,"of":12,"metrics":{"Accuracy":"90.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM 2-L (one-shot)","rank_in_archive_order":5,"of":56,"metrics":{"EM":"86.1"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM 2-M (one-shot)","rank_in_archive_order":10,"of":56,"metrics":{"EM":"81.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM 2-S (one-shot)","rank_in_archive_order":22,"of":56,"metrics":{"EM":"75.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM 2-L (one-shot)","rank_in_archive_order":25,"of":37,"metrics":{"EM":"28.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM 2-M (one-shot)","rank_in_archive_order":26,"of":37,"metrics":{"EM":"26.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM 2-S (one-shot)","rank_in_archive_order":31,"of":37,"metrics":{"EM":"21.8"},"uses_additional_data":false},{"leaderboard":"/sota/sarcasm-detection-on-big-bench-snarks","task":"Sarcasm Detection","dataset":"BIG-bench (SNARKS)","model":"PaLM 2(few-shot, k=3, CoT)","rank_in_archive_order":1,"of":8,"metrics":{"Accuracy":"84.8"},"uses_additional_data":false},{"leaderboard":"/sota/sarcasm-detection-on-big-bench-snarks","task":"Sarcasm Detection","dataset":"BIG-bench (SNARKS)","model":"PaLM 2 (few-shot, k=3, Direct)","rank_in_archive_order":2,"of":8,"metrics":{"Accuracy":"78.7"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM 2-L (1-shot)","rank_in_archive_order":16,"of":89,"metrics":{"Accuracy":"87.4"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM 2-M (1-shot)","rank_in_archive_order":18,"of":89,"metrics":{"Accuracy":"86.7"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM 2-S (1-shot)","rank_in_archive_order":22,"of":89,"metrics":{"Accuracy":"85.6"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-x-sum","task":"Text Summarization","dataset":"X-Sum","model":"PaLM 2-L (one-shot)","rank_in_archive_order":16,"of":18,"metrics":{"ROUGE-2":"23.2"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-x-sum","task":"Text Summarization","dataset":"X-Sum","model":"PaLM 2-M (one-shot)","rank_in_archive_order":17,"of":18,"metrics":{"ROUGE-2":"17.2"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-x-sum","task":"Text Summarization","dataset":"X-Sum","model":"PaLM 2-S (one-shot)","rank_in_archive_order":18,"of":18,"metrics":{"ROUGE-2":"16.9"},"uses_additional_data":false},{"leaderboard":"/sota/toxic-comment-classification-on-civil","task":"Toxic Comment Classification","dataset":"Civil Comments","model":"PaLM 2 (few-shot, k=10)","rank_in_archive_order":21,"of":22,"metrics":{"AUROC":"0.8535"},"uses_additional_data":false},{"leaderboard":"/sota/toxic-comment-classification-on-civil","task":"Toxic Comment Classification","dataset":"Civil Comments","model":"PaLM 2 (zero-shot)","rank_in_archive_order":22,"of":22,"metrics":{"AUROC":"0.7596"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"PaLM 2-L (one-shot)","rank_in_archive_order":13,"of":37,"metrics":{"Accuracy":"66.8"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"PaLM 2-M (one-shot)","rank_in_archive_order":28,"of":37,"metrics":{"Accuracy":"52.0"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"PaLM 2-S (one-shot)","rank_in_archive_order":31,"of":37,"metrics":{"Accuracy":"50.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.10403","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}