{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/palm-scaling-language-modeling-with-pathways-1","title":"PaLM: Scaling Language Modeling with Pathways","arxiv_id":"2204.02311","date":"2022-04-05","proceeding":"Google Research 2022 4","authors":["Aakanksha Chowdhery","Sharan Narang","Jacob Devlin","Maarten Bosma","Gaurav Mishra","Adam Roberts","Paul Barham","Hyung Won Chung","Charles Sutton","Sebastian Gehrmann","Parker Schuh","Kensen Shi","Sasha Tsvyashchenko","Joshua Maynez","Abhishek Rao","Parker Barnes","Yi Tay","Noam Shazeer","Vinodkumar Prabhakaran","Emily Reif","Nan Du","Ben Hutchinson","Reiner Pope","James Bradbury","Jacob Austin","Michael Isard","Guy Gur-Ari","Pengcheng Yin","Toju Duke","Anselm Levskaya","Sanjay Ghemawat","Sunipa Dev","Henryk Michalewski","Xavier Garcia","Vedant Misra","Kevin Robinson","Liam Fedus","Denny Zhou","Daphne Ippolito","David Luan","Hyeontaek Lim","Barret Zoph","Alexander Spiridonov","Ryan Sepassi","David Dohan","Shivani Agrawal","Mark Omernick","Andrew M. Dai","Thanumalayan Sankaranarayana Pillai","Marie Pellat","Aitor Lewkowycz","Erica Moreira","Rewon Child","Oleksandr Polozov","Katherine Lee","Zongwei Zhou","Xuezhi Wang","Brennan Saeta","Mark Diaz","Orhan Firat","Michele Catasta","Jason Wei","Kathy Meier-Hellstern","Douglas Eck","Jeff Dean","Slav Petrov","Noah Fiedel"],"abstract":"Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.","url_abs":"https://arxiv.org/abs/2204.02311v5","url_pdf":"https://arxiv.org/pdf/2204.02311v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/chrisociepa/allamo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/foundation-model-stack/fms-fsdp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/google/paxml","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/lucidrains/CoCa-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/conceptofmind/PaLM-flax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/lucidrains/PaLM-jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"palm-scaling-language-modeling-with-pathways-1","repo_url":"https://github.com/lucidrains/PaLM-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"auto-debugging","task_name":"Auto Debugging"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"cross-lingual-question-answering","task_name":"Cross-Lingual Question Answering"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"hindu-knowledge","task_name":"Hindu Knowledge"},{"task_slug":"known-unknowns","task_name":"Known Unknowns"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"logic-grid-puzzle","task_name":"Logic Grid Puzzle"},{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"math","task_name":"Math"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"multiple-choice-qa","task_name":"Multiple Choice Question Answering (MCQA)"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"strategyqa","task_name":"StrategyQA"},{"task_slug":"winowhy","task_name":"Winowhy"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"multi-query-attention","method_name":"Multi-Query Attention"},{"method_slug":"palm","method_name":"PaLM"},{"method_slug":"parallel-layers","method_name":"Parallel Layers"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"rope","method_name":"Rotary Embeddings"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"swiglu","method_name":"SwiGLU"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"parallel-layers","name":"Parallel Layers","full_name":"Parallel Layers"}],"results":[{"leaderboard":"/sota/auto-debugging-on-big-bench-lite","task":"Auto Debugging","dataset":"Big-bench Lite","model":"PaLM 62B (few-shot, k=5)","rank_in_archive_order":1,"of":3,"metrics":{"Exact string match":"38.2"},"uses_additional_data":false},{"leaderboard":"/sota/auto-debugging-on-big-bench-lite","task":"Auto Debugging","dataset":"Big-bench Lite","model":"PaLM 540B (few-shot, k=5)","rank_in_archive_order":2,"of":3,"metrics":{"Exact string match":"38.2"},"uses_additional_data":false},{"leaderboard":"/sota/auto-debugging-on-big-bench-lite","task":"Auto Debugging","dataset":"Big-bench Lite","model":"PaLM 8B (few-shot, k=5)","rank_in_archive_order":3,"of":3,"metrics":{"Exact string match":"14.7"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"PaLM Coder 540B","rank_in_archive_order":71,"of":99,"metrics":{"Accuracy":"47"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"PaLM 540B","rank_in_archive_order":86,"of":99,"metrics":{"Accuracy":"36.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-known","task":"Common Sense Reasoning","dataset":"BIG-bench (Known Unknowns)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"73.9"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-winowhy","task":"Common Sense Reasoning","dataset":"BIG-bench (Winowhy)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"65.9"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-big-bench-winowhy","task":"Common Sense Reasoning","dataset":"BIG-bench (Winowhy)","model":"PaLM-62B (few-shot, k=5)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"61.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"PaLM 540B (finetuned)","rank_in_archive_order":4,"of":45,"metrics":{"EM":"94.0","F1":"94.6"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM 540B (0-shot)","rank_in_archive_order":15,"of":77,"metrics":{"Accuracy":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM 62B (0-shot)","rank_in_archive_order":21,"of":77,"metrics":{"Accuracy":"77.0"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"PaLM-cont 62B (0-shot)","rank_in_archive_order":22,"of":77,"metrics":{"Accuracy":"77.0"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 540B (fine-tuned)","rank_in_archive_order":1,"of":82,"metrics":{"Accuracy":"100"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 540B (5-shot)","rank_in_archive_order":11,"of":82,"metrics":{"Accuracy":"89.5"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 540B (0-shot)","rank_in_archive_order":12,"of":82,"metrics":{"Accuracy":"89.1"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"PaLM 540B (1-shot)","rank_in_archive_order":16,"of":82,"metrics":{"Accuracy":"86.3"},"uses_additional_data":false},{"leaderboard":"/sota/cross-lingual-question-answering-on-tydiqa","task":"Cross-Lingual Question Answering","dataset":"TyDiQA-GoldP","model":"PaLM-540B (CoT)","rank_in_archive_order":7,"of":11,"metrics":{"EM":"52.9"},"uses_additional_data":false},{"leaderboard":"/sota/extreme-summarization-on-gem-xsum","task":"Extreme Summarization","dataset":"GEM-XSum","model":"PaLM (finetuning)-540B","rank_in_archive_order":2,"of":6,"metrics":{"Parameters":"540 B","ROUGE-2":"21.2"},"uses_additional_data":false},{"leaderboard":"/sota/extreme-summarization-on-gem-xsum","task":"Extreme Summarization","dataset":"GEM-XSum","model":"T5-XXL","rank_in_archive_order":3,"of":6,"metrics":{"ROUGE-2":"21.0"},"uses_additional_data":false},{"leaderboard":"/sota/extreme-summarization-on-gem-xsum","task":"Extreme Summarization","dataset":"GEM-XSum","model":"PaLM (finetuning)-62B","rank_in_archive_order":4,"of":6,"metrics":{"Parameters":"62 B","ROUGE-2":"18.5"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM-540B (Few-Shot)","rank_in_archive_order":1,"of":37,"metrics":{"Accuracy":"89.7"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM-540B (One-Shot)","rank_in_archive_order":9,"of":37,"metrics":{"Accuracy":"81.8"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"PaLM-540B (Zero-Shot)","rank_in_archive_order":15,"of":37,"metrics":{"Accuracy":"77.9"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-strategyqa","task":"Logical Reasoning","dataset":"BIG-bench (StrategyQA)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"73.9"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-on-big-bench-strategyqa","task":"Logical Reasoning","dataset":"BIG-bench (StrategyQA)","model":"PaLM-62B (few-shot, k=5)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"65.4"},"uses_additional_data":false},{"leaderboard":"/sota/memorization-on-big-bench-hindu-knowledge","task":"Memorization","dataset":"BIG-bench (Hindu Knowledge)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"95.4"},"uses_additional_data":false},{"leaderboard":"/sota/memorization-on-big-bench-hindu-knowledge","task":"Memorization","dataset":"BIG-bench (Hindu Knowledge)","model":"PaLM-62B (few-shot, k=5)","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"77.7"},"uses_additional_data":false},{"leaderboard":"/sota/multi-task-language-understanding-on-mgsm","task":"Multi-task Language Understanding","dataset":"MGSM","model":"PaLM 540B","rank_in_archive_order":6,"of":12,"metrics":{"Average (%)":"55.0"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-31","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Novel Concepts)","model":"PaLM-540B (few-shot, k=5)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"71.9"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-31","task":"Multiple Choice Question Answering (MCQA)","dataset":"BIG-bench (Novel Concepts)","model":"PaLM-62B (few-shot, k=5)","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"59.4"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"PaLM 540B (finetuned)","rank_in_archive_order":1,"of":20,"metrics":{"Accuracy":"100","F1":"100"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 540B (fine-tuned)","rank_in_archive_order":2,"of":90,"metrics":{"Accuracy":"95.7%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 540B (5-shot)","rank_in_archive_order":38,"of":90,"metrics":{"Accuracy":"79.6%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 540B (1-shot)","rank_in_archive_order":42,"of":90,"metrics":{"Accuracy":"78.7%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"PaLM 540B (0-shot)","rank_in_archive_order":49,"of":90,"metrics":{"Accuracy":"72.9%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"PaLM 540B (fine-tuned)","rank_in_archive_order":3,"of":65,"metrics":{"Accuracy":"92.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"PaLM 540B (finetuned)","rank_in_archive_order":1,"of":60,"metrics":{"Accuracy":"100"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"PaLM 540B (finetuned)","rank_in_archive_order":1,"of":30,"metrics":{"EM":"69.2","F1":"90.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM-540B (Few-Shot, k=64)","rank_in_archive_order":26,"of":47,"metrics":{"EM":"39.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM-540B (One-Shot)","rank_in_archive_order":36,"of":47,"metrics":{"EM":"29.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"PaLM-540B (Zero-Shot)","rank_in_archive_order":44,"of":47,"metrics":{"EM":"21.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-obqa","task":"Question Answering","dataset":"OBQA","model":"PaLM 540B (zero-shot)","rank_in_archive_order":8,"of":9,"metrics":{"Accuracy":"53.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-obqa","task":"Question Answering","dataset":"OBQA","model":"PaLM 62B (zero-shot)","rank_in_archive_order":9,"of":9,"metrics":{"Accuracy":"50.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM-540B (Few-Shot)","rank_in_archive_order":11,"of":56,"metrics":{"EM":"81.4"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM-540B (One-Shot)","rank_in_archive_order":12,"of":56,"metrics":{"EM":"81.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"PaLM-540B (Zero-Shot)","rank_in_archive_order":17,"of":56,"metrics":{"EM":"76.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM-540B (Few-Shot)","rank_in_archive_order":11,"of":37,"metrics":{"EM":"43.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM-540B (One-Shot)","rank_in_archive_order":30,"of":37,"metrics":{"EM":"22.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"PaLM-540B (Zero-Shot)","rank_in_archive_order":34,"of":37,"metrics":{"EM":"10.6"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"PaLM 540B (zero-shot)","rank_in_archive_order":12,"of":24,"metrics":{"Accuracy (High)":"49.1","Accuracy (Middle)":"68.1"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"PaLM 62B (zero-shot)","rank_in_archive_order":14,"of":24,"metrics":{"Accuracy (High)":"47.5","Accuracy (Middle)":"64.3"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"PaLM 8B (zero-shot)","rank_in_archive_order":19,"of":24,"metrics":{"Accuracy (High)":"42.3","Accuracy (Middle)":"57.9"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM-540B (Few-Shot)","rank_in_archive_order":29,"of":89,"metrics":{"Accuracy":"83.8"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM-540B (1-shot)","rank_in_archive_order":30,"of":89,"metrics":{"Accuracy":"83.6"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"PaLM-540B (0-shot)","rank_in_archive_order":32,"of":89,"metrics":{"Accuracy":"83.4"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"PaLM 540B (finetuned)","rank_in_archive_order":2,"of":37,"metrics":{"Accuracy":"78.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.02311","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.02311"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chrisociepa/allamo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/foundation-model-stack/fms-fsdp","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/paxml","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/PaLM-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/PaLM-jax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/conceptofmind/PaLM-flax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/CoCa-pytorch","reach":null}],"summary":{"ran":20,"ran_violates":1,"ran_draft_wrong":5,"ran_fixture":2,"ran_honours":2,"unverified":7},"by_repo_kind":{"listed":{"samples":37,"ran":30,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54e85922a73fb785","entry":"AllGather","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e85922a73fb785"}},{"code_sha256_prefix":"3e79daee4d06c337","entry":"CrossAttention","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e79daee4d06c337"}},{"code_sha256_prefix":"d8d433ddd03a68b1","entry":"EmbedToLatents","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8d433ddd03a68b1"}},{"code_sha256_prefix":"cbb7647a3b310ee0","entry":"LayerNorm","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbb7647a3b310ee0"}},{"code_sha256_prefix":"c0074f24b1b1dddf","entry":"LayerNorm","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0074f24b1b1dddf"}},{"code_sha256_prefix":"f77c04c0d47070c9","entry":"LayerNorm","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f77c04c0d47070c9"}},{"code_sha256_prefix":"036d204fef6ec731","entry":"PaLM","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"036d204fef6ec731"}},{"code_sha256_prefix":"901f254b57206879","entry":"PaLM","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"901f254b57206879"}},{"code_sha256_prefix":"75aea79ee837ee93","entry":"PaLM","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75aea79ee837ee93"}},{"code_sha256_prefix":"25a9d6da0d4b31f2","entry":"ParallelTransformer","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25a9d6da0d4b31f2"}},{"code_sha256_prefix":"0b005776897ae6b9","entry":"ParallelTransformerBlock","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b005776897ae6b9"}},{"code_sha256_prefix":"7b94868665d25696","entry":"ParallelTransformerBlock","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b94868665d25696"}},{"code_sha256_prefix":"7214c60748cdc2fb","entry":"ParallelTransformerBlock","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7214c60748cdc2fb"}},{"code_sha256_prefix":"4cf5ff951dbba496","entry":"PreNorm","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cf5ff951dbba496"}},{"code_sha256_prefix":"5fab2dfc0a4ccb06","entry":"Residual","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fab2dfc0a4ccb06"}},{"code_sha256_prefix":"b7ffedd1471cf6f7","entry":"Residual","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7ffedd1471cf6f7"}},{"code_sha256_prefix":"c327a17241c2969e","entry":"RotaryEmbedding","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c327a17241c2969e"}},{"code_sha256_prefix":"231817c8d194a68b","entry":"RotaryEmbedding","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"231817c8d194a68b"}},{"code_sha256_prefix":"84cdb1c0d54b1ef4","entry":"SwiGLU","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84cdb1c0d54b1ef4"}},{"code_sha256_prefix":"8d2e371b5f3aa503","entry":"SwiGLU","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d2e371b5f3aa503"}},{"code_sha256_prefix":"37ffe94c2d21b89e","entry":"apply_rotary_pos_emb","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37ffe94c2d21b89e"}},{"code_sha256_prefix":"fd592af960c2230e","entry":"apply_rotary_pos_emb","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd592af960c2230e"}},{"code_sha256_prefix":"6d7a8eee9f74827c","entry":"apply_rotary_pos_emb","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d7a8eee9f74827c"}},{"code_sha256_prefix":"1e2e827e2a6e8c99","entry":"apply_rotary_pos_emb","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e2e827e2a6e8c99"}},{"code_sha256_prefix":"78821f282a7ccfa2","entry":"fixed_pos_embedding","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78821f282a7ccfa2"}},{"code_sha256_prefix":"18a3c5b8f897961c","entry":"jax_unstack","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18a3c5b8f897961c"}},{"code_sha256_prefix":"d3d5bf1b20109dcd","entry":"pad_dim_to","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3d5bf1b20109dcd"}},{"code_sha256_prefix":"6aa7304ec358cbea","entry":"rotate_every_two","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa7304ec358cbea"}},{"code_sha256_prefix":"ca6a06a57596c8b4","entry":"rotate_half","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca6a06a57596c8b4"}},{"code_sha256_prefix":"08213c6683993d3b","entry":"rotate_half","repo":"lucidrains/PaLM-pytorch","repo_kind":"listed","path":"palm_pytorch/palm_pytorch.py","file_url":"https://github.com/lucidrains/PaLM-pytorch/blob/HEAD/palm_pytorch/palm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08213c6683993d3b"}},{"code_sha256_prefix":"c4b1f7fca53ae6f0","entry":"CoCa","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4b1f7fca53ae6f0"}},{"code_sha256_prefix":"51d0ce838f5af6d3","entry":"ParallelTransformerBlock","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51d0ce838f5af6d3"}},{"code_sha256_prefix":"f9d416af20c5e200","entry":"SwiGLU","repo":"conceptofmind/PaLM-flax","repo_kind":"listed","path":"palm_flax/palm_flax.py","file_url":"https://github.com/conceptofmind/PaLM-flax/blob/HEAD/palm_flax/palm_flax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9d416af20c5e200"}},{"code_sha256_prefix":"5079742c167ede96","entry":"all_gather_variable_batch","repo":"lucidrains/CoCa-pytorch","repo_kind":"listed","path":"coca_pytorch/coca_pytorch.py","file_url":"https://github.com/lucidrains/CoCa-pytorch/blob/HEAD/coca_pytorch/coca_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5079742c167ede96"}},{"code_sha256_prefix":"c0b06c9deb2f2918","entry":"is_checkpoint_asset","repo":"google/paxml","repo_kind":"listed","path":"paxml/checkpoint_paths.py","file_url":"https://github.com/google/paxml/blob/HEAD/paxml/checkpoint_paths.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0b06c9deb2f2918"}},{"code_sha256_prefix":"835cdf8db74062ed","entry":"metadata_exists","repo":"google/paxml","repo_kind":"listed","path":"paxml/checkpoint_metadata.py","file_url":"https://github.com/google/paxml/blob/HEAD/paxml/checkpoint_metadata.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"835cdf8db74062ed"}},{"code_sha256_prefix":"73f6618e94cff797","entry":"restore_metadata","repo":"google/paxml","repo_kind":"listed","path":"paxml/checkpoint_metadata.py","file_url":"https://github.com/google/paxml/blob/HEAD/paxml/checkpoint_metadata.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73f6618e94cff797"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}