{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-models-are-few-shot-learners","title":"Language Models are Few-Shot Learners","arxiv_id":"2005.14165","date":"2020-05-28","proceeding":"NeurIPS 2020 12","authors":["Tom B. Brown","Benjamin Mann","Nick Ryder","Melanie Subbiah","Jared Kaplan","Prafulla Dhariwal","Arvind Neelakantan","Pranav Shyam","Girish Sastry","Amanda Askell","Sandhini Agarwal","Ariel Herbert-Voss","Gretchen Krueger","Tom Henighan","Rewon Child","Aditya Ramesh","Daniel M. Ziegler","Jeffrey Wu","Clemens Winter","Christopher Hesse","Mark Chen","Eric Sigler","Mateusz Litwin","Scott Gray","Benjamin Chess","Jack Clark","Christopher Berner","Sam McCandlish","Alec Radford","Ilya Sutskever","Dario Amodei"],"abstract":"Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.","url_abs":"https://arxiv.org/abs/2005.14165v4","url_pdf":"https://arxiv.org/pdf/2005.14165v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/openai/gpt-3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EightRice/atn_GPT-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/gpt-neo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/EleutherAI/lm_evaluation_harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/NVIDIA/NeMo-Curator","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/RUCAIBox/LLMBox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Samyu0304/thought-propagation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Sypherd/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/VachanVY/gpt.jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ai21labs/lm-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/allenai/macaw","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/asahi417/lmppl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/asahi417/relbert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/bigscience-workshop/Megatron-DeepSpeed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/bigscience-workshop/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/codedotal/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/contextlab/abstract2paper","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/crazydigger/Callibration-of-GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ethanjperez/true_few_shot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/facebookresearch/anli","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/fywalter/label-bias","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ggerganov/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ggml-org/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/gmum/dl-mo-2021","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/grantslatton/llama.cpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/haiyang-w/git","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hazyresearch/ama_prompting","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hilberthit/gpt-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/hojjat-mokhtarabadi/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/insait-institute/lm-evaluation-harness-bg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/juletx/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/karpathy/build-nanogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/karpathy/llm.c","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/lambert-x/prolab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/longhao-chen/aicas2024","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ltruncel/Microsoft_Azure_50daysofudacity","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ncoop57/gpt-code-clippy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/neuralmagic/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/nlx-group/overlapy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/openbiolink/promptsource","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/opengptx/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/postech-ami/smile-dataset","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/ramanakshay/nanogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/roberttwomey/machine-imagination-isea","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/roberttwomey/machine-imagination-workshop","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/sambanova/lm-evaluation-harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/scrayish/ML_NLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/shreyashankar/gpt3-sandbox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/smarton-empower/smarton-ai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/smile-data/smile","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/tonyzhaozh/few-shot-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/turkunlp/megatron-deepspeed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/um-arm-lab/efficient-eng-2-ltl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/vilm-ai/viet-llm-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/volcengine/vegiantmodel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/x-lance/neusym-rag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/zphang/lm_evaluation_harness","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/Mind23-2/MindCode-138","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/gpt-3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/abhaskumarsinha/Corpus2GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/abhaskumarsinha/MinimalGPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/kyegomez/GPT3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/milmor/GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"language-models-are-few-shot-learners","repo_url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/gpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"unsupervised-machine-translation","task_name":"Unsupervised Machine Translation"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"answerability-prediction","task_name":"answerability prediction"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"GPT-3 175B (1 shot)","rank_in_archive_order":31,"of":54,"metrics":{"Accuracy":"53.2"},"uses_additional_data":true},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"GPT-3 175B (0-shot)","rank_in_archive_order":34,"of":54,"metrics":{"Accuracy":"51.4"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"GPT-3 175B (1 shot)","rank_in_archive_order":30,"of":47,"metrics":{"Accuracy":"71.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-easy","task":"Common Sense Reasoning","dataset":"ARC (Easy)","model":"GPT-3 175B (0-shot)","rank_in_archive_order":38,"of":47,"metrics":{"Accuracy":"68.8"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":15,"of":45,"metrics":{"EM":"82.1"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"GPT-3 175B (0-shot)","rank_in_archive_order":39,"of":77,"metrics":{"Accuracy":"70.2"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-winogrande","task":"Common Sense Reasoning","dataset":"WinoGrande","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":58,"of":77,"metrics":{"Accuracy":"57.4"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"GPT-3 175B (few-shot)","rank_in_archive_order":21,"of":82,"metrics":{"Accuracy":"80.1"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-learning-on-medconceptsqa","task":"Few-Shot Learning","dataset":"MedConceptsQA","model":"gpt-3.5-turbo","rank_in_archive_order":2,"of":12,"metrics":{"Accuracy":"41.476"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":3,"of":37,"metrics":{"Accuracy":"86.4","Perplexity":"1.92"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 175B (Zero-Shot)","rank_in_archive_order":18,"of":37,"metrics":{"Accuracy":"76.2","Perplexity":"3.00"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 13B (Zero-Shot)","rank_in_archive_order":20,"of":37,"metrics":{"Accuracy":"72.5","Perplexity":"3.56"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 6.7B (Zero-Shot)","rank_in_archive_order":23,"of":37,"metrics":{"Accuracy":"70.3","Perplexity":"4.00"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GPT-3 2.7B (Zero-Shot)","rank_in_archive_order":28,"of":37,"metrics":{"Accuracy":"67.1","Perplexity":"4.60"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-penn-treebank-word","task":"Language Modelling","dataset":"Penn Treebank (Word Level)","model":"GPT-3 (Zero-Shot)","rank_in_archive_order":1,"of":43,"metrics":{"Params":"175000M","Test perplexity":"20.5"},"uses_additional_data":true},{"leaderboard":"/sota/natural-language-inference-on-anli-test","task":"Natural Language Inference","dataset":"ANLI test","model":"GPT-3","rank_in_archive_order":13,"of":25,"metrics":{"A1":"36.8","A2":"34","A3":"40.2"},"uses_additional_data":true},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":13,"of":20,"metrics":{"Accuracy":"75.6"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-commitmentbank","task":"Natural Language Inference","dataset":"CommitmentBank","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":20,"of":20,"metrics":{"F1":"52"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":58,"of":90,"metrics":{"Accuracy":"69%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":34,"of":65,"metrics":{"Accuracy":"76.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"GPT-3 75B (0-shot)","rank_in_archive_order":57,"of":65,"metrics":{"Accuracy":"60.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":11,"of":60,"metrics":{"Accuracy":"92"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (0-shot)","rank_in_archive_order":15,"of":60,"metrics":{"Accuracy":"91"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 175B (1-shot)","rank_in_archive_order":23,"of":60,"metrics":{"Accuracy":"87"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 13B (few-shot, k=32)","rank_in_archive_order":27,"of":60,"metrics":{"Accuracy":"86"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":46,"of":60,"metrics":{"Accuracy":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-coqa","task":"Question Answering","dataset":"CoQA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":7,"of":9,"metrics":{"Overall":"85"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-drop-test","task":"Question Answering","dataset":"DROP Test","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":15,"of":16,"metrics":{"F1":"36.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":13,"of":30,"metrics":{"F1":"75.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-natural-questions","task":"Question Answering","dataset":"Natural Questions","model":"GPT-3 175B (Few-Shot, k=64)","rank_in_archive_order":35,"of":47,"metrics":{"EM":"29.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-obqa","task":"Question Answering","dataset":"OBQA","model":"GPT-3 175B (zero-shot)","rank_in_archive_order":5,"of":9,"metrics":{"Accuracy":"57.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-openbookqa","task":"Question Answering","dataset":"OpenBookQA","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":29,"of":45,"metrics":{"Accuracy":"65.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"GPT-3 175B (0-shot)","rank_in_archive_order":29,"of":67,"metrics":{"Accuracy":"81.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":52,"of":67,"metrics":{"Accuracy":"72.9"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-peerqa","task":"Question Answering","dataset":"PeerQA","model":"GPT-3.5-Turbo-0613-16k","rank_in_archive_order":6,"of":6,"metrics":{"AlignScore":"0.1378","Prometheus-2 Answer Correctness":"3.0408","Rouge-L":"0.2414"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quac","task":"Question Answering","dataset":"QuAC","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":2,"of":2,"metrics":{"F1":"44.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-race","task":"Question Answering","dataset":"RACE","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":6,"of":7,"metrics":{"RACE-m":"58.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-race","task":"Question Answering","dataset":"RACE","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":7,"of":7,"metrics":{"RACE-h":"46.8"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-story-cloze","task":"Question Answering","dataset":"Story Cloze","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"87.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-storycloze","task":"Question Answering","dataset":"StoryCloze","model":"GPT-3 Large 760M (zero-shot)","rank_in_archive_order":19,"of":23,"metrics":{"Accuracy":"72.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":31,"of":56,"metrics":{"EM":"71.2"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"Few-shot","rank_in_archive_order":9,"of":37,"metrics":{"EM":"44.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (Few-Shot)","rank_in_archive_order":18,"of":37,"metrics":{"EM":"41.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (One-Shot)","rank_in_archive_order":29,"of":37,"metrics":{"EM":"25.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-webquestions","task":"Question Answering","dataset":"WebQuestions","model":"GPT-3-175B (Zero-Shot)","rank_in_archive_order":33,"of":37,"metrics":{"EM":"14.4"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"GPT-3 175B (0-shot)","rank_in_archive_order":18,"of":24,"metrics":{"Accuracy (Middle)":"58.4"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"GPT-3 175B (zero-shot)","rank_in_archive_order":24,"of":24,"metrics":{"Accuracy (High)":"45.5"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":45,"of":89,"metrics":{"Accuracy":"79.3"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 (0-shot)","rank_in_archive_order":48,"of":89,"metrics":{"Accuracy":"78.9"},"uses_additional_data":false},{"leaderboard":"/sota/sentence-completion-on-hellaswag","task":"Sentence Completion","dataset":"HellaSwag","model":"GPT-3 Large 760M (0-shot)","rank_in_archive_order":65,"of":89,"metrics":{"Accuracy":"51.0"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2014-2","task":"Unsupervised Machine Translation","dataset":"WMT2014 English-French","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":5,"of":7,"metrics":{"BLEU":"32.6"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2014-1","task":"Unsupervised Machine Translation","dataset":"WMT2014 French-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"39.2"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016","task":"Unsupervised Machine Translation","dataset":"WMT2016 English-German","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"29.7"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-2","task":"Unsupervised Machine Translation","dataset":"WMT2016 English-Romanian","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":3,"metrics":{"BLEU":"21"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-1","task":"Unsupervised Machine Translation","dataset":"WMT2016 German-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":7,"metrics":{"BLEU":"40.6"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-machine-translation-on-wmt2016-3","task":"Unsupervised Machine Translation","dataset":"WMT2016 Romanian-English","model":"GPT-3 175B (Few-Shot)","rank_in_archive_order":1,"of":3,"metrics":{"BLEU":"39.5"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"GPT-3 175B (few-shot, k=32)","rank_in_archive_order":36,"of":37,"metrics":{"Accuracy":"49.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-medconceptsqa","task":"Zero-Shot Learning","dataset":"MedConceptsQA","model":"gpt-3.5-turbo","rank_in_archive_order":2,"of":13,"metrics":{"Accuracy":"37.058"},"uses_additional_data":false},{"leaderboard":"/sota/answerability-prediction-on-peerqa","task":"answerability prediction","dataset":"PeerQA","model":"GPT-3.5-Turbo-0613-16k","rank_in_archive_order":3,"of":6,"metrics":{"Macro F1":"0.3304"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.14165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.14165"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tonyzhaozh/few-shot-learning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigscience-workshop/promptsource","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Samyu0304/thought-propagation","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ggml-org/llama.cpp","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EleutherAI/lm-evaluation-harness","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-ai/models/tree/master/official/nlp/gpt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai21labs/lm-evaluation","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/juletx/lm-evaluation-harness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shreyashankar/gpt3-sandbox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/um-arm-lab/efficient-eng-2-ltl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhaskumarsinha/Corpus2GPT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haiyang-w/git","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neuralmagic/lm-evaluation-harness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ltruncel/Microsoft_Azure_50daysofudacity","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EightRice/atn_GPT-3","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EleutherAI/gpt-neo","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fywalter/label-bias","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhaskumarsinha/MinimalGPT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/ama_prompting","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hojjat-mokhtarabadi/promptsource","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RUCAIBox/LLMBox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/gpt-3","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/macaw","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/gpt-3","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/crazydigger/Callibration-of-GPT","reach":{"status":"unanswered"}}],"summary":{"ran_violates":1,"ran":8,"ran_draft_wrong":4,"ran_fixture":1,"ran_honours":1,"unverified":50},"by_repo_kind":{"listed":{"samples":61,"ran":15,"repositories":17}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f0fc0ac7590a2d39","entry":"FLOP","repo":"abhaskumarsinha/Corpus2GPT","repo_kind":"listed","path":"models/GPT.py","file_url":"https://github.com/abhaskumarsinha/Corpus2GPT/blob/HEAD/models/GPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0fc0ac7590a2d39"}},{"code_sha256_prefix":"57e732507bad40db","entry":"escaped_split","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57e732507bad40db"}},{"code_sha256_prefix":"53354f082d9addc6","entry":"get_result","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/gguf.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/models/gguf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53354f082d9addc6"}},{"code_sha256_prefix":"00f5e7d9ee6636ba","entry":"get_sample_size","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00f5e7d9ee6636ba"}},{"code_sha256_prefix":"5b315604fb608f3b","entry":"get_sample_size","repo":"opengptx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/opengptx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b315604fb608f3b"}},{"code_sha256_prefix":"0ee5d4d20cf990bd","entry":"handle_arg_string","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ee5d4d20cf990bd"}},{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"cc82794b6ec72130","entry":"pad_vocab","repo":"karpathy/llm.c","repo_kind":"listed","path":"train_gpt2.py","file_url":"https://github.com/karpathy/llm.c/blob/HEAD/train_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc82794b6ec72130"}},{"code_sha256_prefix":"400ea4f64ccce5ef","entry":"parse_eval_args","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/__main__.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/__main__.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"400ea4f64ccce5ef"}},{"code_sha256_prefix":"e60f912106425ef4","entry":"positional_function","repo":"abhaskumarsinha/MinimalGPT","repo_kind":"listed","path":"GPT.py","file_url":"https://github.com/abhaskumarsinha/MinimalGPT/blob/HEAD/GPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e60f912106425ef4"}},{"code_sha256_prefix":"809bc9208dd99dce","entry":"prepare_print_tasks","repo":"opengptx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/opengptx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"809bc9208dd99dce"}},{"code_sha256_prefix":"de43c7cdbf869350","entry":"remove_none_pattern","repo":"sambanova/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/logging_utils.py","file_url":"https://github.com/sambanova/lm-evaluation-harness/blob/HEAD/lm_eval/logging_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de43c7cdbf869350"}},{"code_sha256_prefix":"c44830a7722ff5f7","entry":"request_caching_arg_to_dict","repo":"Sypherd/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator.py","file_url":"https://github.com/Sypherd/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c44830a7722ff5f7"}},{"code_sha256_prefix":"dfed6e9291316231","entry":"simple_parse_args_string","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/utils.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfed6e9291316231"}},{"code_sha256_prefix":"3bb8fcfcb4c12121","entry":"add_byte_giant_model_customize_args","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/initialize.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/initialize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3bb8fcfcb4c12121"}},{"code_sha256_prefix":"e66a197775436632","entry":"bps_recv","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e66a197775436632"}},{"code_sha256_prefix":"7ca8b4cd5f558e02","entry":"bps_send","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ca8b4cd5f558e02"}},{"code_sha256_prefix":"9dd643ee2669464c","entry":"build_model","repo":"VachanVY/gpt.jax","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/VachanVY/gpt.jax/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dd643ee2669464c"}},{"code_sha256_prefix":"904fac6af5735d5a","entry":"column_parallel_load_hook","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/module/dense.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/module/dense.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"904fac6af5735d5a"}},{"code_sha256_prefix":"64087979238e1d41","entry":"column_serial_load_hook","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/module/dense.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/module/dense.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64087979238e1d41"}},{"code_sha256_prefix":"451bc0edacfc317d","entry":"configure_optimizers","repo":"ramanakshay/nanogpt","repo_kind":"listed","path":"src/algorithm/optimizer.py","file_url":"https://github.com/ramanakshay/nanogpt/blob/HEAD/src/algorithm/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"451bc0edacfc317d"}},{"code_sha256_prefix":"3055a2ccdae837c7","entry":"cosine_similarity","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3055a2ccdae837c7"}},{"code_sha256_prefix":"3e6ff0c8581ee623","entry":"cosine_similarity","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/relation_mapping.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/relation_mapping.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e6ff0c8581ee623"}},{"code_sha256_prefix":"ffcac685bcb61388","entry":"create_image_prompt","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffcac685bcb61388"}},{"code_sha256_prefix":"a20ac4af0aec2beb","entry":"create_image_prompt","repo":"EleutherAI/lm_evaluation_harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/EleutherAI/lm_evaluation_harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a20ac4af0aec2beb"}},{"code_sha256_prefix":"ecbbe3133a0f68b3","entry":"create_image_prompt","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/api_models.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/models/api_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecbbe3133a0f68b3"}},{"code_sha256_prefix":"5307f83d52fe15d1","entry":"custom_prompter","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/lm.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/lm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5307f83d52fe15d1"}},{"code_sha256_prefix":"03edfbaef45e2c28","entry":"default_gen_kwargs","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/defaults.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/defaults.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03edfbaef45e2c28"}},{"code_sha256_prefix":"e1d89d7579eaaeec","entry":"em_accuracy_helper","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e1d89d7579eaaeec"}},{"code_sha256_prefix":"3866cbd2c9f5fd58","entry":"euclidean_distance","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3866cbd2c9f5fd58"}},{"code_sha256_prefix":"7753000f9482cac5","entry":"eval_accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7753000f9482cac5"}},{"code_sha256_prefix":"34f65890e3e58ec4","entry":"evaluate_analogy","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/analogy_questions.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/analogy_questions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34f65890e3e58ec4"}},{"code_sha256_prefix":"87cfebcf19ab803b","entry":"evaluate_classification","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/lexical_relation_classification.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/lexical_relation_classification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87cfebcf19ab803b"}},{"code_sha256_prefix":"c7271f2f2ce3c4fa","entry":"evaluate_relation_mapping","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/evaluation/relation_mapping.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/evaluation/relation_mapping.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7271f2f2ce3c4fa"}},{"code_sha256_prefix":"c56b548854fce3e6","entry":"flatten_list","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/list_keeper.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/list_keeper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c56b548854fce3e6"}},{"code_sha256_prefix":"f6825b4eb5fe78c8","entry":"get_commit_from_path","repo":"sambanova/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/logging_utils.py","file_url":"https://github.com/sambanova/lm-evaluation-harness/blob/HEAD/lm_eval/logging_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6825b4eb5fe78c8"}},{"code_sha256_prefix":"306b51e6dcfb30ef","entry":"get_distribution_scope","repo":"abhaskumarsinha/Corpus2GPT","repo_kind":"listed","path":"training_utils/distribution_utils.py","file_url":"https://github.com/abhaskumarsinha/Corpus2GPT/blob/HEAD/training_utils/distribution_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"306b51e6dcfb30ef"}},{"code_sha256_prefix":"77347b697708c203","entry":"get_label_probs","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"77347b697708c203"}},{"code_sha256_prefix":"b1e0531f8c17d5d6","entry":"get_lr","repo":"ramanakshay/nanogpt","repo_kind":"listed","path":"src/algorithm/optimizer.py","file_url":"https://github.com/ramanakshay/nanogpt/blob/HEAD/src/algorithm/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e0531f8c17d5d6"}},{"code_sha256_prefix":"d09226e56c2085ea","entry":"get_p_content_free","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d09226e56c2085ea"}},{"code_sha256_prefix":"8e3012dd13fbb68a","entry":"get_readme","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/relbert_cl/readme_template.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/relbert_cl/readme_template.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e3012dd13fbb68a"}},{"code_sha256_prefix":"5decf8cec7692d54","entry":"get_result","repo":"insait-institute/lm-evaluation-harness-bg","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/insait-institute/lm-evaluation-harness-bg/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5decf8cec7692d54"}},{"code_sha256_prefix":"89b22f3da55a855c","entry":"get_result","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89b22f3da55a855c"}},{"code_sha256_prefix":"0db35c379a280bd8","entry":"get_sample_size","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0db35c379a280bd8"}},{"code_sha256_prefix":"0fb487fd2495f33f","entry":"get_sample_size","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/evaluator_utils.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/evaluator_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fb487fd2495f33f"}},{"code_sha256_prefix":"26519d22d8af6d48","entry":"hash_args","repo":"neuralmagic/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/neuralmagic/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26519d22d8af6d48"}},{"code_sha256_prefix":"72f5b387b64e9d9b","entry":"hash_args","repo":"EleutherAI/lm_evaluation_harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/EleutherAI/lm_evaluation_harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72f5b387b64e9d9b"}},{"code_sha256_prefix":"3e9d38ecacd03fa7","entry":"hash_args","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/api/model.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9d38ecacd03fa7"}},{"code_sha256_prefix":"2e6ad295f92e3d9d","entry":"is_even","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/engine.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e6ad295f92e3d9d"}},{"code_sha256_prefix":"deba15e38b692b23","entry":"list_split","repo":"nlx-group/overlapy","repo_kind":"listed","path":"overlapy.py","file_url":"https://github.com/nlx-group/overlapy/blob/HEAD/overlapy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"deba15e38b692b23"}},{"code_sha256_prefix":"4ff6d4e326d3c0db","entry":"load_model","repo":"mbzuai-paris/lm-evaluation-harness-atlas-chat","repo_kind":"listed","path":"lm_eval/models/nemo_lm.py","file_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat/blob/HEAD/lm_eval/models/nemo_lm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ff6d4e326d3c0db"}},{"code_sha256_prefix":"2ee40684ab58dcb9","entry":"load_object","repo":"VachanVY/gpt.jax","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/VachanVY/gpt.jax/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ee40684ab58dcb9"}},{"code_sha256_prefix":"fdde66f5d1999801","entry":"loss_nce","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/util.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdde66f5d1999801"}},{"code_sha256_prefix":"bd67f40255aa71d3","entry":"loss_triplet","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/util.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd67f40255aa71d3"}},{"code_sha256_prefix":"7cacc3b113e10007","entry":"oa_chat_completion","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/openai_completions.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/openai_completions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cacc3b113e10007"}},{"code_sha256_prefix":"63b45afd720a808e","entry":"rand_sample","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/trainer.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63b45afd720a808e"}},{"code_sha256_prefix":"99b9d9d7a4d6ca29","entry":"restore_list","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/list_keeper.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/list_keeper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99b9d9d7a4d6ca29"}},{"code_sha256_prefix":"9d752127437e6b8b","entry":"run_inference_one_model","repo":"vilm-ai/viet-llm-eval","repo_kind":"listed","path":"lm_eval/models/vllm_causallms.py","file_url":"https://github.com/vilm-ai/viet-llm-eval/blob/HEAD/lm_eval/models/vllm_causallms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d752127437e6b8b"}},{"code_sha256_prefix":"c7a7a9dca42a1347","entry":"safe_json_load","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/relbert_cl/push_to_hub.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/relbert_cl/push_to_hub.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7a7a9dca42a1347"}},{"code_sha256_prefix":"2fe668161976f857","entry":"send","repo":"volcengine/vegiantmodel","repo_kind":"listed","path":"src/veGiantModel/engine/p2p.py","file_url":"https://github.com/volcengine/vegiantmodel/blob/HEAD/src/veGiantModel/engine/p2p.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fe668161976f857"}},{"code_sha256_prefix":"5036094f3af0dc56","entry":"setup_distributed_environment","repo":"mbzuai-paris/lm-evaluation-harness-atlas-chat","repo_kind":"listed","path":"lm_eval/models/nemo_lm.py","file_url":"https://github.com/mbzuai-paris/lm-evaluation-harness-atlas-chat/blob/HEAD/lm_eval/models/nemo_lm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5036094f3af0dc56"}},{"code_sha256_prefix":"cabc905de3f72170","entry":"steer","repo":"juletx/lm-evaluation-harness","repo_kind":"listed","path":"lm_eval/models/hf_steered.py","file_url":"https://github.com/juletx/lm-evaluation-harness/blob/HEAD/lm_eval/models/hf_steered.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cabc905de3f72170"}},{"code_sha256_prefix":"11b1822611370e59","entry":"to_tensor","repo":"asahi417/relbert","repo_kind":"listed","path":"relbert/trainer.py","file_url":"https://github.com/asahi417/relbert/blob/HEAD/relbert/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11b1822611370e59"}},{"code_sha256_prefix":"8cb749113ad26cc3","entry":"wrap_constant_batch_size","repo":"insait-institute/lm-evaluation-harness-bg","repo_kind":"listed","path":"lm_eval/models/neuron_optimum.py","file_url":"https://github.com/insait-institute/lm-evaluation-harness-bg/blob/HEAD/lm_eval/models/neuron_optimum.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cb749113ad26cc3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}