Papers › LLaMA: Open and Efficient Foundation Language Models

LLaMA: Open and Efficient Foundation Language Models

27 Feb 2023arXiv 2023 2arXiv:2302.13971archive 2025-07-28

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

PaperPDFConference PDFCodeCode Syntology ranMethod as code

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

No sample from the archive's official repository (facebookresearch/llama) is linked to this paper; the 56 samples below come from 15 community repositories, and 2 more are identical to code Syntology harvested from another paper's repository. Syntology harvested code from facebookresearch/llama for another paper.

facebookresearch/llama: Syntology's graph links these samples to On the Influence of Gender and Race in Romantic Relationship Prediction from Large... (arXiv:2410.03996; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method.

Syntology 58 code samples: 56 from 15 repositories linked to this paper, 2 identical to code harvested from another paper's repository. Across all 58: 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified. Where Syntology's instrument failed: 11 where our contract draft was wrong; 1 where the contract check could not run; failures of Syntology's instrument, not of the code.

By repository: community (archive-listed): 56 samples from 15 repositories, 35 ran (24 with no instrument failure; 11 where Syntology's instrument failed); 2 identical to code first harvested elsewhere. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

No sample is linked to this paper from 26 of its repositories that Syntology harvested for other papers:

Levels written between 2026-09-22 and 2026-09-25 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed. 4 of the 37 samples that ran have no recorded date.

57 repositories listed; official and paper-mentioned ones first.

facebookresearch/llama officialmentioned in papermentioned on GitHubpytorchNOASSERTIONSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to On the Influence of Gender and Race in Romantic Relationship Prediction from Large... (arXiv:2410.03996; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
aethercortex/llama-x mentioned on GitHubpytorchApache-2.0Syntology: 3 samples, 2 ran (1 instrument-failed) · 1 unverified report
aozhongzhang/magr mentioned on GitHubpytorchMITSyntology: 1 sample, 0 ran · 1 unverified report
batsresearch/alfred mentioned on GitHubpytorchBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Alfred: A System for Prompted Weak Supervision (arXiv:2305.18623; that paper's own run record: 3 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 3 samples that ran constructed an object rather than computing a result); nothing checks that they implement this paper's method. report
beomi/koalpaca mentioned on GitHubpytorchApache-2.0Syntology: 3 samples, 3 ran (2 instrument-failed) · 0 unverified report
chaoyi-wu/finetune_llama mentioned on GitHubjaxSyntology: not harvested report
chaoyi-wu/pmc-llama mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to PMC-LLaMA: Towards Building Open-source Language Models for Medicine (arXiv:2304.14454; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
ecnu-icalk/educhat mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
ecolab-postech/owq mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to OPT: Open Pre-trained Transformer Language Models (arXiv:2205.01068; that paper's own run record: 14 ran (of which 2 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified) and GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323; that paper's own run record: 5 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified); nothing checks that they implement this paper's method. report
facebookresearch/chai mentioned on GitHubpytorchGPL-3.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CHAI: Clustered Head Attention for Efficient LLM Inference (arXiv:2403.08058; that paper's own run record: 0 ran · 1 unverified); nothing checks that they implement this paper's method. report
facico/chinese-vicuna mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Self-Instruct: Aligning Language Models with Self-Generated Instructions (arXiv:2212.10560; that paper's own run record: 10 ran (of which 3 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report
fajri91/indommlu mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive... (arXiv:2310.04928; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
flagalpha/llama2-chinese mentioned on GitHubpytorchSyntology: not harvested report
freedomintelligence/huatuogpt mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to HuatuoGPT, towards Taming Language Model to Be a Doctor (arXiv:2305.15075; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
fsoft-ai4code/codecapybara mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
ganjinzero/rrhf mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RRHF: Rank Responses to Align Language Models with Human Feedback without tears (arXiv:2304.05302; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
ggerganov/llama.cpp mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (arXiv:2306.00978; that paper's own run record: 16 ran (of which 1 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 10 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
ggml-org/llama.cpp mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation (arXiv:2609.01315; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
grantslatton/llama.cpp mentioned on GitHubMITSyntology: 5 samples, 5 ran (0 instrument-failed) · 0 unverified report
greenbitai/low_bit_llama mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (arXiv:2308.13137; that paper's own run record: 10 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
guinmoon/llmfarm mentioned on GitHubMITSyntology: not harvested report
hamishivi/easylm mentioned on GitHubjaxApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (arXiv:2406.09279; that paper's own run record: 15 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 13 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report
huggingface/transformers mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
icalk-nlp/educhat mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education (arXiv:2308.02773; that paper's own run record: 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
kayvr/token-hawk mentioned on GitHubpytorchMITSyntology: not harvested report
kbressem/medalpaca mentioned on GitHubpytorchGPL-3.0Syntology: not harvested report
krafton-ai/korani mentioned on GitHubpytorchApache-2.0Syntology: not harvested report
llamafamily/llama-chinese mentioned on GitHubpytorchSyntology: not harvested report
longhao-chen/aicas2024 mentioned on GitHubpytorchMITSyntology: 1 sample, 1 ran (0 instrument-failed) · 0 unverified report
ntunlplab/traditional-chinese-alpaca mentioned on GitHubpytorchApache-2.0Syntology: 4 samples, 2 ran (0 instrument-failed) · 2 unverified report
ofa-sys/expertllama mentioned on GitHubpytorchSyntology: 1 sample, 0 ran · 1 unverified report
ohadrubin/rpt mentioned on GitHubjaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval (arXiv:2306.13421; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
phoebussi/alpaca-cot mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685; that paper's own run record: 51 ran (of which 19 constructed an object rather than computing a result; 43 with no instrument failure: 1 honoured, 0 violated, 42 with no contract checked; 8 where Syntology's instrument failed) · 33 unverified) and ACL Anthology id 2023.findings-emnlp.269 (that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
qwopqwop200/GPTQ-for-LLaMa mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (arXiv:2210.17323; that paper's own run record: 5 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified); nothing checks that they implement this paper's method. report
replicate/cog_stanford_alpaca mentioned on GitHubpytorchApache-2.0Syntology: not harvested report
squeezeailab/squeezellm mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to SqueezeLLM: Dense-and-Sparse Quantization (arXiv:2306.07629; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288; that paper's own run record: 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified); nothing checks that they implement this paper's method. report
stanfordbdhg/llama.cpp mentioned on GitHubMITSyntology: 1 sample, 1 ran (0 instrument-failed) · 0 unverified report
tatsu-lab/stanford_alpaca mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering (arXiv:2605.17352; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified) and Modelpedia: A Catalog of Model Findings for the Meta-Science of AI (arXiv:2609.01090; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
teelinsan/camoscio mentioned on GitHubpytorchApache-2.0Syntology: 2 samples, 1 ran (0 instrument-failed) · 1 unverified report
vcskaushik/LLMzip mentioned on GitHubpytorchGPL-3.0Syntology: not harvested report
xiaoman-zhang/PMC-VQA mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (arXiv:2305.10415; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
xusenlinzy/api-for-open-llm mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to A Survey of Large Language Models (arXiv:2303.18223; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 1 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report
xvyaward/owq mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to OPT: Open Pre-trained Transformer Language Models (arXiv:2205.01068; that paper's own run record: 14 ran (of which 2 constructed an object rather than computing a result; 13 with no instrument failure: 0 honoured, 0 violated, 13 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified); nothing checks that they implement this paper's method. report
xzhang97666/alpacare mentioned on GitHubApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to AlpaCare:Instruction-tuned Large Language Models for Medical Application (arXiv:2310.14558; that paper's own run record: 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
young-geng/easylm mentioned on GitHubjaxSyntology: 15 samples, 9 ran (5 instrument-failed) · 6 unverified report
yuanmu97/secure-transformer-inference mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Secure Transformer Inference Protocol (arXiv:2312.00025; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
zihanzhaosjtu/librisqa mentioned on GitHubSyntology: not harvested report
2023-MindSpore-4/Code12 mindsporeSyntology: not harvested report
Lightning-AI/lit-llama pytorchSyntology: 8 samples, 4 ran (0 instrument-failed) · 4 unverified report
MS-P3/code5 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via... (arXiv:2405.18386; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
MS-P3/code5 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via... (arXiv:2405.18386; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
Mind23-2/MindCode-140 mindsporeSyntology: not harvested report
MindSpore-scientific-2/code-14 mindsporeSyntology: not harvested report
abhaskumarsinha/Corpus2GPT pytorchSyntology: 1 sample, 1 ran (0 instrument-failed) · 0 unverified report
akanyaani/miniLLAMA pytorchSyntology: 9 samples, 6 ran (3 instrument-failed) · 3 unverified report
meta-llama/llama pytorchNOASSERTIONSyntology: not harvested report
nebuly-ai/nebullvm pytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

58 samples harvested: 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

3ran · honoured contract
11ran · our draft was wrong
1ran; the contract check could not run
22ran
21unverified

Licence: 4 of the 58 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from 15 repositories linked to this paper, official or community; the list below is grouped by repository, each group headed by its name and what it is to the paper. 1 sample whose link to this paper records no repository is listed under one of this paper's own repositories, because identical code was harvested from it for another paper. 2 samples are listed last, in a group without a repository. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. For 3 samples that ran, the graph records a signature this page does not show, each row saying why in its tooltip: for 2, for a class the fixture built without calling, the harness called the object with inputs it made up itself, so the recorded signature is the harness's probe, not the fixture's arguments; for 1, an argument's recorded type or element kind is a printout of an object in Syntology's harness (such as a mock object, a class or a pandas printout), not the type of an input, so it does not describe the call's shape. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.

aethercortex/llama-x community (archive-listed) 3 samples, 2 ran (1 instrument-failed) · 1 unverified
generate_prompt in 110 papers src/generate.py ran fingerprinted Apache-2.0 (permissive) · 96db588eb784d7e4 · report fixture: signature instruction:str/0/str, input:str/0/str · returned str · level recorded 2026-09-25
jload in 59 papers src/utils.py ran · our draft was wrong Apache-2.0 (permissive) · d07d04439cd1d44f · report fixture: signature f:str/0/str, mode:str/0/str · returned dict · level recorded, date not recorded
get_conv_template in 72 papers src/conversation.py unverified Apache-2.0 (permissive) · 614f3f063d0ec7f0 · report
aozhongzhang/magr community (archive-listed) 1 sample, 0 ran · 1 unverified
llama_sequential in 10 papers llama.py identical code Syntology harvested from this repository for another paper; this paper's link records no repository, path or licence of its own unverified licence of this copy not recorded · 8a49f06400cfa533 · report
beomi/koalpaca community (archive-listed) 3 samples, 3 ran (2 instrument-failed) · 0 unverified
encode_prompt in 54 papers generate_instruction.py ran · our draft was wrong Apache-2.0 (permissive) · 9b2dd0265fc6e6b6 · report fixture: signature prompt_instructions:list/1/dict · returned str · level recorded, date not recorded
find_word_in_string in 12 papers generate_instruction.py ran · our draft was wrong fingerprinted Apache-2.0 (permissive) · 8ca6a5f9f77053c2 · report fixture: signature w:str/0/str, s:str/0/str · returned Match · level recorded, date not recorded
post_process_gpt3_response in 12 papers generate_instruction.py ran · honoured contract Apache-2.0 (permissive) · a5e4ad4e11cf2143 · report fixture: signature num_prompt_instructions:int/0/int, response:dict/0/? · returned list · level recorded, date not recorded
ecnu-icalk/educhat community (archive-listed) 1 sample, 0 ran · 1 unverified
initialize_model_and_tokenizer demo/educhat_api.py unverified no licence file found · pointer only · 50924ee98cd1539e · report
fsoft-ai4code/codecapybara community (archive-listed) 1 sample, 0 ran · 1 unverified
load_model in 479 papers deploy/run_hf.py unverified Apache-2.0 (permissive) · bb9d66258a238b9c · report
grantslatton/llama.cpp community (archive-listed) 5 samples, 5 ran (0 instrument-failed) · 0 unverified
bf16_to_fp32 convert.py ran MIT (permissive) · 3af74b5c95d41acd · report fixture: signature bf16_arr:ndarray/1/int · returned ndarray · level recorded 2026-09-25
many grammar.py ran MIT (permissive) · e1e39d01d0f3f81a · report fixture: signature parser:callable/0/? · returned function · level recorded 2026-09-25
none_of grammar.py ran MIT (permissive) · 24f233cbf074bc95 · report fixture: signature chars:str/0/str · returned function · level recorded 2026-09-25
one_of grammar.py ran MIT (permissive) · 42cd9e49542e405f · report fixture: signature chars:str/0/str · returned function · level recorded 2026-09-25
translate_tensor_name convert-lora-to-ggml.py ran MIT (permissive) · 17761b6cb6d0876d · report fixture: signature t:str/0/str · returned str · level recorded 2026-09-25
longhao-chen/aicas2024 community (archive-listed) 1 sample, 1 ran (0 instrument-failed) · 0 unverified
handle_metadata in 5 papers convert-llama-ggml-to-gguf.py ran MIT (permissive) · 6f0f4989a4b70aeb · report fixture: returned tuple · level recorded 2026-09-25
ntunlplab/traditional-chinese-alpaca community (archive-listed) 4 samples, 2 ran (0 instrument-failed) · 2 unverified
generate_prompt in 110 papers code/finetune.py ran Apache-2.0 (permissive) · 911a8a5064b9a121 · report fixture: signature data_point:dict/0/? · returned str · level recorded 2026-09-25
generate_prompt in 110 papers code/inference.py ran fingerprinted Apache-2.0 (permissive) · d7d073a5b1729e58 · report fixture: signature instruction:str/0/str, input:str/0/str · returned str · level recorded 2026-09-25
generate_and_tokenize_prompt in 8 papers code/finetune.py unverified Apache-2.0 (permissive) · ec1a43f948c4bfb1 · report
tokenize in 176 papers code/finetune.py unverified Apache-2.0 (permissive) · f54de390493549bb · report
ofa-sys/expertllama community (archive-listed) 1 sample, 0 ran · 1 unverified
gen in 18 papers model/gen_demo.py unverified Apache-2.0 (permissive) · ef609b4fb59efa8a · report
stanfordbdhg/llama.cpp community (archive-listed) 1 sample, 1 ran (0 instrument-failed) · 0 unverified
get_op_by_name awq-py/awq/apply_awq.py ran MIT (permissive) · d06d467f98e77d68 · report fixture: signature module:callable/0/?, op_name:str/0/str · returned SimpleSubModule · level recorded 2026-09-25
teelinsan/camoscio community (archive-listed) 2 samples, 1 ran (0 instrument-failed) · 1 unverified
generate_prompt in 110 papers eval/src/utils.py ran fingerprinted Apache-2.0 (permissive) · efa79bea90c5fd5a · report fixture: signature instruction:str/0/str, input:str/0/str · returned str · level recorded 2026-09-25
openai_call eval/qa_em_gpt_eval.py unverified Apache-2.0 (permissive) · 690c9b43da9c18a8 · report
young-geng/easylm community (archive-listed) 15 samples, 9 ran (5 instrument-failed) · 6 unverified
Carry EasyLM/models/llama/llama_model.py ran Apache-2.0 (permissive) · a066e3b4457453fd · report fixture: returned Carry · level recorded 2026-09-22
FlaxLLaMABlockCollection EasyLM/models/llama/llama_model.py ran · metamorphic tier: deterministic Apache-2.0 (permissive) · 2b160cc445933c46 · report fixture: returned FlaxLLaMABlockCollection · level recorded 2026-09-22
FlaxLLaMAMLP EasyLM/models/llama/llama_model.py ran · metamorphic tier: deterministic Apache-2.0 (permissive) · a4fb09d490cd772f · report fixture: returned FlaxLLaMAMLP · level recorded 2026-09-22
RMSNorm in 61 papers EasyLM/models/llama/llama_model.py ran · metamorphic tier: deterministic Apache-2.0 (permissive) · aaa8a447973c0f68 · report fixture: returned RMSNorm · level recorded 2026-09-22
_chunk_attention_bias EasyLM/models/llama/llama_model.py ran · our draft was wrong Apache-2.0 (permissive) · d81e74d07a61f182 · report fixture: signature query_chunk_size:int/0/int, key_chunk_size:int/0/int, bias:ArrayImpl/4/float, deterministic:bool/0/bool, attn_dropout:ArrayImpl/4/float, attn_pdrop:float/0/float, causal:bool/0/bool, dtype:callable/0/?, query_chunk_idx:int/0/int, key_chunk_idx:int/0/int · returned ArrayImpl · level recorded 2026-09-22
blockwise_attn EasyLM/models/llama/llama_model.py ran · our draft was wrong Apache-2.0 (permissive) · 4932ae36f745aea2 · report fixture: signature query:ArrayImpl/4/float, key:ArrayImpl/4/float, value:ArrayImpl/4/float, bias:ArrayImpl/4/float, deterministic:bool/0/bool, dropout_rng:ArrayImpl/1/int, attn_pdrop:float/0/float, causal:bool/0/bool, query_chunk_size:int/0/int, key_chunk_size:int/0/int, dtype:callable/0/?, policy:bool/0/bool, precision:NoneType/0/?, float32_logits:bool/0/bool, prevent_cse:bool/0/bool · returned ArrayImpl · level recorded 2026-09-22
get_gradient_checkpoint_policy EasyLM/models/llama/llama_model.py ran · our draft was wrong Apache-2.0 (permissive) · a9b09c32ed58621c · report fixture: signature name:str/0/str · returned function · level recorded 2026-09-22
get_names_from_parition_spec EasyLM/models/llama/llama_model.py ran · our draft was wrong Apache-2.0 (permissive) · dad72196bdb2301a · report fixture: signature partition_specs:list/1/str · returned list · level recorded 2026-09-22
with_sharding_constraint EasyLM/models/llama/llama_model.py ran · our draft was wrong fingerprinted Apache-2.0 (permissive) · 6c8893718b619284 · report fixture: signature x:ArrayImpl/2/float, partition_specs:str/0/str · returned ArrayImpl · level recorded 2026-09-22
FlaxLLaMAAttention EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · 733d82a9306baa8b · report
FlaxLLaMABlock EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · 7e8dcebe492702bb · report
FlaxLLaMAModule EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · 9238f425f871bc11 · report
apply_rotary_emb in 121 papers EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · f280056a639ae634 · report
blockwise_ffn EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · 4833ab3506125762 · report
names_in_current_mesh EasyLM/models/llama/llama_model.py unverified Apache-2.0 (permissive) · a8530c4a22be1de7 · report
Lightning-AI/lit-llama community (archive-listed) 8 samples, 4 ran (0 instrument-failed) · 4 unverified
LLaMAConfig lit_llama/model.py ran Apache-2.0 (permissive) · 0d0deeba6c2a15d7 · report fixture: returned LLaMAConfig · level recorded 2026-09-22
MLP in 375 papers lit_llama/model.py ran Apache-2.0 (permissive) · 76a887e44ebc75d5 · report fixture: returned MLP · level recorded 2026-09-22
RMSNorm in 61 papers lit_llama/model.py ran fingerprinted Apache-2.0 (permissive) · 133fab2290f71687 · report fixture: returned RMSNorm · level recorded 2026-09-22
build_rope_cache in 25 papers lit_llama/model.py ran · honoured contract Apache-2.0 (permissive) · 8a187cb05b43141d · report fixture: signature seq_len:int/0/int, n_elem:int/0/int, dtype:dtype/0/?, device:device/0/?, base:int/0/int · returned Tensor · level recorded 2026-09-22
CausalSelfAttention in 28 papers lit_llama/model.py unverified Apache-2.0 (permissive) · 5a58d3796ebd39f1 · report
LLaMA lit_llama/model.py unverified Apache-2.0 (permissive) · cad096e5b0761afb · report
abhaskumarsinha/Corpus2GPT community (archive-listed) 1 sample, 1 ran (0 instrument-failed) · 0 unverified
GPT in 24 papers models/GPT.py ran Apache-2.0 (permissive) · ebfbcce998fad801 · report fixture: returned GPT · level recorded 2026-09-22
akanyaani/miniLLAMA community (archive-listed) 9 samples, 6 ran (3 instrument-failed) · 3 unverified
FeedForward in 198 papers model.py ran MIT (permissive) · 6efcd693d3b4dc58 · report fixture: signature x:Tensor/3/float · returned Tensor · level recorded 2026-09-24
RMSNorm in 61 papers model.py ran · metamorphic tier: invariant MIT (permissive) · 0ea811a4ad9027b8 · report fixture: signature hidden_states:Tensor/3/float · returned Tensor · level recorded 2026-09-22
RotaryEmbedding in 27 papers model.py ran · metamorphic tier: deterministic fingerprinted MIT (permissive) · fd513b4604a55972 · report fixture: returned RotaryEmbedding · level recorded 2026-09-22
apply_rotary_pos_emb in 393 papers
census: 14 implementations from 15 papers, 8 distinct outputs across 7 buckets, one shared input per bucket
model.py ran · our draft was wrong fingerprinted MIT (permissive) · 705bfc1394637339 · report fixture: signature q:Tensor/3/float, k:Tensor/3/float, cos:Tensor/2/float, sin:Tensor/2/float, position_ids:Tensor/2/int, unsqueeze_dim:int/0/int · returned tuple · level recorded 2026-09-22
create_masks in 17 papers model.py ran · our draft was wrong fingerprinted MIT (permissive) · 31c156187744eaa5 · report fixture: signature inp:Tensor/2/int, device:NoneType/0/? · returned Tensor · level recorded 2026-09-22
repeat_kv in 244 papers
census: 15 implementations from 60 papers, 2 distinct outputs on one shared input this sample: bucket 1 of 1, arg 1: rank 4, kind float, dtype float32
model.py ran; the contract check could not run (RAISES) fingerprinted MIT (permissive) · 31333a36df31a820 · report fixture: signature hidden_states:Tensor/4/float, n_rep:int/0/int · returned Tensor · level recorded 2026-09-22
DecoderBlock in 30 papers model.py unverified MIT (permissive) · dba8af448d5b7b39 · report
LLAMA model.py unverified MIT (permissive) · 10092db6d72af6ab · report
Repository not located 2 samples, 2 ran (1 instrument-failed) · 0 unverified
2 samples are identical to code Syntology harvested from another paper's repository; this paper's own copy was not located, so no repository is named and no licence is recorded.
find_multiple in 31 papers repository not located ran · honoured contract fingerprinted licence of this copy not recorded · aba86ffbed27ba65 · report fixture: signature n:int/0/int, k:int/0/int · returned int · level recorded 2026-09-22
get_llama in 21 papers repository not located ran · our draft was wrong licence of this copy not recorded · 2c201f017173954b · report fixture: signature model:str/0/str · returned MockLlamaForCausalLM · level recorded 2026-09-22

Method as code Syntology

What Syntology's graph records as this paper's claimed method, lifted into code: each assembly names its bricks (functions and classes), each with its own role and its own verification level. An assembly is not a check that the code implements the method correctly or reproduces the paper's results. Levels are each brick's own verification level in Syntology's graph (level 0: no recorded run), listed brick by brick and never combined into one level; “ran” means the brick executed on a synthesized input, not that its output is correct, and the instrument figure counts failures of Syntology's instrument, not of the code. Where a model read the code against the paper's claim, each assembly's label verdict is one model's judgement, kept in this page's JSON twin; it is not calibrated, so no card prints it. Siblings are assemblies lifted from this paper in the same batch for the same method: alternative readings kept side by side; none replaces or supersedes another.

9 assemblies on this paper: 9 harvested, 0 generated, listed in 1 groups (siblings lifted in one batch form one group).

9 sibling assemblies for LLaMA (harvested), lifted in the same batch on 2026-09-09: alternative readings kept side by side; none replaces or supersedes another.

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. gen entry · level 0 · unverified · ef609b4fb59efa8a

harvested Assembly for LLaMA (graph method: LLaMA). 10 bricks; individually on a fixture: 7 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. LLAMA entry · level 0 · unverified · 10092db6d72af6ab
  2. DecoderBlock model · level 0 · unverified · dba8af448d5b7b39
  3. RMSNorm helper · level 1 · ran · 0ea811a4ad9027b8
  4. create_masks helper · level 1 · ran · our draft was wrong · 31c156187744eaa5
  5. Attention model · level 0 · unverified · 00a9dc7ef03e2fcc
  6. FeedForward model · level 1 · ran · 6efcd693d3b4dc58
  7. RotaryEmbedding helper · level 1 · ran · fd513b4604a55972
  8. apply_rotary_pos_emb helper · level 1 · ran · our draft was wrong · 705bfc1394637339
  9. repeat_kv helper · level 1 · ran; the contract check could not run (RAISES) · 31333a36df31a820
  10. rotate_half helper · level 1 · ran · our draft was wrong · e03d53ba9d4f9ae5

harvested Assembly for LLaMA (graph method: LLaMA). 15 bricks; individually on a fixture: 9 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. FlaxLLaMAModule entry · level 0 · unverified · 9238f425f871bc11
  2. FlaxLLaMABlockCollection model · level 1 · ran · 2b160cc445933c46
  3. RMSNorm model · level 1 · ran · aaa8a447973c0f68
  4. FlaxLLaMABlock model · level 0 · unverified · 7e8dcebe492702bb
  5. get_gradient_checkpoint_policy helper · level 1 · ran · our draft was wrong · a9b09c32ed58621c
  6. FlaxLLaMAAttention model · level 0 · unverified · 733d82a9306baa8b
  7. FlaxLLaMAMLP model · level 1 · ran · a4fb09d490cd772f
  8. blockwise_ffn helper · level 0 · unverified · 4833ab3506125762
  9. with_sharding_constraint helper · level 1 · ran · our draft was wrong · 6c8893718b619284
  10. apply_rotary_emb model · level 0 · unverified · f280056a639ae634
  11. blockwise_attn helper · level 1 · ran · our draft was wrong · 4932ae36f745aea2
  12. get_names_from_parition_spec helper · level 1 · ran · our draft was wrong · dad72196bdb2301a
  13. names_in_current_mesh helper · level 0 · unverified · a8530c4a22be1de7
  14. Carry helper · level 1 · ran · a066e3b4457453fd
  15. _chunk_attention_bias helper · level 1 · ran · our draft was wrong · d81e74d07a61f182

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result. The assembly has not been run end-to-end.

one of 9 siblings

  1. GPT entry · level 1 · ran · ebfbcce998fad801

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. llama_sequential entry · level 0 · unverified · 8a49f06400cfa533

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. get_llama entry · level 1 · ran · our draft was wrong · 2c201f017173954b

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. initialize_model_and_tokenizer entry · level 0 · unverified · 50924ee98cd1539e

harvested Assembly for LLaMA (graph method: LLaMA). 9 bricks; individually on a fixture: 5 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. LLaMA entry · level 0 · unverified · cad096e5b0761afb
  2. Block model · level 0 · unverified · d24315a7d247960d
  3. LLaMAConfig helper · level 1 · ran · 0d0deeba6c2a15d7
  4. RMSNorm model · level 1 · ran · 133fab2290f71687
  5. build_rope_cache helper · level 1 · ran · honoured contract · 8a187cb05b43141d
  6. CausalSelfAttention model · level 0 · unverified · 5a58d3796ebd39f1
  7. MLP model · level 1 · ran · 76a887e44ebc75d5
  8. find_multiple helper · level 1 · ran · honoured contract · aba86ffbed27ba65
  9. apply_rope helper · level 0 · unverified · 7137105772c52074

harvested Assembly for LLaMA (graph method: LLaMA). 1 brick; individually on a fixture: 0 ran · 1 unverified. The assembly has not been run end-to-end.

one of 9 siblings

  1. load_model entry · level 0 · unverified · bb9d66258a238b9c

Each brick ends with its code_sha256, Syntology's identity for that exact code; click one to copy the get_code call that returns it. For agents, on Syntology's MCP service: assembly(arxiv_id="2302.13971") returns these assemblies with each brick's role and wiring (how to connect).

Tasks

Arithmetic ReasoningCode GenerationCommon Sense ReasoningFew-Shot LearningMath Word Problem SolvingMulti-task Language UnderstandingQuestion AnsweringSentence CompletionStereotypical Bias AnalysisZero-Shot Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Arithmetic Reasoning GSM8K LLaMA 65B-maj1@k Accuracy 69.7 #104 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 65B-maj1@k Parameters (Billion) 65 #104 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 33B-maj1@k Accuracy 53.1 #129 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 33B-maj1@k Parameters (Billion) 33 #129 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 65B Accuracy 50.9 #135 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 65B Parameters (Billion) 65 #135 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 33B Accuracy 35.6 #143 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 33B Parameters (Billion) 33 #143 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 13B-maj1@k Accuracy 29.3 #147 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 13B-maj1@k Parameters (Billion) 13 #147 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 7B (maj1@k) Accuracy 18.1 #152 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 7B (maj1@k) Parameters (Billion) 7 #152 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 13B Accuracy 17.8 #155 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 13B Parameters (Billion) 13 #155 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 7B Accuracy 11.0 #159 of 164 Archive leaderboard report
Arithmetic Reasoning GSM8K LLaMA 7B Parameters (Billion) 7 #159 of 164 Archive leaderboard report
Code Generation MBPP LLaMA 65B (0-shot) Accuracy 37.7 #85 of 99 Archive leaderboard report
Code Generation MBPP LLaMA 33B (0-shot) Accuracy 30.2 #91 of 99 Archive leaderboard report
Code Generation MBPP LLaMA 13B (0-shot) Accuracy 22 #96 of 99 Archive leaderboard report
Code Generation MBPP LLaMA 7B (0-shot) Accuracy 17.7 #99 of 99 Archive leaderboard report
Common Sense Reasoning ARC (Challenge) LLaMA 33B (zero-shot) Accuracy 57.8 #27 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Challenge) LLaMA 65B (zero-shot) Accuracy 56.0 #29 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Challenge) LLaMA 13B (zero-shot) Accuracy 52.7 #32 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Challenge) LLaMA 7B (zero-shot) Accuracy 47.6 #40 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Easy) LLaMA 33B (0-shot) Accuracy 80.0 #15 of 47 Archive leaderboard report
Common Sense Reasoning ARC (Easy) LLaMA 65B (0-shot) Accuracy 78.9 #19 of 47 Archive leaderboard report
Common Sense Reasoning ARC (Easy) LLaMA 13B (0-shot) Accuracy 74.8 #25 of 47 Archive leaderboard report
Common Sense Reasoning ARC (Easy) LLaMA 7B (0-shot) Accuracy 72.8 #27 of 47 Archive leaderboard report
Common Sense Reasoning WinoGrande LLaMA 65B (0-shot) Accuracy 77.0 #23 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LLaMA 33B (0-shot) Accuracy 76.0 #25 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LLaMA 13B (0-shot) Accuracy 73.0 #33 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LLaMA 7B (0-shot) Accuracy 70.1 #41 of 77 Archive leaderboard report
Few-Shot Learning MedConceptsQA meta-llama/Meta-Llama-3-8B-Instruct Accuracy 25.653 #3 of 12 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 65B (maj1@k) Accuracy 20.5 #102 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 65B (maj1@k) Parameters (Billions) 65 #102 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 33B-maj1@k Accuracy 15.2 #108 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 33B-maj1@k Parameters (Billions) 33 #108 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 65B Accuracy 10.6 #116 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 65B Parameters (Billions) 65 #116 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 13B-maj1@k Accuracy 8.8 #119 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 13B-maj1@k Parameters (Billions) 13 #119 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 33B Accuracy 7.1 #120 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 33B Parameters (Billions) 33 #120 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 7B-maj1@k Accuracy 6.9 #121 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 7B-maj1@k Parameters (Billions) 7 #121 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 13B Accuracy 3.9 #131 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 13B Parameters (Billions) 13 #131 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 7B Accuracy 2.9 #133 of 135 Archive leaderboard report
Math Word Problem Solving MATH LLaMA 7B Parameters (Billions) 7 #133 of 135 Archive leaderboard report
Multi-task Language Understanding MML LLaMA 65B (fine-tuned) Average (%) 68.9 #13 of 44 Archive leaderboard report
Multi-task Language Understanding MML LLaMA 65B (5-shot) Average (%) 63.4 #15 of 44 Archive leaderboard report
Multi-task Language Understanding MML LLaMA 33B (5-shot) Average (%) 57.8 #20 of 44 Archive leaderboard report
Question Answering BoolQ LLaMA 65B (0-shot) Accuracy 85.3 #17 of 65 Archive leaderboard report
Question Answering BoolQ LLaMA 33B (0-shot) Accuracy 83.1 #23 of 65 Archive leaderboard report
Question Answering BoolQ LLaMA 13B (zero-shot) Accuracy 78.1 #29 of 65 Archive leaderboard report
Question Answering BoolQ LLaMA 7B (zero-shot) Accuracy 76.5 #32 of 65 Archive leaderboard report
Question Answering Natural Questions LLaMA 65B (few-shot, k=64) EM 39.9 #25 of 47 Archive leaderboard report
Question Answering Natural Questions LLaMA 65B (few-shot, k=5) EM 35.0 #29 of 47 Archive leaderboard report
Question Answering Natural Questions LLaMA 65B (one-shot) EM 31.0 #34 of 47 Archive leaderboard report
Question Answering Natural Questions LLaMA 33B (zero-shot) EM 24.9 #42 of 47 Archive leaderboard report
Question Answering OBQA LLaMA 65B (zero-shot) Accuracy 60.2 #3 of 9 Archive leaderboard report
Question Answering OBQA LLaMA 33B (zero-shot) Accuracy 58.6 #4 of 9 Archive leaderboard report
Question Answering OBQA LLaMA 7B (zero-shot) Accuracy 57.2 #6 of 9 Archive leaderboard report
Question Answering OBQA LLaMA 13B (zero-shot) Accuracy 56.4 #7 of 9 Archive leaderboard report
Question Answering PIQA LLaMA 65B (0-shot) Accuracy 82.8 #17 of 67 Archive leaderboard report
Question Answering PIQA LLaMA 33B (0-shot) Accuracy 82.3 #20 of 67 Archive leaderboard report
Question Answering PIQA LLaMA 13B (0-shot) Accuracy 80.1 #33 of 67 Archive leaderboard report
Question Answering PIQA LLaMA 7B (0-shot) Accuracy 79.8 #34 of 67 Archive leaderboard report
Question Answering SIQA LLaMA 65B (zero-shot) Accuracy 52.3 #18 of 24 Archive leaderboard report
Question Answering SIQA LLaMA 13B (zero-shot) Accuracy 50.4 #21 of 24 Archive leaderboard report
Question Answering SIQA LLaMA 33B (zero-shot) Accuracy 50.4 #22 of 24 Archive leaderboard report
Question Answering SIQA LLaMA 7B (zero-shot) Accuracy 48.9 #23 of 24 Archive leaderboard report
Question Answering TimeQuestions Llama3 P@1 17.8 #20 of 21 Archive leaderboard report
Question Answering TriviaQA LLaMA 65B (few-shot, k=64) EM 73.0 #23 of 56 Archive leaderboard report
Question Answering TriviaQA LLaMA 65B (few-shot, k=5) EM 72.6 #25 of 56 Archive leaderboard report
Question Answering TriviaQA LLaMA 65B (one-shot) EM 71.6 #28 of 56 Archive leaderboard report
Question Answering TriviaQA LLaMA 65B (zero-shot) EM 68.2 #34 of 56 Archive leaderboard report
Question Answering TruthfulQA LLaMA 65B % info 53 #25 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 65B % true 57 #25 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 33B % info 48 #26 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 33B % true 52 #26 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 13B % info 41 #27 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 13B % true 47 #27 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 7B % info 29 #28 of 33 Archive leaderboard report
Question Answering TruthfulQA LLaMA 7B % true 33 #28 of 33 Archive leaderboard report
Reading Comprehension RACE LLaMA 65B (zero-shot) Accuracy (High) 51.6 #13 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 65B (zero-shot) Accuracy (Middle) 67.9 #13 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 33B (zero-shot) Accuracy (High) 48.3 #15 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 33B (zero-shot) Accuracy (Middle) 64.1 #15 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 13B (zero-shot) Accuracy (High) 47.2 #16 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 13B (zero-shot) Accuracy (Middle) 61.6 #16 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 7B (zero-shot) Accuracy (High) 46.9 #17 of 24 Archive leaderboard report
Reading Comprehension RACE LLaMA 7B (zero-shot) Accuracy (Middle) 61.1 #17 of 24 Archive leaderboard report
Sentence Completion HellaSwag LLaMA 65B (0-shot) Accuracy 84.2 #28 of 89 Archive leaderboard report
Sentence Completion HellaSwag LLaMA 33B (0-shot) Accuracy 82.8 #35 of 89 Archive leaderboard report
Sentence Completion HellaSwag LLaMA 13B (0-shot) Accuracy 79.2 #47 of 89 Archive leaderboard report
Sentence Completion HellaSwag LLaMA 7B (0-shot) Accuracy 76.1 #51 of 89 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Age 70.1 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Disability 66.7 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Gender 70.6 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Nationality 64.2 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Overall 66.6 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Physical Appearance 77.8 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Race/Color 57.0 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Religion 70.6 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Sexual Orientation 81.0 #4 of 4 Archive leaderboard report
Stereotypical Bias Analysis CrowS-Pairs LLaMA 65B Socioeconomic status 71.5 #4 of 4 Archive leaderboard report
Zero-Shot Learning MedConceptsQA meta-llama/Meta-Llama-3-8B-Instruct Accuracy 25.840 #4 of 13 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: LLaMA

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDropoutGPT-3LLaMALayer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections