{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-survey-of-large-language-models","title":"A Survey of Large Language Models","arxiv_id":"2303.18223","date":"2023-03-31","proceeding":null,"authors":["Wayne Xin Zhao","Kun Zhou","Junyi Li","Tianyi Tang","Xiaolei Wang","Yupeng Hou","Yingqian Min","Beichen Zhang","Junjie Zhang","Zican Dong","Yifan Du","Chen Yang","Yushuo Chen","Zhipeng Chen","Jinhao Jiang","Ruiyang Ren","YiFan Li","Xinyu Tang","Zikang Liu","Peiyu Liu","Jian-Yun Nie","Ji-Rong Wen"],"abstract":"Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.","url_abs":"https://arxiv.org/abs/2303.18223v10","url_pdf":"https://arxiv.org/pdf/2303.18223v10.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/rucaibox/llmsurvey","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/MansMeg/IntroML","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/juyongjiang/codeup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/little-misfit/survey-of-virtual-try-on","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/xusenlinzy/api-for-open-llm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-survey-of-large-language-models","repo_url":"https://github.com/zhen-tan-dmml/llm4annotation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"survey","task_name":"Survey"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.18223","atlas_url":"https://app.syntology.ai/?focus=2303.18223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.18223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tloen/alpaca-lora","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/peft","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MansMeg/IntroML","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/little-misfit/survey-of-virtual-try-on","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xusenlinzy/api-for-open-llm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhen-tan-dmml/llm4annotation","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/llmsurvey","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/juyongjiang/codeup","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/RUCAIBox/LLMSurvey","reach":{"status":"ok"}}],"summary":{"ran_honours":2,"ran":6,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":9,"ran":6,"repositories":1},"found_in_text":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"32380a050ca033d4","entry":"b2mb","repo":"huggingface/peft","repo_kind":"found_in_text","path":"examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","file_url":"https://github.com/huggingface/peft/blob/HEAD/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32380a050ca033d4"}},{"code_sha256_prefix":"162fa636ec9c2a11","entry":"get_bool_env","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/config.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/config.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"162fa636ec9c2a11"}},{"code_sha256_prefix":"bdad3d08a6241ede","entry":"get_closest_label","repo":"huggingface/peft","repo_kind":"found_in_text","path":"examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","file_url":"https://github.com/huggingface/peft/blob/HEAD/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdad3d08a6241ede"}},{"code_sha256_prefix":"0ad95ef49376f579","entry":"get_env","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/config.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/config.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ad95ef49376f579"}},{"code_sha256_prefix":"14c9fb02d60eea83","entry":"get_package_version","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/adapter/patcher.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/adapter/patcher.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14c9fb02d60eea83"}},{"code_sha256_prefix":"2e71330e34fbbfd7","entry":"is_package_available","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/adapter/patcher.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/adapter/patcher.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e71330e34fbbfd7"}},{"code_sha256_prefix":"e6f6857273f4b6df","entry":"jsonify","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/common.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6f6857273f4b6df"}},{"code_sha256_prefix":"d267d33e45b87bcd","entry":"levenshtein_distance","repo":"huggingface/peft","repo_kind":"found_in_text","path":"examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","file_url":"https://github.com/huggingface/peft/blob/HEAD/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d267d33e45b87bcd"}},{"code_sha256_prefix":"a11cc5c5954610b6","entry":"model_validate","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/common.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a11cc5c5954610b6"}},{"code_sha256_prefix":"6794c98718b27e2e","entry":"annotate","repo":"rucaibox/llmsurvey","repo_kind":"official","path":"Experiments/HumanAlignment/HaluEval/claude_halu.py","file_url":"https://github.com/rucaibox/llmsurvey/blob/HEAD/Experiments/HumanAlignment/HaluEval/claude_halu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6794c98718b27e2e"}},{"code_sha256_prefix":"ee8b1f74b93de1f9","entry":"dictify","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/common.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee8b1f74b93de1f9"}},{"code_sha256_prefix":"2f84f57fc15961bb","entry":"infer_optim_dtype","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/adapter/patcher.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/adapter/patcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f84f57fc15961bb"}},{"code_sha256_prefix":"37c3d7ab0a5f9367","entry":"load_model_and_tokenizer","repo":"xusenlinzy/api-for-open-llm","repo_kind":"listed","path":"api/adapter/loader.py","file_url":"https://github.com/xusenlinzy/api-for-open-llm/blob/HEAD/api/adapter/loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37c3d7ab0a5f9367"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}