{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-changes-can-large-scale-language-models","title":"What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers","arxiv_id":"2109.04650","date":"2021-09-10","proceeding":"EMNLP 2021 11","authors":["Boseop Kim","HyoungSeok Kim","Sang-Woo Lee","Gichang Lee","Donghyun Kwak","Dong Hyeon Jeon","Sunghyun Park","Sungju Kim","Seonhoon Kim","Dongpil Seo","Heungsub Lee","Minyoung Jeong","Sungjae Lee","Minsub Kim","Suk Hyun Ko","Seokhun Kim","Taeyong Park","Jinuk Kim","Soyoung Kang","Na-Hyeon Ryu","Kang Min Yoo","Minsuk Chang","Soobin Suh","Sookyo In","Jinseong Park","Kyungduk Kim","Hiun Kim","Jisu Jeong","Yong Goo Yeo","Donghoon Ham","Dongju Park","Min Young Lee","Jaewook Kang","Inho Kang","Jung-Woo Ha","WooMyoung Park","Nako Sung"],"abstract":"GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.","url_abs":"https://arxiv.org/abs/2109.04650v2","url_pdf":"https://arxiv.org/pdf/2109.04650v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-changes-can-large-scale-language-models","repo_url":"https://github.com/kakaobrain/kogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"what-changes-can-large-scale-language-models","repo_url":"https://github.com/naver-ai/hypermix","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04650","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04650"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/kogpt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver-ai/hypermix","reach":null}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"12e3c05aa5de866a","entry":"TMixBertModel","repo":"naver-ai/hypermix","repo_kind":"listed","path":"models/tmix.py","file_url":"https://github.com/naver-ai/hypermix/blob/HEAD/models/tmix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12e3c05aa5de866a"}},{"code_sha256_prefix":"4619a31b371ea239","entry":"BertEncoder4Mix","repo":"naver-ai/hypermix","repo_kind":"listed","path":"models/tmix.py","file_url":"https://github.com/naver-ai/hypermix/blob/HEAD/models/tmix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4619a31b371ea239"}},{"code_sha256_prefix":"d62eabcffff15fb3","entry":"KoGPTInference","repo":"kakaobrain/kogpt","repo_kind":"listed","path":"kogpt/inference.py","file_url":"https://github.com/kakaobrain/kogpt/blob/HEAD/kogpt/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d62eabcffff15fb3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}