{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/megatron-lm-training-multi-billion-parameter","title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","arxiv_id":"1909.08053","date":"2019-09-17","proceeding":null,"authors":["Mohammad Shoeybi","Mostofa Patwary","Raul Puri","Patrick Legresley","Jared Casper","Bryan Catanzaro"],"abstract":"Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%).","url_abs":"https://arxiv.org/abs/1909.08053v4","url_pdf":"https://arxiv.org/pdf/1909.08053v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/NVIDIA/Megatron-LM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/THUDM/ProteinLM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/ezelikman/STaR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/facebookresearch/fairscale","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/kingoflolz/mesh-transformer-jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/nvidia/transformerengine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/qhduan/CPM-LM-TF2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/Beacontownfc/MegatronBert_paddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok"}},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/PaddlePaddle/FleetX/tree/develop/examples/hybrid_parallelism","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"megatron-lm-training-multi-billion-parameter","repo_url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/megatronbert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"lambada","task_name":"LAMBADA"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Megatron-LM","rank_in_archive_order":3,"of":89,"metrics":{"Number of params":"8300M","Test perplexity":"10.81"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-piqa","task":"Question Answering","dataset":"PIQA","model":"MT-NLG 530B (0-shot)","rank_in_archive_order":23,"of":67,"metrics":{"Accuracy":"82.0"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"Megatron-BERT (ensemble)","rank_in_archive_order":2,"of":24,"metrics":{"Accuracy":"90.9","Accuracy (High)":"90.0","Accuracy (Middle)":"93.1"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-race","task":"Reading Comprehension","dataset":"RACE","model":"Megatron-BERT","rank_in_archive_order":4,"of":24,"metrics":{"Accuracy":"89.5","Accuracy (High)":"88.6","Accuracy (Middle)":"91.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1909.08053","atlas_url":"https://app.syntology.ai/?focus=1909.08053","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.08053"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/Megatron-LM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ezelikman/STaR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nvidia/transformerengine","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Beacontownfc/MegatronBert_paddle","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qhduan/CPM-LM-TF2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/megatronbert","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUDM/ProteinLM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/fairscale","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kingoflolz/mesh-transformer-jax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/FleetX/tree/develop/examples/hybrid_parallelism","reach":null}],"summary":{"ran":3,"ran_fixture":2,"ran_honours":4,"ran_draft_wrong":3,"unverified":35},"by_repo_kind":{"listed":{"samples":47,"ran":12,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"cf02ea13122d66ef","entry":"_CopyToModelParallelRegion","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf02ea13122d66ef"}},{"code_sha256_prefix":"0b452ebdf5b90909","entry":"_GatherFromModelParallelRegion","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0b452ebdf5b90909"}},{"code_sha256_prefix":"3e9321ad8928f17d","entry":"apply_rotary_pos_emb","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9321ad8928f17d"}},{"code_sha256_prefix":"b333830ef6a60a60","entry":"copy_to_model_parallel_region","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b333830ef6a60a60"}},{"code_sha256_prefix":"04bccd9db93a140d","entry":"copy_to_model_parallel_region","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04bccd9db93a140d"}},{"code_sha256_prefix":"33de97c702d46371","entry":"divide","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33de97c702d46371"}},{"code_sha256_prefix":"53dcb3e03bcd8ac6","entry":"divide_and_check_no_remainder","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"53dcb3e03bcd8ac6"}},{"code_sha256_prefix":"3d74408951f86d6b","entry":"fixed_pos_embedding","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d74408951f86d6b"}},{"code_sha256_prefix":"c3a8d35e5b99ad30","entry":"get_cuda_rng_tracker","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3a8d35e5b99ad30"}},{"code_sha256_prefix":"5efa020fabe4b20b","entry":"rotate_every_two","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5efa020fabe4b20b"}},{"code_sha256_prefix":"787ffed12fc28a46","entry":"split_tensor_along_last_dim","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"787ffed12fc28a46"}},{"code_sha256_prefix":"c7a00a98e71fad14","entry":"split_tensor_along_last_dim","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7a00a98e71fad14"}},{"code_sha256_prefix":"3a3e6cd309d80ca5","entry":"BertParallelSelfAttention","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a3e6cd309d80ca5"}},{"code_sha256_prefix":"9fafe439d9f687d9","entry":"CausalTransformerShard","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fafe439d9f687d9"}},{"code_sha256_prefix":"764045860ccf534a","entry":"ColumnParallelLinear","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"764045860ccf534a"}},{"code_sha256_prefix":"288f7749514cc2bd","entry":"ColumnParallelLinear","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"288f7749514cc2bd"}},{"code_sha256_prefix":"fc4bdb2e28201578","entry":"CudaRNGStatesTracker","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc4bdb2e28201578"}},{"code_sha256_prefix":"176d216b0475ad82","entry":"EmbeddingShard","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"176d216b0475ad82"}},{"code_sha256_prefix":"2a11a065886e73ae","entry":"ProjectionShard","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a11a065886e73ae"}},{"code_sha256_prefix":"ae952da8d1f3ecfc","entry":"RMSNorm","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae952da8d1f3ecfc"}},{"code_sha256_prefix":"356e9a6604faf993","entry":"RelativePositionEmbs","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"356e9a6604faf993"}},{"code_sha256_prefix":"4e4deb5427de1770","entry":"ReplicatedLayerNorm","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e4deb5427de1770"}},{"code_sha256_prefix":"f1f29dd1feac61bc","entry":"TransformerLayerShard","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f29dd1feac61bc"}},{"code_sha256_prefix":"daaecc67e19dee51","entry":"_CopyToModelParallelRegion","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daaecc67e19dee51"}},{"code_sha256_prefix":"d074b0cdd29844e9","entry":"_GatherFromModelParallelRegion","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d074b0cdd29844e9"}},{"code_sha256_prefix":"b63c4a93c8fbaec8","entry":"_gather","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b63c4a93c8fbaec8"}},{"code_sha256_prefix":"e042d40151c8e5b5","entry":"_gather","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e042d40151c8e5b5"}},{"code_sha256_prefix":"6c2eb620fc5d8483","entry":"_initialize_affine_weight","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6c2eb620fc5d8483"}},{"code_sha256_prefix":"5bdf5f6af94d14be","entry":"_initialize_affine_weight","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bdf5f6af94d14be"}},{"code_sha256_prefix":"2bae7a1462d53b78","entry":"_reduce","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2bae7a1462d53b78"}},{"code_sha256_prefix":"a8bcff2e094dfd4a","entry":"_reduce","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8bcff2e094dfd4a"}},{"code_sha256_prefix":"26c4d45293254bda","entry":"_set_cuda_rng_state","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26c4d45293254bda"}},{"code_sha256_prefix":"e798817bd7481060","entry":"_split","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e798817bd7481060"}},{"code_sha256_prefix":"93b792270121fe82","entry":"_split","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93b792270121fe82"}},{"code_sha256_prefix":"3bd26074ee7c9889","entry":"ensure_divisibility","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3bd26074ee7c9889"}},{"code_sha256_prefix":"158e905c29f23472","entry":"ensure_divisibility","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"158e905c29f23472"}},{"code_sha256_prefix":"b5f626934ffb18b4","entry":"f_psum","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5f626934ffb18b4"}},{"code_sha256_prefix":"0687254bf74e83b3","entry":"g_psum","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0687254bf74e83b3"}},{"code_sha256_prefix":"3e7b5da4deb87f39","entry":"gather_from_model_parallel_region","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e7b5da4deb87f39"}},{"code_sha256_prefix":"f9991e714de95be5","entry":"gather_from_model_parallel_region","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9991e714de95be5"}},{"code_sha256_prefix":"2929645262f955ef","entry":"get_model_parallel_group","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2929645262f955ef"}},{"code_sha256_prefix":"6bb3803691d991da","entry":"get_model_parallel_group","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bb3803691d991da"}},{"code_sha256_prefix":"cc03593755c3ba09","entry":"get_model_parallel_rank","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cc03593755c3ba09"}},{"code_sha256_prefix":"f6251a861544b65e","entry":"get_model_parallel_rank","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6251a861544b65e"}},{"code_sha256_prefix":"e2db563887482b11","entry":"get_model_parallel_world_size","repo":"facebookresearch/fairscale","repo_kind":"listed","path":"fairscale/nn/model_parallel/layers.py","file_url":"https://github.com/facebookresearch/fairscale/blob/HEAD/fairscale/nn/model_parallel/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e2db563887482b11"}},{"code_sha256_prefix":"e40c745e214477b2","entry":"get_model_parallel_world_size","repo":"qhduan/CPM-LM-TF2","repo_kind":"listed","path":"CPM-Generate/mpu/transformer.py","file_url":"https://github.com/qhduan/CPM-LM-TF2/blob/HEAD/CPM-Generate/mpu/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e40c745e214477b2"}},{"code_sha256_prefix":"e905d7fbdd56818f","entry":"getnorm","repo":"kingoflolz/mesh-transformer-jax","repo_kind":"listed","path":"mesh_transformer/transformer_shard.py","file_url":"https://github.com/kingoflolz/mesh-transformer-jax/blob/HEAD/mesh_transformer/transformer_shard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e905d7fbdd56818f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}