{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/indictrans2-towards-high-quality-and","title":"IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages","arxiv_id":"2305.16307","date":"2023-05-25","proceeding":null,"authors":["Jay Gala","Pranjal A. Chitale","Raghavan AK","Varun Gumma","Sumanth Doddapaneni","Aswanth Kumar","Janki Nawale","Anupama Sujatha","Ratish Puduppully","Vivek Raghavan","Pratyush Kumar","Mitesh M. Khapra","Raj Dabre","Anoop Kunchukuttan"],"abstract":"India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people. 22 of these languages are listed in the Constitution of India (referred to as scheduled languages) are the focus of this work. Given the linguistic diversity, high-quality and accessible Machine Translation (MT) systems are essential in a country like India. Prior to this work, there was (i) no parallel training data spanning all 22 languages, (ii) no robust benchmarks covering all these languages and containing content relevant to India, and (iii) no existing translation models which support all the 22 scheduled languages of India. In this work, we aim to address this gap by focusing on the missing pieces required for enabling wide, easy, and open access to good machine translation systems for all 22 scheduled Indian languages. We identify four key areas of improvement: curating and creating larger training datasets, creating diverse and high-quality benchmarks, training multilingual models, and releasing models with open access. Our first contribution is the release of the Bharat Parallel Corpus Collection (BPCC), the largest publicly available parallel corpora for Indic languages. BPCC contains a total of 230M bitext pairs, of which a total of 126M were newly added, including 644K manually translated sentence pairs created as part of this work. Our second contribution is the release of the first n-way parallel benchmark covering all 22 Indian languages, featuring diverse domains, Indian-origin content, and source-original test sets. Next, we present IndicTrans2, the first model to support all 22 languages, surpassing existing models on multiple existing and new benchmarks created as a part of this work. Lastly, to promote accessibility and collaboration, we release our models and associated data with permissive licenses at https://github.com/AI4Bharat/IndicTrans2.","url_abs":"https://arxiv.org/abs/2305.16307v3","url_pdf":"https://arxiv.org/pdf/2305.16307v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"indictrans2-towards-high-quality-and","repo_url":"https://github.com/ai4bharat/indictrans2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.16307","atlas_url":"https://app.syntology.ai/?focus=2305.16307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.16307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai4bharat/indictrans2","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"336749dd7d699ef6","entry":"create_position_ids_from_input_ids","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/modeling_indictrans.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/modeling_indictrans.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"336749dd7d699ef6"}},{"code_sha256_prefix":"9dfcf8b13847c65f","entry":"shift_tokens_right","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/modeling_indictrans.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/modeling_indictrans.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dfcf8b13847c65f"}},{"code_sha256_prefix":"a29de23a02373ce2","entry":"compute_metrics_factory","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/train_lora.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/train_lora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a29de23a02373ce2"}},{"code_sha256_prefix":"f0c8343794ef8b5f","entry":"initialize_model_and_tokenizer","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/example.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c8343794ef8b5f"}},{"code_sha256_prefix":"4345abd3e91fa2b1","entry":"load_and_process_translation_dataset","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/train_lora.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/train_lora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4345abd3e91fa2b1"}},{"code_sha256_prefix":"e7e94d76c8167997","entry":"make_linear_from_emb","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/convert_indictrans_checkpoint_to_pytorch.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/convert_indictrans_checkpoint_to_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e94d76c8167997"}},{"code_sha256_prefix":"71b64e8cc27d0701","entry":"predict","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"baseline_eval/m2m100_inference.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/baseline_eval/m2m100_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71b64e8cc27d0701"}},{"code_sha256_prefix":"60615acd2c44cb5d","entry":"preprocess_fn","repo":"ai4bharat/indictrans2","repo_kind":"official","path":"huggingface_interface/train_lora.py","file_url":"https://github.com/ai4bharat/indictrans2/blob/HEAD/huggingface_interface/train_lora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60615acd2c44cb5d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}