Papers › CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised...
CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computing
Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Silvia Severini, Florian Matthes, Burkhard Rost
Currently, a growing number of mature natural language processing applications make people's life more convenient. Such applications are built by source code - the language in software engineering. However, the applications for understanding source code language to ease the software engineering process are under-researched. Simultaneously, the transformer model, especially its combination with transfer learning, has been proven to be a powerful technique for natural language processing tasks. These breakthroughs point out a promising direction for process source code and crack software engineering tasks. This paper describes CodeTrans - an encoder-decoder transformer model for tasks in the software engineering domain, that explores the effectiveness of encoder-decoder transformer models for six software engineering tasks, including thirteen sub-tasks. Moreover, we have investigated the effect of different training strategies, including single-task learning, transfer learning, multi-task learning, and multi-task learning with fine-tuning. CodeTrans outperforms the state-of-the-art models on all the tasks. To expedite future works in the software engineering domain, we have published our pre-trained models of CodeTrans. https://github.com/agemagician/CodeTrans
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| API Sequence Recommendation | DeepAPI | CodeTrans-MT-TF-Large | BLEU-4 | 73.39 | #1 of 1 | Archive leaderboard | report |
| Code Comment Generation | DeepCom | CodeTrans-TF-Large | Smoothed BLEU-4 | 39.50 | #1 of 1 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - Go | CodeTrans-TF-Large | Smoothed BLEU-4 | 19.54 | #7 of 7 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - Java | CodeTrans-MT-Large | Smoothed BLEU-4 | 21.87 | #1 of 8 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - JavaScript | CodeTrans-TF-Large | Smoothed BLEU-4 | 18.98 | #2 of 8 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - Php | CodeTrans-MT-Base | Smoothed BLEU-4 | 26.23 | #1 of 8 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - Python | CodeTrans-MT-Base | Smoothed BLEU-4 | 20.39 | #1 of 7 | Archive leaderboard | report |
| Code Documentation Generation | CodeSearchNet - Ruby | CodeTrans-MT-Base | Smoothed BLEU-4 | 15.26 | #1 of 7 | Archive leaderboard | report |
| Git Commit Message Generation | CommitGen | CodeTrans-TF-Large | BLEU-4 | 44.41 | #1 of 1 | Archive leaderboard | report |
| Program Synthesis | AlgoLisp | CodeTrans-MT-TF-Small | Accuracy | 90.31 | #1 of 1 | Archive leaderboard | report |
| Source Code Summarization | Summarizing Source Code using a Neural Attention Model - C# | CodeTrans-MT-Large | Smoothed BLEU-4 | 23.57 | #1 of 1 | Archive leaderboard | report |
| Source Code Summarization | Summarizing Source Code using a Neural Attention Model - Python | CodeTrans-MT-Base | Smoothed BLEU-4 | 13.37 | #1 of 1 | Archive leaderboard | report |
| Source Code Summarization | Summarizing Source Code using a Neural Attention Model - SQL | CodeTrans-MT-TF-Large | Smoothed BLEU-4 | 19.98 | #1 of 1 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections