Papers › How Does Naming Affect LLMs on Code Analysis Tasks?
How Does Naming Affect LLMs on Code Analysis Tasks?
Zhilong Wang, Lan Zhang, Chen Cao, Nanqing Luo, Xinzhi Luo, Peng Liu
The Large Language Models (LLMs), such as GPT and BERT, were proposed for natural language processing (NLP) and have shown promising results as general-purpose language models. An increasing number of industry professionals and researchers are adopting LLMs for program analysis tasks. However, one significant difference between programming languages and natural languages is that a programmer has the flexibility to assign any names to variables, methods, and functions in the program, whereas a natural language writer does not. Intuitively, the quality of naming in a program affects the performance of LLMs in program analysis tasks. This paper investigates how naming affects LLMs on code analysis tasks. Specifically, we create a set of datasets with code containing nonsense or misleading names for variables, methods, and functions, respectively. We then use well-trained models (CodeBERT) to perform code analysis tasks on these datasets. The experimental results show that naming has a significant impact on the performance of code analysis tasks based on LLMs, indicating that code representation learning based on LLMs heavily relies on well-defined names in code. Additionally, we conduct a case study on some special code analysis tasks using GPT, providing further insights.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Code Generation | MBPP | GPT-4 (ChatGPT Plus) | Accuracy | 87.5 | #10 of 99 | Archive leaderboard | report |
| Code Generation | MBPP | GPT-3.5 Turbo (ChatGPT) | Accuracy | 83.2 | #15 of 99 | Archive leaderboard | report |
| Code Generation | MBPP | GPT-4 (Bing Chat) | Accuracy | 82 | #19 of 99 | Archive leaderboard | report |
| Code Generation | MBPP | Bard (PaLM 2/chat-bison-001) | Accuracy | 76.2 | #26 of 99 | Archive leaderboard | report |
| Code Generation | MBPP | Claude | Accuracy | 71.4 | #28 of 99 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections