</> Code with Papers by Syntology

Papers › Optimizing Large Language Models for OpenAPI Code Completion

Optimizing Large Language Models for OpenAPI Code Completion

24 May 2024arXiv:2405.15729archive 2025-07-28

Bohdan Petryshyn, Mantas Lukoševičius

Recent advancements in Large Language Models (LLMs) and their utilization in code generation tasks have significantly reshaped the field of software development. Despite the remarkable efficacy of code completion solutions in mainstream programming languages, their performance lags when applied to less ubiquitous formats such as OpenAPI definitions. This study evaluates the OpenAPI completion performance of GitHub Copilot, a prevalent commercial code completion tool, and proposes a set of task-specific optimizations leveraging Meta's open-source model Code Llama. A semantics-aware OpenAPI completion benchmark proposed in this research is used to perform a series of experiments through which the impact of various prompt-engineering and fine-tuning techniques on the Code Llama model's performance is analyzed. The fine-tuned Code Llama model reaches a peak correctness improvement of 55.2% over GitHub Copilot despite utilizing 25 times fewer parameters than the commercial solution's underlying Codex model. Additionally, this research proposes an enhancement to a widely used code infilling training technique, addressing the issue of underperformance when the model is prompted with context sizes smaller than those used during training. The dataset, the benchmark, and the model fine-tuning code are made publicly available.

PaperPDFCode

Code

BohdanPetryshyn/code-llama-fim-fine-tuning officialmentioned in papermentioned on GitHubpytorchMIT report
BohdanPetryshyn/openapi-completion-benchmark officialmentioned in papermentioned on GitHub report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Code CompletionCode GenerationOpenAPI code completionPrompt Engineering

Datasets

Introduced by this paper, per the archive.

OpenAPI completion refined

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned with document splitting Correctness, avg., % 34 #1 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned with document splitting Correctness, max., % 42 #1 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned with document splitting Validness, avg., % 69.1 #1 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned with document splitting Validness, max., % 76 #1 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned at 4096 tokens Correctness, avg., % 32 #2 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned at 4096 tokens Correctness, max., % 45 #2 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned at 4096 tokens Validness, avg., % 63.1 #2 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B, fine-tuned at 4096 tokens Validness, max., % 84 #2 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B Correctness, avg., % 31.1 #3 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B Correctness, max., % 36 #3 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B Validness, avg., % 60.7 #3 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined Code Llama 7B Validness, max., % 64 #3 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined GitHub Copilot Correctness, avg., % 29 #4 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined GitHub Copilot Correctness, max., % 29 #4 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined GitHub Copilot Validness, avg., % 68 #4 of 4 Archive leaderboard report
OpenAPI code completion OpenAPI completion refined GitHub Copilot Validness, max., % 68 #4 of 4 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

LLaMASET

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections