Papers › Insights from Benchmarking Frontier Language Models on Web App Code Generation
Insights from Benchmarking Frontier Language Models on Web App Code Generation
Yi Cui
This paper presents insights from evaluating 16 frontier large language models (LLMs) on the WebApp1K benchmark, a test suite designed to assess the ability of LLMs to generate web application code. The results reveal that while all models possess similar underlying knowledge, their performance is differentiated by the frequency of mistakes they make. By analyzing lines of code (LOC) and failure distributions, we find that writing correct code is more complex than generating incorrect code. Furthermore, prompt engineering shows limited efficacy in reducing errors beyond specific cases. These findings suggest that further advancements in coding LLM should emphasize on model reliability and mistake minimization.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Code Generation | WebApp1K-React | gpt-4o-2024-08-06 | pass@1 | 0.885 | #3 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | claude-3.5-sonnet | pass@1 | 0.8808 | #4 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | mistral-large-2 | pass@1 | 0.7804 | #6 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | deepseek-coder-v2-instruct | pass@1 | 0.7002 | #7 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | llama-v3p1-405b-instruct | pass@1 | 0.302 | #8 of 8 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections