Papers › A Case Study of Web App Coding with OpenAI Reasoning Models
A Case Study of Web App Coding with OpenAI Reasoning Models
Yi Cui
This paper presents a case study of coding tasks by the latest reasoning models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling number of tasks and test cases. The new benchmark causes the o1 model performances to decline significantly, falling behind Claude 3.5. Moreover, they consistently fail when confronted with atypical yet correct test cases, a trap non-reasoning models occasionally avoid. We hypothesize that the performance variability is due to instruction comprehension. Specifically, the reasoning mechanism boosts performance when all expectations are captured, meanwhile exacerbates errors when key expectations are missed, potentially impacted by input lengths. As such, we argue that the coding success of reasoning models hinges on the top-notch base model and SFT to ensure meticulous adherence to instructions.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Code Generation | WebApp1K-React | o1-preview | pass@1 | 0.952 | #1 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | o1-mini | pass@1 | 0.939 | #2 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1K-React | deepseek-v2.5 | pass@1 | 0.834 | #5 of 8 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | claude-3-5-sonnet | pass@1 | 0.679 | #1 of 6 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | o1-mini | pass@1 | 0.667 | #2 of 6 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | o1-preview | pass@1 | 0.652 | #3 of 6 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | gpt-4o-2024-08-06 | pass@1 | 0.531 | #4 of 6 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | deepseek-v2.5 | pass@1 | 0.49 | #5 of 6 | Archive leaderboard | report |
| Code Generation | WebApp1k-Duo-React | mistral-large-2 | pass@1 | 0.449 | #6 of 6 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections