Papers › The Unreasonable Effectiveness of Eccentric Automatic Prompts
The Unreasonable Effectiveness of Eccentric Automatic Prompts
Rick Battle, Teja Gollapudi
Large Language Models (LLMs) have demonstrated remarkable problem-solving and basic mathematics abilities. However, their efficacy is highly contingent on the formulation of the prompt. This study endeavors to quantify the influence of incorporating "positive thinking" into the system message of the prompt, then compare that to systematic prompt optimization. We assess the performance of 60 combinations of system message snippets, tested with and without Chain of Thought prompting, across three models with parameters ranging from 7 to 70 billion on the GSM8K dataset. Our findings reveal that results do not universally generalize across models. In most instances, the inclusion of "positive thinking" prompts positively affected model performance. Notably, however, Llama2-70B exhibited an exception when not utilizing Chain of Thought, as the optimal system message was found to be none at all. Given the combinatorial complexity, and thus computation time, of experimenting with hand-tuning prompts for large black-box models, we then compared the performance of the best "positive thinking" prompt against the output of systematic prompt optimization. We show that employing an automated prompt optimizer emerges as the most effective method for enhancing performance, even when working with smaller open-source models. Additionally, our findings reveal that the highest-scoring, automatically-optimized prompt exhibits a degree of peculiarity far beyond expectations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Arithmetic Reasoning | GSM8K | Llama-2 70B (on 100 first questions, 4-shot, auto-optimized prompting) | Accuracy | 61 | #115 of 164 | Archive leaderboard | report |
| Arithmetic Reasoning | GSM8K | Llama-2 70B (on 100 first questions, 4-shot, auto-optimized prompting) | Parameters (Billion) | 70 | #115 of 164 | Archive leaderboard | report |
| Arithmetic Reasoning | GSM8K | Llama-2 13B (on 100 first questions, 4-shot, auto-optimized prompting) | Accuracy | 43 | #137 of 164 | Archive leaderboard | report |
| Arithmetic Reasoning | GSM8K | Llama-2 13B (on 100 first questions, 4-shot, auto-optimized prompting) | Parameters (Billion) | 13 | #137 of 164 | Archive leaderboard | report |
| Arithmetic Reasoning | GSM8K | Mistral 7B (on 100 first questions, 4-shot, auto-optimized prompting) | Accuracy | 41 | #139 of 164 | Archive leaderboard | report |
| Arithmetic Reasoning | GSM8K | Mistral 7B (on 100 first questions, 4-shot, auto-optimized prompting) | Parameters (Billion) | 7 | #139 of 164 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections