Papers › Do we still need Human Annotators? Prompting Large Language Models for Aspect...
Do we still need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Prediction
Nils Constantin Hellwig, Jakob Fehle, Udo Kruschwitz, Christian Wolff
Aspect sentiment quadruple prediction (ASQP) facilitates a detailed understanding of opinions expressed in a text by identifying the opinion term, aspect term, aspect category and sentiment polarity for each opinion. However, annotating a full set of training examples to fine-tune models for ASQP is a resource-intensive process. In this study, we explore the capabilities of large language models (LLMs) for zero- and few-shot learning on the ASQP task across five diverse datasets. We report F1 scores slightly below those obtained with state-of-the-art fine-tuned models but exceeding previously reported zero- and few-shot performance. In the 40-shot setting on the Rest16 restaurant domain dataset, LLMs achieved an F1 score of 52.46, compared to 60.39 by the best-performing fine-tuned method MVP. Additionally, we report the performance of LLMs in target aspect sentiment detection (TASD), where the F1 scores were also close to fine-tuned models, achieving 66.03 on Rest16 in the 40-shot setting, compared to 72.76 with MVP. While human annotators remain essential for achieving optimal performance, LLMs can reduce the need for extensive manual annotation in ASQP tasks.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Aspect-Based Sentiment Analysis (ABSA) | ASQP | Gemma-3-27B (50-shot, self-consistency learning) | F1 (R15) | 41.74 | #8 of 12 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | ASQP | Gemma-3-27B (50-shot, self-consistency learning) | F1 (R16) | 51.54 | #8 of 12 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | ASQP | Gemma-3-27B (10-shot, self-consistency learning) | F1 (R15) | 39.95 | #9 of 12 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | ASQP | Gemma-3-27B (10-shot, self-consistency learning) | F1 (R16) | 46.23 | #9 of 12 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | TASD | Gemma-3-27B (50-shot, self-consistency learning) | F1 (R15) | 62.12 | #6 of 11 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | TASD | Gemma-3-27B (50-shot, self-consistency learning) | F1 (R16) | 68.53 | #6 of 11 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | TASD | Gemma-3-27B (10-shot, self-consistency learning) | F1 (R15) | 54.37 | #9 of 11 | Archive leaderboard | report |
| Aspect-Based Sentiment Analysis (ABSA) | TASD | Gemma-3-27B (10-shot, self-consistency learning) | F1 (R16) | 66.75 | #9 of 11 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections