Papers › Towards AI-Assisted Protocol Analysis in Design Research: Automating Question...
Towards AI-Assisted Protocol Analysis in Design Research: Automating Question Labelling with GPT-4 According to Eris’ (2004) Taxonomy
Ahmed Shahriar Sakib, Ada Hurst, Frank Safayeni
This study explores the potential of large language models (LLM)-based tools, specifically GPT-4 – a state-of-the-art language processing model - to assist in the analysis of verbal protocols of design. We focus on Eris’ taxonomy, a well-established framework that classifies questions asked by participants in a design-focused task according to three broad categories: low-level, deep reasoning, and generative design questions. Using a large dataset of pre-classified questions from design review meetings, a series of experiments test GPT-4’s capability in the categorization task and evaluate how different factors influence its precision. Results indicate that GPT-4 matches performance by human coders – a promising result for design researchers who can benefit from this tool with little prior natural language processing expertise. Overall, findings offer insights into the strengths and limitations of LLMs in this context and suggest directions for future research into the use of LLM-based tools in qualitative analyses of design activity.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections