Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild
Authors’ abstract
How do product teams evaluate LLM-powered products? As or- ganizations integrate large language models (LLMs) into digital products, their unpredictable nature makes traditional evaluation approaches inadequate, yet little is known about how practitioners navigate this challenge. Through interviews with nineteen practi- tioners across diverse sectors, we identify ten evaluation practices spanning informal ‘vibe checks’ to organizational meta-work. Be- yond confirming four documented challenges, we introduce a novel fifth we call the results-actionability gap, in which practitioners gather evaluation data but cannot translate findings into concrete improvements. Drawing on patterns from successful teams, we contribute strategies to bridge this gap, supporting practitioners’ formalization journey from ad-hoc interpretive practices (e.g., vibe checks) toward systematic evaluation. Our analysis suggests these interpretive practices are necessary adaptations to LLM character- istics rather than methodological failures. For HCI researchers, this presents a research opportunity to support practitioners in system- atizing emerging practices rather than developing new evaluation frameworks.
Paper
- Authors
- W. Van Der Maden, M. Sadek, Z. Xiao, A. Mottelson, Q.V. Liao, J. Zhu
- Publication date
- 2026
Transcript
A transcript has not been published for this episode yet.
Licensing
- Source license
- Creative Commons Attribution 4.0 International
- Recording license
- CC-BY-4.0
The authors and publisher do not sponsor or endorse this recording.