Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild
Authors’ abstract
How do product teams evaluate LLM-powered products? As or- ganizations integrate large language models (LLMs) into digital products, their unpredictable nature makes traditional evaluation approaches inadequate, yet little is known about how practitioners navigate this challenge. Through interviews with nineteen practi- tioners across diverse sectors, we identify ten evaluation practices spanning informal ‘vibe checks’ to organizational meta-work. Be- yond confirming four documented challenges, we introduce a novel fifth we call the results-actionability gap, in which practitioners gather evaluation data but cannot translate findings into concrete improvements. Drawing on patterns from successful teams, we contribute strategies to bridge this gap, supporting practitioners’ formalization journey from ad-hoc interpretive practices (e.g., vibe checks) toward systematic evaluation. Our analysis suggests these interpretive practices are necessary adaptations to LLM character- istics rather than methodological failures. For HCI researchers, this presents a research opportunity to support practitioners in system- atizing emerging practices rather than developing new evaluation frameworks.
Paper
- Authors
- Willem van der Maden, Malak Sadek, Ziang Xiao, Aske Mottelson, Q. Vera Liao, Jichen Zhu
- Journal
- Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
- Publication date
- April 13, 2026
Transcript
A transcript has not been published for this episode yet.
Licensing
- Source license
- Creative Commons Attribution 4.0 International
- Recording license
- CC-BY-4.0
The authors and publisher do not sponsor or endorse this recording.