1 More Paper.
Brief00:01:47

Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild

Authors’ abstract

How do product teams evaluate LLM-powered products? As or- ganizations integrate large language models (LLMs) into digital products, their unpredictable nature makes traditional evaluation approaches inadequate, yet little is known about how practitioners navigate this challenge. Through interviews with nineteen practi- tioners across diverse sectors, we identify ten evaluation practices spanning informal ‘vibe checks’ to organizational meta-work. Be- yond confirming four documented challenges, we introduce a novel fifth we call the results-actionability gap, in which practitioners gather evaluation data but cannot translate findings into concrete improvements. Drawing on patterns from successful teams, we contribute strategies to bridge this gap, supporting practitioners’ formalization journey from ad-hoc interpretive practices (e.g., vibe checks) toward systematic evaluation. Our analysis suggests these interpretive practices are necessary adaptations to LLM character- istics rather than methodological failures. For HCI researchers, this presents a research opportunity to support practitioners in system- atizing emerging practices rather than developing new evaluation frameworks.

Paper

Authors
Willem van der Maden, Malak Sadek, Ziang Xiao, Aske Mottelson, Q. Vera Liao, Jichen Zhu
Journal
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
Publication date
April 13, 2026

Read the original paper

Transcript

A transcript has not been published for this episode yet.

Licensing

Source license
Creative Commons Attribution 4.0 International
Recording license
CC-BY-4.0

The authors and publisher do not sponsor or endorse this recording.