Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations
Authors’ abstract
Understanding what people use AI assistants for in the workplace is essential for assessing their impact and informing future design. We present a novel framework HILTON: Human-In-the-Loop Topic mOdeling of multi-turn coNversations that combines the summarization capabilities of large language models (LLMs) with unsupervised, non-parametric clustering. The framework first sum-marizes multi-turn conversations between users and AI assistants, then clusters these summaries into coherent topics, which are fur-ther organized hierarchically, and dynamically updated as new data arrives. To ensure quality and practical relevance, we employ a human-in-the-loop process: annotators post-process discovered clusters and provide feedback, which is then used to refine topics and create evaluation data for future iterations. We further train a lightweight statistical classifier that enables efficient topic infer-ence for unseen conversations, making the pipeline suitable for real-world deployment. Our results show that our approach pro-duces coherent topic structures suitable for workplace analytics of AI assistants. We open-source our code for the application.1
Paper
- Authors
- Y. Chen, S. Archasantisuk, J. Bjerva
- Publication date
- 2026
Transcript
A transcript has not been published for this episode yet.
Licensing
- Source license
- Creative Commons Attribution 4.0 International
- Recording license
- CC-BY-4.0
The authors and publisher do not sponsor or endorse this recording.