Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations

You’re listening to “Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations,” by Y. Chen, S. Archasantisuk, and J. Bjerva. Published in 2026.

Aalborg Universitet

Towards Understanding Professional AI Assistant Use

Human-in-the-Loop Topic Modeling of Multi-turn Conversations

Chen, Yiyi; Archasantisuk, Sukhumarn; Bjerva, Johannes

Published in:

WWW Companion 2026 - Companion Proceedings of the ACM Web Conference 2026

Creative Commons License CC BY 4.0

Publication date: 2026

Document Version

Publisher's PDF, also known as Version of record

Link to publication from Aalborg University

Citation for published version (APA):

Chen, Y., Archasantisuk, S., & Bjerva, J. (2026). Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations. In WWW Companion 2026 - Companion Proceedings of the ACM Web Conference 2026: WWW Companion '26 (pp. 379-390). Association for Computational Linguistics (ACL). the linked source

General rights and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

Take down policy

If you believe that this document breaches copyright please contact us at the email address providing details, and we will remove access to the work immediately and investigate your claim.

Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations

Yiyi Chen Sukhumarn Archasantisuk the email address the email address Aalborg University Independent Consultant Copenhagen, Denmark Tokyo, Japan

Abstract.

Understanding what people use AI assistants for in the workplace is essential for assessing their impact and informing future design. We present a novel framework HILTON: Human-In-the-Loop Topic mOdeling of multi-turn coNversations that combines the summarization capabilities of large language models (LLMs) with unsupervised, non-parametric clustering. The framework first sum-marizes multi-turn conversations between users and AI assistants, then clusters these summaries into coherent topics, which are fur-ther organized hierarchically, and dynamically updated as new data arrives. To ensure quality and practical relevance, we employ a human-in-the-loop process: annotators post-process discovered clusters and provide feedback, which is then used to refine topics and create evaluation data for future iterations.

We further train a lightweight statistical classifier that enables efficient topic infer-ence for unseen conversations, making the pipeline suitable for real-world deployment. Our results show that our approach pro-duces coherent topic structures suitable for workplace analytics of AI assistants. We open-source our code for the application.1

CCS Concepts.

• Computing methodologies → Discourse, dialogue and prag-matics; Information extraction; Natural language generation; • In-formation systems → Clustering.

Multi-turn Conversations, Topic Modeling, Language Models, Human-in-the-Loop

ACM Reference Format.

Yiyi Chen, Sukhumarn Archasantisuk, and Johannes Bjerva. 2026. Towards Understanding Professional AI Assistant Use: Human-in-the-Loop Topic Modeling of Multi-turn Conversations. In Companion Proceedings of the ACM Web Conference 2026 (WWW Companion ’26), April 13–17, 2026, Dubai, United Arab Emirates. ACM, New York, NY, USA, 12 pages. the linked source 10.1145/3774905.3794645 1GitHub: the linked source. Disclaimer: This research was conducted during an internship at an anonymous commercial entity. Due to confidentiality agreements, the identity of the organization, the data used, and any details regarding its distribution cannot be disclosed.

Johannes Bjerva the email address Aalborg University Copenhagen, Denmark 1 Introduction

Large Language Models (LLMs) are being increasingly deployed in workplaces worldwide, supporting a range of tasks both profes-sionally and privately. Prior work has explored this topic in general settings, however, there is still limited understanding of how employees actually use AI assistants in practice in professional tasks. Gaining systematic insights into these usage patterns is es-sential for optimizing workflows, promoting efficiency, and guiding the design of future AI systems. We approach this as an exploratory study that seeks to discover emergent topics from conversations between employees and AI assistants, rather than applying an ex-isting taxonomy to classify the conversations as in prior work.

To this end, we propose a human-in-the-loop topic modeling frame-work that discovers topics from multi-turn conversations with AI assistants, as illustrated in a general schema of such tasks in Fig. 1. The framework integrates LLM-based summarization with unsuper-vised clustering, and incorporates annotator feedback to refine and evaluate results. This design enables both robust topic discovery and continual adaptation as new data arrives.

Topic modeling is an increasingly critical task in natural lan-guage processing (NLP) as textual data grows in volume, diversity and complexity. Topic models automatically organize large corpora into interpretable structures, facilitating downstream tasks such as summarization, retrieval, and dialogue understanding. In real-world applications, such as customer service logs, user feedback, or conversational AI systems, topic modeling helps surface patterns and categories without requiring manual annotation. Topic model-ing can be tailored to the specific requirements of a particular task. In our case, the topic modeling pipeline is designed for summariz-ing tasks for each conversation turn, while considering previous conversation turns as context.

Classic approaches of topic modeling include Latent Dirichlet Allocation and neural topic modeling. While effective in word-level settings, these models fail to capture the semantic nuances in conversations. Moreover, they lack the expressive power of contextualized embeddings derived from pre-trained language models. More recent neural topic models leverage sentence embeddings; however, they remain static and lack mechanisms for iterative refinement or real-time update. With the growing popularity of LLMs, they are also pervasively deployed for end-to-end topic modeling tasks, which is very costly and also less interpretable. To address these limitations and challenges, our contributions are as follows:

• We propose a novel topic modeling framework that com-bines LLM-based summarization with unsupervised, non-parametric clustering on contextual sentence embeddings, enabling the discovery of meaningful topics from multi-turn conversations.

• We introduce a dynamic update mechanism using intra-cluster Min-Max sampling and inter-cluster stratified sam-pling to preserve representativeness of the clusters and adapt to incoming data efficiently.

• We show that a lightweight statistical classifier is sufficient to enable topic inference for unseen inputs, making the pipeline suitable for real-world deployment.

• We perform human-in-the-loop to validate both the quality of generated task summaries and the semantic coherence of the resulting topic clusters.

Overall, our method is scalable and adaptable over time, making it applicable to industrial settings with evolving user conversations and human-in-the-loop.

2 Related Work 2.1 Topic Modeling

Topic modeling has evolved from probabilistic bag-of-words formu-lations to embedding-aware, LLM-assisted pipelines. Classical methods such as LDA model texts as mixtures over latent topics, while matrix-factorization approaches offer fast, inter-pretable decompositions. Neural variational models then use encoders rather than hand-crafted features, which tie topics to word embeddings and better capture semantics over time. With transformers, contextualized topic models and related variants leverage sentence/document embeddings to boost coherence, specifically for short multilingual texts. In practice, cluster-ing on sentence embeddings, such as Hierarchical Density-based Spatial Clustering of Application with Noise (HDBSCAN), has become widely used.

Furthermore, BERTopic becomes a strong, scalable baseline, often combined with human-in-the-loop for in-terpretability, and widely applied in various domains. Evaluation has also shifted from generic topic coherence and diversity toward purpose-specific tasks and lightweight LLM judges that approximate human assessments. Most recently,

LLMs serve both as tools to label, merge, and refine topics and as models that can induce topics directly from corpora, trading off reproducibility and cost for flexibility. In comparison, we adopt a hybrid framework that combines robust, scalable clustering, LLM-guided naming and deduplication, and human-in-the-loop control and propulsion of analysis in an adaptable way.

2.2 Dialogue Text Summarization

Dialogue summarization has become essential for managing and comprehending large-scale conversations across various domains, including the medical domain, consumer reviews, and on-line tutorials. Handling long dialogue texts is particularly chal-lenging, which motivates the efficient attention and long-context architectural models, such as Longformer-Encoder-Decoder and GOVReport. In addition, multi-document news summarization is investigated by Multi-News at a scale. Within multi-turn dialogue specifically, datasets like DialogSum and SAMSum estab-lish multi-speaker, turn-based settings, while QMSum formalizes a locate-then-summarize paradigm for query-focused meeting sum-maries. With the growing availability of instruction-tuned LLMs, summarization quality improves dramatically in zero-and few-shot settings.

Specifically for our work, we treat the conversation turns with previous conversations as context and leverage LLMs to generate and evaluate task summaries for further analysis, which humans then validate.

3 HILTON

Our proposed framework HILTON consists of seven main compo-nents, as illustrated in Fig. 2: 1) Conversation turn extraction; 2) Task summarization with LLM few-shot prompting; 3) Initial clus-tering with an iterative refinement approach; 4) Naming the initial clusters, and hierarchization and deduplication of the clusters to build a hierarchical cluster; 5) Human evaluation of text summariza-tion and clustering; 6) Training and evaluating a statistical classifier, and further 7) Updating existing clusters and the built-in classifier. We experimented on a set of data provided by the company, and the metadata and statistics of the data remain non-disclosed due to a confidentiality agreement.

3.1 Conversation Turn Extraction

The threads of conversations between users and the AI assistant are extracted for a certain period of time. Initializing a thread of conversation, the user can define a system message that serves as a context for the entire thread.

For example, if the user defines “You are a professional translator in translating Spanish text into English”, the user is likely to conduct translation tasks throughout the thread of conversations. We define each conversation turn as a basic unit for further analysis, which consists of a request from a user, and the subsequent response(s) from the AI assistant, as illustrated in Fig. 2. Each thread between a user and the AI assistant consists of at least one conversation turn to be analyzed.

3.2 Task Summarization

To capture the task being requested in each conversation turn, while leveraging the previous conversation turns and system message as context, we use LLMs to implement few-shot prompting with templatic examples, to extract information such as conversation continuation (e.g., whether this conversation turn is the continua-tion of the given previous turns, if there is any), and identifying the user’s overall task for the AI assistant.2 To balance the input token length and offer enough context, we manually select a threshold of 4 for extracting previous conversation turns for task summariza-tion of the current conversation turn. We have epxerimented with three proprietary LLMs, i.e., GPT-4.1-Nano (GPT-Nano), GPT-4o-mini (GPT-mini), and Claude-haiku (Claude) (cf. Section 5.1 for results).

3.3 HDBSCAN Clustering with Voronoi Iterations

After obtaining about 50k task summaries from the conversation turns, we employ HDBSCAN3 to cluster the task representations. A sentence transformer MPNET4 is used to embed the task summaries to enable the vectors used in the clustering, with a dimension of 768. The embeddings are then normalized by subtracting the mean and scaling to unit variance,5 and the dimension is reduced to 50 with Uniform Manifold Approximation and Projection (UMAP) with Euclidean as distance metric.

The processed embeddings are then fit into HDBSCAN to obtain the initial clusters, as illustrated in Algorithm 1. HDBSCAN is a density-based clustering method well-suited for high-dimensional embeddings, enabling the discovery of clusters of varying densi-ties without requiring a predefined number of clusters. However, its performance is highly sensitive to the choice of hyperparam-eters, in particular the minimum samples and minimum cluster size, which control the local density threshold and the minimum number of points needed to form a cluster. To select the optimal hyperparameters, a grid search is conducted, and eventually we select 150 for minimum cluster size and 5 for minimum sample size. Despite this tuning, HDBSCAN often labels a substantial number of data points as outliers, especially when the data is sparse or the density contrast between clusters is low.

A resulting clustering is shown in Fig. 2, where clusters are color-coded accordingly.

Algorithm 1 Iterative HDBSCAN with Noise Refinement 39: end procedure

To improve cluster coverage and better utilize these data points, we implement subsequent Voronoi iterations to reclassify the out-liers by assigning them to the nearest cluster center, with the fol-lowing constraints, as illustrated in Fig. 3. Let cluster centers be {ck }K k =1 under Euclidean distance d (·, ·), the admissible region for cluster k ∈ K is the constrained Voronoi cell:

Vk = {x: d (x,ck) ≤ d (x,c j)∀ j } ∩{x: d (x,ck) ≤ Rk } and an outlier x is assigned to cluster k if and only if x ∈ Vk; otherwise it remains an outlier. At each iteration, we evaluate using the appropriate clustering metrics to decide whether the outliers should be reassigned (cf. Section 5.2).

3.4 Naming Clusters and Hierarchization

Let E = {e1...,en } ⊂ Rm denote a set of embeddings, partitioned k =1 with E = ÒK into K disjoint cluster sets {Ek }K k =1 Ek and distance d (·, ·). For few-shot cluster naming with LLM prompting, we con-struct a distance-based contrastive sampler for each cluster k as follows:

1) We randomly sample up to t points Pk ⊆ Ek as positive sam-.

ples, where |Pk | ≤ t;

2) for each p ∈ Pk, let Ok = E \ Ek and choose the t nearest.

elements of Ok to p: Nk (p) denotes the set of the t nearest elements to p in Ok under d (·, ·);

3) form the union over positives, deduplicate, and keep at most.

t negatives Nk, which is min(t, | Ð p ∈Pk Nk (p)|)-sized subset of Ð Nk (p). p ∈Pk

We rely on embeddings to automatically curate the semantically relevant texts, ensuring representative and coherent names. We experimented with sampling sizes t ∈, and after man-ually evaluating the results, we select the names generated with t = 200 and K = 44. We extract the corresponding texts of the re-sulting contrastive set (Pk, Nk) for each k as few-shot exemplars in the prompt for naming each cluster With few-shot prompting, we further build a hierarchy of the base clusters with cluster merging and deduplication, see detailed prompts in GitHub repo.

3.5 Human-in-the-Loop Evaluation and Curation

We stratifiedly sample 7 instances from each sub-cluster, resulting in a dataset of 308 samples. Each sample consists of a conversa-tional turn and its context, the LLM-generated task summary, the assigned sub-cluster, and its higher-level cluster. For each sample, we annotate whether the conversational turn is (i) consistent with the task summary, (ii) relevant to the sub-cluster name, and (iii) higher-level cluster name. All samples are independently annotated by two experts, following the annotation guideline as stipulated in our GitHub repository.6 After evaluating the annotation, the annotators manually modify the task summary, sub-cluster name and higher-level accordingly based on full agreement. Eventually, we have a modified dataset of 308 samples as our ground-truth dataset for further evaluation and development (cf. Section 5.3).

3.6 KNN Classifier and Updating Database

Let the whole set of embeddings be denoted as E and the cor-responding embeddings of the processed ground truth dataset as ET, we take the difference set ED = E \ ET as the database for training a statistical K-Nearest Neighbour (KNN) classifier7 f (·), and verify it using ET. When new task summary embed-dings, noted as EN, come in, the trained classifier is used to perform inference, assigning tentative cluster membership probabilities to each new embedding. We evaluate the newly tentative embed-ding set EN ∪ ED regulated by the probabilitiy threshold p with the embedding integrity metrics to select the optimal p, and further filter the EN by p. A newly merged databset EM = ENp ∪ ED is curated for further processing.

Hoewever, when there is a continuously large volume of new input coming in, the database can increase exponentially over time,

Davies Bouldin Score (DB), Calinski Harabasz Score (CH) and Standard Deviation of Cluster Sizes (SDCS) are used to evaluate the quality, cohension, and balance of the resulting clusters, the value ranges are shown in Table 1. SIL, BD and CH are widely used metrics that measure how well-separated and compact the clusters are: SIL evaluates both intra-cluster similarity and inter-cluster separation, DB penalizes clusters that are close together or have large variance, and CH favors dense and well-separated clusters. SDCS is also used to assess the uniformity of cluster sizes, helping to detect whether the clustering is dominated by a few large or small groups — particularly important when cluster balance is desired for downstream applications.

Inter-annotator Agreement (IAA). For measuring the inter-annotator agreement, we use Krippendorff’s alpha (K’s α), average ob-served agreement (AOA)8.

Evaluating Classifier. To evaluate the performance of the trained KNN classifier and further verify the sampling strategy, we use accuracy (ACC) and weighted F1 9.

5 Results & Insights 5.1 Task Summarization w. LLMs

To evaluate the cross-LLMs consistency of generated task sum-maries, we perform a pairwise comparison where the output from one LLM is treated as prediction and the others are treated as ref-erence. We use string matching metrics - BLEU1, BLEU2, Rouge-1, and Rouge-L - as surrogates to assess how closely the summaries from one LLM align with those produced by others. As shown in Table 2, GPT-mini consistently achieves the highest scores across all metrics, when evaluated against other LLMs’ outputs. These re-sults suggest that the task summaries generated from GPT-mini are semantically rich, lexically precise, and most aligned with broader summarization patterns of other LLMs, demonstrating its ability to produce outputs that are both distinctive and generalizable. As a result, we use the task summaries generated from GPT-mini for further analysis.

5.2 Clustering with Voronoi Iterations

As described in Section 3.3, we implement a grid search on hy-perparameters, for initializing HDBSCAN, with min cluster size in the range of and min sample size in. Evaluated with the best scores of SIL, DB, SDCS, we choose 150 as the minimal cluster size and 5 as the minimal sample size, which leaves 37.5% data points as outliers. To assign the outliers to the initial clusters, we conduct Voronoi iterations. For each iteration, we reevaluate the embedding space using SIL, DB, and CH scores. The iteration ends when SIL and CH stop increasing, and DB stops decreasing, which leaves less than 1% outliers after six rounds of refinement, as results shown in Table 3.

5.3 Human-in-the-Loop Evaluation and Curation

To assess inter-annotator reliability and agreement, we report K’s alpha and AOA across three dimensions for each conversational turn: consistency of task summary, relevance of the sub-cluster, and the higher-level cluster. As shown in Table 4, the highest agree-ment was observed for the consistency of task summaries, with K’sα of 0.7348 and AOA of 0.9221, indicating substantial agree-ment between the annotators. Agreement on sub-cluster relevance is moderate, reflecting increased boundary ambiguity due to finer granularity and semantic overlap among sub-clusters. In contrast, inter-annotator agreement in higher-level cluster relevance shows stronger reliability, consistent with the expectation that coarser categories are easier to distinguish than fine-grained distinctions.

In detail, we show IAA for Relevance of Higher-Level Cluster Name per higher-level cluster in Fig. 6 and 5. As an example, shown in Fig. 5, the annotators show stronger agreement on the relevance of task summaries, compared to the agreement on the correspond-ing clusters. Moreover, for certain clusters, the annotators agree nearly perfectly for higher-level clusters such as Language Transla-tion and Concept Explanation, while the annotations in clusters such as Business Analysis and Strategic Planning and Project and Coordi-nation Management Efficiency are more divergent. Through manual investigation, we found that the inconsistency of the relevance of clusterings can mainly be attributed to the incorrect task summa-rization. For example, the LLM tends to focus on the task the user requests, rather than the actual task.

For example, when the user asks the AI assistants to translate a manga or a word game through-out the conversation turns, even given previous turns as context, the LLM will judge the task to be about the story or the word game rather than translation. For instance, when a user asks an assistant to translate a manga or a word game across multiple turns, despite the access to the prior context, the LLM often infers the content (the story or game) as the task, rather than translation. As shown in Fig. 7, after manual post-processing, many clusters are modified to Coding, Debugging, Testing and Deployment, Content Creation and Enhancement, and Language Translation and Concept Explanation. The distribution of modified sub-clusters is in Appendix B.

5.4 Train KNN Classifier and Updating Database

To incorporate newly embedded task summaries in the database, we first adjust the threshold of the assigned probabilities of the new data using the trained classifier, as illustrated in Fig. 4. Table 5 shows the evaluation results for incorporating new data for (a) higher-level and (b) sub-level cluster classification, regulated by the probabilities p. While higher probability thresholds (e.g. p = 1.0) result in better clustering scores such as SIL and DB, they do so at the cost of significantly reducing the new data size, a reduction of 16% at p = 1.0. To balance the data coverage and database integrity, we select p = 0.6 as the threshold for further analysis, to maintain a broader input distribution for robustness and generalizability.

Afterwards, we further update the existing database with the newly filtered data, leveraging Min-max and stratified sampling to preserve embedding space integrity, mitigate bias, as well as maintain a reasonable number of overall samples. As shown in Table 6(a), the sampling strategy enables better embedding integrity even when the updated data samples are only at 20% in terms of SIL and DB, with even better classification performance than whole data. Moreover, for sub-cluster classifier, merely 50% of the data is necessary to deliver on-par results compared to the baseline, as shown in Table 6(b).

6 Discussion and Conclusion 6.1 Continuous Deployment at a Large Scale

Certain design choices can be made to further optimize the pipeline. The most costly component of the pipeline is the task summariza-tion with LLM prompting. It takes GPT-mini 98 hours to process all the 50k turns sequentially with system message and four previ-ous turns as context, GPT-nano 37 hours, and Claude-haiku 241 hours. To mitigate such cost, we can (i) keep the prompt short and structured for a specific analysis; (ii) summarize conversations incre-mentally using a rolling per-thread summary (noting this approach depends on per-thread summarization quality and can introduce bias); (iii) sample in production by inserting lightweight summa-rization calls into live traffic at random intervals. Together, these design choices enable the collection of anonymized summaries and avoid a long, sequential offline pass.

6.2 Privacy-preserving analytics

Leakage of personally identifiable information (PII) is a substantial risk in, e.g., task summarization. Prior work shows that PII-aware prompting, by explicitly instructing LLMs not to output PII, can re-duce this risk, while clustering can further obfuscate the remaining PII. In the same vein and beyond, HILTON conducts cluster-ing and subsequent KNN classification with adaptable databases fully leverage the standalone embeddings, without accessing the underlying text. This embedding-only pathway reduces the surface for adversarial attacks and is more robust to embedding inversion attacks.

Moreover, Chatterji et al. demonstrates more privacy-preserving approaches by using stricter sampling, automatically excluding users’ metadata for exclusion, and prohibiting human scrutiny by relying on LLM to scrub PII automatically and then classifying according to classifiers defined over a controlled label space. In comparison, our approach is exploratory in nature, as there is no predefined taxonomy, and discovering the label space is part of the analysis. Consequently, we adopt human-in-the-loop to cureate gold-standard datasets for evaluation and quality control.

6.3 Conclusion

We introduce HILTON, a human-in-the-loop topic modeling frame-work analyzing professional, multi-turn conversations between users and AI assistants. By combining LLM-based contextual sum-marization with embedding-driven clustering and lightweight clas-sification, HILTON discovers coherent, interpretable, and dynam-ically updated topics at scale. Our evaluation shows that this ap-proach yields high-quality topic structures while remaining cost-effective and privacy-preserving, as it operates primarily on em-beddings without accessing raw text. The integration of human feedback ensures semantic coherence and practical relevance, bal-ancing automation with expert oversight. This work lays the foun-dation for scalable, transparent, and privacy-aware conversational analytics, enabling a deeper understanding of how AI assistants are used in real-world professional contexts.

Acknowledgements.

Y.C. and J.B. are funded by the Carlsberg Foundation, under the Semper Ardens: Accelerate programme (project no. CF21-0454).

Ethical Statement

We comply with the ACM Code with Ethics and Professional Con-duct. We also comply with the confidentiality of the commercial entity.

Limitations.

Non-disclosure of the data can limit reproducibility. However, we argue that the proposed pipeline can be generally applied to any topic modeling setting that leverages the LLM’s capacity for sum-marization and unsupervised, non-parametric clustering. We also open-source our code to improve reproducibility.