1 More Paper.
Brief00:04:00

A Framework for Evaluating GenAI Adoption and Use in Software Engineering

Authors’ abstract

Generative Artificial Intelligence (GenAI) is increas-ingly integrated into software products to enable new features and user capabilities, from early exploration to operational deployment. GenAI adoption as a component within a soft-ware system introduces quality risks because GenAI outputs are probabilistic, prompt-sensitive, and may drift after release. Organizations, therefore, need to decide what to evaluate, when to evaluate, and who owns quality evaluation activities across software design, development, and operations. ISO/IEC 25059 standard distinguishes between software product quality (e.g., usability) and quality-in-use (e.g., satisfaction) for AI-enabled software, yet it provides limited operational guidance for these evaluation activities. We therefore investigate how industrial software teams adopt and use GenAI models in the software systems they build and operate, and how they evaluate system qualities when deciding to adopt GenAI during development and after deployment. We do not benchmark the underlying GenAI model itself. In this study, we conducted 19 semi-structured inter-views in two software development companies. We triangulated the interviews with archival data (15 internal documents and 184 internal wiki/web pages) to capture GenAI adoption steps, quality concerns, evaluation practices, and role responsibilities. Our findings describe a three-phase adoption process – Ideation, Development, and Operation – highlighting where quality eval-uations occur, which criteria are used, and how evaluation responsibilities are distributed. Based on observed practices and using ISO/IEC 25059 as an organizing lens, we synthesize a process-oriented quality evaluation framework. This framework maps metrics to explicit gatekeeping, validation, and monitoring checkpoints, bridging abstract ISO quality characteristics with engineering workflows. We applied the framework in a GenAI-enabled software product (SE4AI) use case and reported how it supported structured evaluation activities. We also observed that quality evaluations span legal, security, development, QA, and operations, but ownership is fragmented across phases. We therefore propose a GenAI Quality Lead responsibility (often assignable to an existing senior role) to coordinate criteria, evidence, and traceability across quality evaluation activities. The results contribute to Software Engineering for AI (SE4AI) by clarifying how teams can measure qualities when building software that adopts and uses GenAI. Liang Yu and Emil Alégroth are with the Department of Software Engi-neering, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden (e-mail: liang.yu@bth.se; emil.alegroth@bth.se). Panagiota Chatzipetrou is with Örebro University, 701 82 Örebro, Sweden (e-mail: panagiota.chatzipetrou@oru.se). Tony Gorschek is with Blekinge Institute of Technology, 371 79 Karlskrona, Sweden, and also with Fortiss, 80805 Munich, Germany (e-mail: tony. gorschek@bth.se).

Paper

Authors
Liang Yu, Emil Alegroth, Panagiota Chatzipetrou, Tony Gorschek
Journal
IEEE Transactions on Software Engineering
Publication date
August 2026

Read the original paper

Transcript

A transcript has not been published for this episode yet.

Licensing

Source license
Creative Commons Attribution 4.0 International
Recording license
CC-BY-4.0

The authors and publisher do not sponsor or endorse this recording.