Understanding Human-Centred AI: a review of its defining elements and a research agenda

You’re listening to “Understanding Human-Centred AI: a review of its defining elements and a research agenda,” by S. Schmager, I.O. Pappas, and P. Vassilakopoulou. Published in 2025.

ISSN: 0144-929X (Print) 1362-3001 (Online) Journal homepage: the linked source

Stefan Schmager, Ilias O. Pappas & Polyxeni Vassilakopoulou

To cite this article: Stefan Schmager, Ilias O. Pappas & Polyxeni Vassilakopoulou (2025) Understanding Human-Centred AI: a review of its defining elements and a research agenda, Behaviour & Information Technology, 44:15, 3771-3810, DOI: 10.1080/0144929X.2024.2448719

© 2025 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group

Published online: 16 Feb 2025.

Submit your article to this journal

Article views: 29092

View related articles

View Crossmark data

Citing articles: 137 View citing articles

Understanding Human-Centred AI: a review of its defining elements and a research agenda a,b and Polyxeni Vassilakopoulou a, Ilias O. Pappas a Stefan Schmager aDepartment of Information Systems, University of Agder, Kristiansand, Norway; bDepartment of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway

ABSTRACT.

The rapid advancements in artificial intelligence (AI) have ushered in a new era of innovative applications, while also prompting concerns regarding risks and adverse consequences. In light of the growing interest in comprehending AI’s impact on society and its alignment with human values and needs, Human-Centred Artificial Intelligence (HCAI) has emerged as a potential approach to address questions and concerns. In this Systematic Literature Review, we aim to contribute to conceptual clarity around the definition, conceptualisation, and implementation of HCAI. The first part of our review addresses how HCAI is defined in the existing literature, culminating in a novel comprehensive HCAI definition. Subsequently, we delve into the identified constituent elements of HCAI, namely ‘purpose’, ‘values’, and ‘properties’. Purposes include augmentation, AI autonomy, and automation.

Values relate to ethics, safety, and performance. Properties cover oversight, comprehension, and integrity. The third part of the review explores Human-Centred Design processes, methods, and tools and their applicability for HCAI. In conclusion, we discuss the characteristics and critiques of HCAI and provide a research agenda. This literature review contributes to advancing the discourse on HCAI, thus enhancing human welfare and societal well-being.

Abbreviations: AI: artificial intelligence; AI-HLEG: high-level expert group on artificial intelligence; GenAI: generative AI; HCAI: human-centred artificial intelligence; HCD: human-centred design; HCI: human-computer interaction; ISO: international organization for standardization; OECD: organisation for economic co-operation and development.

1. Introduction.

Artificial intelligence (AI) is reshaping society and our daily lives. AI technologies are advancing rapidly, enabling novel applications but also raising concerns about potential risks and negative ramifications. In recent years, the capabilities and prevalence of AI have expanded dramatically, heightening both fascination with and anxiety about AI’s consequences. In response, different stakeholders, including policymakers, researchers, and practitioners are exploring how to deploy AI in ways that maximise benefits while mitigating risks and aligning with human needs and values. The European Commission’s High-Level Expert Group on Artificial Intelligence (AI-HLEG) (2019) has called for ethically aligned design, transparency, and accountability.

Researchers have argued that AI’s rapid advances create new requirements for responsible design and deployment (Dignum 2019; Schmager 2022;

Vassilakopoulou et al. 2022). Major tech companies have released frameworks and principles for responsible and trustworthy AI. As AI technologies become further integrated into social functions, new design considerations are needed to ensure transparency and predictability. A crucial first step is for AI designers, developers, and policymakers to begin envisioning how AI systems can be shaped to support human wel-fare, enhance human well-being and augment - rather than diminish – what it means to be human.

Creating technology – by humans, together with humans, and with the ultimate goal of human avail – is nothing new. Enhancing human abilities with the help of technology and simultaneously ensuring appro-priate levels of automation, supervision, and decision-making are well-established objects of inquiry. Bannon and Schmidt (1989) noted that by changing the allo-cation of functions between humans and their implements, changes in technology induce changes in work organisation. The longstanding debates about whether technology is neutral or politically charged are more topical than ever. Unlike the implementation of traditional technologies, AI systems introduce several unique challenges due to their unprecedented level of complexity, autonomy and personalised adaptive behaviour. AI can learn, adapt, and make independent decisions, often in ways that are not easily interpretable.

This ‘black box’ nature can make it more and more difficult for humans to com-prehend and potentially challenge AI-driven outcomes. This necessitates a shift in approaches to the design and development of AI systems, ensuring they are well-aligned with human needs and focusing on aspects such as explainability, transparency and data privacy. One approach that has received increasing attention in the discourse about the responsible design and develop-ment of AI systems, can be found under the name of ‘Human-Centred Artificial Intelligence’ (HCAI). Simi-lar to the Human-Centred Systems approach described by Cooley (2000), HCAI rejects the mechanistic para-digm of technological and societal development.

Although the idea of human-centeredness for AI does gain more and more traction within different research landscapes as shown in a recent mapping of the area, due to its relative novelty, there still is a lack of conceptual clarity. This is not surprising, since the pace of new academic output for this highly relevant topic is only exceeded by the number of technological breakthroughs it tries to exam-ine. Liikkanen (2019) describes that human-centred design will be crucial in further defending humans, par-ticularly underprivileged users at risk of being mis-treated by AI, while Yvonne Rogers calls HCAI ‘the new zeitgeist’ (2022). Different authors and professions construe their conceptions into various, maybe similar yet still diverse meanings.

Even if such a lack of consen-sus is accepted, the question remains if HCAI just describes the combination of Human-Centred Design and Artificial Intelligence, or if it constitutes something greater than the sum of its parts.

Against the backdrop of ambiguity and the grow-ing attention and research calls towards the adoption of HCAI, the objective of this Systematic Literature Review (SLR) is to contribute to HCAI conceptual clarity by answering the following research question:

RQ: How can Human-Centered Artificial Intelligence be defined, conceptualized, and implemented?

To answer this overarching research question, three sub-questions will be answered:

SRQ1: How is Human-Centered AI defined in the exist-ing literature?

SRQ2: What are the elements constituting Human-Centered AI systems?

SRQ3: What are the processes, methods, and tools for the development and deployment of Human-Centered AI?

The remainder of the paper is structured as follows. First, the conceptual background of Human-Centred Design and Artificial Intelligence which serve as the defining constructs of this research is presented followed by the research method. Then, the findings of the literature review are presented in three parts in accordance with the three sub-questions. First, the answer to SRQ1 is pro-vided by examining how HCAI is defined in the litera-ture and proposing a novel comprehensive HCAI definition. In the second part of the findings, we analyze the literature on HCAI constituent elements of ‘purpose’, ‘values’ and ‘properties’ (SRQ2). In the third part, we identify processes, methods, and tools for HCAI devel-opment and deployment (SRQ3). Before concluding, we discuss our findings and provide a research agenda for future research to advance the HCAI discourse.

2. Background.

2.1. Human-Centred design.

Human-centred design (HCD) is commonly referred to as a creative approach to problem-solving that starts with understanding the people you’re trying to reach and designing around their needs and perspectives. The term ‘human-centred design’ is often used synony-mously with the idea of ‘user-centred design’; however, HCD emphasises that it impacts a number of stake-holders, not just those typically considered as users. IDEO, a design and consultancy company that is often attributed as one of the organisations coining the term, describes human-centred design as being about cultivating deep empathy with the people you’re design-ing with, generating ideas, building different prototypes, sharing what you’ve made together and eventually put-ting your innovative new solution out in the world (IDEO.org 2023).

A similar notion of Human-Centred Design is brought forward by the Stanford d.school, describing HCD as a process, mindset, and approach to identify meaningful challenges and creatively solve complex problems. HCD as an approach guides under-standing and responding to the needs of specific people, questioning assumptions and reframing problems, and experimenting to advance their solutions. Human-Centred Design has its roots in fields such as ergonomics, psychology, computer science, anthropology, engineering, and arts. The echoes of this historical origin can be noted in the inter-national standard ISO 9241-210:2019: ‘Ergonomics of human-system interaction – Human-centered design for interactive systems’.

There, HCD is defined as an ‘approach to systems design and development that aims to make interactive systems more usable by focusing on the use of the system and applying human factors/ergo-nomics and usability knowledge and techniques’.

Each of these definitions highlights the importance of putting people at the centre of the design process and understanding their needs and perspectives to create effec-tive and meaningful solutions. According to the ISO stan-dard, using a human-centred approach to design and develop interactive systems has substantial economic and social benefits for users, employers, and suppliers. Using a human-centred foundation in the design and develop-ment of systems, services, and products can increase the productivity of users, as well as the operational efficiency of organisations. It can help to reduce training and support costs, improve accessibility and user experience, reduce discomfort, and stress, and contribute towards sustainabil-ity objectives as well as well-being (ISO - International Organisation for Standardisation 2019).

2.2. Artificial intelligence.

Artificial Intelligence (AI) has been studied for decades and is still - today maybe more than ever – one of the most elusive subjects in contemporary debates. The term itself was coined at the Dartmouth Summer Research Project on Artificial Intelligence by John McCarthy in 1956, where AI was described as the ability of a machine or computer program to learn and think’. However, even almost 70 years later, there is still no officially agreed definition of Artifi-cial Intelligence. Various scholars have attempted to define AI. Bellman (1978) describes AI as the automation of activities we associate with human thinking (i.e. cog-nitive activities). According to Poole and Mackworth (2010), AI is the field that studies the synthesis and analy-sis of computational agents that act intelligently.

Accord-ing to Rai, Constantinides, and Sarker (2019), AI refers to machines performing the cognitive functions typically associated with humans, including perceiving, reason-ing, and learning. The OECD (2019) defines AI as a machine-based system that is capable of influencing the environment by producing an output (predictions, recommendations, or decisions) for a given set of objec-tives. It uses machine and/or human-based data and inputs to (i) perceive real and/or virtual environments; (ii) abstract these perceptions into models through analysis in an automated manner (e.g. with machine learning), or manually; and (iii) use model inference to formulate options for outcomes. AI systems are designed to operate with varying levels of autonomy.

Demis Hassa-bis,theCo-founder andCEOof DeepMind, anAIresearch laboratory that serves as a subsidiary of Google, describes AI in a ‘The Guardian’ article as the science of making machines smart, and the art of making them come alive. The High-Level Expert Group on Artificial Intelligence (AI-HLEG) (2018) defined AI as systems designed by humans that act in the physical or digital world by perceiving their environment, interpreting struc-tured or unstructured data, reasoning on the knowledge derived from this data and deciding the best actions to achieve a given goal. AI systems can also be designed to learn to adapt their behaviour by analysing how the environment is affected by their previous actions. Further they also describe AI as a scientific discipline, AI includes several approaches and techniques, such as machine learn-ing, machine reasoning, and robotics.

Almost all the definitions and descriptions above mention humans or human traits, in one form or another. It becomes apparent that humans have a relevant and significant stake in AI technologies, which calls for a closer investigation of a human-centred perspective.

3. Research method.

For this systematic literature review, we applied the methodological framework by Kitchenham (2004), fol-lowing her structured literature review process. The three steps within the framework consist of planning-, conducting-, and reporting the review. In the first step, we developed a detailed search protocol, defining specific search terms as well as inclusion/exclusion criteria. In the second step, the review was conducted. This includes identification, selection, appraisal of quality, evaluation, and synthesis of the literature. In the last step, the findings of the literature review were summarised and reported.

For this literature review, database searches were conducted in the Scopus research database as well as the Web of Science database. The literature search cov-ers published research up to April 2024. Scopus is renowned for its extensive coverage across a wide array of disciplines. According to Mongeon and Paul-Hus (2016), Scopus boasts a larger journal coverage in all fields compared to other databases, making it an unparalleled resource for accessing a broad spectrum of scholarly work. This breadth was considered a crucial aspect to identify relevant literature across disciplines, including computer science, psychology, sociology, and design, among others. The ability of Scopus to provide a diverse set of literature ensures that our review compre-hensively captures the multifaceted nature of Human-Centred AI.

However, to ensure a comprehensive literature coverage, we also conducted a database search in the Web of Science (WoS) database, which identified additional lit-erature that was not covered by the Scopus search.

We conducted searches with the search string: TITLE-ABS-KEY (‘human cent AI’ OR ‘human cent artificial intelligence’) without any further parameter or criteria. By this, we ensured the search did consider American English as well as British English spellings of the search terms. The search was not limited by a time frame since the goal of the study was to acquire a full overview of HCAI literature until this point in time. Therefore, a time limitation was not deemed expedient. At the same time, however, the search was limited to the explicit men-tion of the terms ‘human-centred AI’ or ‘human cent AI’. This limitation excludes literature that discusses related concepts but uses different terminology. This was a deliberate search strategy choice, as we see this lit-erature analysis as a building block toward a HCAI-specific conceptual unraveling.

The number of publi-cations concerning AI-related topics is constantly increasing, using a vast number of different concepts and terminology. A clear distinction between these terms and understanding is key to facilitate further struc-tured discourse. With this broad approach, we aimed to catch the largest number of resources on the specific topic of human-centred AI, without diluting the results and focus on the overarching goal of the study. To ensure a high degree of relevance in the literature review corpus, the following exclusion criteria have been defined before the initial search and screening phases:

. Topic overviews not related to a conceptual under-standing of AI.

. Studies discussing purely technical improvements

. No AI relation, or AI only as an auxiliary aspect of the research

Table 1 provides an overview of the process that was followed.

The selection process was performed on database exports in the form of a spreadsheet, including infor-mation about Authors, Title, Year, Source, Abstract, and Keywords. In the first screening stage, all abstracts from the initial list of 736 sources were read, which eliminated 496 sources as they matched the exclusion criteria. In the second screening phase, the remaining 240 sources have been fully read and assessed according to their suitability for the literature review. A total of 164 eligible, non-duplicate documents related to HCAI were identified.

The analysis consisted of two main parts. In the first part, we examined whether a paper includes a definition for HCAI and if yes, if it reuses a pre-existing definition of HCAI or if it introduces a new one. If existing definitions were used, the respective references were marked in the spreadsheet. This coding was performed for all the papers in the corpus analysed. This led to the identification of pat-terns and groupings within the literature, identifying the most used definitions (Table 2), various combinations of definitions, and common concepts within the different definitions as well as the discovery that a significant num-ber of publications don’t use a definition at all.

In the second part of the analysis, we utilised the principles suggested by Webster and Watson (2002), alongside the grounded theory approach by Wolfswin-kel, Furtmueller, and Wilderom (2013), to sort, analyse and synthesise the included articles. According to Web-ster and Watson’s principles, we followed a structured, iterative, and concept-centric approach to identify core concepts, merge similar ones, and synthesise over-arching concept groups. Such conceptual reviews are particularly well-suited for focusing on a single concept and examining how the concept and its core attributes have been defined in the literature (de Guinea and Paré 2017). Furthermore, a grounded approach pro-vided a robust framework for open, axial, and selective coding to systematically refine the conceptual matrix and capture emergent themes.

Open coding allowed us to identify a broad range of concepts creating a prelimi-nary conceptual map. During the axial coding we grouped the identified concepts into broader categories, and interrelated subcategories. Selective coding was then employed to integrate the identified concepts into cohesive themes, which link back to the overarch-ing research questions and goals. The goal is to provide a comprehensive and balanced understanding of the existing research and its implications. Based on the analysis we created a concept matrix (Appendix 1) that provides an overview of the literature reviewed.

4. HCAI definitions in the literature.

A key premise of HCAI is that by placing human beings at the centre, it is possible to create AI systems that are more inclusive, trustworthy, and aligned with human values and goals. However, a widely agreed-upon definition of HCAI has not yet been reached. Different researchers from various disciplines have attempted to formulate their perspectives on HCAI introducing different definitions. Table 2 provides an overview of definitions HCAI found in the literature, highlighting their variety of emphases and, their commonalities.

4.1. Definitions in conjunction.

Several of the papers reviewed, combine more than one definition of HCAI. For instance, Herrmann (2022) employs the HCAI definitions of Shneiderman (2020c) and the framework by Xu (2019) in research on interaction modes for promoting human capabili-ties. The identified interaction modes highlight both human and AI strengths. Examples include the pro-vision of explanations and possibilities for exploration, testing, and re-training with human involvement and keeping humans in control by allowing for interven-tion and vetoing. Another example of a combination of definitions is the research by Yang et al. (2021). Yang and colleagues in their conceptual work on smart learning environments state that HCAI can be interpreted from two perspectives. The first is ‘AI control’, under human describing the interplay between human control and AI automation.

The other perspective is ‘AI on the human condition’, which relates to having explainable and interpretable computation and judgment processes and continuous adjustments of AI to societal phenom-ena.

4.2. Definition synthesis.

In his seminal work ‘Human-Centred AI: A New Synthesis’, Shneiderman (2021a) states that building AI-driven technologies that serve human needs requires combining AI-based algorithms with human-centred design thinking. According to the International Stan-dard ISO 9241-210:2019; 3.7 on Human-Centred Design, the extent to which products are usable and accessible depends on an understanding of the context of use, which includes the involved stakeholder groups, their tasks, and goals as well as the values and needs of these stakeholders. The context of use is described in ISO 9241-210:2019; 3.2 as the combination of users, goals, and tasks, resources, and environments. This component comprises multiple expedient aspects, high-lighting the importance of considering the plurality of stakeholders of a system as well the technical, physical, social, cultural, and organisational circumstances.

Cooley (2000) describes Human-Centred Systems as the appropriate context to leverage linguistic, cultural, and geographical diversity as a source of information.

Similarly, according to Riedl (2019), HCAI systems must be designed with the awareness that they are part of a larger system, which also encompasses human stakeholders. Stakeholders are individuals or organisations with a right, share, claim, or interest in a system or in its possession of characteristics that meet the individual or collective needs, goals, and duties (ISO 9241-210: 2019; 3.11). Dignum and Dignum (2020) describe that any task of an HCAI system is not done in isolation, but should be done for someone, in some context. The advent of HCAI systems also raises the general question of necessity and beneficial intents.

As part of the overarching goal for a human-centred AI future, the High-Level Expert Group on AI High-Level Expert Group on Artificial Intelligence (AI-HLEG) (2020), a group of experts appointed by the European Commission to provide advice on its artificial intelli-gence strategy, mentions that an HCAI system needs to intend to improve human welfare and freedom. Therefore, one defining factor within the context of use can be found in the consideration of the Purpose of the system.

Another crucial aspect within the context of use is the identification and comprehensive understanding of the existing needs and values of different stakeholders. They are a major source of information for establishing requirements and an essential input to the HCD pro-cess. Needs can be individual, considering the immedi-ate users of a system, as well as collective. The development of HCAI systems entails the identification of social values, deciding on a moral deliberation approach, and linking values to formal system require-ments and concrete functionalities. The Stanford Institute for Human-Centred Artificial Intelligence includes the addressing of societal needs in their HCAI definition. And according to Dignum and Dignum (2020), an HCAI perspective enables deliberate considerations of whether the actions of an AI system affect people directly or indirectly.

The HAI framework by Xu (2019) uses the term ethically aligned design, meaning AI solutions that avoid discrimination, and maintain fairness and justice. Similarly, Holzinger et al. (2022) defines HCAI as a synergistic approach to aligning AI solutions with human values and ethical principles. By considering these attributes and by aiming to make beneficial contri-butions to human well-being as well as organisational and system performance, HCAI systems are driven by stakeholder Values.

Further dimensions within HCAI that require close attention are the interactions among humans and other elements of a system. Within the ISO standard, this is described as human factors and ergonomic con-siderations. It concerns the extent to which a system, product, or service can be used by humans to achieve their goals with effectiveness, efficiency, and satisfaction in a specific context of use, and can be understood as the result of the combination of users, goals, and context of use (ISO 9241-210:2019; 3.13). To achieve a usable

HCAI system, there are different Properties that can contribute to this goal. Xu’s HAI framework includes human factors design to ensure that AI solutions are explainable, comprehensible, useful, and usable.

Finally, Human-Centred Design practices, processes, and tools, are building the backbone of human-centred design. They entail continuous and active involvement of users throughout the design and development (ISO 9241-210:2019; 5.3) to maintain a clear understanding of user and task requirements (ISO 9241-210:2019; 5.2). The ISO standard states that the involvement of users provides a valuable source of information, and that the effectiveness of involvement increases as the level of interaction between developers and users increases. According to Auernhammer (2020), the par-ticipation of stakeholders in the design and implementation of HCAI systems can provide essential perspectives in the re-design of social organisation arrangements such as work systems.

Including many diverse groups throughout the design and development process allows the design of the AI system to be acces-sible and meaningful for a wider group of people. User-centred evaluation (ISO 9241-210:2019; 5.4) and iterative processes are essential, as many of the needs and expectations of users and other stakeholders that will impact the design of the interaction only emerge in the course of development, as the designers refine their understanding of the users and their tasks, and as users are better able to express their needs in response to potential solutions (ISO 9241-210:2019; 5.5).

4.3. A new HCAI definition.

The literature review reveals significant conceptual overlap among the multiple different definitions ident-ified. At the same time, the review also highlights the diversity in emphases and approaches toward an under-standing of what Human-Centred Artificial Intelligence could entail. Based on the analysis of the identified exist-ing takes on HCAI, and a synthesis with the ISO definition of Human-Centred Design, we propose a new comprehensive definition of HCAI to encompass the richness of the scholarly understanding and a deeply ingrained Human-Centred Design core:

Human-Centered AI (HCAI) focuses on understand-ing purposes, human values, and desired AI properties in the creation of AI systems by applying Human-Centered Design practices. HCAI seeks to augment human capabilities while maintaining human control over AI systems, by considering the necessity, context, and ethical and legal conditions of the AI system as well as promoting individual and societal well-being.

5. The elements of HCAI in the literature:.

purposes, values, and AI properties

In this section, we answer SRQ2 ‘What are the elements constituting Human-Centred AI systems?’ The HCAI definition suggested in 4.2.1 serves as an analytical framework for the further analysis of the literature. It provides a taxonomy for classifying HCAI concepts into: ‘Purpose’, ‘Values’, and ‘Properties’ (Figure 1).

5.1. Purpose.

Considering the purpose of use of a product, service, or system is a crucial aspect of Human-Centred Design. The same is true for Human-Centred AI systems. These are fundamentally socio-technical systems that include the social context where they are developed, used, and acted upon. Taking the variety of stakeholders, institutions, cultures, norms, and spaces into account is fundamental to recog-nising that the technical component cannot be separ-ated from the socio-technical system when considering the effects and the governance of AI tech-nology, or the artifact that embeds that technology. A similar understanding is put forward by Riedl (2019), stating that HCAI systems must be designed with the awareness that they are part of a larger system consisting of human stakeholders, such as users, operators, clients, and other people in close proximity.

A common differentiation within HCAI exists between intelligence imitation and intelligence amplifi-cation. Intelligence imi-tation is often the cause of fear and dystopian perspectives, whereas intelligence amplification takes a more practical, human-centred side. Where ‘intelligence imitation’ considers the ideas of AI autonomy and automation, ‘intelligence amplifica-tion’ carries the notion of human augmentation and enhancement of human capabilities. Although the lines are not always clear-cut, we suggest that the differ-entiation between Augmentation, AI-Autonomy, and Automation provides a useful taxonomy for defining factors when thinking about the purpose or motivation of the technology (Figure 2; Table 3).

5.1.1. Augmentation.

The core objective of Augmentation lies in the idea of using AI technology to enhance human capabilities, rather than replacing or diminishing them. Intelligence augmentation is an idea originating in the works by Douglas Engelbart at the Stanford Research Institute as well as the Defense Advanced Research Projects Agency (DARPA) in the early 1950s and 1960s. Instead of viewing AI as a standalone tool, the idea of symbiotic cooperation between humans and machines is a fundamental belief of Human-Centred AI with the shared maxim of ‘augmentation instead of replacement’. This is based on the under-standing that technology is created with the purpose of supporting humans, not rendering them redundant. Ford et al. (2015) use the term ‘Cognitive Orthoses’ to describe the enhancement of human capabilities by AI technology.

Augmentation describes one of the core Human-Centred AI aims to augment human intelligence, skills, and decision-mak-ing processes, leading to improved performance and outcomes. Fischer (2021) states, that HCAI is closely related to the idea of intelligence augmentation by focusing on improving the quality of life of humans by creating systems that amplify, aug-ment, and enhance human performance in ways that make systems reliable, safe and trustworthy. The differentiation is still used in contem-porary HCAI literature. A study about the perceptions and perceived threat of AI technology towards their work conducted by Wang et al. (2019) within IBM with Data Science Specialists also discussed the conceptual ‘Augmentation vs. Automation’ approaches toward Data Science.

But in general, Augmentation is scope-agnostic, as it can take place in every field where the cooperation of humans and AI systems can augment the capabilities of each entity alone. Augmentation has been found in the medical field, policy-making, citizen science, education, automotive, medical affairs, finan-cial inclusion and other areas.

Another investigated facet within augmentation is the variable of human cognitive effort when interacting with the AI system. Gajos and Mamykina (2022) exam-ined that the level of cognitive engagement could be used to conceptualise human engagement with AI sys-tems. They propose to distinguish between passive engagement, meaning humans following AI recommen-dations without any reflection or assessment, and dee-per forms of engagement, critically examining these recommendations in relation to their own existing knowledge and judgment. A related issue that has been a subject of research is the willingness to use the AI system. Calisto et al. (2021) show that a high level of acceptance of AI techniques from radiologists can result in a significant reduction of cognitive workload and improvement in diagnosis execution.

One common and frequent use case for AI systems is in the Decision-Making context. Within decision-making, human capabilities can be enhanced by ‘Decision-Support Sys-tems’ where algorithmic systems are leveraged to pro-vide assistance to human actors. Weizenbaum (1976) made a distinction between making ‘decisions’ (by the computer) and making ‘choices’ (by a human user). Increasingly, decision-making algorithms are involved in decisions influencing multiple stakeholders in gov-ernment institutions, private organisations, and com-munity services.

5.1.2. AI autonomy.

Another purpose that has been identified in the Human-Centred AI literature is AI-Autonomy. It describes the capability of an AI system to make decisions and take actions without direct human intervention or control. In order to be fully autonomous, a system must have the capability to independently compose and select among different courses of action to accomplish goals based on its knowledge and understanding of the world, itself and the situation. AI Autonomy is often also understood as Intelligence imitation and often causes fear and dystopian perspectives – in contrast to intelligence amplification, which takes a more practical, human-enhancement side. Although seemingly counter-intuitive, AI-Autonomy can be a viable and sensible purpose for HCAI.

Some features and systems can be made auton-omous if they require rapid action, e.g. airbag deploy-ment, anti-lock brakes, pacemakers or implantable defibrillators. In these applications, there is no time for human intervention or control. Because the price of failure is so high, these applications require extremely careful design, extensive testing, and monitoring during usage at scale to refine designs.

However, Autonomy can also occur in the form of Human-AI collaboration. Human-AI collaboration refers to the interaction between humans and AI sys-tems with the objective of achieving common goals. The concept recognises that humans and AI have comp-lementary strengths and can work together to address complex challenges. On a human-AI team, the AI sys-tems can work autonomously alongside human counterparts as members of hybrid teams and carry out the fundamentals of teamwork and taskwork. As interest in human-AI interaction and collaboration is exploding within the HCI community and beyond, the concept of Human-AI collaboration, and its related concepts ‘Human-AI teaming’ and ‘Hybrid Intelligence’ are among the most commonly discussed concepts within the HCAI literature (Appendix 1).

Human-AI teaming (HAT) is fundamentally an emerging way of working with autonomous AI, which distinguishes it from Automation. HAT can occur in various forms and compositions, ranging from sensing, data collection, and processing to communication, con-trol, and monitoring. An appropriate allocation of func-tion between users and technology, should not be a simple matter of identifying what the technology can do autonomously and then allocating all other activities to the human. Instead, this decision should be based on several factors including the relative competence of technology and humans in terms of reliability, flexibility of response, and user well-being.

One Human-AI teaming constellation exists in sys-tems that use their computational power to generate information or recommendations by using algorithmic models which are trained on historical data and then present a generated output to human actors. This syner-getic cooperation can improve human performance, as well as algorithmic. AI can also use real-time human data as input into the system. Bacciu et al. (2021) put forward a human-centric perspective on cyber-physical system intelligence which leverages the physiological, emotional, and cognitive state of users as a driver for the adaptation and optimisation of an autonomous application for informative and depend-able teaching information. Prior research has also described ‘supervised autonomy’ and ‘assured auton-omy’.

These concepts describe autonomous systems where the performance is moni-tored by humans to enable them to intervene in a timely manner to ensure vital outcomes. Lundberg, Arvola, and Palmerius (2021) portray the role of human oper-ators as changing due to the increased intelligence and autonomy of computer systems. The interaction between humans and autonomous systems will happen at a more overarching level or only in specific situations, e.g. when necessary to ensure safe outcomes. This involves learning new practices and changing habitual ways of thinking and acting, including reconsidering human autonomy in relation to autonomous systems. But autonomy can also be used for creative purposes, such as co-creating public spaces, however, the level of creative autonomy that is appropriate for co-creative AI agents is heavily dependent on context.

Human collaborators tend to want more control over the behaviour and actions of co-creative AI when they are using AI as a perform-ance technology or creativity support tool. To avoid giv-ing rise to hostile attitudes towards advanced technology, He et al. (2022) identify four aspects of trustworthiness for trustworthy Robots and

Autonomous Systems (RAS) that need to be investi-gated thoroughly to promote the implementation of fully trustworthy RAS, namely Security, Safety, Health, and Human-Machine interaction. Schmidt (2020) raises the question of whether AI autonomy can lead to more consistent behaviour, and therefore to less variety (e.g. texts become more and more similar as keyboards suggest next words or provide template texts for responses).

5.1.3. Automation.

A third possible purpose of an HCAI system identified within the HCAI literature is Automation. Automation is not an AI-specific purpose since it can be achieved also without AI. However, the novel possibilities artificial unlocked by intelligence and especially, machine learning technologies, enable new forms of automation. According to Xu et al. (2023), automation refers to the outsourcing of repetitive, mundane, or time-consuming tasks that can be performed by AI sys-tems more efficiently. The demarcation between Auton-omy and Automation is rather blurry, as an autonomous system can also perform automated tasks. A useful distinction is provided by Shneiderman (2022), who states that systems governed by prescriptive rules that permit no deviations are automated, but not autonomous.

Xu et al. (2023) align with such differen-tiation by describing Automation as being deterministic whereas Autonomous behaviour as non-deterministic.

Frequently, the reason for automation, within an AI context but also beyond, is efficiency gain. In her study analysing B2B factory automation use cases, Heier (2021) identified optimisation as the main driver for AI implementation. However, within the context of Human-Centred AI, auto-mation is also discussed rather critically. Cheng et al. (2022) argue that Automation should be implemented with caution, especially in contexts with existing racial disparities and biases. Blindly automating decisions in high-stake, real-world social environments can exacerbate these issues. The authors show that human judgment can reduce racial disparities by allowing for productive dis-agreement with algorithmic recommendations.

There-fore, even partial or ‘soft’ automation should be carefully designed to ensure it does not discourage human intervention, which can play a crucial role in miti-gating bias and ensuring fair outcomes.

5.2. Values.

The second element of HCAI identified in the literature is Values. Values refer to fundamental beliefs and ideals that shape individuals’ and societies’ judgment and behaviour. Values serve as a guiding framework for individuals and groups, to define what is important, just, and meaningful. Developing HCAI systems entails processes such as identifying social values, deciding on a moral deliberation approach, and linking values to for-mal system requirements and concrete functionalities. A prominent example of how to work with and for values can be found in the Value-Sensitive Design approach by Friedman et al. (2013). Similar to Value-Sensitive Design, Renz and Vladova (2021) intro-duce Design for Values, which is based on a method-ology aimed at incorporating moral values as part of technological design, research, and development.

Within our analysis of the HCAI literature, three sub-categories of values emerged, namely ‘Ethical Values, ‘Protection Values’ and ‘Performance Values’ (Figure 3; Table 4).

5.2.1. Ethical values.

A common term that is used not only within the realm of HCAI but also much more broadly in the debate on AI technology is Ethical AI or AI Ethics. In 2011, Bos-trom and Yudkowsky (2011) concluded that although AI would offer only a few new ethical issues, which are not already present in the design of other technol-ogies, new challenges are foreseeable. No more than a few years later, this assessment seems almost like an understatement considering the incredible development of AI technologies and the related ethical challenges. According to Coeckelbergh (2020), the recent spectacu-lar breakthroughs in AI have created a sense of urgency on the part of ethicists and policymakers.

Bietti (2020) argues that ethics and morality in relation to technology are at risk of being instrumentalized, either by the industry in the form of ‘ethics washing’, or by scholars and policymakers in the form of ‘ethics bashing’. If ethics are instrumentalized as a performative façade, they are voided of their value and purpose. Van Wyns-berghe and Robbins (2014) state that ethical consider-ations need to be integrated at an earlier stage into the design process before a product or service is getting developed or even introduced. In their perspective, ethics ought to be pragmatic and the ethicist should be considered a designer in the process of technology development, who subscribes to a pragmatic view of ethics in order to bring ethics into the research and design of artifacts.

Another issue is that current analyses of AI in a global context are biased toward Western per-spectives and that there is a lack of research, especially outside the U.S. and Western Europe. For a more holis-tic approach to AI ethics, they need to be addressed with a cross-cultural understanding. Ethical AI promotes the development of AI sys-tems that adhere to ethical principles and guidelines. Therefore, it needs to be understood not as a specific value, but rather as an overarching goal to create ethi-cally sound technology. The idea of AI ethics has emerged as a response to the abundance of societal and individual perils for individuals and society, caused by the abuse, misuse, poor design, or unintended conse-quences of AI systems.

By incorporating ethical con-siderations into the design and implementation of AI systems, Human-Centred AI aims to safeguard against potential negative consequences and ensure that AI technology operates in a manner aligned with human values and societal well-being. In the following para-graphs, we elaborate on three key ethical values ident-ified in the HCAI literature: Dignity, Fairness, and Justice.

Dignity. Dignity describes the respect for the inherent worth and value of individuals throughout the develop-ment, deployment, and use of HCAI systems. It involves ensuring that AI systems uphold human rights and do not undermine or dehumanize individuals, promoting a sense of dignity and preserving the integrity of human experiences. As Shneiderman (2023) describes in the ‘107th NOTE on Human-Centred AI’ mailing list, science, engineering, design, and ethical foundations that are aligned with human values, social justice, and individual dignity will lead to technologies that advance human welfare and preserve the environment. Nagitta et al. (2022) found that irresponsible and unethical AI has shown the potential to abuse the dignity of people and distorts the value systems of the most vulnerable for whom AI solutions are intended.

In their study exploring the sociotechnical bases of human autonomy, Laitinen and Sahlgren (2021) describe autonomy as an important constituent of human dignity. In her paper titled ‘Technology as Infrastructure for Dehumanisation’, Oviatt (2021) states that respect for the autonomy of others is widely viewed as a societal obligation affording dignity, a basic human value and ethical principle. According to the EU Charter by the European Commis-sion (2012), the fundamental right of dignity is under-stood as a human-centric value which is an immutable principle that no AI system should break at any time in its life cycle.

Fairness. Fairness refers to the equitable treatment and absence of discrimination or bias in the design, development, and deployment of AI systems. It involves ensuring that AI systems do not dis-proportionately favour, or harm individuals or groups based on characteristics such as race, gender, age, or socioeconomic status, promoting equal opportunities and outcomes for all. The importance of fairness, not only in HCAI but in software development in general, received attention as a dedicated topic in software engineering research. Within the context of HCAI, Fairness is an important value in governing algorithms as they can perpetuate unfair treatment of different populations or stakeholders. Existing research on fair machine learning has primarily focused on fairness at the level of pre-defined groups.

This group fairness approach aims to fix a small collection of groups defined by protected attributes first, e.g. race or gender, and then asks for approximate equality of some statistic of the predictor, such as positive classification rate or false positive rate. Another approach to determining algorithmic fairness goes beyond pre-defined groups by using diverse human inputs in the design of algorithmic models. Since not all fairness prin-ciples can be guaranteed simultaneously, making tra-deoff decisions is necessary. To determine which fairness definitions an algorithm should use, Lee et al. (2019) found that human decision-makers need to be involved. This approach improves both procedural fair-ness and the distributive outcomes of algorithms, rais-ing participants’ algorithmic awareness, and helping identify inconsistencies in human decision-making.

Justice. The increasing interest in topics on ethical, social, economic, and responsible design as well as human rights and social justice can be understood as a positive indication for those who wish to see HCAI applied for social good. Justice has been mentioned as a key aspect of AI Ethics in various AI Ethics guidelines. For instance, the information and com-munications technology company ‘Fujitsu’, specified Justice in their AI ethics principles, saying that the benefits of Al should be distributed widely to all people. Compensation for damages including provisions for recovery and relief should be stipulated. However, defining justice is not an obvious and straightforward undertaking. Qadir, Islam, and Al-Fuqaha (2022) remark that the adoption of ‘decolo-nial’ thinking has been suggested to minimise the hege-mony of any one group in directing the trajectory and development of AI.

They highlight the importance of asking critical ques-tions about where AI design and development is hap-pening, and who is doing it. To promote justice and beneficence, everyone, especially marginalised groups need to be included and given opportunities to influence the process, as this context has fundamental influences on power structures and culture embedded in a system.

5.2.2. Protection values.

Another group of values identified in the HCAI litera-ture is values that carry a protective meaning. Protection values such as Privacy, Safety, and Prevention of Harm concern the observation of fundamental human rights, the safeguarding of physical and mental health, as well as environmental protection (The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems 2019; United Nations General Assembly 2015). In the context of HCAI, they play a pivotal role in promoting and war-ranting trust, societal acceptance, and a beneficial impact of an AI system.

Privacy. Privacy considerations involve providing clear information about data collection, processing, and sharing and obtaining user consent. Human-Centred AI allows users to understand and manage their privacy preferences, empowering them to make informed decisions about sharing their data. HCAI recognises the importance of safeguarding individuals’ personal data. Privacy considerations involve imple-menting measures to collect, store, and process data in a manner that respects privacy regulations and individ-uals’ consent. Human-Centred AI strives to design AI systems that minimise the collection and use of person-ally identifiable information and ensure secure handling of data throughout its lifecycle. HCAI promotes the use of privacy-preserving techniques to protect sensitive information while enabling AI capabilities.

Techniques such as differential privacy, federated learning, and homomorphic encryp-tion allow AI systems to extract meaningful insights from data without compromising individual privacy. By adopting such techniques, Human-Centred AI strikes a balance between data utility and privacy pro-tection. Further, HCAI aims to prevent unintended dis-closure of sensitive or personal information. Privacy considerations involve carefully designing AI systems to avoid inadvertent leakage of private data through outputs, recommendations, or system behaviour. Measures such as anonymization, de-identification, and secure data handling protocols are employed to protect against unintended disclosure.

Safety. Another concept that emerged from the HCAI literature is Safety. It refers to the state or con-dition of being protected from harm, danger, or injury from AI. It involves taking measures and implementing precautions to minimise risks and hazards while an HCAI system is used. Safety measures focus on creating an environment or situation where potential harm is mitigated or eliminated altogether. Safety often encom-passes a broad range of aspects, such as physical safety, emotional safety, cybersecurity, occupational safety, and public safety. HCAI takes into account potential risks associated with AI deployment and seeks to mitigate them. Safety considerations involve identifying and addressing risks such as inappropriate decision-making, privacy breaches, or unintended consequences of AI systems.

Shneiderman (2020c), defines safety as one of the three key requirements featuring HCAI. He portrays a conceptual understanding that revolves around safety culture through business management strategies as one of three levels of organisational structures. This level includes leadership commitment to safety, hiring, and training oriented to safety, extensive reporting of fail-ures and near misses, internal review boards for pro-blems and future plans, and alignment with industry standards and accepted best practices. By incorporating different safety measures, such as rigorous testing, vali-dation, and risk assessments, HCAI aims to minimise harm and ensure safety.

Prevention of Harm. A similar value to Safety can be found in the idea of Prevention of Harm. Whereas Safety emphasises protective measures to ensure well-being which are often reactive in nature and are designed to respond to identified risks, Prevention of Harm centers on proactive actions to avert harm from occurring in the first place. In an HCAI context, Preven-tion of Harm is primarily focused on identifying the root causes, risks, or contributing factors that could lead to harm by the AI system and implementing strat-egies or interventions to address them. A term with a similar meaning that has been found in the literature is Risk Mitigation. This may involve implementing pol-icies, regulations, or educational programs to promote awareness, change behaviours, or improve HCAI sys-tems.

The High-Level Expert Group set up by the Euro-pean Commission, identified Prevention of Harm as one of four ethical principles in their Guidelines on Trust-worthy AI. Consequently, the recently agreed European Union’s AI Act adopts a risk-based approach to the regulation of AI systems, categorising them according to the level of risk they pose. This seeks to proactively mitigate risks and by that prevent harm. Because the nature of AI makes many of the implications of using AI as a new technology unknown, such a careful approach to defining the principle of Pre-vention of Harm as part of the basis of a European con-sumer protection policy is consequential. By implementing effective measures and adopting pre-ventive strategies, organisations, communities, and individuals can work towards creating environments that minimise the occurrence of harm and foster a cul-ture of well-being.

5.2.3. Performance values.

One of the most obvious and remarkable strengths of AI technology is the improvement of performance, whether by augmenting human capabilities, autonomous action, or automation. Within the reviewed literature we identified three performance values related to HCAI, namely Efficiency, Effectiveness, and Accuracy. Each of these values contributes to HCAI systems to maximise performance benefits for individ-uals and societies individually but also in interdepen-dence as they are also interconnected and mutually reinforcing. An efficient AI system can enhance effec-tiveness by delivering results promptly and optimising resource utilisation. An accurate system ensures that the outcomes produced are usable and meaningful, and by that also contribute to overall effectiveness.

While optimising efficiency can lead to faster and more economical processes, it should not come at the expense of accuracy. An effective AI system ensures that it reliably meets the intended objectives and pro-vides meaningful benefits to users while utilising resources efficiently. There might be cases where AI sys-tems increase capabilities but add risks, exhibiting bias or marginalising subgroups of people. An HCAI approach needs to aim for achiev-ing the right balance among these values. This balance is essential for HCAI, as it influences individual and societal benefits.

Efficiency. Efficiency plays a significant role in HCAI as it focuses on optimising the use of resources, time, and effort. Efficient AI systems help minimise waste, reduce computational costs, and streamline processes, leading to increased productivity and improved resource allocation. By prioritising efficiency, AI systems can enhance user experiences, minimise wait times, and enable faster decision-making, ultimately improving pro-ductivity and reducing unnecessary burdens on users. In the realm of HCAI, efficiency can be seen as a human-inspired objective. Efficiency refers to the ability to accomplish tasks or goals with minimal wasted effort, time, or resources. In the context of AI, efficiency often relates to the automation and augmentation of human tasks through intelligent systems.

Therefore, it is not sur-prising that efficiency has been found to be an important value in the HCAI literature across many different domains. In their study on infrastructure inspection, Kar-aaslan, Bagci, and Catbas (2019) demonstrate how collec-tive human–AI intelligence has greater efficiency than conventional or fully automated inspection. Bellet et al. (2021) aim to improve efficiency in human-machine transitions within vehicle automation and Fu et al. (2021) explored Human-in-the-Loop reasoning to improve the efficiency of conversational reasoning.

Effectiveness. Another performance value identified in the HCAI literature is Effectiveness. It is closely related to the value of efficiency but can carry different meanings. Effectiveness can pertain to the ability of AI systems to achieve their intended goals and deliver valu-able outcomes in general. For Morrison et al. (2021a) effectiveness is an important measure in their study on enabling meaningful use of AI-infused educational technologies for children with blindness. It is also used to evaluate the performance of explainability measures, although evaluating the effectiveness of expla-nations remains a challenging task. By focusing on effectiveness, AI systems can ensure that their outputs are impactful and valuable to users, gener-ating the desired impact and meeting expectations.

Effectiveness is a key performance value for HCAI sys-tems as it is a foundational requirement to create benefits for users and stakeholders. For instance, in their study on the impact of AI assistance on incidental learning, Gajos and Mamykina (2022) assessed the effectiveness of different recommendation and expla-nation combinations while Bove et al. (2022) explored the effectiveness of four different versions of an Explain-able AI (XAI) system, in the insurance context.

Accuracy. When it comes to accuracy – the degree to which AI systems produce correct and precise results – computational systems usually have a significant per-formance advantage. Especially deep learning systems, which are using multi-level neural networks, have been proven to exceed the performance of experts in different contexts. Accuracy is another essential requirement for an HCAI system, as it is a prerequisite for informed decision-making and the provision of precise rec-ommendations. In sensitive domains such as healthcare, finance, or government, accuracy is especially critical. Accu-racy also plays a vital role in maintaining user trust, as users rely on AI systems to provide reliable and trust-worthy information or assistance. Accuracy can also be understood as an enabler of other values, e.g. efficiency, effectiveness, or fairness.

However, even if accuracy is highly relevant, it is not always the most important aspect. In their study on Human-AI Collab-oration in Data Science, Wang et al. (2019) found a dis-crepancy between the primary goal of the evaluated system (AutoAI) – producing models with the highest prediction accuracy – and the data scientists’ goals in the real world. While sometimes the data scientist’s goal is to produce the most accurate model, other times the data scientist aims to understand relationships in the data. In those cases, models are developed as a means to uncover insights into the relationships between those features, rather than as an end in achiev-ing high predictive accuracy.

5.3. AI properties.

The third element of HCAI systems that has been ident-ified in the literature relates to ensuring specific AI prop-erties (Figure 4). Some of these properties concern oversight aspects, addressing questions around account-ability and responsibility. Furthermore, the property of controllability is considered key and is linked to mean-ingful human control over AI systems. Sustainability emerges as another crucial aspect, emphasising the far-sightedness of HCAI systems concerning environ-mental, social, and economic sustainability goals to ensure the longevity of HCAI applications. Other ident-ified properties concern comprehension.

These relate to making AI interpretable and understandable to users, stakeholders, and regulatory bodies whereas transpar-ency and traceability are also pivotal and concern dis-closure of data usage, algorithmic logic, and decision-making to build trust and minimise the risk of hidden biases or unfair practices. And finally, the properties of reliability and robustness relate to the consistent and dependable performance of HCAI systems under diverse conditions to instill user confidence and facili-tate safe AI deployment. Overall, each of the identified properties contributes to the overarching objective of human and societal well-being (Table 5).

5.3.1. Oversight properties.

The first type of properties is classified as Oversight Properties. In the context of HCAI, they relate to moni-toring, reviewing, and regulating AI systems and their impact. Through oversight, the design and development of HCAI systems can enable trust and beneficial impact.

identified Accountability. Various studies have Accountability as one of the important themes in the development of HCAI (Fjeld et al. 2020; Garfinkel et al. 2017; High-Level Expert Group on Artificial Intel-ligence (AI-HLEG) 2020). Xu et al. (2023) and Shnei-derman (2020a) tie the property of accountability to the dimension of human control in their HCAI and HAI frameworks. But within the literature, there are different perspectives on accountability. For some, accountability is understood as a form of responsibility. For others, accountability refers to the obligation of individuals or organisations to be answerable for their actions, decisions, and impacts related to the design, development, deploy-ment, and use of AI systems. Further, accountability can be closely related to liability focusing on sanctioning abilities when AI systems work in unacceptable ways.

To explore who and to what extent is accountable for the outcome of decisions in flexible autonomous sys-tems, Yazdanpanah et al. (2021) suggest accountability ascription methods for human-agent autonomous sys-tems as well as accountability reasoning tools as mech-anisms to provide explanations for outcomes. Such methods are expected to be expressive to reason about task coordination, delegation, and shared control in HCAI systems. Ahuja et al. (2020) explore how established software engineering practices could be leveraged to incorporate accountability into an HCAI develop-ment process.

Controllability. Human-Centred AI recognises the importance of human oversight and control over AI sys-tems. Controllability can be seen as a property to address the value of Safety, involving designing AI sys-tems that allow humans to intervene, override, or guide the technology when necessary. Controllability empow-ers users to have a level of control over AI systems, enabling them to set boundaries, define constraints, or customise system behaviour. This helps prevent AI sys-tems from making harmful or inappropriate decisions and ensures that humans retain ultimate responsibility over critical tasks and decisions. Cunningham and Kular (2019) demonstrate that their study participants desired the ability to manually override automatic actions for building and maintaining trust with the sys-tem.

He et al. (2022) state that no matter which level a system’s autonomy is at, humans should be able to interrupt the system, and human interaction should be built upon tangible and attentive user interface prin-ciples. Human judgment and decisions should take pre-cedence over those of Al systems. Humans maintain the right to decide over, to know the reason for, and to not obey any judgment, decision, instruction, or command given by Al. The right to entrust decision-making power to Al solely lies in the person concerned, and that power can be revoked at any time. As San-karan et al. (2020) urge, it is the responsibility of researchers, designers, and developers of HCAI systems to continuously reflect upon human aspects such as autonomy along the process.

Macher et al. (2021) are adapting the control strategy to the human in the loop, hence fulfilling the promise of a human-centric approach to driving automation where improvements of the machine itself depend on the human state. Laiti-nen and Sahlgren (2021) propose a multi-dimensional model of human autonomy, evaluating how HCAI sys-tems can support or hinder it, including the constitutive aspects of autonomy such as social recognition, self- respect, and exercise as well as contrast cases ranging from maldevelopment via misrecognition to lack of self-respect and heteronomous performance.

Responsibility. While there exists a conceptual over-lap between Responsibility and Accountability, both concepts also carry distinct nuances. Responsibility refers to the moral duty and obligation to act in con-sideration of individual and societal interests, rights, and well-being as well as the potential impacts and con-sequences of AI technologies on them. It includes addressing biases, promoting transparency, safeguard-ing privacy, and mitigating potential negative social, economic, or environmental effects. Responsibility in the context of HCAI systems entails acting in consideration of an ethical framework ensur-ing that the overall system can be trusted by society. Responsibility means that individuals, organisations, and AI system stakeholders act in a manner that upholds ethical principles and respects human rights throughout the AI lifecycle.

Although initiatives to gain public support for human rights and corporate social responsibility are helpful, technology decisions by software engineers, managers, and review boards must be guided by clear principles and actionable rec-ommendations. As Dignum and Dignum (2020) highlight, HCAI guidelines, principles, and strategies must understand HCAI systems as socio-technical systems. The focus should not only be on the AI artifact itself. It is the social component of the sociotechnical system that needs to take responsibil-ity. Responsible AI doesn’t mean attributing responsibility to machines for their actions and decisions, and in consequence, discharging people and organisations of their moral duty.

It rather requires responsibility from the people and organisa-tions involved for the decisions and actions of the AI applications and for their own decision of using AI in each application context. According to Riedl (2019), the term HCAI is used by some AI researchers and prac-titioners to refer to intelligent systems that are designed with social responsibility in mind. This demonstrates that responsibility is understood as a broader perspec-tive on the design, development, and deployment of HCAI systems.

Sustainability. The concept of Sustainability in AI focuses on long-term viability, considering environ-mental, societal, and economic impacts to ensure that AI systems align with sustainable development goals, including those related to food, health, water, education, and energy. Vinuesa et al. (2020) demonstrate that AI can have an enabling impact in fields such as agriculture, medicine, education, energy, circular economy, and more. But at the same time, AI can also be an inhibitor, e.g. by its high energy use, per-petuating and exacerbating biases and inequalities, or the misuse of personal information. Sustainability in HCAI also includes ensuring that AI systems are adapt-able and future-proof, capable of evolving to meet chan-ging needs and avoiding negative dependencies or lock-in effects.

Akula and Garibay (2021) include sustainabil-ity as one of the key goals in their definition of ‘Ethical AI for Social Good’, which postulates that the design, development, and deployment of AI systems need to prevent, mitigate, or resolve problems that negatively impact human life and the well-being and enable socially preferable and environmentally sustainable developments. This definition is conceptually very close to the understanding of HCAI. Relating to the concept of ‘Prevention of Harm’, Walz (2017) sees such a preventive approach as the basis for more sus-tainable development and use of HCAI systems to ulti-mately benefit humanity.

5.3.2. Comprehension properties.

Another type of properties for HCAI systems are Com-prehension Properties. The properties of this type share the objective of supporting and enhancing human understanding, decision-making, and control over AI systems. The HCAI property of being explainable con-cerns the general capability of a system to be explicable. Being intelligible ensures that AI outputs are presented in an understandable and accessible manner. The com-prehension properties constitute another key aspect of HCAI systems.

Explainability. The core idea of Explainability is meaning making, concerning the ability to make sense of the inner workings of an AI system. This becomes particularly important in situations when the under-standing of failures and unexpected AI behaviour is necessary, answering questions like ‘How does it work?’, ‘Why did it do that? and ‘What mistakes can it make?’. Doran, Schulz, and Besold (2017) assert that to achieve trustworthiness and an evaluation of the ethical and moral standards of a machine, explanations should provide insight into the rationale of the AI system and enable users to draw conclusions based on them. Bove et al. (2022) show that the contextualisation of explanations can improve user satisfaction and the user’s objective under-standing.

But as Gajos and Mamykina (2022) found in their study on incidental learning, it may not be sufficient to just include explanations together with AI-generated recommendations and assume that people engage carefully with the AI-provided information. When designing a HCAI system, the first step is to fully understand the context of use, i.e. users, stakeholders, and the environment of the system. Ehsan and Riedl (2019) propose a socio-technological approach in which the social and the technical part of an HCAI system co-evolve in an iterative process. They describe a cycle of altering the system to the needs of the user and evaluating the effect. Steels (2020) argues that HCAI, with its implications of explainability, transpar-ency, and robustness, will only be possible when AI comes to grips with meaning and understanding.

Shnei-derman (2020c) states that explainability, among others, is a helpful consideration for the development of general HCAI principles and his HCAI framework does lay a foundation for the development of such. Especially since generative AI (GenAI) technologies are advancing and maturing, applying established HCD methods can provide useful tools.

Intelligibility. Within the HCAI literature, Intellig-ibility, with its semantically close terms Interpretability and Understandability, refers to the property of an HCAI system to present its outputs or results in ways that enable humans to understand and interpret the reasoning, processes, and outputs. It focuses on the clarity and compre-hensibility of the information or insights provided by HCAI systems. Intelligibility enables human users to understand how the AI system arrived at its decisions, fostering trust, and facilitating effective collaboration between humans and AI. According to Akula and Gar-ibay (2021), the need for understandability is an essen-tial ethical concept in the development of HCAI systems. The authors argue that AI for Social Good initiatives should aim for explainability and intelligibil-ity in relation to the specific needs of the recipient group.

This consideration of user needs constitutes a tenet of Human-Centered AI. Böckle, Yeboah-Antwi, and Kouris (2021) argue that there is a clear need for a human-centered approach to helping users to under-stand the characteristics and output of the AI algorithm in order to address the AI black-box problem.

Transparency. The concept of Transparency relates to the openness and potential to have insight on the decision-making processes, algorithms, data sources, and intentions behind HCAI systems and can be described as an umbrella term encompassing several overlapping and associated concepts. It involves provid-ing understandable and accessible information about how AI systems operate, enabling human users to make sense of and evaluate the factors that influence AI-generated outputs, and fostering trust and account-ability. According to Fjeld et al. (2020), transparency is the assertion that AI systems should be designed and implemented in such a way that oversight is possible. In order to do this, an HCAI system must pro-vide explanations for outputs together with transpar-ency of the underlying system processes and data allowing stakeholders to inspect it as needed.

In a frame-work to assess system transparency, Hepenstal, Zhang, Kodagoda, et al. (2021) and Hepenstal, Zhang and Wong (2021) show that an understanding of context is crucial. This rhymes with the HCD requirement of understanding the context of use. Wang et al. (2019) show in their study on the perception of data scientists toward the use of AI systems in their work, that trans-parency is a major component of trust.

5.3.3. Integrity properties.

A third type of AI properties has been defined as Integ-rity Properties. These properties ensure that HCAI sys-tems operate consistently and dependably in a trustworthy manner. They address technical and struc-tural soundness, as well as the emergence of trust-worthiness by providing auditability features to review the alignment and adherence with human and societal goals. Considering and incorporating integrity proper-ties in HCAI systems is a crucial building block to build-ing trust and confidence in the use of AI technologies.

Reliability. Reliability refers to the consistency and dependability of AI systems to perform as intended. In Shneiderman’s understanding of HCAI, reliability emerges from appropriate software design. This includes continuous reviews via benchmark tests for data quality, bias, and testing to cater to shifting contexts. A reliable HCAI system consistently produces dependable outputs and main-tains appropriate performance across various con-ditions. Reliability is essential for building trust in AI systems, as users need to rely on the system’s outputs for decision-making or task completion. A reliable AI system ensures that it consistently meets user expec-tations, performs reliably under normal operating con-ditions, and avoids unexpected or unreliable behaviours.

The apparent poor dependability of AI in critical decision-making environments is one of the key causes of the low level of acceptance of and trust in new technologies. Thus, there is a need to demonstrate and inform the community of reliable approaches for AI and their benefits. Ozmen Garibay et al. (2023) call to expand the under-standing of usability in the context of HCAI by consid-ering the unique features and impacts of AI such as dynamic reliability.

Robustness. To handle unexpected or adversarial conditions, inputs, or situations and maintain perform-ance and effectiveness, an HCAI system needs to pro-vide a level of Robustness. The technical robustness of AI systems is a central integrity property. Ozmen Garibay et al. (2023), describe techni-cal robustness as the interoperable agility and resistance against attack, which is fundamental to sustaining the rapid evolution of any AI-enabled system. The technical robustness of HCAI systems is central to their reliability. While performing well in their main performance metrics, safety, robustness, and resilience metrics often remain open challenges. Therefore, some authors pro-pose to leverage established software engineering tech-niques to assess the technical robustness of AI systems against adversarial inputs. He et al. (2022) also discuss robustness in relation to data quality.

In the cyber security context, machine learning techniques represent potential sol-utions for the automation of cyber security but require correct and trustworthy data to be available. Further-more, the AI system could be hacked, to produce unex-pected consequences. They conclude that the robustness and security of HCAI systems and machine learning models need to be investigated in the system design.

Trustworthiness. The construct of Trustworthiness emerges from an interplay of multiple aspects. Accord-ing to the Ethics Guidelines for Trustworthy Artificial Intelligence (2019), an AI system can be considered trustworthy, if it lawful, meaning it respects all appli-cable laws and regulations; ethical by respecting ethical principles and robust, both from a technical perspective while taking into account its social environment. It involves demonstrating consistent performance, trans-parency, fairness, and accountability in the design, oper-ation, and outcomes of AI systems, and ensuring that a system aligns with stakeholder values and is legally com-pliant. Thiebes, Lins, and Sunyaev (2021) claim that individuals, organ-isations, and societies will only be able to realise the full potential of AI if trust can be established in its develop-ment, deployment, and use.

Francesca Rossi (2018), a member of the European Commission’s High-level Expert Group on AI (AI-HLEG) claims that to fully gauge its potential benefits, a system of trust needs to be established, both in the technology itself as well as in those who produce it. Glikson and Woolley (2020) state that the success of integrating AI into the organis-ational context critically depends on workers’ trust in AI technology. They define Trust as particularly relevant to human-AI relationships because of the perceived AI risks due to the complexity and non-determinism of AI behaviours. According to Xu (2019), reflexive skepti-cism can affect users’ trust and decision-making efficiency, which in turn will also affect the adoption of AI solutions. However, there are also critical voices regarding the idea of trustworthiness in relation to

HCAI. Ferrario, Loi, and Viganò (2020) are worried that the perceived reliance on trust is not providing a full picture of the challenges of HCAI adoption. They pro-pose an incremental model of trust that can be applied to both human-human and human-AI interactions. Ryan (2020) even argues that AI doesn’t have the prop-erties to be trusted since it doesn’t possess any emotive states and cannot be held responsible for actions. He suggests changing the term or removing it altogether, as trust can only occur between trusted parties, whereas AI is just a systematic group of techniques.

Traceability. Traceability, and the related terms Audit-ability and Falsifiability, refer to the ability to track and understand the decision-making processes, data sources, and actions of an HCAI system. It involves capturing and documenting information about the inputs, and algorithms via audit trails to examine system failures and unexpected behaviour. Traceability enables users, stakeholders, or regulators to understand how AI decisions were made and to identify potential biases, errors, or ethi-cal concerns. Traceability also supports accountability and the ability to investigate or rectify any adverse outcomes of unintended consequences arising from the AI system’s deployment. Ensuring audibility can facilitate the identifi-cation of risks before AI gets deployed in production and causes harm.

Bringing all stakeholders together through appropriate technology can improve traceability, which is an important goal of HCAI. In a workshop to define an HCAI framework and research agenda, Degen and Ntoa (2021) defined traceability as an important research direction and proposed Human-Centred Design together with cause–effect analyses as potential approaches to research the concept further.

6. HCAI processes, methods, and tools in.

the literature

The last part of this literature review answers SRQ3: ‘What are the processes, methods, and tools for the devel-opment and deployment of Human-Centered AI? The answer to this question provides insights into the Human-Centred Design practices that are part of the HCAI foundation. This work uses the ISO 9241-210:2019 standard on HCD, particularly the principles of human-centred design defined in sections 5.2–5.5 to structure the related HCAI literature. The ISO stan-dard is not tied to a specific design process or method-ology. Rather, it provides a more general human-centred perspective that can be integrated into various design and development approaches in context-appro-priate ways (ISO 9241-210:2019; 5.1).

A common perspective on HCAI processes, methods, and tools in the literature is the relevance of the HCI discipline. The authors highlight specifically the intersection of AI and HCI and how HCI approaches and methods can be applied to ensure Human-Centred AI develop-ment. A similar proposal has been also brought forward by Harper (2019), claiming that ‘the future is HCI, not AI’. Capel and Brereton (2023) found that the design of HCAI systems has been approached through various methods, including participatory design, focus groups, interviews, usability studies, and observations.

There exist already multiple methods and tools for the design of HCAI systems. Some of them are previously established methods that are transferred and applied to the AI context. However, there are also new processes and methods being specifically developed for the design and development of HCAI systems. Ozmen Garibay et al. (2023) ask whether existing HCI design methods and processes scale up to accommodate a wide variety of users’ charac-teristics and contexts of use in order to create AI-enabled systems that are universally accessible and uni-versally usable? In their work on new challenges and opportunities for HCI professionals to enable HCAI, Xu et al. (2023) highlight that the context of use of single user-artifact interaction needs to be expanded to study the impacts of complex ecosystems of artifacts, services, and data in a distributed contexts of use.

Overall, we identified two key HCD principles that are important for HCAI: (a) continuous user involvement and (b) an iterative process for understanding, design and develop-ment, and evaluation (Figure 5).

6.1. User involvement.

Continuous user involvement constitutes the procedural bedrock of HCAI. In a research panel discussing the role of HCI in the conception, design, and implementation of HCAI, Schmidt (2020) state that HCAI emphasises the human side of the interaction between people and AI and focuses on creating a positive impact on human users and society, rather than highlighting algo-rithmic performance. In section 5.2, the ISO 9241-210:2019 standard defines that a human-centred design needs to be based upon an explicit understanding of users, tasks, and environments, i.e. the context of use. Products, systems, and services should be designed to take account of the people who will use them as well as other stakeholder groups, including those who will be directly or indirectly affected.

For Dignum and Dignum (2020), the idea of a human-centred system design is manifested in having humans always in focus for deliberation. This means that an HCAI system should not be understood in isolation, but that it acts for someone, in a specific context of use.

To gain a deep understanding of their values and needs, the involvement of all concerned stakeholders is inevitable (ISO 9241-210:2019; 5.3). To grasp the vast complexity of an algorithmic system, the continuous inclusion of the people who are and will be affected by the envisioned HCAI system is crucial. By involving people who have the capabilities, characteristics, and experience that reflect the range of stakeholders it is possible to identify their values and needs for certain properties, which are necessary elements in the HCAI framework. Cheng et al. (2021) propose a framework for eliciting stakeholders’ subjective fairness notions. Lee et al. 2019, created a collec-tive participatory framework that enables people to build algorithmic policies for their communities.

Andersen, Gjølstad, and Mørch (2022) applied end-user development and domain-oriented design environments to ensure user involvement.

Stakeholder participation can help operationalise moral values and their associated trade-offs, such as fairness and efficiency. As Cooley (2000) states in his chapter on HCD, Human-Centred Systems reject the notion of ‘one best way’ and he suggests forms of technology that are culturally specific. Riedl (2019) proposes to differentiate between systems that understand humans in relation to sociocultural context and meaning, as well as aspects that help humans to understand the system and how both natures affect the goals of such a system. A great emphasis is put on the importance of diversity, providing a motivation to reflect and enhance cultural, educational, and pro-duct diversity. Degen and Ntoa (2021) identified the diversity of end users and context as one of the key challenges to trust in AI systems, which they define as a requirement for HCAI.

It regards the social and cultural shaping of technology as central to the design and development of future technological systems and society as a whole. Specific attention needs to be paid to recognising the needs and values of marginalised communities to avoid reinforcing existing biases and social imbalances. HCD methods should rely on and elevate the lived experiences and needs of end-users, particularly users who have been historically excluded. Input from these users can inform inclusive data definitions and mitigate concerns of failing to capture and account for important nuance and complexity. A practical example of applying a human-centred practice in the context of AI is shown by Waschull and Emmanouilidis (2022), where via co-cre-ation methods the integration of an AI system in a manufacturing context was ideated and developed together with the affected workers.

Long, Jacob, and Magerko (2019) used co-creation methods in the con-text of AI for public spaces, engaging a broad range of participants to broaden public AI literacy.

To assess whether a proposed HCAI system addresses identified needs, adheres to elicited values, and provides the necessary properties, evaluation is key. Evaluating HCAI systems against real-world scen-arios with stakeholders and improving them based on feedback provides an effective means of minimising the risk of a system not meeting stakeholder needs (ISO 9241-210:2019; 5.4). Desolda et al. (2022) present SERENE, a platform that uses AI technology for semi-automatic UX evaluation. The AI model collects user data automatically and translates them into emotional predictions presented to the UX expert. A primary issue with current AI development is the significant lack of understanding about how AI works in the real world and a further lack of understanding about how AI works in the Global South.

With AI applications increasingly adopted for use in low-resource environ-ments, it becomes even more important to test and evaluate these technologies with the communities and in the environments in which they are expected to be integrated. In cases where AI is evaluated in medical contexts, previous work has found significant issues of bias and irreproducibility. This raises serious concerns about the state of ML research in healthcare, and researchers should be wary of deploying AI-enabled technologies in environments that have limited regulat-ory oversight.

In line with ISO 9241-210:2019, 5.7, HCAI research needs to integrate diverse views including computer science, psychology, and more to be able to understand the context of use as well as the situated values and needs of stakeholders in the appropriate levels of breadth and depth. However, studies show that within companies, real end-user involvement still doesn’t seem to be common practice in AI development. Participatory design methods show promise, but the associated expense may inhibit their use especially in contexts with few resources.

6.2. Iterative process.

The second HCD principle identified in the HCAI lit-erate is Iteration. Continuous iteration helps to pro-gressively eliminate uncertainty during the development of HCAI systems. It implies that descrip-tions, specifications, and prototypes are constantly revised and refined when new information is obtained in order to minimise the risk of the system under devel-opment failing to meet user requirements (ISO 9241-210:2019; 5.5). Iteration allows the designers and devel-opers to refine their understanding of the involved sta-keholders since many of the values and needs of stakeholders will only emerge during the design and development of an HCAI system. In their five-step HCAI process, Cooke, Demir, and Huang (2020) define the development of a synthetic task environment with Wizard of Oz capability as well as human subject experimentation to ensure iterative evaluation and learning.

Such evaluation allows preliminary design sol-utions to be tested and the results to be fed back into the design and development process. Through iterative design and testing, Yang et al. (2018) discovered that designers found it more effective to work in collabor-ation with data scientists, rather than become machine learning experts themselves. Calisto et al. (2021) applied iterative HCD principles to explore the behaviour of clinicians when an AI module is present in a diagnostic breast cancer system and identify a high level of accep-tance of AI techniques from radiologists.

Iterative design and development are established practices in software development, which can be adapted to the context of AI. A key concept for an iterative HCAI development is the idea of proto-typing. The ISO standard 9241-210:2019; 3.9 defines a prototype as a representation of all or part of an interac-tive system, that, although limited in some way, can be used for analysis, design, and evaluation. Evaluating rough prototypes and mock-ups of HCAI systems helps to obtain a deeper understanding of user needs, as well as providing initial feedback on the design con-cepts. These activities can also be applied during revi-sions to an interactive system and can be useful in evaluating systems in routine operation (ISO 9241-210:2019; 7.1). Due to the learning and evolving charac-ter of AI systems, iteration becomes especially impor-tant, yet considerably different to prototype for.

Subramonyam, Seifert, and Adar (2021a) explored the use of AI technology as a prototyping tool that enables designers to get a better understanding of potential model outcomes, as well as allows for direct manipu-lation of the model inputs via an API. Costabile et al. (2022) explore how they can support designers in creat-ing AI-based systems by conceptualising three strategies of interaction with AI systems, namely Clarification, Negotiation, and Reconfiguration. However, also exist-ing usability heuristics are still applicable in an AI con-text. He et al. (2022) found, that the two principles of Norman‘s seven principles for HCI design ‘Design for Error’ and ‘When all else fails, standardise’, should be applied to ensure the system keeps the bottom line of safety and reliability.

7. Discussion.

The growing interest on HCAI signifies a shift towards AI systems that prioritise human needs and values. This paper makes a strong argument that established methods from the field of Human-Centred Design can be leveraged to face and address the novel challenges of contemporary AI technologies providing a compre-hensive and systematic overview of prior research in the domain. The paper contributes to conceptual clarity definition, around the conceptualisation, and implementation of HCAI suggesting an integrative HCAI framework and identifies areas for further research to advance knowledge on creating AI systems that genuinely benefit humanity.

7.1. Integrative HCAI framework.

This section provides an integrative HCAI framework synthesising all findings from this review. The integra-tive HCAI framework brings together the strong Human-Centred Design foundation with the three key AI elements identified: Purpose, Values, and Proper-ties – to help guide AI design, development and use for benefitting humanity. Purpose focuses on defining clear objectives for the AI system. Having a well-defined purpose helps ensure the system is developed and applied appropriately. Values are the ethical, pro-tection and performance related considerations that should be respected during design, development and use of the AI. Core values like privacy, fairness, safety need to be embedded from the start. Finally, AI Proper-ties relate to the system capabilities for enabling over-sight, comprehension and integrity.

The framework utilises iterative Human-Centred Design cycles of understanding, designing and developing appropriate solutions, and evaluating. Most critically, the framework emphasises giving people a voice throughout AI lifecycles.

One key finding of this literature review is that there is a lack of conceptual clarity around the HCAI term. This is probably due to its novelty and the rapidly evolving nature of AI technologies. We suggest a new comprehen-HCAI definition sive clarifying the concept. The definition indicates that HCAI is rooted in understanding and shaping purposes, values, and desired AI properties by applying Human-Centred Design practices. In the previous sections, we mapped out different aspects of these three main elements of HCAI. Purposes include augmentation, AI autonomy, and automation. Values relate to ethics, safety, and performance. Properties cover oversight, comprehension, and integrity.

The paper also identifies and dissects the key Human-Centred Design practices, which foreground the continuous user involvement and iterations of understanding, design and development, and evaluation needed for HCAI.

The very first appearance of the term HCAI can be found in Garcia (1999), exploring how human-centred perspectives can address systems complexity. Another early occurrence of HCAI was found in the 2015 article ‘Cognitive Orthoses: Toward Human-Centred AI’ by Ford et al. (2015). It reflects a human-centred vision for applied AI that is less about intelligence imitation and more about amplified intelligence. The terms cogni-tive orthoses, or cognitive prostheses, are meant as metaphors describing the enhancement of human capa-bilities. A technological device, whose purpose is to improve the life of its users, by augmenting, enabling, and enhancing their abilities, while acknowledging the importance of understanding them. But although the term Human-Centred AI was new, the idea was much less so.

Since the beginning of the AI discipline, the con-cept of intelligence augmentation has always co-existed as a perspective on potential directions of development. Douglas Engelbart’s vision for the use of AI technologies evolved at the same time as the gen-eral discourse on AI, making a strong argument for keeping the human in the loop and for augmenting the human intellect rather than for automation. However, we argue that despite augmentation being identified as the purpose most associated with HCAI purposes, AI autonomy and auto-mation are equally valid purposes for HCAI systems. Degen and Ntoa (2021) state that it becomes clear that the distinction between intelligence imitation and intel-ligence amplification is in many cases no longer helpful.

The questions shift to how agency and control are implemented and how this is manifested in the inter-action between humans and smart technologies and tools. AI technologies provide means for the augmentation or automation of cognitive and perceptual processes which makes the border between augmentation, autonomy, and automation rather blurred. Therefore, the initial requirement for an HCAI system is not to be confined to a specific type of purpose but to acknowledge the existence of different types of purposes and deliberately define a purpose in the first place. A deliberate consideration of a certain purpose will manifest a system’s raison d’être and pro-vide an anchor point to steer the further process, the eli-citation and evaluation of values, and guide the exploration to understand a context of use.

AI systems are socio-technical systems; the social context of these systems’ development and use needs to be a fundamental consideration (ISO 9241-210:2019; 5.6). This means, that for a Human-Centred approach to AI, the technical component cannot be sep-arated from the rest of the socio-technical system. This understanding of the context of use, its stakeholders, and their values is essential for a human-centred design and development of AI systems (ISO 9241-210:2019; 5.2). A multidisciplinary approach involving experts from various domains, including social sciences, is necessary to inform the development of AI systems and their introduction to use. Inclusive stakeholder involvement is essential for identifying values and needs, particularly those of marginalised communities, to prevent the reinforcement of biases and social imbalances.

Recognising that no AI system will be perfect from the outset, continuous evaluation and iteration are fundamental. Unintended conse-quences and flaws may arise, making it vital to be open to investigation, feedback, and criticism. Providing secure and safe sandbox environments for testing and evaluating AI systems enables transparent design and iterative assessment of the identified values and proper-ties. The effectiveness of stakeholder involvement increases as the interaction between different roles intensifies.

This literature review identified three types of values for HCAI systems – Ethical, Safety, and Performance. However, the found values within the different types should only be understood as pointers, rather than normative guidelines. While Performance Values like Efficiency have established definitions and ways to be measured, Ethical values like Dignity, Fairness, and Jus-tice are immensely context-dependent and can carry different meanings in different environments. This paper can only provide a small range of examples for each value, emphasising its relevance, but claims by no means to be exhaustive. To fully understand what these values mean in a specific context of use, close interaction and collaboration with all involved stake-holders is key.

The nature and frequency of the involve-ment can vary throughout design and development, but the effectiveness of stakeholder involvement increases as the interaction between the different roles increases (ISO 9241-210:2019; 5.3). HCAI projects can benefit from the interaction and collaboration of stakeholders who, collectively form an extensive skill base. Addition-ally, a multidisciplinary and multi-perspective approach helps to uncover stakeholder values but also raises awareness of the constraints and realities of the other disciplines. Multiple authors are highlighting the neces-sity to ensure not only stakeholder diversity but also multidisciplinary skills and perspectives within the design and development team.

In Human-Centred AI discourses, and more broadly in Human–Computer Interaction research, different initiatives have been pro-posed to engage experts from various domains of social science in determining how AI should reach our societies, predominantly through informing the adoption policies.

Based on the results of the SLR, we are proposing an integrative HCAI framework, encapsulating the three identified elements – AI Purpose, Values, and Proper-ties as well as the Human-Centred Design foundation of iterative Understanding, Design & Development, and Evaluation through constant User Involvement (Figure 6).

Interestingly, going through prior literature on HCAI we identified that there are papers claiming a human-centred approach for AI but providing little proof or documentation of it. This could be seen as an uncon-structive attempt to cover up a loaded term by connect-ing it through language use with human-centred chains of equivalence that popularly go under the name ‘ethical AI’. In such papers, the terms ‘human-centred’ or ‘ethi-cal AI’ are included as buzzwords without actualising them in a meaningful way. Such papers tend to focus primarily on technical solutions with limited user invol-vement or consideration of human values and empha-sise performance without rooting performance metrics to human needs. This could be considered a form of ‘Potemkin village’ – giving the appearance of something without substance.

Utilising the term HCAI holds merit in terms of raising awareness and directing attention towards the concept, however, actual progress toward more humanistic AI depends on researchers and practitioners backing up claims with rigorous approaches that place human needs and values at the core of AI system design and development.

Human-centred design has been both lauded and cri-ticised for its ability to actively solve problems with affected communities. Criticisms include the inability of human-centred design to push the boundaries of avail-able technology by solely tailoring to the demands of pre-sent-day solutions, rather than focusing on possible future solutions. Whilst users are very important for some types of innovation (namely incre-mental innovation), focusing too much on users may result in producing an outdated or no longer necessary product or service. Critics have pointed out that the insights generated from studying stakeholders today are insights that are related to the needs and values of today and the environment they live in today. Users may develop new preferences, wants, and needs as time goes by. Furthermore, choosing who to involve matters.

Okolo (2022) suggested that centreing humans in the development of artificially intelligent systems shows promise for improving current systems but will only be useful when the distinct needs of marginalised commu-nities are also met. Dignum and Dignum (2020) highlight the dilemma of balancing between the good of the com-munity or society and that of individuals. To address this, ‘Humanity-Centred AI’ they propose a approach focusing on a bigger picture to arrive at common under-standings and agreements. The approach aims to join forces to facilitate innovation and tackle wicked problems in relation to complex human-digital networks. Another criticism of HCD has been formulated by different scho-lars suggesting a ‘More-Than-Human design’ approach, which extends the universe of design beyond solely human, or humanity needs and values.

Similar arguments have been made by Mellamphy (2021), who problematises discourses on HCAI through a posthumanism perspective. ‘Design’ as a discipline has also been criticised from within its own ranks. Mon-teiro (2019) provides an extensive collection of examples where (human-centred) design has failed or even has been identified as the culprit. Furthermore, the focus on understanding human ‘needs’ may be too narrow, putting attention on utilitarian aspects with limited acknowledgment of more critical or philosophical aspects of human well-being, flourishing, and freedom. Albeit we appreciate, recognise, and understand most of these points of criticism, we believe that the foundational insights and clarification of this work is essential.

It is a first step towards creating the fer-tile ground for further exploration of these aspects and to advance a critical and reflective discourse on HCAI.

The paper acknowledges some limitations. The definitions and elements of HCAI identified are based on the analysed literature and may not capture all poss-ible perspectives on the topic. By focusing only on peer-reviewed published articles, other sources of relevant information like company reports, white papers, guide-lines were excluded. Including gray literature can sur-face additional insights as these sources often provide timely perspectives not (yet) present in academic publi-cations. Furthermore, the analysis of purposes, values and properties provides a high-level overview and does not provide an in-depth examination of each con-cept. The paper charts the HCAI landscape rather than providing an exhaustive treatment of each element. Additional conceptual refinement of specific purposes, values, and properties through focused investigations can further develop the framework.

In the paragraphs that follow we propose a research agenda that leverages this review identifying areas for further research to advance the field.

7.2. Research agenda.

This section provides a research agenda to guide further HCAI research. We’re not claiming to have created an exhaustive list of research questions, but rather present a list of themes, to advance HCAI research. Specifically, we define eight research themes that are presented and discussed in the following paragraphs.

7.2.1. Theme 1: purposes.

A crucial question to ask when designing and developing any Human-Centred artifact, is ‘Why?’. The answer to this question defines the purpose and provides a first glimpse into the justification for the existence of an AI system. But who or what has the authority and obligation to define a purpose? Identifying the stakeholders involved in defining the purpose of AI systems, and exploring their roles and responsibilities is a critical part of the HCAI process to prevent undue influence or biases. It empow-ers users and communities to have a say in the AI systems that impact them, promoting user agency and meaning-ful engagement. Clarifying the role of the involved stake-holders promotes a responsible path forward in HCAI development and helps to align with human needs and values.

Further-more, more research is needed on the conceptual idea of augmentation which has been around since the begin-ning of AI. An important topic of investigation is a comparative analysis of the defined purposes of augmentation, automation and autonomy. On one hand, augmentation implies working with humans to enhance and extend human capabilities, keeping humans ‘in the loop’. This seems distinct from full autonomy where AI acts independently, or auto-mation where AI substitutes human work. However, the boundaries may blur in practice. For example, an ‘autonomous’ AI assistant that takes over routine tasks could simultaneously be augmenting by freeing up time for more creative work within the same job. Looking at the relations between these purposes can provide a more nuanced perspective.

Additional theorising and practical case studies are needed to untangle the relation-ships between augmentation, autonomy and automation in HCAI systems. This has important ramifications for the design of AI systems and relates to beliefs that it is universally pertinent to strike a balance between human control and AI capabilities, ensuring that AI enhances human capabilities without compromising human autonomy.

7.2.2. Theme 2: values.

HCAI systems possess the potential to have a significant influence on society. But societies are diverse and dynamic being affected by various demographic, tech-nological, cultural, political, economic, and educational factors. This makes it essential to understand how values differ different in environments and to identify necessary cultural, ethical, and societal considerations for responsible AI design and development. How does the AI context influence prevalent value frameworks? A better understanding of the interplay between pur-poses and values is needed to appreciate how different purposes of HCAI systems impact the prioritisation and adaptation of values, emphasising the need for value alignment across AI applications. Further, since HCAI seeks to prioritise human well-being, more research is needed on how conflicting stakeholder values in the HCAI context should be handled.

Explor-ing conflicting stakeholder values helps to design AI sys-tems that consider diverse perspectives and avoid favoring specific groups over others. Studying the impact of different AI purposes on values ensures that AI aligns with human-centred goals, upholds fairness, and avoids unintended ethical consequences. We need to develop frameworks, methods, and tools for handling conflicting stakeholder values, exploring mechanisms to find common ground and mitigate potential ethical dilemmas. Additionally, the relationships and interdependencies between values present a com-pelling avenue for further research. For instance, Akula and Garibay (2021), in their work on Ethical AI for Social Good, highlight how respect for privacy serves as a fundamental prerequisite for upholding human dig-nity.

Similarly, while the concepts of safety and preven-tion of harm are distinct, they are inherently interconnected. Implementing robust safety measures can significantly reduce or even prevent the occurrence of harm.

7.2.3. Theme 3: properties.

One of the fundamental characteristics of AI systems, is their dynamic nature. AI systems learn and evolve over time, which necessitates continuous evaluation and adaptation of HCAI proper-ties. Investigating if and how the evolving character of AI systems affects HCAI properties over time due to learning, adaptation, or user feedback presents an inter-esting research avenue. How does accountability change, what explainability and intelligibility may mean with an increase in AI lit-eracy, and what does ‘Meaningful Human Control over AI systems’ look like with advancing automation and AI autonomy (Santoni de Sio and Van den Hoven 2018; Shneiderman 2020a)?

Defining and operationalising ‘human control over AI systems’, studying how to ensure that users maintain authority and understanding over AI decisions can help to avoid situations where AI decisions become inscrutable or uncontrollable, empowering users to understand and influence AI outcomes. Furthermore, traditional usability properties need to be reevaluated and compared with novel AI-specific properties, to understand how AI impacts attributes like trust, trans-parency, and fairness. Additionally, a comparative analysis of the identified properties could further improve the definitions and provide shar-per delimitations between related or similar concepts.

7.2.4. Theme 4: processes, methods, and tools.

This literature review shows that appropriate processes, methods, and tools are essential to ensure responsible HCAI design and development. The applicability of existing HCD and HCI methods in the AI context is currently still insufficiently studied. Eval-uating existing methods can act as a procedural starting point to integrate critical perspectives, diversity, and user involvement into the HCAI process. But stake-holder involvement can also create tensions as can be seen with OpenAI’s decision to make a publicly avail-able version of their GTP large language model without proper initial guardrails in place. This raises questions about safe sandbox environments and continuous iter-ations of AI systems.

Tailoring existing HCD processes but also developing and adopting novel tools and methods for the AI age provides a large research area to ensure that during the design and development of HCAI systems relevant and diverse stakeholders are involved and humans are at the focal point of the pro-cess. This also brings up questions about the need for new roles and knowledge for HCAI design and develop-ment, integrating ethical expertise and perspectives in AI design to address ethical dilemmas and societal implications proactively Considering the three identified HCAI elements of purpose, values, and AI properties also raises ques-tions about the interplay between these elements. For instance, whether there is a potential implicit hierarchy among these elements and how the elements affect each other.

7.2.5. Theme 5: Human-Centred design as the.

methodological foundation

On a more fundamental level, the Human-Centred Design approach also needs to be critically scrutinised. Although HCD emphasises empowerment, empathy, and inclusion, its focus is limited to humans and the human benefit. However, AI introduces unique chal-lenges that require adaptation and integration of a human-centred perspective into the HCAI process. This involves questioning if an HCD approach is still appropriate and contempor-ary to address these challenges. As has been highlighted by Giaccardi and Redström (2020), Nicenboim et al. (2020), and Coskun et al. (2022), an exclusive focus on the human perspective might be too limiting. Con-sidering the pivotal capabilities AI technology possesses, a broader examination could be necessary to fully understand the breadth and depth of the implications AI can have.

7.2.6. Theme 6: required conditions for HCAI.

realization

Another research direction that needs deeper explora-tion is the broad spectrum of social, economic, and transformational preconditions necessary for an effec-tive realisation of HCAI. Simply calling for a human-centred approach to the design and development of AI systems is no self-sufficient course of action. More research is needed on the required environments and boundary conditions that provide the necessary con-ditions for HCAI. Furthermore, more research on the inherently interdisciplinary foundation of HCAI is needed. As it has been shown, HCAI draws from a diverse intellectual background, the cross-disciplinary nature needs to become ingrained in both academic as well as industry propositions to dismantle any existing or looming mindcuffs.

This includes a thorough evalu-ation and understanding of possible conflicts of interest and prevailing power dynamics, which could potentially derail such initiatives. Identifying and addressing such risks at the earliest stages is critical to prevent HCAI from failing right at the start.

7.2.7. Theme 7: analysis of other dimensions within.

the HCAI literature

Beyond the elicitation of concepts and the exploration of human-centred design methods and processes, our review has uncovered several additional analytical dimensions within the literature. These dimensions include for instance, the empirical domain, the geo-graphical and cultural context, the type of AI technology. We invite researchers to pick up this analysis and explore these dimensions further, as we believe they hold great potential for enriching the understanding of HCAI. Investigating the domain-specific appli-cations of HCAI can reveal unique challenges and opportunities within different fields. Examining the geographical and cultural contexts of the research can uncover how regional values and societal norms influence the development and implementation of HCAI systems.

Furthermore, an analysis of experi-ences with different types of AI technologies (e.g. gen-erative AI, discriminative AI) can also provide valuable insights. By exploring these dimensions further, a more comprehensive and nuanced understanding of HCAI could be achieved.

7.2.8. Theme 8: conceptual clarity on related terms.

This study intentionally focuses on the term ‘Human-Centred Artificial Intelligence’. While a primary objec-tive of such foundational investigations is to establish a basic, shared understanding of terminology, it is also critical to analyze how this term intersects with, over-laps, and differs from related concepts. However, this comparative analysis necessitates preliminary conceptual clarification of these related terms as well. Conse-quently, we invite fellow researchers to lay similar groundwork for associated terminologies such as User-Centred AI (UCAI), Responsible AI (RAI), or Ethical AI and engage in comparisons to enhance scho-larly discourse.

8. Conclusion.

This work provides conceptual clarity on HCAI by answering the overarching research question ‘How can Human-Centred Artificial Intelligence be defined, con-ceptualised, and implemented?’ To answer this overarch-ing research question, three sub-questions have been formulated and investigated.

SRQ1 (How is Human-Centered AI defined in the existing literature?) is answered by examining the HCAI literature for existing definitions of HCAI, iden-tifying commonalities and differences among these definitions. Further, it provides a new comprehensive definition, that captures the identified crucial aspects of augmenting human capabilities while maintaining human control, and considering the necessity, context, and ethical and legal conditions to promote individual and societal well-being.

SRQ2 (What are the elements constituting Human-Centered AI systems?) identified and further unpacked the constituent elements of HCAI emerging from the analysis. Within the literature three purposes, namely Augmentation, AI Autonomy and Automation have been identified. Furthermore, the analysis revealed three types of values, described as Ethical Values, Pro-tection Values and Performance Values. Lastly, the review found three types of properties in the litera-ture, which have been described as Oversight Proper-ties, Comprehension Properties and Integrity Properties.

Answering SRQ3 (What are the processes, methods, and tools for the development and deployment of Human-Centered AI?) points to the fundamental role of Human Centered Design methodological approaches, describing processes, methods, and tools for HCAI. Established Human-Centred Design methods do offer a constructive starting point for navigating the complex issues posed by AI. With deliberate refinement and expansion, these methods can guide the design of AI systems that benefit humanity by strengthening rather than eroding what makes us human.

Lastly, the literature review outlines a research agenda, pointing to major themes warranting further investigation to advance knowledge on creating AI sys-tems that genuinely benefit humanity.

A mutually beneficial relationship between humans and AI will not happen by itself but will require a serious commitment to transformations that empower all stake-holders. Changes in complex environ-ments are not primarily dictated by technology but rather, the result of a shift in human behaviour and social organisation. However, it is acknowledged that a human-centred approach to AI systems can be met with opposi-tion, especially when profit-oriented perspectives are challenged by HCAI. Nevertheless, we are confident that the provided comprehensive definition, the conceptual clarity and the emphasis on Human-Centred Design process and tools can provide helpful insights and argu-ments towards a beneficial use of AI technologies beyond sole commercial interest. While the growth of AI tech-nologies is certain, the inevitability of any particular future is not.

How AI reshapes work, education, health-care and societies at large is contingent on the approaches we will take for AI design, development and deployment. Ensuring that AI aligns with human priorities through a human-centered approach requires intentional focus on diversity, equity and the needs of humans throughout AI lifecycles.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Funding

This research is funded by the Norwegian Research Council through the program IKTPLUSS-IKT og digital innovasjon, project AI4Users, project number 311680. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conceptualization: S.S., I.P., P.V.; Methodology: S.S., P.V.; Data collection: S.S. Analysis: S.S., P.V Writing original draft: S.S, I.P, P.V.; Writing—review & editing: S.S., I.P., P.V.

Data availability statement

Data can be made available to interested researchers upon request by email to the corresponding author.