Responsible artificial intelligence governance: A review and research framework

You’re listening to “Responsible artificial intelligence governance: A review and research framework,” by Emmanouil Papagiannidis, Patrick Mikalef, and Kieran Conboy. Published in The Journal of Strategic Information Systems in June 2025.

Abstract.

The widespread and rapid diffusion of artificial intelligence (AI) into all types of organizational activities necessitates the ethical and responsible deployment of these technologies. Various na- tional and international policies, regulations, and guidelines aim to address this issue, and several organizations have developed frameworks detailing the principles of responsible AI. Nevertheless, the understanding of how such principles can be operationalized in designing, executing, moni- toring, and evaluating AI applications is limited. The literature is disparate and lacks cohesion, clarity, and, in some cases, depth. Subsequently, this scoping review aims to synthesize and critically reflect on the research on responsible AI.

Based on this synthesis, we developed a conceptual framework for responsible AI governance (defined through structural, relational, and procedural practices), its antecedents, and its effects. The framework serves as the foundation for developing an agenda for future research and critically reflects on the notion of responsible AI governance.

Introduction.

Following a surge in data and computational capability, companies have increasingly turned to artificial intelligence (AI) to achieve a competitive edge. This has precipitated an explosion of academic research in this area, with numerous papers, special issues, conferences, and tracks emerging. Although the literature has noted several benefits of AI adoption, for most organizations, AI has several potential ramifications and unexpected and unwanted outcomes. This study defines AI as “the ability of a system to identify, interpret, make inferences, and learn from data to achieve predetermined organizational and societal goals”.

The development of responsible principles to minimize AI’s negative and unintended consequences has been a central point of discussion over the past years (Council of Europe, 2018; European Commission, 2019; Floridi et al., 2021; Hagendorff, 2020; M ̈okander & Floridi, 2021). Generally, these principles provide a guide and set of targets for designing and deploying AI to ensure that the technology is fair, equitable, ethical, and generally “good” for all those affected by it. Despite extensive efforts to define the dimensions of responsible AI principles, research has largely focused on high-level guidelines, notably lacking the elements of governance. Consequently, a gap exists in understanding how AI technologies are governed responsibly throughout their life cycles.

This review is motivated by the fact that adhering to responsible AI principles is generally deprioritized or considered an ancillary task during the actual implementation and management of AI projects (M ̈antym ̈aki et al., 2022). For instance, Meske et al. (2022) and Mannes (2020) demonstrate that organizations need certain trade-offs to find the right equilibrium between performance, transparency, and ethical conduct. While doing so provides some insight into the choices that organizations must make in response to developing and deploying AI applications, it does not provide a holistic or comprehensive understanding of how responsible AI governance practices are formulated and enacted. Hence, a significant challenge exists in translating theoretical principles into practical implementation approaches (M ̈antym ̈aki et al., 2022).

We aim to develop a more coherent understanding of how responsible AI governance can be comprehended and implemented in research and practice. Specifically, we address the lack of guidance needed to translate high-level abstract principles into deployable practices throughout the AI project lifecycle. We argue that bridging this gap facilitates a more comprehensive approach to responsibly developing and deploying AI technologies. Additionally, we understand how responsible AI governance is shaped depending on the context and its effects on organizations and their environments.

We achieve this by: (i) synthesizing prior research on the definition and principles of responsible AI; (ii) proposing a concept of responsible AI governance based on seven key principles of responsible AI spanning three types of organizational practices (structural, procedural, and relational); and (iii) discussing the broader context wherein responsible AI governance is developed and utilized, highlighting its key antecedents and effects, both internal and external. Placing responsible AI governance in a framework that highlights the broader context wherein it is developed and deployed enables us to uncover key assumptions and highlights important areas for future research. This paper concludes with important issues underpinning research and practice and develops a set of research questions to help guide future studies.

Research methodology

We conducted a systematic literature review comprising a sequence of steps to identify the relevant research work. In the following sub-sections, we present the protocol development process, the inclusion and exclusion criteria, the data sources and strategy for searching articles, and how we quality-checked the pool of papers and extracted data from them.

Protocol development

The systematic literature review was based on well-established procedures to ensure all relevant publications’ inclusion. We followed a scoping review approach (Par ́e et al., 2015), outlining how the primary research would be conducted, the search phrases used, and the sites to consult when gathering literature. The review was conducted in five steps. Two researchers collaboratively examined the papers to mitigate bias during the selection process. Step one involved gathering information, resulting in the identification of 1,080 documents. The documents were iteratively filtered according to their relevance. All titles were reviewed, and 494 papers were excluded because of irrelevance or because the same paper appeared twice.

Step two involved reviewing the abstracts, which resulted in the exclusion of 378 papers after carefully reading each to determine the inclusion or exclusion criteria. Step three, which involved critically viewing the approach in the studies, excluded 157 papers. Step four involved the extraction of data, which were then organized into a spreadsheet. Three papers did not conform to the inclusion criteria, resulting in a final sample of 48 papers. Step five included data synthesis, which involved using the concept matrix to structure the content of the articles. A concept matrix was used to draw connections between the different research articles. Fig. 1 depicts the steps for conducting this review.

Inclusion and exclusion criteria

To define this systematic literature review’s scope, various inclusion and exclusion criteria were used to ensure that the selected studies aligned closely with the research objectives. Our criteria were designed to encompass a broad spectrum of research on AI’s integration and impact in business and organizational contexts. These criteria include studies examining how AI contributes to digital transformation within businesses and how organizations leverage AI to address operational challenges. First, we examined whether there was a focus on AI in the organizational context. Second, we checked the publication dates. Considering AI’s rapid evolution and increasing adoption in recent years, only studies published from 2017 onwards were considered. Third, we included only papers published in English to ensure accessibility and comprehension for a wider audience.

Fourth, we filtered based on publication type, ensuring the inclusion of peer-reviewed journal articles and providing a substantial depth of analysis. For the exclusion criteria, we first examined the technical focus of the articles. Studies that primarily focused on AI’s technical aspects, such as architectural infrastructure or model benchmarking, were excluded. Second, we excluded certain publications – such as book series, disseminated articles, and webpages – from the selection process. Third, the articles were examined based on their publication status; pre-publication or under-review studies were excluded to maintain the included studies’ integrity and quality.

Data sources and search strategy

In the first phase, a series of search strings were developed. The first group of terms (see Table 1) contained keywords linked to AI and related technologies, whereas the second set focused on organizational viewpoints. The terms used were treated as exact keywords; thus, to increase the number of search strings, keywords from both sets were concatenated to generate a search string using wildcard symbols. Thereafter, the search phrases were used in Scopus, Business Source Complete, Emerald, Taylor & Francis, Springer, Web of Knowledge, ABI/Inform Complete, IEEE Xplore, and the Association of Information Systems (AIS) libraries – as well as in other electronic databases, including ScienceDirect, JSTOR, Digital Bibliography & Library Project, and Google Scholar. This was done to ensure that the index contained all the relevant items.

Data collection commenced in November 2022, followed by further refinement in December 2023.

Quality assessment

All documents were subjected to quality evaluation to improve their internal and external validity and eliminate bias. Each article was evaluated for relevance using three fundamental values. The first was to examine the biases in the data collection process of the identified articles; we assessed the validity of the research methodology employed therein. Second, we checked for internal validity,

which refers to how well the study’s design and execution prevented systematic mistakes, suggesting that the examination would yield positive findings. Nevertheless, it may lack intrinsic validity; therefore, internal validity was further investigated by examining the research’s discussion and conclusion. Following the eligibility check, two coauthors independently reviewed the papers and evaluated their quality using multiple criteria. These criteria include study design and methodology, sample size and representativeness, data quality, ethical considerations, data interpretation, results, and reporting.

When the two coders disagreed, a third researcher provided additional perspective and expertise to help mediate and resolve disagreements. The third researcher contributed by offering insights, conducting further analyses, and facilitating discussions between the first two researchers to reach a consensus on the papers’ quality. This collaborative approach ensured a more robust and unbiased evaluation. The scientific rigor and relevance of these studies were also reviewed. Specifically, we examined the credibility and impact of the retained studies. These criteria helped us produce reliable findings with less bias that can be generalized to broader populations or contexts.

Data extraction and synthesis of findings

The first step in data extraction was identifying all the relevant materials within the final pool of papers. A concept matrix was developed to classify the investigations and combine data. The matrix includes information regarding the data extraction date, title, authors, journal, publishing details, and topic-specific information. The extraction was conducted following the guidelines of Kitchenham (2004). Furthermore, arranging the data on a spreadsheet made comparing the data from each article easier. The studies were analyzed based on 15 distinct values. Table 2 presents each of these values and the key categories and themes they belong to. The need for sub-views arose because each responsible principle may encompass diverse viewpoints and techniques. These perspectives were used as labels to depict the perspective of each article.

Nevertheless, some labels may not apply to all responsible principles, resulting in missing values.

The data were synthesized after inserting and analyzing 48 entries into the concept matrix. During this stage, we systematically combined and summarized data from multiple sources to draw meaningful conclusions and generate new knowledge. This study aimed to fulfill the standards of scoping synthesis and provide an in-depth overview of the available evidence. Additionally, we utilized a descriptive synthesis that helped map materials from several studies and presented them to research streams. Furthermore, we examined the consistencies and inconsistencies of responsible AI governance and compared different studies on the same topic.

Definitional aspects of responsible AI

In the last decade, how AI should be developed and AI deployment based on responsible principles have received significant attention. Questions such as what responsible AI is and what governance practices should be applied to enact it remain unanswered. In recent years, reports have provided conceptualizations and descriptions of responsible AI principles (de Almeida et al., 2021; Freiman, 2023; IBM, 2019; Singapore Government, 2020; Smuha, 2021). This trend toward an increased number of articles on responsible AI is largely attributable to the growing number of incidents in which the use of AI leads to unforeseen or undesirable repercussions. For instance, Amazon has been developing AI applications to automate the process of analyzing resumes to identify top vacancy candidates.

In 2015, Amazon’s machine learning (ML) experts found that its AI-powered recruitment tool discriminated against women when recruiting technical professionals, such as software developers. These algorithms were partially trained on resumes submitted to the corporation over the previous 10 years, during which, most successful resumes were disproportionately from male applicants. Such cases have spurred policymakers,

Themes extracted from the concept matrix.

researchers, and practitioners to consider how AI development and use should adhere to “responsible” norms. Therefore, discussing the core principles of responsible AI is critical.

Responsible AI principles

Governments, researchers, and corporations are increasingly focusing on AI-related ethical standards and principles. Numerous reports have been published outlining the significance of these principles and the reasons behind their importance. However, determining what constitutes responsible AI is a work in progress, with several organizational bodies and researchers aiming to provide complete and coherent conceptualizations. While previous research has focused on AI’s specific aspects – such as bias elimination, the explainability of AI outcomes, and safety and security (Hern ́andez-Orallo et al., 2020) – recent years have observed a shift toward a more holistic understanding of responsible AI’s constituents.

The European Commission recently requested an independent expert body – the High-Level Expert Group on Artificial Intelligence (AI HLEG) – to develop an integrated framework for responsible and trustworthy AI. Meanwhile, the Singapore government (2020) recognized the forthcoming AI difficulties regarding discrimination, biased outcomes, and concerns linked to consumer awareness and understanding of AI engagement in decision outcomes in the ethical, legal, and governance sectors. Simultaneously, there is a push for independent bodies to certify responsible AI best-practice advocacy from corporations, such as Google.

Regarding responsible AI practices, the word responsible can be interpreted in various ways. For example, the AI HLEG promotes trustworthy AI with three main necessary components: the system in question should be lawful, complying with all applicable laws and regulations; ethical, ensuring adherence to ethical principles and values; and robust, having the ability to withstand and adapt to different challenges and disruptions in the environment that encompasses both social and technical elements. Similarly, Singapore Government (2020) framework is based on two high-level guiding concepts that foster AI trust. The first concerns companies that use AI to make decisions and ensure transparent, explainable, and fair processes.

Although absolute explainability, transparency, and fairness are difficult to achieve, companies should invest every effort to ensure these values, thereby contributing to AI development. The second category comprises human-centered AI solutions. Human interests, including well-being and safety, should be key considerations in designing, developing, and deploying AI, as it augments human skills. Therefore, businesses should ensure that human-centric decision-making processes adhere to ethical norms.

A recent Harvard report highlighted 38 similar corporate and group efforts. An underlying agreement exists that responsible AI represents a set of principles assuring ethical, transparent, and accountable usage of AI technology per user expectations, corporate values, and societal laws and conventions based on responsible AI’s emerging consensus (Flavi ́an & Casal ́o, 2021). In this sense, responsible AI encompasses a wide range of standards that must be satisfied throughout the lifecycle of AI applications. Winfield and Jirotka (2018) describe responsible principles as a collection of processes, procedures, cultures, and beliefs that ensure the highest levels of conduct. They emphasize AI governance’s ethical side, suggesting that responsible AI transcends principles and instills ethical behaviors in individuals and companies.

Winfield and Jirotka (2018) claim that these are critical components of responsible research and development, which “entails an approach, rather than a mechanism; hence, they seek to tackle ethical issues before they arise in a principled manner rather than waiting until a problem surfaces and dealing with it in an ad-hoc way”.

Building on these concentrated efforts, responsible AI principles can be divided into accountability, diversity, non-discrimination and fairness, human agency and oversight, privacy and data governance, technical robustness and safety, transparency, and social and environmental well-being. Several of these principles appear in different reports using various terms. For instance, “transparency” might appear as “transparency and explainability,” while “human agency and oversight” might be noted as “human control of technology.” The terms describe the same principles, with no major differences, signifying an ongoing conceptualization process. Through an open consultation process, these guidelines describe the key components of responsible AI principles.

The discussion pertains to the need to establish a set of responsible principles emerging from challenges specific to AI technologies. These include effectively governing and controlling autonomous intelligent systems, establishing responsibility and accountability for algorithms, and ensuring privacy and data security in opaque and multilayered systems. Responsible AI has gained traction at the policymaking level as a testament to its importance, with several countries defining responsible AI’s fundamental principles. At the national level, the AI readiness index measures the degree to which countries are implementing AI technologies; they now include a new sub-index that quantifies the degree to which responsible AI principles are adopted.

When considering more in-depth the issues potentially arising when using AI, several situations may be preempted using responsible AI principles. The complexity of AI renders it difficult to comprehend and interpret the final outcomes, frequently rendering the results opaque. This phenomenon is commonly called a “black box,” whereby AI may implement unforeseeable actions or suggest outcomes that are difficult to trace. This may be exacerbated when AI gains the autonomy to pursue its own objectives, even if it unintentionally harms others. Such instances raise concerns regarding AI transparency in decision-making processes and accountability for the outcomes of AI use. Consequently, responsibility and accountability are important concepts in governance and regulation.

Operators or developers of AI systems cannot accurately predict with exact certainty all actions and results generated by the self-learning ability of AI algorithms at any given time. Therefore, carefully assessing the actors and regulation of transparent and explainable AI systems is necessary.

Mass data reuse and ubiquitous digitalization have become global drivers of competitiveness. Furthermore, AI has been widely described as having the potential to vastly improve efficiency across all domains and sectors and help resolve humanity’s greatest challenges, such as the United Nations Sustainable Development Goals. Nevertheless, the swift and extensive adoption of narrow AI – with notable attributes such as efficiency, scalability, performance, decision automation, and speculative progression toward general AI – has raised significant concerns, which have prompted extensive research into the concepts of human dignity and existence.. Human dignity can be jeopardized in situations involving job displacement, loss of privacy and surveillance, and loss of control or autonomy over AI systems.

This landscape of direct threats and structural imbalances increases the urgency to develop appropriate governance solutions. The coordination of the development and implementation of responsible AI principles is not mutually exclusive. They complement each other in building trustworthy and ethical AI systems. Therefore, understanding how the design, development, and implementation of AI applications can be infused with the key principles of responsibility is necessary.

Responsible AI governance

Responsible AI governance has been conceptualized as a framework that encapsulates the practices that organizations must implement in their AI design, development, and implementation to ensure AI systems’ trustworthiness and safety. Responsible AI governance concerns delegating authority and control over data and exercising authority through data-related decision-making. Awareness and understanding of AI’s impact are crucial for effective governance, as AI-educated individuals are essential pillars of any successful AI system. Thus, responsible AI governance should incorporate incentives and sanctions to encourage desirable data collection, administration, and utilization behaviors. Furthermore, responsible AI governance relies on collaboration between firms and individuals who comprise the system and extends beyond a single company (M ̈antym ̈aki et al., 2022).

This multi-organizational context necessitates trusted frameworks to ensure dependable data sharing among organizations while adhering to the General Data Protection Regulation (GDPR) and other applicable laws and regulations. Accordingly, we define responsible AI governance as follows:

A set of practices for developing, deploying, and monitoring AI applications in a safe, trustworthy, and ethical manner that ensures appropriate functionality of AI over the entire lifecycle.

Although clear definitions of responsible AI exist, the literature uses terms that are synonymous or largely overlapping (see Table 3). For example, a large stream of research refers to “trustworthy AI,” while others use the term “principled AI.” We argue that this divergence in naming stems from the immaturity of responsible AI as a research concept. Researchers have built on various definitions and themes that encompass responsible AI practices and dimensions. This conceptual opaqueness makes it challenging to determine what responsible AI governance should include.

The overarching objective of frameworks is to maximize the value of AI while simultaneously lowering the associated risks and

Description of artificial intelligence (AI) governance terms.

unintended consequences. However, those aiming to implement responsible AI governance risks are undermined in two ways. First, the diversity and breadth of responsible AI principles render it challenging for organizations to implement practices that cover a variety of goals. Thus, a principles-first approach may prove counterproductive and incompatible with the organizational modus operandi. Second, responsible AI principles are in a continuous state of flux, whereby every new type of emergent AI technology comes into a new set of conditions that must be considered.

An indicative example is the recent release of Open AI’s ChatGPT, which poses a novel dilemma regarding human creativity and innovation.1 Collectively, these issues suggest that we must reconsider how we conceptualize and approach responsible AI governance, with the caveat that the notion entails actionable practices for AI applications’ design, deployment, and oversight throughout their lifecycle.

Synthesis of responsible AI principles

Based on the current discussion around frameworks for responsible AI, the following sub-sections present a synthesis of research findings based on the seven pillars of key principles and their underlying sub-dimensions (Table 4).

Accountability

Accountability involves the implementation of mechanisms and processes to ensure responsibility and auditability during and after AI development and deployment. Auditability is fundamental to every project and establishes the foundation for data selection and system architecture. Vollmer et al. (2020) propose that auditability should be assessed during data accumulation, focusing on data inspection concerning its usage and addressing questions regarding dataset distribution and sample representation. Establishing error-reporting mechanisms is critical because it allows data comparison at different stages. Similar approaches have been proposed for data collection in autonomous vehicles. These procedures may not guarantee ongoing accountability but ensure that the data trace possible deviations and those responsible for them.

Although AI can produce accurate analytical findings, implementing accountability procedures is typically challenging because of its “black-box” nature. Hence, technologies for detecting weaknesses and critically evaluating AI systems are essential. Screening CVs as part of the recruitment process exemplifies the challenge of embedding accountability principles in AI systems. Unlike human recruiters, AI systems cannot be held personally responsible for filtering candidates. The question of who is accountable for a judgment made by an AI system is difficult to resolve and multilayered. Some businesses early in implementing AI for such processes have found it challenging to account for the decisions taken. Following these cases, organizations have been criticized for lacking accountability in using AI systems (Schl ̈ogl et al., 2019).

Specifically, accurately capturing where responsibility lies and auditing such systems transparently and easily have been a key focus.

Diversity, Non-Discrimination, and fairness

Another fundamental principle of responsible AI is ensuring that systems do not reproduce discrimination or unfairness. Recent real-life applications have highlighted how AI systems produce discriminating results based on their training data (Varona & Su ́arez, 2022). Two popular areas are credit ratings and criminal sentences. The European Commission (2019) recommends that AI systems use appropriate mathematical and statistical methodologies to uncover unintended behavior. According to Korinek (2020), the European Commission has developed the only method for eliminating algorithmic bias whereby fairness primarily entails accessibility and the absence of unjust prejudice. Accordingly, systems should be user-centric, allowing all individuals to utilize AI products irrespective of age, gender, abilities, and characteristics.

Other forms of prejudice and discrimination concern language because it is highly complicated and includes features such as word grouping and ordering rules. Using natural language datasets to train models precipitates various biases, and detecting this prejudice may be challenging. For example, word groups of men and women, adjectives associated with them, and the frequency or order in which they appear in a list may all encourage bias in the dataset and alter the model. Hence, one can comprehend the potential of placing a biased dataset into a “black-box” system, causing unpredictable and unjust consequences. Bias awareness can be realized by addressing record distribution and ensuring relevant updated data. Furthermore, data processing and analysis should be based on core ethical values to reduce any disparities or prejudice as much as possible.

Human agency and oversight

The principles of human agency and oversight guarantee that AI systems adhere to democratic, prosperous, and equitable societal values. The user’s knowledge and interpretation of AI system outcomes are the focus of human agency. By contrast, human review refers to the presence of humans in an AI’s decision-making process. The most widely utilized strategies for supervision are planning oversight, continuous monitoring, and retrospective disaster analysis. Planning oversight entails assessing proposals ahead of time, allowing for an examination of the chosen technologies and an understanding of their influence in the context

Responsible artificial intelligence (AI) principles.

of use before they are adopted. Continuous monitoring refers to the ongoing inspection and correction of a system at regular intervals, which is beneficial for systems that operate in dynamic and uncertain environments. Finally, retrospective disaster analysis refers to a detailed system examination following a serious incident.

The extent to which humans govern and monitor AI systems is a contention among enterprises. According to Tolmeijer (2022), two ways of engagement exist. First, “human in the loop” refers to AI suggesting recommendations while a human makes the final choice—also termed “assisted intelligence.” Second, “human out of the loop” follows the logic whereby the system makes the final decisions, with developers tweaking the models to obtain the desired result. Autonomous intelligence is a term used to characterize such systems (e.g., self-driving cars and automated stock market trading systems). Overseeing systems is a proposed mitigation approach for ensuring system anticipation – accomplished by creating a system wherein stakeholders voice ethical concerns and such concerns are incorporated into the revised models of the AI system.

However, using AI as a support system might precipitate a moral dilemma because people may exploit the system to absolve their moral responsibility for their actions. This tendency to place responsibility for system malfunctions is particularly evident in systems lacking transparency in their inner workings and with limited auditability.

Privacy and data governance

Privacy and data governance refers to the principles for managing the availability and usability (data access), integrity (data quality), and security (data privacy) of enterprise data when developing AI systems. Building on formalized processes is one possible means to ensure privacy concerns regarding data gathering that can be accomplished in different ways. First, developers who create algorithms should provide documentation about the data life cycle; moreover, a continuous exploration and sensitivity evaluation of both the data-gathering methods and the data itself is critical. Second, without a legal compliance assessment, firms might struggle to adhere to various laws regarding data collection and processing, as they are typically subject to different national and international directives. This can have significant repercussions on the maturation and use of AI models.

Finally, cultural variations should be considered during data collection. This is especially important when considering country variations; in certain contexts, a stronger or weaker link exists between private enterprises and government organizations, which has ripple effects on what data are collected and how they are used.

Technical robustness and safety

Technical robustness – closely linked to damage prevention – involves creating AI systems that are proactive in their approach to risks and consistently perform as intended while minimizing unintentional and unanticipated harm and preventing unacceptable harm. This should also apply to potential changes in operating environments and the presence of other agents (both human and artificial) that may interact antagonistically with AI models. Additionally, AI systems must protect (resilience to attacks) against flaws that could allow adversaries to exploit them or gain unauthorized access. Data poisoning (a malicious or adversarial attack in which an attacker manipulates the training data used to train an AI or ML model)), model leakage, and attacks on the underlying infrastructure (both software and hardware) pose risks to AI applications’ proper functioning.

When AI systems are targeted, data and system behavior can be altered, precipitating significant deviations in outcomes and the underlying logic of operation. Hence, AI systems must have a contingency plan (general safety) for when such attacks occur to ensure continuous operation. The extent to which safety precautions are necessary is determined by the scale of the risk posed by an AI system, which, in turn, is determined by the system’s capabilities and the consequences’ severity. When the development process or system itself is anticipated to offer exceptionally high risks, establishing and testing safety measures in advance is critical; therefore, accuracy is crucial. The capacity of AI to make accurate judgments, such as classifying information into appropriate categories or making precise forecasts and suggestions, varies depending on its application.

The unintended risks from faulty forecasts can be supported, mitigated, and corrected through well-formed development and review processes. In situations where occasional incorrect predictions cannot be prevented, the system must demonstrate probability errors, particularly when human life is at risk. Thus, the AI system’s results must be both reproducible and dependable.

Transparency

A prevalent concern regarding AI systems is that, despite their potential widespread use in everyday applications, knowledge about their functioning is limited. Consequently, there is a lack of understanding and, consequently, a lack of trust among users, who may be hesitant to use AI. A key aspect of transparency is explainability – the ability to explain both the technological processes of an AI system and the human judgments resulting from these procedures. Owing to these procedures, a new subfield called explainable AI (XAI) has emerged to provide human-comprehensible models and interpretations of complex machine-based calculations. Nevertheless, trade-offs may be required to improve the explainability of a system (at the expense of accuracy) and increase its accuracy (at the expense of explainability).

When an AI system substantially influences people’s lives, an appropriate explanation of the system’s decision-making process should be available on demand. Such explanations should be timely and tailored to stakeholders’ knowledge (e.g., laypersons, regulators, or researchers).

Transparency is necessary to address any subsequent need for traceability. The datasets and procedures that lead to the AI system’s conclusion, such as data collection and labeling and the algorithms utilized, should be documented to the highest degree of feasibility. This also holds for AI system decisions and allows the discovery of the logic driving an incorrect AI judgment, which may aid in preventing future errors (Mezg ́ar, 2021). Additionally, users should be aware and clearly informed when interacting with an AI agent rather than a human actor. This necessitates that AI systems be identified explicitly. Furthermore, the option to opt out of AI interaction in favor of human interaction should be offered to respect the fundamental rights of communication.

Additionally, the AI system’s capabilities and limits should be conveyed to end users suitably, including sharing advanced information on the AI system’s accuracy and limitations. Collectively, these form the principles of communication.

Social and environmental Well-Being

The sustainability and ecological responsibility of AI systems are also elements of responsible AI use that have been highlighted frequently. Furthermore, AI applications should be built on the logic of addressing global challenges, such as ensuring social well-being and protecting the environment, and can potentially solve some of society’s most pressing problems. However, they must be designed in an environmentally friendly manner. In this regard, a system’s development, deployment, and usage processes should be examined through critical studies on resource use and energy consumption. Concerns have been raised regarding how AI may lead to job displacement and how the replacement of jobs by AI systems may precipitate the emergence of new ways of social organization.

These issues are sensitive and complex, as different societies may have different views on how to solve them; thus, these issues should be addressed locally and as needed.

Additionally, exposure to social AI systems in various contexts may alter beliefs regarding social agency and affect social interactions (social well-being). For example, AI can replace people in hazardous occupations, such as mining and quarrying. This may be perceived as a threat to certain professions. However, it may also increase workplace safety if such tasks are performed by robotic agents built on AI. Furthermore, some solutions may result in “cold care” when people are replaced by AI agents. Hence, AI systems can help to provide care for those in need, but simultaneously, they may also degrade the level of social interaction. Moreover, new threats might emerge because AI could be misused in democratic processes, such as political and electoral decision-making contexts. Consequently, these systems’ effects must be thoroughly considered and planned.

Critical reflection & research framework

In the previous sections, we briefly reviewed the key principles on which responsible AI should be built. Nevertheless, a consolidated framework for understanding how these principles are implemented and how they shape and are shaped by society is lacking (Sepp ̈al ̈a et al., 2021). Therefore, we build on the notion of responsible AI governance as a central concept for the effective infusion of AI systems with responsibility principles. The research agenda in this section discusses the antecedents, key components, and effects of responsible AI governance (Fig. 2).

The key components of responsible AI governance are based on the works of Van Grembergen et al. (2004) and Tallon et al. (2013), who highlight the structural, procedural, and relational practices that organizations must consider as part of their governance. Structural practices include assigning roles and responsibilities for decision-making around AI. Procedural practices concern how organizations execute responsible AI governance and include aspects that concern a series of actions at different levels. Relational practices concern the different links among employees within and outside the organization, as well as the means for developing the skills and knowledge of human capital. This distinction is made because it enables the identification of different types of practices that are relevant at various levels within the organization.

Table 5 summarizes the future research questions on responsible AI governance.

Antecedents of responsible AI governance

The antecedents of responsible AI governance can be divided into three broad, interdependent themes. The first refers to societal expectations and norms, which are unwritten rules of behavior shared by society and adopted by organizations, as well as more formalized regulations and directives. The second group concerns the organizational values that predate the use of AI and characterize corporate culture. Finally, the third group of antecedents involves the responsible AI principles of organizations. As these are contextual, fluid, and constantly changing, they represent key directives toward which responsible AI governance must be aligned. In Fig. 2, we also visually depict the relationship between these groups of antecedents, where broader societal and normative beliefs indirectly impact the prioritized responsible AI principles that firms decide to adhere to.

These factors are mediated by how organizational values and culture adopt external stimuli and the importance they assign to adhering to responsible AI principles.

Societal expectations and norms can be considered the starting point of the antecedents of responsible AI governance. Social norms shape the AI principles that should be followed, especially those perceived as ethical and acceptable in a social context. Organizations have shifted their operating paradigms through various top-down and bottom-up efforts. This reevaluation of core beliefs aims to project or maintain a good public image, as organizations need to foster a good reputation within their operational context. Additionally, evolving regulations and directives such as the AI Act influence which aspects of governance are prioritized and how practices are designed and implemented.2 Nevertheless, an issue that aligns with societal norms and expectations is that they are in constant flux.

Therefore, organizations must develop appropriate mechanisms to identify external stimuli quickly and accurately and interpret how they affect their AI applications. An example of such a case is the controversy surrounding Google’s Gemini image generator, which faced backlash from many users who accused the AI service of being “woke”.3 Such societal signals are often challenging to identify and require different mechanisms to act upon formalized regulations and directives. Organizations must develop different mechanisms to identify emerging external pressures and create appropriate channels for filtering and adaptation.

Apart from external signals and feedback, an essential part of what organizations implement in their responsible AI governance frameworks relates to internal corporate values and capabilities (J ̈ohnk et al., 2021; Papagiannidis et al., 2022). Organizations must foster a culture where individuals are encouraged to acknowledge and respond to external stimuli that are perceived as significant. Several internal factors mediate the extent to which such signals are leveraged – including the style of decision-making, concentration of power, level of democratic and inclusive organization in the firm, and other contextual and industry-specific aspects. These contingencies mediate how effective and adaptive organizations are in identifying changing external stimuli and incorporating them into their principles of AI governance.

An example of this is an organization’s capability to capture data influenced by norms and redefine its responsible AI principles in a relatively short period. Nevertheless, we still have limited knowledge concerning how organizational path dependencies, culture, and other internal factors influence an organization’s ability to be receptive and adaptive to such signals and, therefore, to effectively roll out appropriate responsible AI governance schemes.

Another important issue that organizations must consider when formulating responsible AI governance practices is cultural and ethical variations when deploying systems or services to different countries or populations. One of the generally held assumptions of many responsible AI frameworks is that there is uniformity in the elements and importance of aspects noted as key pillars. However, a challenge with such an approach is the considerable diversity among cultures and even segments of user groups concerning what is ethical and appropriate. Thus, responsible AI governance should consider the requirement to accommodate such variation. Key decision-makers should be conscious of potential user groups and how they may perceive the elements of use and design of AI systems with which they will interact.

Responsible AI governance

Structural practices

Structural practices in the context of responsible AI governance describe the key decision-makers and the rights and responsibilities of individuals and user groups within an organization. Such structural practices, in turn, impact various phases of AI development and deployment, as well as different levels within the organization. Within the context of structural practices, AI governance committees are responsible for overseeing AI initiatives within an organization. Structural practices should outline the criteria for selecting committee members, ensuring that key decision-makers with relevant domain knowledge and strategic insights are included. Furthermore, practitioners should clarify roles, responsibilities, and decision-making by establishing clear approaches for effective oversight and fostering alignment with organizational objectives and ethical standards.

Simultaneously, structural practices include the implementation of decision-making protocols within AI governance structures that provide consistency in the decision-making process. Such practices should also specify the roles and responsibilities of individuals and user groups involved in decision-making to ensure accountability and transparency. By establishing clear decision-making structures, organizations can enhance agility, responsiveness, and integrity in their AI governance, thus driving better outcomes by mitigating risks.

Structural practices articulate the rights and responsibilities of stakeholders engaged in AI development and deployment. These practices should provide a clear and concise framework for understanding the roles, obligations, and expectations of the individuals and user groups involved in AI initiatives. Within this set of practices, documentation should be provided for drafting, reviewing, and approving rights and responsibilities, ensuring alignment with organizational values, regulatory requirements, and ethical guidelines. Moreover, the procedures should establish mechanisms for regular reviews and update the documentation of rights and responsibilities to accommodate evolving organizational needs and industry standards.

By formulating robust documentation on rights and responsibilities, organizations can promote transparency, accountability, and trust in AI-related activities, thus fostering a culture of ethical and responsible AI governance. Nevertheless, there is limited knowledge of how organizations should define their structural practices within and outside firm boundaries. Within organizational boundaries, there is a lack of research identifying how rights and responsibilities should be assigned for vertical decision-making and how control and power dynamics influence the effectiveness of AI projects. Additionally, there is a limited understanding of how horizontal coordination and decision-making are enacted and how different departments within organizations coordinate actions at the respective levels.

Similarly, the extended stakeholder ecosystem for formulating responsible AI governance has received limited attention. Understanding how different stakeholders can optimally provide input and exert influence on AI projects is critical to ensuring that AI applications meet the requirements and are ethically aligned.

Issues and research agenda on responsible artificial intelligence (AI) governance.

Procedural practices

Procedural practices encompass processes crucial for data and model management, pipeline evaluation, and human-AI interaction. In data management, these practices ensure the organization, security, and accessibility of data through classification and encryption protocols in a manner that minimizes bias and checks for errors. Procedural practices streamline workflows, optimize efficiency, and facilitate the continuous improvement of models and data throughout their lifecycles. Furthermore, procedural practices in human-AI interaction focus on designing ethical and transparent AI systems, emphasizing user trust and accountability. These practices serve as essential frameworks across diverse domains – ensuring operational efficiency, reliability, and ethical integrity by aligning AI activities with applicable laws, regulations, and industry guidelines.

By upholding responsible AI governance processes through robust compliance monitoring and enforcement, organizations can mitigate legal and ethical risks, build trust with stakeholders, and safeguard themselves against the potential harm arising from AI-related activities. Furthermore, organizations can develop robust incident responses and crisis management procedures to address AI-related issues promptly and effectively. These procedures should encompass protocols for detecting and assessing incidents, defining escalation pathways, coordinating response efforts, and communicating with stakeholders. Organizations can mitigate risks, minimize potential damage, and maintain trust and confidence in their AI governance frameworks by implementing comprehensive incident response and crisis management procedures.

Despite a broad set of practices and processes for implementing responsible AI governance, knowledge concerning strategic planning is limited. Organizations often struggle to harmonize their competitive strategies with the aims and ambitions of responsibly managing AI applications. A common assumption is that being responsible for AI deployment is incompatible with gaining a competitive edge when leveraging such technology. Thus, it is crucial to understand the processes at the strategic level that facilitate the responsible deployment of AI technologies to be utilized as a competitive strategy, and vice versa. At the same time, many of the defined processes around responsible AI governance concern the operational levels of decision-making, with limited well-defined practices concerning strategic and tactical levels.

Exploring how responsible AI principles can be translated into concrete processes at these levels and how they can be utilized to proactively mitigate negative and unintended consequences is critical for ensuring that organizations remain competitive by leveraging AI and doing so responsibly and ethically.

Relational practices

Relational practices within the context of responsible AI governance involve establishing links within and beyond the organization, training and educating stakeholders, and ensuring that appropriate links are formed to facilitate the responsible development and use of AI. To facilitate cross-functional collaboration, organizations establish systematic procedures to encourage and enhance communication and teamwork among various departments. These procedures outline clear communication channels, establish regular collaboration meetings, and promote knowledge-sharing and collaboration platforms to facilitate the exchange of ideas and expertise. By fostering a culture of collaboration, organizations can ensure that AI initiatives are aligned with overarching organizational values and norms, leveraging diverse perspectives and skill sets to drive innovation and achieve strategic goals.

Relational practices define the modes of stakeholder participation in the development process. For example, participation can be inclusive throughout the entire data collection process – from strategic planning through identifying data needs, choosing and testing a suitable collection approach, collecting the data, and finally storing, disseminating, analyzing, and interpreting it. Such approaches are useful for avoiding conflicts of interest, unethical uses of data (where data collected for one purpose are employed for an entirely different purpose), unauthorized data ownership or access (determining the true owner of the data), and misuse of data (sharing information with third parties that the user is not aware of). To this end, organizations must identify all relevant stakeholder groups and envision ways to complement the design, development, and use of AI.

Nevertheless, an inclusive and expansive view of stakeholders in AI development can potentially slow assimilation. To prevent risk management from stifling innovation, organizations must carefully consider their internal and external relational practices. Interest in responsible AI literacy has been growing as it becomes evident that much of what organizations do regarding responsible AI practices hinges on employees’ knowledge. Thus, it becomes increasingly important to understand what skills and competencies employees at different roles and levels need to be equipped with, as well as how individual and collectively responsible AI literacy is developed. Doing so requires organizations to empower their employees to raise questions or concerns about AI systems to effectively control technology without limiting innovation.

Additionally, research should focus on facilitating a greater sense of belonging by creating inclusion and diversity strategies. Hence, research on responsible AI practices should identify opportunities to act in areas where AI meets human ingenuity. Doing so relies as much on formal control mechanisms from the organization as on informal and self-organizing mechanisms in stakeholder groups. Thus, relational practices in responsible AI governance should also be examined through the informal channels through which they are diffused and executed. Additionally, ensuring that external stakeholders are involved through appropriate formal and informal approaches and that there are effective control mechanisms to orchestrate and govern such relationships is critical for project outcomes.

Effects of responsible AI governance

The diffusion of AI applications and services has opened new opportunities for organizations, promising a competitive edge over their rivals. However, the lack of comprehensive responsible AI governance practices in organizations and limited knowledge about implementing such principles in practice can be problematic. The rationale for adopting responsible AI governance is tied to avoiding possible negative repercussions and formulating competitive strategies that build on key pillars. Similar to how corporate social responsibility has been linked to positive returns for firms that practice such approaches, so can responsible AI governance be tied to corresponding outcomes.

A recent study by Minkkinen et al. (2024) indicated that responsible AI governance practices are increasingly important in the context of environmental, social, and governance (ESG) assessments, which are key indicators of the corporate reputation of external actors and investors.

Recent commentaries have suggested that adopting responsible AI governance can generate value for companies by legitimizing their presence in a broader societal context and improving morale, productivity, and employee loyalty (de Laat, 2021; Martin & Waldman, 2023). At the organizational level, deploying and using responsible AI governance practices can enhance organizations’ sense of external legitimacy in broader ecosystems. Organizations that apply responsible AI governance across their various applications are often perceived as more reliable and trustworthy business partners, enhancing their external image among customers. Knowing the secure, transparent, and reliable use of data and AI applications can result in positive customer trust and an overall corporate image return.

With the prevalence of digital responsibility, cultivating a positive image and fostering appropriate practices to support it can increase the likelihood of other stakeholders engaging in business partnerships. This positive image of responsibility for using AI agents can lead to higher levels of profitability, greater customer retention and satisfaction, and greater ease of entering into strategic alliances and partnerships that benefit the focal organization.

Conversely, a lack of responsible AI practices has been associated with low job performance in different occupations and industries and AI-induced anxiety. When employees perceive that AI applications have been introduced in a manner that diminishes their autonomy and threatens their occupations, it can lead to lower levels of career satisfaction and organizational commitment. Nevertheless, effective, responsible AI governance can limit the feelings of threat that individuals in organizations perceive. Additionally, it can enhance synergy and seamless cooperation, eventually resulting in higher job satisfaction, productivity, and overall well-being. Nevertheless, we still lack empirical evidence on the effects of adopting responsible AI governance on internal operations and how it affects organizations’ competitive positioning in broader ecosystems.

A generally held assumption is that responsible AI is implemented primarily to adhere to societal expectations and norms; however, the role of the individual within an organization has been largely overlooked. Recent advancements in Generative AI technologies have shown that deploying AI applications in organizational operations can significantly impact the work life and well-being of employees. Ethical considerations for deploying AI in the workplace are manifold and raise questions about human identity, autonomy, and anxiety. In this regard, responsible AI governance must incorporate practices that prevent such occurrences and emphasize the well-being of employees.

As AI-based services and products are increasingly diffused in everyday life, their adoption by end users will affect the social perspective and shape the current norm around AI use. During this process, different AI service offerings are gauged by society, not only in terms of functionality and use but also in terms of their fit to societal norms and expectations. Organizations will, therefore, need to find suitable approaches to communicate their AI offerings and develop mechanisms to develop trust to facilitate adoption by end users. Research to date has provided limited insight into how organizations can foster a sense of trust and effective communication with the end users of their AI services.

Examples, such as Meta’s attempts to leverage user data to train AI models, have shown that insufficient communication practices and opaque approaches to data handling can lead to backlash from end users.4 Simultaneously, the relationship between AI offerings and societal perceptions and expectations is dynamic, with what is considered responsible and normatively aligned continuously evolving. Thus, not only do social norms and expectations shape what AI applications are deployed, but the way we interact with such technologies’ daily shapes are also considered normative and ethical. While users are aware of data privacy concerns related to AI applications, in numerous circumstances, they are willing to overlook such problems when the perceived value they realize through such applications is sufficiently high.

Consequently, deploying more sophisticated and complex AI applications that often go unnoticed in our interactions can lead us to re-evaluate what we consider to be the ethical and responsible use of AI. Thus, future research must examine the relationship between AI offerings from organizations and societal norms and expectations, as well as how developing tensions influence the evolution of responsible AI governance.

Conclusions.

In this study, we examine the notion of responsible AI and synthesize current knowledge from academic research. Additionally, we illustrate the significance of responsible principles and their applications in organizations and society. Based on this synthesis, we developed an extended conceptual framework (Fig. 2) for responsible AI governance that illustrates how organizations can apply responsible AI principles in their AI applications. We discuss the key dimensions of responsible AI governance and identify the key antecedents and outcomes. In conclusion, we propose a set of actionable research themes that can help expand our understanding of how responsible AI governance can be deployed and its impact on organizations and society.

Our study had certain limitations. First, while we applied a thorough search strategy, the resulting papers and the corresponding AI principles identified were predominantly derived from institutions and organizations in Europe and the USA; the resulting ethical and cultural norms that are present in such frameworks may partially reflect the formulation of responsible AI governance. Nevertheless, our framework and the proposed approach to implementing responsible AI governance are principally agnostic and thus can be applied in different contexts. Second, although we have synthesized a diverse and fragmented body of literature, our arguments and assumptions must be tested empirically, and the relationships we propose require further exploration. Finally, although we aimed to provide a set of themes for future research, the proposed research questions were neither exhaustive nor complete.

Many other open questions and essential themes will likely emerge as AI diffusion accelerates. Similarly, the emergence of regulations and directives, such as those of the AI Act, imposes ordinances on values that must be prioritized and conditions how responsible AI governance is perceived and thought of. How such regulations affect organizations’ responsibility for AI governance implementation and how society and end users perceive their effect remain unclear.

CRediT authorship contribution statement

Emmanouil Papagiannidis: Writing – review & editing, Writing – original draft, Project administration, Methodology, Conceptualization. Patrick Mikalef: Writing – review & editing, Writing – original draft, Supervision, Resources, Project administration, Conceptualization. Kieran Conboy: Writing – review & editing, Writing – original draft, Supervision, Conceptualization.

Declaration of competing interest influence the work reported in this paper.

Acknowledgements.

The authors have no conflict of interest, and the research did not receive any funding.