Charting Competence: A Holistic Scale for Measuring Proficiency in Artificial Intelligence Literacy

Abstract

The rapid evolution of AI technologies has reshaped our daily lives. As AI systems become increasingly prevalent, AI literacy, the ability to comprehend and engage with these technologies, becomes paramount in modern society. However, existing research has yet to establish a comprehensive framework for AI literacy. This study aims to fill this gap by developing a holistic AI literacy scale. Three levels of dimensions are considered: individual, interactive, and sociocultural. The scale includes cognitive, behavioral, and normative competencies. After rigorous reliability and validity assessments, the final AI literacy scale comprises six dimensions: AI features, AI processing, algorithm influences, user efficacy, ethical consideration, and threat appraisal. Detailed scale development, validation, and dimension-specific items are thoroughly explained. This comprehensive scale equips individuals with the competencies needed to navigate and critically engage with AI in today’s multifaceted AI landscape.

Keywords

artificial intelligence AI literacy AI competence assessment multi-level construct scale development

Introduction

In recent years, the rapid advancements in artificial intelligence (AI) technologies have transformed various aspects of our lives, profoundly impacting how we interact, communicate, and make decisions. AI technologies have been increasingly integrated into various systems, leading to their widespread adoption across online platforms, social networking sites, e-commerce websites, and beyond (Hancock et al., 2020; Stige et al., 2023; Zarouali et al., 2021). These AI-mediated systems employ specific applications or implementations of AI technology that simulates human intelligence in machines for purposes like recommendation or personalization to enhance user experience (Stige et al., 2023). As AI continues to shape our societies, it has become increasingly crucial for individuals to possess the necessary skills and knowledge to navigate and engage effectively with these AI-mediated technologies to understand AI functionalities, discern its impact, and make informed decisions (Fast & Horvitz, 2017; Ng et al., 2021, 2021b). This critical need has given rise to the concept of AI literacy, which can be defined as a multi-level capacity to comprehend the derivation of AI technologies from data processing to algorithm training, critically assess their applications and consequences, and proficiently interact with AI systems (Hancock et al., 2020; Long & Magerko, 2020; Ng et al., 2021, 2021b).

AI literacy overlaps with various literacies such as digital, media, information, algorithm, and data literacies, each of which addresses aspects of technological proficiency (Dogruel et al., 2022; Hargittai et al., 2020). However, they fall short in comprehensively capturing the complexities of AI literacy. While digital literacy focuses on basic technological skills, media literacy emphasizes critical evaluation of information; information literacy overlaps with media literacy to some degree and focuses on the access, use, creation, and evaluation of information; algorithm and data literacies center on understanding algorithms and data management, respectively (Brodsky et al., 2020; Claes & Philippette, 2020; Dogruel et al., 2022; Hargittai et al., 2020; Zarouali et al., 2021). None of these literacies holistically addresses the nuances of AI literacy, which extends beyond technical competence and information evaluation to encompass the ability to navigate complex AI-driven environments effectively (Hancock et al., 2020; Pinski & Benlian, 2023). In other words, AI literacy sets itself apart from algorithm or data literacies as it does not necessitate individuals to become experts in the underlying theory and model developments related to AI. Rather, being AI literate should indicate being proficient and having reasonable capability in using AI products effectively and ethically (Ng et al., 2021, 2021b; Wang et al., 2023).

The current approach to AI literacy predominantly centers on the cognitive aspect or confines itself to educational settings or specific use cases (Brodsky et al., 2020; Claes & Philippette, 2020; Dogruel et al., 2022; Hallaq, 2013; Hargittai et al., 2020; Hornberger et al., 2023; Kandlhofer et al., 2016; Kong et al., 2021; Laupichler et al., 2022; Wang et al., 2023). While these efforts are valuable in understanding AI within targeted contexts, they overlook the broader vision of AI literacy, which should encompass a more general and holistic perspective. AI literacy is not limited to mere technical understanding or restricted to particular environments; rather, it should embrace a multidimensional framework that encompasses individual, interactive, and sociocultural dimensions (Hancock et al., 2020; Long & Magerko, 2020). By taking a comprehensive and multi-level approach, AI literacy effectively addresses the intricacies of human-AI interaction and the evolving sociocultural impact of AI. This equips users with critical competencies for navigating diverse AI scenarios and making informed decisions in this evolving AI landscape (Chiu et al., 2024; Stolpe & Hallström, 2024).

This paper presents a significant contribution to the field by addressing the critical need for a comprehensive AI literacy scale. Recognizing the limitations of existing measurement approaches, our study endeavors to fill this gap by developing and validating a scale that captures the multidimensional nature of AI literacy. By delineating the individual, interactive, and sociocultural dimensions of AI literacy, our scale aims to offer a comprehensive grasp of the cognitive, behavioral, and normative competencies crucial for individuals to adeptly navigate the AI-mediated landscape. Notably, our study advances the rigor and generalizability of AI literacy scale validation by utilizing a national random sample, which ensures representation across diverse demographic factors such as gender, age, and region. Unlike prior research relying on convenience or student samples, our approach enhances the applicability and relevance of our findings to the broader societal context.

Literature Review

Media and Information Literacies

The term “literacy” originally refers to the ability to read and write (Gee, 1989). However, with the development of media and technologies, “media literacy” and “information literacy” were developed to understand the abilities needed to utilize media and information. Media literacy is related to the ability to access, analyze, evaluate, and create messages (Heins & Cho, 2003; Livingstone, 2004). Media literacy emphasizes the empowerment of audiences. Some scholars focused on information processing when referring to media literacy. For example, Potter (2004) proposed the Cognitive Model of Media Literacy and emphasized the delivering and processing of information. Four elements were proposed in this cognitive model, including knowledge structure, decisions motivated, information-processing tools, and the flow of information-processing tasks. The structure of the model illustrates the similarity between media literacy and information literacy. Some scholars adopted media and information literacy or new media literacy to refer to the consumption and utilization of media or information content (e.g., Datu et al., 2021; Schofield et al., 2023).

On the other hand, according to the Library and Information Association (2018), information literacy refers to the ability to think critically and make balanced and appropriate judgments about the information we search and use. This updated meaning of information literacy emphasizes the empowerment of users. McLure (2023) argues that information literacy includes traditional literacy, computer literacy, network literacy, and media literacy, with a focus on the information problem-solving skills. However, scholars have not yet reached an agreement on the definitions and relationship between media and information literacies.

As AI technologies are used to cope with “information” and “messages,” media and information literacies are by nature related to AI literacy. However, neither media literacy nor information literacy can capture the comprehensive meaning of AI literacy. Therefore, in this study, we also apply some concepts from media and information literacies to develop the AI literacy scale.

Defining AI Literacy

AI literacy is related to data and algorithm literacy as it entails the understanding and competence to navigate the interplay between data-driven technologies and the algorithms that drive AI systems. Scholars have proposed a concept of algorithm literacy that encompasses two key aspects: awareness of algorithm usage in online applications, platforms, and services, as well as comprehension of algorithmic workings, which involves understanding the various types, functions, and scopes of algorithmic systems on the internet (Dogruel et al., 2022; Hargittai et al., 2020). Algorithm-literate individuals are defined as those who possess the capacity to utilize strategies that allow them to alter predefined settings within algorithmically curated environments, such as social media newsfeeds or search engines, while protecting their privacy in the digital landscape (Dogruel et al., 2022). In other words, algorithm literacy refers to users’ awareness of the data extraction process, which means they are cognizant of the fact that data is collected from them, even if they might not fully understand the intricate details of how it happens or where it goes (Dogruel et al., 2022; Hargittai et al., 2020). This is similar to another neighboring concept, data literacy, which describes competences centering on users’ technical proficiency in retrieving or producing information as well as scrutinizing the politics of data and employing tactics to bypass negative influences from the recommender systems (Claes & Philippette, 2020).

In contrast, AI literacy goes beyond mere awareness, comprehension, or critical use of data and algorithms; instead, it encompasses users’ understanding of the entire data generation process and the likely ways in which this data is handled and processed by artificial intelligence systems (Chiu et al., 2024; Hancock et al., 2020; Long & Magerko, 2020; Stolpe & Hallström, 2024; Wang et al., 2023). So, while algorithm literacy is about being aware of data extraction, AI literacy delves deeper into understanding the entire data lifecycle and the role of AI in it. It is important to distinguish AI literacy from related data and algorithm literacy in that tools that employed AI algorithms permeate in our daily lives that go beyond recommender systems; moreover, AI algorithms are distinct from other algorithms due to their ability to learn from data, adapt, and make decisions based on patterns and experiences in solving complex problems (Carolus et al., 2023; Chiu et al., 2024; Laupichler et al., 2023; Stolpe & Hallström, 2024; Wang et al., 2023). The ability to learn from data empowers AI to iterate and optimize autonomously. However, it is critical to recognize that this capability is closely tied to the availability and quality of data.

When users lack a comprehensive understanding of AI and its functioning, several consequences may arise. They may misinterpret AI-generated results, leading to misguided decisions or even make decisions based on biases introduced by AI (Yuan et al., 2023). Worse, unaware of inherent AI biases, users may unknowingly perpetuate discrimination or cause ethical dilemmas (Madaio et al., 2020; Sokol, 2019). In the data handling process, privacy and security concerns may emerge as users remain unaware of AI modeling practices (Mittelstadt, 2019). Moreover, a lack of trust in AI systems can develop, hindering their adoption (Abdul et al., 2018; Miller, 2019b; Wang et al., 2019). Overall, promoting AI literacy is crucial to empower users, enabling them to make informed and responsible decisions while maximizing AI’s benefits and minimizing potential pitfalls. While there is widespread recognition of the importance of AI literacy (Druga et al., 2019; Long & Magerko, 2020; Ng et al., 2021b), a standardized measure of AI literacy has yet to be established.

In order to promote a scale for AI literacy, our endeavor goes beyond previous efforts by introducing a validated measure that comprehensively captures users’ knowledge, awareness, and practices revolving around AI and AI-mediated technologies at the individual, interactive, and sociocultural levels. This holistic perspective ensures the scale’s applicability across diverse contexts, not limited to any specific application.

Existing Approaches towards AI Literacy

AI literacy is an emerging concept driven by the rapid advancement and widespread commodification of AI technologies. From the technical perspective, to enhance human-AI interaction, various public and private organizations have proposed fairness guidelines, including the Organization for Economic Cooperation and Development¹, European Union², Microsoft³, and Google⁴. These guidelines commonly emphasize AI models’ fairness, accountability, transparency, and privacy (Jobin et al., 2019). However, researchers highlight that these guidelines may ignore users’ domain-, context-, or system-specific practices (Madaio et al., 2020). Moreover, the derivation and practical usefulness of some guidelines remain unclear and unsupported, with certain guidelines being based solely on researchers’ intuition (Miller, 2019b; Wang et al., 2019).

Subsequently, researchers delved into the conceptualization of AI literacy, focusing on educators’ and learners' perspectives (Hornberger et al., 2023; Kong et al., 2021; Laupichler et al., 2022; Ng et al., 2021b; Wang et al., 2023) or in specific contexts, such as curriculum development for courses, grades, or fields (Deuze & Beckett, 2022; Kandlhofer et al., 2016; Perchik et al., 2023; Zhang et al., 2022). Other studies focused on general users’ AI literacy on social network sites or embedded newsfeed curation (Brodsky et al., 2020; Schwartz & Mahnke, 2021). This conceptual exploration has incorporated both technical and social aspects of AI literacy, which encompasses three crucial dimensions: technological knowledge, involving the use of AI tools; pedagogical knowledge, encompassing learning styles related to AI; and content knowledge, which involves awareness and ethics pertaining to AI (Ng et al., 2021b).

Despite making significant contributions to the conceptual understanding of AI literacy, the majority of these studies were qualitative analyses centered around user perceptions of algorithmic curation within specific services (Brodsky et al., 2020; Deuze & Beckett, 2022; Kandlhofer et al., 2016; Laupichler et al., 2022; Ng et al., 2021b). Those studies conducted in educational settings may be constrained in their sample (Carolus et al., 2023; Hornberger et al., 2023; Kong et al., 2021; Wang et al., 2023). Also, the context- and population-specific measures while on the one hand stay closely tied to the content or context of the evaluation, it may weaken their generalizability for a diverse population or AI applications (Carolus et al., 2023).

Other endeavors include the attitudinal aspect of human-AI interaction, focusing on users’ general attitudes towards AI (Schepman & Rodway, 2020; Sindermann et al., 2021) or the perceived anxiety in relation to AI use (Wang & Wang, 2022). While these scales are relevant for understanding users’ stances or attitudes towards AI and its applications, it is essential to note that they do not directly address the meaning and definition of AI literacy itself.

Not until more recently have scholars started to tackle the competences users may need to equip when interacting with AI (Chiu et al., 2024). Among them, an awareness of identifying and comprehending AI while using it, an ability to leverage various AI-mediated technologies to accomplish tasks, a critical evaluation to analyze and select proper AI-mediated technologies and their outcomes, and knowledge about AI-related risks and ethical concerns (Carolus et al., 2023; Laupichler et al., 2023; Pinski & Benlian, 2023; Wang et al., 2023) were included. Expanding beyond the previously mentioned dimensions, Carolus et al. (2023) integrated theoretical frameworks like the theory of planned behavior and self-efficacy to propose the inclusion of self-management, attitudes towards AI, and willingness to adopt AI in a meta-AI literacy scale. Weber et al. (2023) distinguished between the social and technical aspects of human-AI interaction, and they also accounted for different actors involved in the process, including creators, evaluators, and average users in their scale.

The previous studies predominantly concentrate on the cognitive aspect of AI literacy (Laupichler et al., 2023; Pinski & Benlian, 2023; Wang et al., 2023) or include exogenous and endogenous aspects of AI or different actors for scale development instead of focusing on individuals’ psychometric values of AI literacy (Carolus et al., 2023; Weber et al., 2023). By drawing on pertinent theories and frameworks related to both key dimensions of literacy and AI development, the AI literacy scale proposed in this study distinguishes ourselves by concentrating on universal AI literacy for average users at the individual, interactive, and sociocultural levels. This comprehensive approach offers a holistic understanding of individuals’ proficiency and capabilities in interacting with AI technologies, taking into account their cognitive abilities, interactive responses towards AI, and critical evaluations of AI’s broader impacts.

The Current Study: Proposed Dimensions of AI Literacy

In this study, we define AI literacy across three levels: individual, interactive, and sociocultural level, each corresponding to a specific competence, namely cognitive, behavioral, and normative competence, essential for proficiency required within each level.

Individual Level: Cognitive Dimensions

Bloom (1956) presented a diverse set of competencies that served as a universal language for learning, organized based on the underlying educational objective and independent of the specific domain. Consequently, this comprehensive framework has been instrumental in categorizing and evaluating diverse cognitive abilities and educational objectives across a wide range of learning contexts (Weber et al., 2023) and guided the development of various literacy scales. Among the competences proposed by Bloom (1956) and contextualized in AI literacy, an understanding of what AI is, what AI can do, and how AI works constitutes core cognitive dimensions at the individual level (Laupichler et al., 2023; Long & Magerko, 2020; Wang et al., 2023; Weber et al., 2023).

In the rapidly evolving technological landscape, where AI-mediated technologies are becoming increasingly prevalent, the ability to discern which technologies incorporate AI and which do not holds paramount importance. This skill is related to individuals’ knowledge about what AI can do and how AI works, which taken together can empower individuals to make well-informed decisions, comprehend potential implications, and engage responsibly with AI-driven systems that significantly impact various aspects of their daily lives (Laupichler et al., 2023; Long & Magerko, 2020; Wang et al., 2023; Weber et al., 2023).

Having an awareness of algorithm use and knowledge about algorithms are regarded as meta-skills and essential prerequisites for acquiring additional proficiencies in the domain of modern technology and data-driven environments (Dogruel et al., 2022; Hargittai et al., 2020; Schwartz & Mahnke, 2021). Here knowledge refers to the particular prerequisites necessary for comprehension, such as declarative knowledge and factual information (Wilson et al., 2015). With knowledge, individuals may develop skills essential for effectively utilizing and applying this knowledge (Wilson et al., 2015). These foundational competences provide individuals with the ability to navigate, understand, and interact effectively with various AI-driven systems, creating opportunities for further growth and development in the dynamic digital landscape.

Additionally, Bloom suggests to incorporate more contextualized capability where individuals know how to make applications by implementing procedures in given contexts (Bloom, 1956). Artificial intelligence is often applied in content filtering and automated decision making (Zarouali et al., 2021). Content filtering involves algorithms used to curate and present relevant subsets of content from a vast corpus due to an explosive growth and diversity of information on the internet. For example, to avoid overwhelming users with choices and to enhance relevance (Schreiner et al., 2019), social media often leverage AI technologies to filter newsfeed or posts, e-commerce websites recommend relevant product offerings, streaming platforms display interested video overviews, and search engines orient results (Hancock et al., 2020). Then, while AI has increasingly been used to select content for end-users, the results are used for assisting users to make decisions or even replacing human-based decisions with efficient and optimized algorithmic processes (Van Dijck et al., 2018). Having the knowledge of how AI can be applied and make influences in technologies that individuals encounter is essential in comprehending how AI shapes online environments and crucial in developing AI literacy.

Taken together, we propose to include “What AI is,” “What AI can do,” “How AI works,” “content curation,” and “automated decision-making” for cognitive dimensions of AI literacy (see Table 1).

Table 1.

Proposed Dimensions of AI Literacy.

	Level	Dimensions	Competencies
1	Individual	Cognitive	1. What AI is
			2. What AI can do
			3. How AI works
			4. Content curation
			5. Automated decision-making
2	Interactive	Behavioral	1. Human-AI interactions
2	Interactive	Behavioral	2. Response efficacy
3	Sociocultural	Normative	1. Threat appraisal
3	Sociocultural	Normative	2. Ethical considerations over AI use

Interactive Level: Behavioral Dimensions

Apart from the cognitive dimensions, scholars argue that AI literacy also encompasses individuals’ capacity to critically assess AI technologies, effectively communicate and collaborate with AI, and proficiently utilize AI in various contexts (Long & Magerko, 2020; Pinski & Benlian, 2023; Weber et al., 2023). In other words, behavioral dimensions have been structured in line with the socio-technical perspective at the operational level, underscoring the importance of users’ interactions with AI and their ability to adapt when encountering various AI-mediated outcomes, even in biases or errors. The variations in interactions reflect users' mental models when dealing with AI and AI-mediated technologies (Wang et al., 2023).

The efficacy to respond and interact with AI emerges as a fundamental capacity in human-AI interaction, encompassing the ability to effectively address and navigate AI-mediated issues, such as biases or anomalies, ensuring responsible and informed engagement with AI technologies (Ashrafi & Easmin, 2023). According to the social cognitive theory (Stajkovic & Luthans, 1998) and in the context of human-AI interaction, self-efficacy plays a crucial role in determining individuals’ confidence and belief in their ability to effectively engage with AI technologies. People who can respond to AI with higher efficacy may have confidence and belief in their ability to effectively interact with AI technologies. Higher levels of response efficacy in AI usage can lead to greater engagement, exploration, and mastery of AI-related tasks, fostering a deeper understanding of AI systems and functionalities. At the interactive level, we include the dimensions of “human-AI interactions,” and individuals’ “response efficacy” in responding to vairous situations when applying AI.

Sociocultural Level: Normative Dimensions

AI technologies play a pivotal role in mediating both social and technical interactions (Weber et al., 2023; Ågerfalk, 2020). Therefore, it becomes essential to uphold social norms and values that emphasize risk-free, ethical, and transparent AI modeling and application during the deployment of AI and in human-AI interactions (Miller, 2019a). These principles are vital to ensure individuals to understand responsible and equitable use of AI technologies in various societal contexts (Wang et al., 2023).

The importance of contextualizing technologies and evaluating their results or applications has long been emphasized in Bloom’s core competencies and appropriated in various kinds of literacy, such as digital and media literacy (Bloom, 1956). While media and information literacy primarily centers on evaluating information quality (Hallaq, 2013), in the context of AI literacy, evaluation also involves users forming accurate opinions about AI applications and products, including how the applications may yield biases, have risk concerns, or present partial perspectives. In the realm of AI technologies, the inherent risks of biased outcomes due to data processing and model training highlight the importance of individuals possessing the capacity to evaluate potential AI-related risks, spanning from biases to privacy concerns. This capability is crucial for fostering a heightened awareness of the broader societal impact of AI (Mitchell et al., 2019; Yuan et al., 2023). The recognition of AI biases goes beyond technical considerations, intertwining with ethical dimensions across diverse cultural and social contexts (Jobin et al., 2019; Ng et al., 2021a; Zhang et al., 2022). Understanding the ethical implications and potential threats associated with AI use cultivates a sense of responsibility among users, encouraging ethical behavior and decision-making (Hancock et al., 2020; Zhang et al., 2022). Additionally, users who possess the ability to evaluate an AI application or product typically demonstrate a rich experience of using various AI applications or products. In sum, we include “threat appraisal” and “ethical considerations over AI use” at the sociocultural level.

Methods

Development of the AI Literacy Scale

The instrument development for assessing AI literacy follows a series of steps: (1) generating an item pool based on literature review; (2) consulting AI and literacy experts to ensure content validity; (3) selecting and modifying the items based on experts’ opinions; (4) conducting cognitive interviews to evaluate readability; (5) administering the survey; (6) analyzing the data and revising the framework. These steps are explained in detail below.

Step 1: Item Generation

To generate items based on our theoretical framework, we drew on prior works to define the main dimensions and associated items regarding what AI is, what AI can do, how AI works, content curation, automated decision-making, human-AI interaction, response efficacy, threat appraisal, and ethical considerations over AI use (Gupta et al., 2021; Long & Magerko, 2020; Zarouali et al., 2021). Finally, the item pool consisted of 38 items rated on a 5-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree (see Table 2). Given that this study was conducted in Taiwan, all items were back-translated into Mandarin Chinese.

Table 2.

Item Development, Selection, and Modifications.

Dimension	Original definition	Modification	Revised item
Individual level: cognitive dimensions
a. What AI is	1. Distinguish between technological artifacts that use and do not use AI.		AL1. I Can distinguish between technological artifacts that use and do not use AI.
	2. Critically analyze and discuss features that make an entity “intelligent”, including discussing differences between human, animal, and machine intelligence.		AL2. I Can identify which features make a technology product “intelligent.”
	3. Recognize that there are many ways to think about and develop “intelligent” machines. Identify a variety of technologies that use AI, including technology spanning cognitive systems, robotics, and ML.		AL3. I Know AI can be applied to different technologies.
	4. Distinguish between general and narrow AI.	Deleted (a)
b. What AI can do	5. Identify problem types that AI excels at and problems that are more challenging for AI.		AL4. I Know what AI can do and what it cannot do yet.
	6. Imagine possible future applications of AI and consider the effects of such applications on the world.		AL5. I Can imagine possible future applications of AI and consider the effects of such applications on the world.
c. How AI works	7. Understand what a knowledge representation is and describe some examples of knowledge representations.	Deleted (b)
	8. Recognize and describe examples of how computers reason and make decisions.		AL6. I Know how AI makes decisions.
	9. Understand the steps involved in machine learning and the practices and challenges that each step entails, including data gathering and preparation, model selection, training, testing, and prediction.		AL7. I Understand the steps involved in machine learning and the practices and challenges that each step entails, including data gathering and preparation, model selection, training, testing, and prediction.
	10. Recognize that humans play an important role in programming, choosing models, and fine-tuning AI systems.		AL8. I Know that humans play an important role in training AI, choosing models, and fine-tuning AI systems.
	11. Understand basic data literacy concepts.		AL9. I Have basic data literacy, including understanding data types and values, evaluating data sources, interpreting data, and managing and making use of data.
	12. Recognize that computers often learn from data (including one’s own data).		AL10. I Know that machine learning deals with identifying patterns and correlations based on data.
	13. Understand that data cannot be taken at face-value and requires interpretation. Describe how the training examples provided in an initial dataset can affect the results of an algorithm.		AL11. I Have the ability to interpret data critically, and I understand how data processing can impact the results generated by machine learning algorithms.
	14. Understand that some AI systems have the ability to physically act on the world. This action can be directed by higher-level reasoning or it can be reactive.	Deleted (a)
	15. Understand what sensors are, recognize that computers perceive the world using sensors, and identify sensors on a variety of devices. Recognize that different sensors support different types of representation and reasoning about the world	Deleted (a)
	16. Understand that agents are programmable.	Deleted (b)
d. Content curation	17. Algorithms are used to recommend [media content] to me on [platform name]	Deleted (b)
	18. Algorithms are used to prioritize certain [media content] above others.		AL16. I Know that algorithms tend to prioritize certain content above others.
	19. Algorithms are used to tailor certain [media content] to me on [platform name].		AL17. I Know that algorithms are used to deliver tailored content to each user.
	20. Algorithms are used to show someone else different [media content] than I get to see on [platform name].		AL18. I Know that algorithms can show diverse content to different people.
e. Automated decision making	21. Algorithms are used to show me [media content] on [platform name] based on automated decisions.		AL19. I Know that the content featured on AI platforms is automatically determined by algorithms.
	22. Algorithms do not require human judgments in deciding which [media content] to show me on [platform name].		AL20. I Know that human judgement is not required when algorithms recommend content.
	23. Algorithms make automated decisions on what [media content] I get to see on [platform name]	Deleted (b)
Interactive level: Behavioral dimensions
f. Human-AI interaction	24. The [media content] that algorithms recommend to me on [platform name] depend on my online behavior on that platform	Deleted (b)
	25. The [media content] that algorithms recommend to me on [platform name] depend on my online behavioral data	Deleted (b)
	26. The [media content] that algorithms recommend to me on [platform name] depend on the data that I make available online	Deleted (b)
g. Response efficacy	27. I Am confident to protect my personal data while using AI.		AL24. I Am confident to protect my personal data while using AI.
	28. When encountering unexpected AI-related problems, I am confident in interpreting the results or solving the issue.		AL25. When encountering unexpected AI-related problems, I am confident in interpreting the results or solving the issue.
	29. When encountering unexpected AI-related problems, I can find someone for help.	Deleted (b)
	30. When encountering unexpected AI-related problems, it takes me much effort to address (reversed).	Deleted (a)
Sociocultural level: Normative dimensions
h. Ethical considerations over AI use	31. Identify and describe different perspectives on the key ethical issues surrounding AI (i.e., privacy, employment, misinformation, the singularity, ethical decision making, diversity, bias, transparency, accountability).		AL12. I Know that biases in data, such as gender, race, and age, can lead to ethical and moral issues in AI applications.
	32. It is not always transparent why algorithms decide to show me certain [media content] on [platform name].		AL13. I Know that the way algorithms recommend content is not always transparent.
	33. The [media content] that algorithms recommend to me on [platform name] can be subjected to human biases such as prejudices and stereotypes.		AL14. I Know that algorithmic recommendations can be influenced by people’s prejudices and stereotypes.
	34. Algorithms use my personal data to recommend certain [media content] on [platform name], and this has consequences for my online privacy.		AL15. I Know that when algorithms use my personal data to recommend content to me, it has consequences for my online privacy.
i. Threat appraisal	35. While using AI, there are privacy related risks to be present in the system	Deleted (b)
	36. While using AI, there is a risk that the system may act in discord with the intentions of its users (Malevolent AI).		AL21. While using AI, there is a risk that the system may act in discord with the intentions of its users.
	37. While using AI, there is a risk that the algorithm may overlook or lack human values.		AL22. I Know that when using AI, there is a risk that the algorithm may overlook or lack human values.
	38. Sharing personal information to AI-based firm(s) can be risky.		AL23. I Know that sharing personal information to AI-based firm(s) can be risky.

Note. (a) Expert consultation stage: I-CVI <.78; (b) Item selection stage.

Ensuring Content Validity

Step 2: Consultation With Experts

Then, we invited seven professors who are experts in the fields of artificial intelligence (n = 3) and information/media literacy (n = 4) to review our item pool and provide feedback on its comprehensiveness and appropriateness. Expert consultation was part of our effort to reach content validity, or theoretical analysis, where we confirmed the adequacy, relevance, and representation of our items to assess the target domain of interest (Boateng et al., 2018; Morgado et al., 2017).

Each expert independently reviewed the entire item set and rated each item on a 4-point ordinal scale where 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant. In addition, they were encouraged to leave comments on each item and to suggest wordings, revise aspects, or propose new items. Each expert was compensated with NTD$2,500 (approximately USD$81.5) for their valuable insights and suggestions.

Following Lynn (1986), we calculated the content validity index for each item (I-CVI). We divided the number of experts who rated it as either 3 or 4 by the total number of experts. According to the item relevancy criteria developed by Lynn (1986), the I-CVI score should meet at least .78 for six to ten experts.

At this stage, four items with less than .78 expert agreement were excluded from the scale (4, 14, 15, 30). Considering the structure of the constructs, we kept item 13 with a score of .67. We also slightly modified the wording of most items based on expert suggestions. The I-CVI of the remaining items ranges from .67 to 1.00.

Step 3: Item Selection and Modification

After generating and revising the item pool based on the literature review and expert consultation, we narrowed the items down to the final set of 25 depending on the content relevance, construct complexity, distinction across items or dimensions, and comprehensibility (see Table 2) (Carpenter, 2018). The aim was to ensure that the selected items adequately represented the various aspects of AI literacy and could effectively differentiate among individuals with different levels of AI literacy.

Based on the comments of the experts, we re-examined the content relevance of the items. Items that did not accurately represent the construct of AI literacy or were too specific to cover various types of AI applications and scenarios were excluded. Next, items were assessed for clarity and complexity. Items that were complex or ambiguous in wordings or concepts were revised or removed to ensure that they could be easily understood by respondents. Finally, items were evaluated to ensure a balance between different dimensions of AI literacy. This ensured that the scale was comprehensive and did not disproportionately focus on one dimension of AI literacy at the expense of others. We also removed redundant items that represent similar ideas to other existing items. The scale was designed to be completed in a reasonable amount of time to minimize participant fatigue and maximize response quality.

Step 4: Cognitive Interview: Readability Evaluations by a Sample of Target Population

Ensuring that scale items are understood by the target population is critical to the reliability and validity of the resulting measures (Desimone & Le Floch, 2004). Evaluation by target population is an important step to evaluate face validity, a core component of content validity, where they assess the appropriateness of the items to the target construct (Boateng et al., 2018). Therefore, we conducted cognitive interviews (n = 8) to elicit participants’ interpretations and responses to survey items and to gain insight into potential problems with the wording, clarity, or relevance of individual items.

Given the intended use of the literacy scale for the general population, we conducted cognitive interviews with a diverse sample of participants reflecting the target demographic. Our sample included four females and four males from various age groups and educational backgrounds. Specifically, we engaged three students - one from junior high school, one from high school, and one from the university level - as participants. Additionally, we included two adults who graduated from high school, two who graduated from university, and one with a doctorate degree. Two of them are aged 13 to 20, another two are between the ages of 21 and 30, an additional two fall within the age range of 31 to 65, and the remaining two are aged 65 years above (see Table 3).

Table 3.

Cognitive Interview Participants.

Participants	Age	Educational backgrounds	Gender
P01	14	Junior high school student	Female
P02	18	High school student	Female
P03	22	University student	Male
P04	23	University degree	Male
P05	41	Doctoral degree	Male
P06	53	High school degree	Female
P07	66	University degree	Female
P08	70	High school degree	Male

During each cognitive interview, the participants were asked to complete the entire AI Literacy Scale and share their thoughts on the items, including their comprehension of the item wording and any difficulties in terms of interpreting the items they encountered. The interviewer also asked follow-up questions to probe participants’ thought processes and gained additional insights regarding wording revisions. The research team then reviewed the respondents' feedback and modified the items to ensure that they were clear, understandable, and accurately reflected the construct of AI literacy.

Step 5: Field Test

Procedure

Empirical data were collected using an online survey tool to assess the AI Literacy Scale from September to October, 2022. To achieve sample representativeness, our study recruited from a panel whose members matched to the demographics of Taiwan census in terms of gender, age, and region according to statistics released by the Taiwanese Ministry of the Interior. Eligible participants were 13–70 years and those who live in Taiwan. Participants received a compensation of NTD$ 100 (USD$3).

In addition to the AI literacy scale, our participants completed information and media literacy and demographic questions such as gender, age, education, region, and occupation. The study was conducted according to the ethical guidelines by the institutional review board approval.

Participants

We recruited a total of 1173 participants from a panel with census-matched quotas to reflect the national Taiwanese population, among whom 50.3% were female and 49.7% were male. Age ranged from 13 to 70 with a mean of 42.88 years old (SD = 15.0). Age distribution was as follows: 13-20 (9.5%), 21-30 (16.4%), 31-40 (18.2%), 41-50 (20.1%), 51-60 (19.3%), and 61-70 (16.5%). In terms of educational level, 3.9% had a junior high school degree or below, 21.7% had a senior high school degree, 61.3% had a college degree, and 13.0% had a graduate degree or above. As for living regions, 47.9% from the northern region, 22.7% from the central region, 26.8% from the southern region, and 2.6% from the eastern region. All participants provided informed consent before proceeding the survey. Data were collected anonymously, and all personal identifiers were removed to protect participants’ privacy and confidentiality.

Results

To test the factorial structure of our measurement instrument, we performed a sequence of confirmatory factor analyses with Mplus 8 (Muthén & Muthén, 2017). The analyses were run in maximum likelihood estimation with robust standard errors (MLR), which does not assume strict normality (see Table 4). In the first step, we loaded all items on the factor AI literacy (model a) (Podsakoff et al., 2003). The single-factor model (unidimentionality) results in an insufficient model fit and indicates that our proposed measurement model was rid of confirmation bias (Shah & Goldstein, 2006). Then, based on the conceptual structure of the items, we tested if the items loaded on six factors (i.e., what AI is, how AI works, ethical consideration, curation and decision making, threat appraisal, and response efficacy), which yielded an insufficient model fit (model b).

Table 4.

Confirmatory Factor Analysis Summary of the Model Fit Indicators.

Model	χ²	df	p	RMSEA	90% CI	CFI	SRMR	χ²-difference test
Model	χ²	df	p	RMSEA	90% CI	CFI	SRMR	TRd	Δdf	p
(a)	2301.78	275	.00	.08	(.08; .08)	.80	.08	--	--	--
(b)	1444.61	260	.00	.06	(.06; .07)	.89	.08	694.79	15	.00
(c)	905.06	237	.00	.05	(.05; .05)	.93	.06	606.32	23	.00
(d)	513.89	219	.00	.03	(.03; .04)	.97	.05	376.37	18	.00

Note. Based on the conventional threshold for each index, a non-significant Satorra-Bentler scaled chi-square value is considered a good model fit, for it suggests the hypothesized model is not different from the observed data. The RMSEA (root mean-square error of approximation) should be less than .05. The lower bound of its 90% CI should be less than .05 and upper bound .10. The CFI (comparative fit index) should be larger than .90 and the SRMR (standardized root mean square residual) smaller than .08 (Kline, 2015).

In the subsequent steps (model c), we made the following changes: (1) We removed item AL20 that showed a low factor loading and unfit with all dimensions; (2) AL21, 22 from the factor decision-making were moved to ethical consideration; (3) AL19 from the factor decision-making was moved to algorithm influences; (4) A separate factor named user efficacy was created with AL24, 25 and the factor curation and decision-making were removed. Last, a sequence of model modifications was pursued in an effort to improve the overall model fit (model d).

Validity: Construct, Discriminant, Convergent, Criterion

For the scale development, we established construct validity through the use of theory and expert consultation. According to the model results, the average dimension extracted (AVE) of each dimension is above .50, suggesting good convergent validity (Fornell & Larcker, 1981). Then discriminant validity was established if the square root of the AVE for a particular dimension is greater than its correlation with all other dimensions, which is supported by our data (see Table 5). Last, we statistically established criterion validity, also called predictive validity, by examining the correlation between information literacy and AI literacy (see Table 6). We expected that people with higher information literacy had higher AI literacy. Results of bivariate correlations between AI literacy and information literacy indicate that there was a positive correlation between the two variables, r (1171) = .52, p < .001.

Table 5.

Composite Validity, Discriminant Validity, and Correlations.

	AVE	Alpha	CR	1	2	3	4	5
Individual level: Cognitive dimensions
AI features	.51	.85	.86
AI processing	.55	.86	.86	.86**
Algorithm influences	.58	.85	.85	.66**	.70**
Interactive level: Behavioral dimensions
User efficacy	.50	.70	.66	.53**	.46**	.30**
Sociocultural level: Normative dimensions
Ethical consideration	.57	.87	.87	.62**	.66**	.81**	.28**
Threat appraisal	.59	.75	.74	.52**	.55**	.73**	.15**	.77**

Note: ** Correlation is significant at the 0.01 level (2-tailed).

Square root of AVE in diagonal to indicate discriminant validity; CR to indicate composite reliability (Fornell & Larcker, 1981).

Table 6.

Criterion-Related Validity.

	Function consumption	Critical consumption	Functional prosumption	Critical prosumption
Individual level: Cognitive dimensions
AI features	.45**	.42**	.42**	.37**
AI processing	.43**	.38**	.43**	.38**
Algorithm influences	.40**	.30**	.39**	.32**
Interactive level: Behavioral dimensions
User efficacy	.22**	.32**	.22**	.23**
Sociocultural level: Normative dimensions
Ethical consideration	.39**	.28**	.34**	.31**
Threat appraisal	.33**	.20**	.26**	.26**

Note: ⁺p < .10, *p < .05, **p < .01, ***p < .001.

**Correlation is significant at the 0.01 level (2-tailed).

Reliability: Composite, Item-Total, Split-Half

We reported the internal consistency with cronbach’s alpha as well as composite reliability derived from the CFA in Table 5. It is suggested that constructs with fewer items tend to result in lower reliability, whereas those with more tend to have higher reliability (Brunner & Süβ, 2005). A decent reliability level is between .60 to .70 for exploratory research (Hair et al., 2021). More specifically, constructs with five to eight items should have meet a minimum threshold of .80 (Netemeyer et al., 2003) but not exceed .90 as it indicates that they are highly identical to others, or called uni-dimensionality (Hair et al., 2021; Nunnally & Bernstein, 1994).

Then, we performed item-total reliability that examined the adjusted correlation of each item with the total score of the scale excluding itself (Boateng et al., 2018). Item-total correlation helps determine the degree to which items of a scale measure similar content. A correlation below .30 is the deletion threshold (see Table 7; numbers in brackets). Last, we computed split-half reliability to compare group differences of scale internal consistency (Hedge et al., 2018). Split-half reliability has been established to produce consistent and trustworthy evaluation of cognitive tasks at the group, instead of individual level, with Spearman-Brown adjusted Pearson correlation coefficient r = .96 (a value above .80 is acceptable) (Pronk et al., 2022).

Table 7.

The Final Framework of the Scale.

Individual level: Cognitive dimensions	AI features	1. I Can distinguish between technological artifacts that use and do not use AI. (.70) [.62]
		2. I Can identify which features make a technology product “intelligent.” (.72) [.66]
		3. I Know AI can be applied to different technologies. (.71) [.69]
		4. I Know what AI can do and what it cannot do yet. (.72) [.66]
		5. I Can imagine possible future applications of AI and consider the effects of such applications on the world. (.71) [.70]
		6. I Know how AI makes decisions. (.63) [.52]
	AI processing	7. I Understand the steps involved in machine learning and the practices and challenges that each step entails, including data gathering and preparation, model selection, training, testing, and prediction. (.72) [.66]
		8. I Know that humans play an important role in training AI, choosing models, and fine-tuning AI systems. (.73) [.68]
		9. I Have basic data literacy, including understanding data types and values, evaluating data sources, interpreting data, and managing and making use of data. (.73) [.68]
		10. I Know that machine learning deals with identifying patterns and correlations based on data. (.77) [.73]
		11. I Have the ability to interpret data critically, and I understand how data processing can impact the results generated by machine learning algorithms. (.75) [.70]
	Algorithm influences	12. I Know that algorithms tend to prioritize certain content above others. (.78) [.65]
		13. I Know that algorithms are used to deliver tailored content to each user. (.77) [.68]
		14. I Know that algorithms can show diverse content to different people. (.78) [.70]
		15. I Know that the content featured on AI platforms is automatically determined by algorithms. (.72) [.67]
Interactive level: Behavioral dimensions	User efficacy	16. I Am confident to protect my personal data while using AI. (.64) [.45]
		17. When encountering unexpected AI-related problems, I am confident in interpreting the results or solving the issue. (.77) [.62]
Sociocultural level: Normative dimensions	Ethical consideration	18. I Know that biases in data, such as gender, race, and age, can lead to ethical and moral issues in AI applications. (.74) [.64]
		19. I Know that the way algorithms recommend content is not always transparent. (.74) [.65]
		20. I Know that algorithmic recommendations can be influenced by people’s prejudices and stereotypes. (.76) [.68]
		21. While using AI, there is a risk that the system may act in discord with the intentions of its users. (.77) [.62]
		22. I Know that when using AI, there is a risk that the algorithm may overlook or lack human values. (.75) [.63]
	Threat appraisal	23. I Know that when algorithms use my personal data to recommend content to me, it has consequences for my online privacy. (.76) [.68]
		24. I Know that sharing personal information to AI-based firm(s) can be risky. (.78) [.36]

Note. factor loadings (in parentheses), and item-total correlations (in brackets).

Discussion

The final AI literacy scale consists of six sub-constructs at three levels (see Table 7), including (1) individual level (cognitive dimensions): AI features, AI processing, algorithm influences; (2) interactive level (behavioral dimensions): user efficacy; (3) sociocultural level (normative dimensions): ethical consideration and threat appraisal. Our scale’s reliability and validity was substantiated through rigorous processes, affirming the appropriateness of the theoretical model grounded in distinct dimensions as the most suitable conceptualization for AI literacy. The robustness of the reliability and validity tests underscored the precision and accuracy of our scale’s design, further enhancing its credibility and relevance in assessing individuals’ AI literacy levels. The developed scale serves practical purposes in both general use and educational settings. For general use, it facilitates the assessment of users’ capabilities in understanding and effectively using AI technologies. In educational settings, the instrument guides the development of AI-related training programs, enhancing students’ proficiency in AI technologies and equipping them for future careers in AI-driven industries.

At the individual level, the core competencies of AI, such as understanding AI features and AI processing, remain consistent as they form the foundational knowledge essential for all types of learning about AI, echoing Bloom’s framework (Bloom, 1956). On the other hand, the contextualized applications, including content curation and automated decision-making, are combined because they represent specific instances of how AI is utilized in different technologies and platforms. These applications may vary and evolve over time as technology advances, making them subject to change depending on the current state of AI development and its applications in various domains. By focusing on core competencies and integrating contextualized applications, our approach provides a balanced and adaptable framework to understand and engage with AI across different technological landscapes.

At the interactive level, items for the human-AI interaction dimension were dropped during the model modification processes. It is possible that the items have a substantial overlap with those of response efficacy, which drew on the social cognitive theory (Stajkovic & Luthans, 1998). Consequently, the final dimension of user efficacy proved to be sufficiently parsimonious in capturing the diverse capabilities required for interacting with AI under various conditions. The consolidation of user efficacy as the primary dimension for evaluating human-AI interaction ensures a concise and effective assessment of users’ proficiency in engaging with AI technologies across different scenarios.

Lastly, at the sociocultural level, our findings highlight ethical consideration and threat appraisal as core dimensions of AI literacy. Recognizing the critical ethical issues associated with AI development and application, as well as understanding the potential risks to users’ privacy, emerges as a crucial capability when engaging with AI technologies. Moreover, by understanding the ethical considerations and risks associated with AI, users can actively engage in discussions and advocate for responsible AI development and regulation. This level of awareness empowers individuals to hold AI developers and organizations accountable for the ethical implications of their technologies (Akata et al., 2020; Gupta et al., 2021). AI technologies often involve the processing and analysis of vast amounts of data, which can raise privacy concerns (Jobin et al., 2019; Sokol, 2019). Being cognizant of the potential risks to one’s privacy when interacting with AI helps users take proactive measures to safeguard their personal information and maintain control over how their data is used.

The utility of the developed AI literacy scale can be extended to various scenarios. For example, in professional environments, organizations can leverage the scale to assess the AI literacy levels within different departments and offer insights into necessary AI training initiatives so as to reduce information asymmetry and facilitate improved inter-departmental cooperation in AI-related projects across divisions. Moreover, the scale finds a pivotal role in evaluating the effectiveness of existing computer-mediated education programs concerning AI. Institutes can use the scale to identify gaps and areas for improvement, ensuring that educational initiatives align with the dynamic landscape of AI technologies. Additionally, the AI literacy scale contributes to educational contexts by guiding the design of AI-related curriculum development. It enhances students’ competence in navigating AI technologies, preparing them for roles in industries increasingly reliant on AI. Beyond career preparedness, the scale facilitates a nuanced exploration of AI literacy in diverse educational contexts, contributing to a more comprehensive understanding of AI’s impact on learning and knowledge acquisition.

Limitations and Future Directions

While our research endeavors to provide a comprehensive AI literacy scale, there are certain limitations that should be acknowledged. Firstly, despite the use of back-translation from English, the scale development and testing were conducted primarily in Mandarin, which may introduce cultural and language-specific nuances. Therefore, to ensure broader applicability and cross-cultural validity, we strongly encourage further validation of the scale in different languages and across diverse countries. Additionally, while we validated the scale with a representative national sample in Taiwan, further generalizability of the findings to more diverse global populations is encouraged. Future research should aim to include participants from various cultural backgrounds and geographical regions to strengthen the scale’s validity and robustness in capturing the multi-faceted nature of AI literacy across different contexts. Thirdly, despite selecting confirmatory factor analysis to test our models, we introduced modifications to the original model during the analysis process. As a result, it is essential to approach our methodology as exploratory, recognizing the iterative nature of our approach and the adjustments made to achieve a more comprehensive understanding of the AI literacy construct. Last, as our research was developed and conducted at a time when there were no published AI literacy scales available, it highlights the pioneering nature of our study in addressing this emerging field. However, with the subsequent emergence of other AI literacy scales, conducting a comparison with these existing instruments regarding their predictive validity becomes even more pertinent. By utilizing these scales to forecast the quality of future AI-related behavior, we can gain valuable insights into the effectiveness and potential contributions of our AI literacy scale. This comparative analysis will not only validate the relevance of our approach but also provide a comprehensive understanding of how our scale complements and enhances the current landscape of AI literacy assessment.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by National Science and Technology Council; 110-2410-H-A49-045, 111-2634-F-002 -022, 112-2221-E-002 -187, 112-2634-F-002 -006.

ORCID iDs

Chien Wen (Tina) Yuan

Hsin-yi Sandy Tsai

Yu-Ting Chen

Notes

Author Biographies

Chien Wen (Tina) Yuan (Ph.D. from Cornell University) is an associate professor at National Taiwan University’s Transdisciplinary Bachelor Degree Program in D-school. Her research primarily focuses on mediated interpersonal relationships, human-computer interaction (HCI), and computer-supported collaborative work (CSCW). Recently, Dr. Yuan has explored topics such as human-AI interaction, the impact of social media, and the dynamics of mediated collaboration.

Dr. Hsin-yi Sandy Tsai is an associate professor in the Department of Telecommunication and Communication Management Science (TCM) and the Institute of Telecommunications Management at National Cheng Kung University (NCKU) in Taiwan. She received her Ph.D. in Media and Information Studies from Michigan State University. Her research interests include the policies, use, and impacts of information and communication technologies. She is especially interested in how to help people improve their quality of life and take full advantage of new technologies. She is currently working on research projects regarding AI literacies, smart cities and well-being, and digital inclusion.

Yu-Ting Chen currently serves as a research assistant in Professor Yuan’s team at National Taiwan University. Her research interests include human-computer interaction (HCI), computer-supported collaborative work (CSCW), and human-AI collaboration. She is particularly focused on creating innovative and inclusive user-driven solutions that leverage technology to mitigate social inequities and enhance information accessibility.

References

Abdul

Vermeulen

Wang

Lim

B. Y.

Kankanhalli

(2018). Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. In: Proceedings of the 2018 CHI conference on human factors in computing systems. ACM.

Ågerfalk

P. J.

(2020). Artificial intelligence as digital agency. European Journal of Information Systems, 29(1), 1–8. https://doi.org/10.1080/0960085x.2020.1721947

Akata

Balliet

De Rijke

Dignum

Eiben

Fokkens

Grossi

Hindriks

Hoos

Hung

Jonker

Monz

Neerincx

Oliehoek

Prakken

Schlobach

van der Gaag

van Harmelen

Welling

(2020). A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. Computer, 53(8), 18–28. https://doi.org/10.1109/mc.2020.2996587

Ashrafi

D. M.

Easmin

(2023). Okay google, good to talk to you… examining the determinants affecting users’ behavioral intention for adopting voice assistants: Does technology self-efficacy matter? International Journal of Innovation and Technology Management, 20(02), Article 2350004. https://doi.org/10.1142/s0219877023500049

Bloom

(1956). Bloom’s taxonomy.

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quiñonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

Brodsky

J. E.

Zomberg

Powers

K. L.

Brooks

P. J.

(2020). Assessing and fostering college students’ algorithm awareness across online contexts. Journal of Media Literacy Education, 12(3), 43–57. https://doi.org/10.23860/jmle-2020-12-3-5

Brunner

Süβ

H.-M.

(2005). Analyzing the reliability of multidimensional measures: An example from intelligence research. Educational and Psychological Measurement, 65(2), 227–240. https://doi.org/10.1177/0013164404268669

Carolus

Koch

Straka

Latoschik

M. E.

Wienrich

(2023). MAILS--Meta AI literacy scale: Development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change-and meta-competencies. arXiv preprint arXiv:2302.09319 .

10.

Carpenter

(2018). Ten steps in scale development and reporting: A guide for researchers. Communication Methods and Measures, 12(1), 25–44. https://doi.org/10.1080/19312458.2017.1396583

11.

Chiu

T. K.

Ahmad

Ismailov

Sanusi

I. T.

(2024). What are artificial intelligence literacy and competency? A comprehensive framework to support them. Computers and Education Open, 6(6), Article 100171. https://doi.org/10.1016/j.caeo.2024.100171

12.

Claes

Philippette

(2020). Defining a critical data literacy for recommender systems: A media-grounded approach. Journal of Media Literacy Education, 12(3), 17–29. https://doi.org/10.23860/jmle-2020-12-3-3

13.

Datu

J. A. D.

Wong

G. S. P.

Rubie-Davies

(2021). Can kindness promote media literacy skills, self-esteem, and social self-efficacy among selected female secondary school students? An intervention study. Computers & Education, 161, Article 104062. https://doi.org/10.1016/j.compedu.2020.104062

14.

Desimone

L. M.

Le Floch

K. C.

(2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22. https://doi.org/10.3102/01623737026001001

15.

Deuze

Beckett

(2022). Imagination, algorithms and news: Developing AI literacy for journalism. Digital Journalism, 10(10), 1913–1918. https://doi.org/10.1080/21670811.2022.2119152

16.

Dogruel

Masur

Joeckel

(2022). Development and validation of an algorithm literacy scale for internet users. Communication Methods and Measures, 16(2), 115–133. https://doi.org/10.1080/19312458.2021.1968361

17.

Druga

S. T.

Likhith

Qiu

(2019). Inclusive AI literacy for kids around the world. In Proceedings of FabLearn 2019 (pp. 104–111). ACM.

18.

Fast

Horvitz

(2017). Long-term trends in the public perception of artificial intelligence. In Proceedings of the AAAI conference on artificial intelligence. AAAI.

19.

Fornell

Larcker

D. F.

(1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. Sage Publications Sage CA.

20.

Gee

J. P.

(1989). What is literacy. Journal of Education, 171(1), 18–25. https://doi.org/10.1177/002205748917100102

21.

Gupta

Kamboj

Bag

(2021). Role of risks in the development of responsible artificial intelligence in the digital healthcare domain. Information Systems Frontiers, 25(6), 2257–2274. https://doi.org/10.1007/s10796-021-10174-0

22.

Hair

J. F.

Hult

G. T. M.

Ringle

C. M.

Sarstedt

(2021). A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publications.

23.

Hallaq

T. G.

(2013). Evaluating online media literacy in higher education: Validity and reliability of the digital online media literacy assessment (DOMLA). Idaho State University.

24.

Hancock

J. T.

Naaman

Levy

(2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89–100. https://doi.org/10.1093/jcmc/zmz022

25.

Hargittai

Gruber

Djukaric

Fuchs

Brombach

(2020). Black box measures? How to study people’s algorithm skills. Information, Communication & Society, 23(5), 764–775. https://doi.org/10.1080/1369118x.2020.1713846

26.

Hedge

Powell

Sumner

(2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. https://doi.org/10.3758/s13428-017-0935-1

27.

Heins

Cho

(2003). Media Literacy: An Alternative to Censorship. Free Expression Policy Project. https://www.academia.edu/81929477/Media_Literacy_An_Alternative_to_Censorship

28.

Hornberger

Bewersdorff

Nerdel

(2023). What do university students know about artificial intelligence? Development and validation of an AI literacy test. Computers and Education: Artificial Intelligence, 5(1), Article 100165. https://doi.org/10.1016/j.caeai.2023.100165

29.

Jobin

Ienca

Vayena

(2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2

30.

Kandlhofer

Steinbauer

Hirschmugl-Gaisch

Huber

(2016). Artificial intelligence and computer science in education: From kindergarten to university. In 2016 IEEE frontiers in education conference (FIE). IEEE.

31.

Kline

R. B.

(2015). Principles and practice of structural equation modeling. Guilford Publications.

32.

Kong

S.-C.

Cheung

W. M.-Y.

Zhang

(2021). Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence, 2(4), Article 100026. https://doi.org/10.1016/j.caeai.2021.100026

33.

Laupichler

M. C.

Aster

Raupach

(2023). Delphi study for the development and preliminary validation of an item set for the assessment of non-experts' AI literacy. Computers and Education: Artificial Intelligence, 4(8), Article 100126. https://doi.org/10.1016/j.caeai.2023.100126

34.

Laupichler

M. C.

Aster

Schirch

Raupach

(2022). Artificial intelligence literacy in higher and adult education: A scoping literature review. Computers and Education: Artificial Intelligence, 3(4), Article 100101. https://doi.org/10.1016/j.caeai.2022.100101

35.

Livingstone

(2004). Media literacy and the challenge of new information and communication technologies. The Communication Review, 7(1), 3–14. https://doi.org/10.1080/10714420490280152

36.

Long

Magerko

(2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI Conference on human factors in computing systems. ACM.

37.

Lynn

M. R.

(1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017

38.

Madaio

M. A.

Stark

Wortman Vaughan

Wallach

(2020). Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In Proceedings of the 2020 CHI Conference on human factors in computing systems. ACM.

39.

McLure

(2023). The thinking frames approach: Improving high school students’ written explanations of phenomena in science. Research in Science Education, 53(1), 173–191. https://doi.org/10.1007/s11165-022-10052-y

40.

Miller

(2019a). “But why?” Understanding explainable artificial intelligence. XRDS: Crossroads, The ACM Magazine for Students, 25(3), 20–25. https://doi.org/10.1145/3313107

41.

Miller

(2019b). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267(2), 1–38. https://doi.org/10.1016/j.artint.2018.07.007

42.

Mitchell

Zaldivar

Barnes

Vasserman

Hutchinson

Spitzer

Raji

I. D.

Gebru

(2019). Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency. ACM.

43.

Mittelstadt

(2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507. https://doi.org/10.1038/s42256-019-0114-4

44.

Morgado

F. F.

Meireles

J. F.

Neves

C. M.

Amaral

Ferreira

M. E.

(2017). Scale development: Ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica, 30, 3. https://doi.org/10.1186/s41155-016-0057-1

45.

Muthén

(2017). Mplus. Chapman and Hall/CRC.

46.

Netemeyer

R. G.

Bearden

W. O.

Sharma

(2003). Scaling procedures: Issues and applications. Sage publications.

47.

D. T. K.

Leung

J. K. L.

Chu

K. W. S.

Qiao

M. S.

(2021a). AI literacy: Definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58(1), 504–509. https://doi.org/10.1002/pra2.487

48.

D. T. K.

Leung

J. K. L.

Chu

S. K. W.

Qiao

M. S.

(2021b). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, Article 100041. https://doi.org/10.1016/j.caeai.2021.100041

49.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychological theory. McGraw-Hill.

50.

Perchik

Smith

Elkassem

Park

Rothenberg

Tanwar

Sturdivant

Tridandapani

Sotoudeh

(2023). Artificial intelligence literacy: Developing a multi-institutional infrastructure for AI education. Academic Radiology, 30(7), 1472–1480. https://doi.org/10.1016/j.acra.2022.10.002

51.

Pinski

Benlian

(2023). AI literacy-towards measuring human competency in artificial intelligence.

52.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J.-Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879

53.

Pronk

Molenaar

Wiers

R. W.

Murre

(2022). Methods to split cognitive task data for estimating split-half reliability: A comprehensive review and systematic assessment. Psychonomic Bulletin & Review, 29(1), 44–54. https://doi.org/10.3758/s13423-021-01948-3

54.

Schepman

Rodway

(2020). Initial validation of the general attitudes towards artificial intelligence scale. Computers in Human Behavior Reports, 1(2), Article 100014. https://doi.org/10.1016/j.chbr.2020.100014

55.

Schofield

Kupiainen

Frantzen

Novak

(2023). Show or tell? A systematic review of media and information literacy measurements. The Journal of Media Literacy education, 15(2), 124–138. https://doi.org/10.23860/JMLE-2023-15-2-9

56.

Schreiner

Rese

Baier

(2019). Multichannel personalization: Identifying consumer preferences for product recommendations in advertisements across different media channels. Journal of Retailing and Consumer Services, 48(C), 87–99. https://doi.org/10.1016/j.jretconser.2019.02.010

57.

Schwartz

S. A.

Mahnke

M. S.

(2021). Facebook use as a communicative relation: Exploring the relation between Facebook users and the algorithmic news feed. Information, Communication & Society, 24(7), 1041–1056. https://doi.org/10.1080/1369118x.2020.1718179

58.

Shah

Goldstein

S. M.

(2006). Use of structural equation modeling in operations management research: Looking back and forward. Journal of Operations Management, 24(2), 148–169. https://doi.org/10.1016/j.jom.2005.05.001

59.

Sindermann

Sha

Zhou

Wernicke

Schmitt

H. S.

Sariyska

Stavrou

Becker

Montag

(2021). Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language. KI - Künstliche Intelligenz, 35(1), 109–118. https://doi.org/10.1007/s13218-020-00689-0

60.

Sokol

(2019). Fairness, accountability and transparency in artificial intelligence: A case study of logical predictive models. In Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society. ACM.

61.

Stajkovic

A. D.

Luthans

(1998). Social cognitive theory and self-efficacy: Goin beyond traditional motivational and behavioral approaches. Organizational Dynamics, 26(4), 62–74. https://doi.org/10.1016/s0090-2616(98)90006-7

62.

Stige

Å.

Zamani

E. D.

Mikalef

Zhu

(2023). Artificial intelligence (AI) for user experience (UX) design: A systematic literature review and future research agenda. In Information technology & people. https://doi.org/10.1108/itp-07-2022-0519

63.

Stolpe

Hallström

(2024). Artificial intelligence literacy for technology education. Computers and Education Open, 6(3), Article 100159. https://doi.org/10.1016/j.caeo.2024.100159

64.

Van Dijck

Poell

De Waal

(2018). The platform society: Public values in a connective world. Oxford University Press.

65.

Wang

Rau

P.-L. P.

Yuan

(2023). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929x.2022.2072768

66.

Wang

Yang

Abdul

Lim

B. Y.

(2019). Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI Conference on human factors in computing systems. ACM.

67.

Wang

Y.-Y.

Wang

Y.-S.

(2022). Development and validation of an artificial intelligence anxiety scale: An initial application in predicting motivated learning behavior. Interactive Learning Environments, 30(4), 619–634. https://doi.org/10.1080/10494820.2019.1674887

68.

Weber

Pinski

Baum

(2023). Toward an objective measurement of AI literacy.

69.

Wilson

Scalise

Gochyyev

(2015). Rethinking ICT literacy: From computer skills to social network settings. Thinking Skills and Creativity, 18(3-4), 65–80. https://doi.org/10.1016/j.tsc.2015.05.001

70.

Yuan

C. W.

Lin

Y.-F.

Tseng

Y.-H.

(2023). Contextualizing user perceptions about biases for human-centered explainable artificial intelligence. In Proceedings of the 2023 CHI Conference on human factors in computing systems. ACM.

71.

Zarouali

Boerman

S. C.

de Vreese

C. H.

(2021). Is this recommended by an algorithm? The development and validation of the algorithmic media content awareness scale (AMCA-scale). Telematics and Informatics, 62(2), Article 101607. https://doi.org/10.1016/j.tele.2021.101607

72.

Zhang

Lee

Ali

DiPaola

Cheng

Breazeal

(2022). Integrating ethics and career futures with technical learning to promote AI literacy for middle school students: An exploratory study. International Journal of Artificial Intelligence in Education, 33, 1–35. https://doi.org/10.1007/s40593-022-00293-3