EBD-enabled Approach to Improving the Efficiency of Developing Information Literacy Assessment Criteria

Abstract

This study proposed an Environment-Based Design (EBDEA) to develop a draft of information literacy assessment criteria (ILAC), to improve the efficiency of developing ILAC. The approach is validated using two methods. Firstly, a case study is conducted to create ILAC for K-12 students by the EBDEA, resulting in four first-tier and 21 s-tier criteria. These were compared with the ILAC from the International Association for Evaluation of Educational Achievement (IEA). The comparison revealed a high degree of consistency between the two sets of ILAC, with the EBDEA-generated ILAC including several additional items that are integral to the criteria but absent in IEA's version. Subsequently, expert evaluation was employed to affirm the effectiveness of the EBDEA, with the majority of experts expressing satisfaction with the ILAC developed via this method. The findings indicate that EBDEA is an efficient approach for developing ILAC, requiring less time and fewer human resources.

Keywords

information literacy assessment criteria environment-Based design (EBD)EBD-enabled approach (EBDEA)IEA

Introduction

Information literacy is a crucial competency for the twenty-first century (Doyle et al., 2016; Henkel et al., 2018; Joseph et al., 2018; Jun et al., 2014; Naito, 2019; Tewell, 2015), which involves the effective use of a variety of information tools and key information resources (Zurkowski, 1974). Information literacy competence and skills are critical contributors to workplace efficiency and success (Tachie-Donkor & Ezema, 2023). Researchers have pointed out that information skills are essential for locating accurate information, and beneficial for high-quality decision-making (Raeis et al., 2013). Rader (1990, 2019) regarded information literacy as the foundation of skills and knowledge vital for lifelong learners. Given its impact on nearly every individual in aspects of life, work, and education, developing appropriate Information Literacy Assessment Criteria (ILAC) is essential for effective educational programs that foster excellence in students’ information literacy (Aliyyah et al., 2020; Bachen et al., 2018; DeLuca et al., 2016; Skedsmo & Huber, 2021).

ILAC refers to the set of standards and measures used to assess an individual's ability to seek, evaluate, use, and create information effectively. These criteria are grounded in academic research and theoretical frameworks that define the essential components of information literacy. ILAC typically covers several dimensions, including awareness of information, use of information, and understanding of legal and ethical issues related to information (Shi et al., 2018). The general process for developing ILAC involves five steps: (1) a researcher panel identifies the objectives and targets of the information literacy assessment; (2) the researcher panel collects information from an expert panel to draft the initial version of ILAC; (3) the expert panel assesses the draft criteria using methods like Delphi, Analytic Hierarchy Process (AHP), and Data Envelopment Analysis (DEA); (4) the researcher panel refines the ILAC draft based on expert feedbacks and comments; and (5) the researcher panel concludes the final results when the experts reach a consensus on the effectiveness of ILAC.

According to the general process to develop the ILAC, it is found that the construction of the initial draft of ILAC is pivotal to subsequent stages. The first draft determines the overall structure and thus the quality of the final ILAC. However, the first draft of ILAC is currently created by the researcher panel based on their expertise and with the help of the experience and knowledge of the expert panel. The success of this process relies significantly on three factors: (1) if the researchers can quickly and accurately collect the necessary information from the expert panel; (2) if experts can provide effective evaluation and feedback in the evaluation stage; and (3) if the researcher panel can make effective use of the expert evaluation and feedback to improve the draft of ILAC. Thus, it is essential to investigate how the researcher panel can effectively gather information from the expert panel and other pertinent sources to develop the ILAC draft.

This paper utilized the Environment-Based Design (EBD) methodology (Zeng, 2004, 2015) to construct the initial draft of ILAC to efficiently and effectively help researchers collect information from experts and other reliable sources. The EBD methodology guides the recursive collection of necessary information during the information-gathering process (Yang et al., 2023; Zeng & Cheng, 1991). Unlike experience-based methods, EBD utilizes domain-independent question-asking and answering algorithms and templates to collect information (Zeng, 2015). EBD categorizes the environment into human, built, and natural components (Zeng, 2015). Consequently, the EBD-enabled approach (EBDEA) enhances the objectivity of outcomes, reduces reliance on experts, shortens information collection time, lowers development costs, and boosts the efficiency of developing ILAC. The EBD methodology has proven its versatility across a wide range of applications, including KPI development (Wang et al., 2024), system evolution analysis (Razavi et al., 2024), policy and criteria design (Chen et al., 2022; Du et al., 2023), literature review methods (Mohammadi et al., 2024), process analysis (Yang et al., 2022), natural language process (Cheligeer et al., 2023).

The rest of the paper is structured as follows: Section 2 outlines the detailed process of the EBDEA in the case of constructing a draft of ILAC for K-12 students in China and validation of the EBDEA. Section 3 presents the EBDEA's results to develop ILAC and the effectiveness validation of the EBD approach. In Section 4, the designed ILAC and the development efficiency are compared with the well-known criteria from the International Association for the Evaluation of Educational Achievement (IEA) (Fraillon et al., 2013). Finally, the conclusion of this study is provided in Section 5.

Methods

There are two parts in the methods section: the construction of an ILAC and the validation of the designed ILAC. Firstly, the EBD methodology was utilized in the construction process to develop criteria for the effective assessment of information literacy among K-12 students in China. Subsequently, the Delphi method was employed to validate the developed ILAC using a questionnaire distributed to experts in related fields, thereby refining the assessment criteria and validating the EBDEA. The overall process of developing the ILAC using the EBDEA is depicted in Figure 1.

Figure 1.

The overall process to develop the ILAC by the EBD-enabled approach.

From a design perspective, if we consider the information literacy assessment to be the product, then the ILAC is part of the environment. One of the three main activities in the EBD methodology, environment analysis, is therefore used to develop the ILAC.

Developing the Draft ILAC by EBD

The construction of an ILAC draft for K −12 students in China is chosen as the case study to illustrate the development process through the EBD methodology. One participant, hailing from a university in Central China with proficient training in information literacy assessment and a strong grasp of the EBD methodology, used this approach to develop the ILAC for K-12 students in China. The environment analysis consists of the following steps: 1) Recursive object model (ROM) diagram creation; 2) Question generation; 3) Question answering; 4) Output the ROM diagram into ILAC.

ROM Diagram Creation

The creation of the ROM diagram stems from the design problem statement, which is to describe the objective of environment analysis (Wang & Zeng, 2009). The design problem statement is summarized as “to develop criteria for effective assessment of information literacy of K-12 students by using EBD approach”.

ROM, which breaks down a sentence into individual words, is a graphic language that conveys the semantic information embedded within a natural language-based design problem description (Zeng, 2008). The ROM diagram consists of two objects and three relationships, as shown in Table 1 (Zeng, 2008, 2015).

Table 1.

Symbols in ROM.

	Type	Description
Object	Object	Everything in the universe is an object
Object	Compound object	It is an object that includes at least two objects
Relation	Connection relation	It is a descriptive, limiting, or particularizing relation of one object to another
	Connection	It is to connect two objects that do not constrain each other
	Predicate relation	It describes an act of an object on another or that describes the states of an object

Following the design problem statement, the information is transformed into two types of graphic symbols. The first type represents the types of objects, including “criteria”, “assessment”, “literacy”, “students”, and “approach”. The second type of graphic symbol represents the relationship between objects. In addition to connection, the other two types of relationships among the objects are identified: constraint relation: “for”, “effective”, “of”, “information”, “K-12”, “by”, “EBD” and predicate relations: “develop”, “using”. Using these graphic symbols, the objects and relations are connected, resulting in the ROM diagram of the problem statement, as shown in Figure 2.

Figure 2.

Environment analysis: the ROM diagram of problem statement.

Question Generation

This step involves asking the right questions based on a question template (Wang & Zeng, 2009). The objects of generating questions include the components and relationships in the ROM diagram. According to the ROM diagram, it is evident that the components “literacy” and “assessment” have the most undefined constraint relations, indicating that “literacy” and “assessment” are the most critical components. Based on the question-generating strategy (Zeng, 2020), the generic questions regarding the objects constraining the critical components of “literacy” and “assessment” should be defined first, followed by the 5W1H (who, what, why, when, where, how) questions to further understand the objects “student” and “develop”, and then “literacy” and “assessment”. The questions for each object are listed in Table 2.

Table 2.

Questions for the Objects Analysis.

No.	Questions
Q1	Why should criteria be developed for the effective assessment of information literacy for students?
Q2	How should criteria be developed for the effective assessment of information literacy for students?
Q3	Who will develop criteria for the effective assessment of information literacy for students?
Q4	When should criteria be developed for the effective assessment of information literacy for students?
Q5	Where should criteria be developed for the effective assessment of information literacy for students?
Q6	What is a K-12 student?
Q7	What is information?
Q8	What is information literacy?
Q9	What is an effective assessment of information literacy for students?

Question Answering

After asking the questions, answers should be provided according to the guidelines to supply sufficient and necessary information at that moment (Zeng, 2015). The answers can be sourced from external resources, such as the internet, dictionaries, and knowledge bases (Wang & Zeng, 2009; Zeng & Gu, 1999). A comprehensive answer to a “what” question should cover the definition and lifecycle of the queried object. It is important to note that all the answers are derived from literature published before 2013 to maintain consistency with the external social environment accessible to the experts who developed the IEA's ILAC. This design lays the groundwork for validating the effectiveness of the EBD methodology. The answers to the questions are collected, as displayed in Table 3.

Table 3.

Answers to Questions in Table 2.

No.	Answer
Q1	To assess the information literacy level of students.
Q2	To use the EBD approach.
Q3	The present designer.
Q4	When educational institutions and organizations want to assess the information literacy level of students.
Q5	A research station.
Q6	Students at the primary education stage from kindergarten to 12^th grade.
Q7	Knowledge communicated or received concerning a particular fact or circumstance. The lifecycle of information includes creation and receipt, storage, distribution and transmittal, access and use, maintenance, disposition, and destruction (Bernard, 2007). For creating and receiving information, the natural environment can be ignored while the built environment is a technological resource, and the human environment consists of the students. For storage, the natural environment can be ignored, and the built environment is the space for storage and storage tools; the human environment is the students. Regarding distribution and transmittal, the natural environment can be ignored, and the built environment is the distributed and transmitted information and tools, the human environment is the students. For accessing and using information, the natural environment can be ignored, the built environment is the resource and tools to access and use the information, and the human environment is the students. For the event of maintenance, the natural environment can be ignored, and the built environment is maintenance resources and tools, the human environment is the students. For the event of disposition and destruction, the natural environment can be ignored, and the built environment is tools and social law, the human environment is the students.
Q8	The abilities to realize when we need the information to solve problems, identify and define, determine, find, create, understand, organize, analyze, interpret and evaluate, communicate and present, utilize, preserve, store, reuse, record, and archive, dispose of, and safeguard information in an ethical and moral manner (Horton, 2007). For the event of identifying and defining information, the natural environment can be ignored, and the built environment is knowledge resources; the human environment is students. For the event of finding information, the natural environment can be ignored, and the built environment is the technological tool; the human environment is students. For the event of creating information, the natural environment can be ignored while the built environment includes creation tools and resources, and the human environment is students. For the event of understanding information, the natural environment can be ignored, and the built environment is social law; the human environment is students. For the event of organizing information, the natural environment can be ignored, and the built environment is the tools for organizing the information; the human environment is students. For the event of analyzing, interpreting, and evaluating the information, the natural environment can be ignored, and the built environment is analysis and evaluation methods; the human environment is students. For the event of communicating and presenting information, the natural environment can be ignored while the built environment is technological tools and knowledge resources, and the human environment is students. For the event of utilizing information, the natural environment can be ignored while the built environment is utilization tools, and the human environment is students. For the event of preserving, storing, reusing, recording, and archiving information, the natural environment can be ignored, and the built environment includes all places for keeping information; the human environment is students. For the event of disposing and safeguarding information, the natural environment can be ignored, while the built environment is technological tools, and the human environment is students.
Q9	An effective judgment about the levels of information literacy.
Q10	Rules or principles for evaluating the levels of information literacy. The life cycle of criteria includes design, formation, use, maintenance, and disposal. For the event of requirement analysis, the natural environment can be ignored while the built environment is the analysis tool and the human environment is students. For the event of design, the natural environment can be ignored while the built environment is the design tool and the human environment is students. For the event of formation, the natural environment can be ignored. The built environment is the ROM diagram while the human environment is students. For the event of using criteria, the natural environment can be ignored, and the built environment is the use of tools; the human environment includes educational institutes and organizations. For the event of maintaining criteria, the natural environment can be ignored; the built environment includes technical resources and maintenance tools, and the human environment is inexperienced students. For the event of disposal, the natural environment can be ignored, while the built environment is technological tools, and the human environment is students.

For a clearer understanding, the ROM diagram can be updated with the following definitions:

- Information literacy is the ability to use information knowledge to realize when we need information, identify and define, determine, find, create, understand, organize, analyze, interpret and evaluate, communicate and present, utilize, preserve, store, reuse, record, and archive, dispose of, safeguard information ethically and morally. Students with no experience in information literacy will develop criteria to assess the information literacy level among students.

- Criteria are the rules or principles for evaluating or testing information literacy levels.

- Educational students will design the criteria using the EBD approach when educational institutions and organizations need to evaluate the information literacy levels of K-12 students.

- The development of criteria includes considering design, manufacture, sales, transportation, use, maintenance, and recycling.

- An assessment is deemed effective if it is efficient, satisfactory, and economical to use.

- Information is knowledge communicated or received concerning a specific fact or circumstance. It includes creation and receipt, storage, distribution and transmittal, access and use, maintenance, disposition, and destruction of information.

The procedure of ROM diagram creation is then applied in this part. After all the questions are answered, the ROM diagram is updated. If a design problem is still considered undefined, sections 2.1.1 to 2.1.3 should be repeated to clarify the design problem. This recursive process continues until no new questions arise, ultimately in a merged ROM diagram with a satisfying problem description. The merged ROM and corresponding new questions will be provided in Section 3.

Output the ROM Diagram into ILAC

The ROM diagrams reveal the relationships between objects. Based on these relationships, the hierarchical structures of the objects can be represented. Consequently, the ILAC can be developed from the final merged ROM diagram.

Conventional Way of Developing ILAC using the Delphi Method

The Delphi method is a structured, iterative, anonymous, yet effective approach for forecasting by converging on a possibility value through the feedback mechanism of multiple rounds of questionnaire results, based on experts’ opinions and judgments regarding complex issues or problems (Khodyakov et al., 2023; Linstone & Turoff, 2011). The objective of the Delphi method is to achieve a consensus among experts on a specific topic or issue while accommodating diverse perspectives and opinions.

The process of conducting a Delphi study typically involves the following steps: (1) Selection of panelists: A diverse group of experts or stakeholders is selected to participate in the study. The panelists should have relevant experience and knowledge of the topic or issue being studied. (2) Development of questionnaires: A series of questions or statements are developed and tailored to the specific topic or issue. The questions should be objective, neutral, and open-ended, allowing for diverse perspectives and opinions to be expressed. (3) Distribution and collection of responses: The questionnaires are distributed to the panelists and collected for review after each round. Typically, multiple rounds are conducted to allow for time for participants to consider their responses and provide feedback on the previous round's results. (4) Analysis and feedback: The collected responses are analyzed and summarized, typically in the form of group discussions or reports. The results are then provided back to the panelists for their review and feedback on the next round's questionnaire, and (5) Iterations and consensus building: The process is repeated until a consensus is reached among the panelists or a predetermined number of rounds has been completed. Depending on the specific objectives of the study, the final results may be used for decision-making, policy development, or other purposes.

The Delphi method is a useful tool for obtaining expert opinions on complex topics or issues that require a broad range of perspectives and inputs. It allows for anonymous participation, reduces personal bias, and provides a structured process for generating consensus among experts. A well-known criteria designed by IEA was used to assess student's level of information literacy.

Validation of the Initial Draft of ILAC

To verify the effectiveness of the EBDEA, the designed ILAC is compared with the IEA's. Additionally, a questionnaire was developed from the initial draft of ILAC (see Appendix B) and sent to 20 experts via email to gauge their satisfaction with each criterion. These experts included local university lecturers and doctoral students whose research focused on information literacy and its assessment. A five-point Likert scale was employed, ranging from “1 = very dissatisfied” to “5 = very satisfied”. Experts returned their completed questionnaires by email. Data analysis could be presented numerically or graphically.

Results

Draft ILAC by EBD

Updated ROM Diagrams, Questions, and Answers

The responses from Table 3 were integrated into the ROM diagram shown in Figure 2, resulting in the updated ROM diagram in Figure 3. To clarify undefined objects in Table 3, more questions were asked, and the questions and answers were described in Appendix A. These responses were incorporated into the ROM diagram in Figure 3, leading to the final ROM diagram depicted in Figure 4.

Figure 3.

The updated ROM diagram.

Figure 4.

The final ROM diagram.

Formation of the ILAC

We simplified the final ROM diagram aiming to have a better review of the main structure, as shown in Figure 5. The hierarchical structure of ILAC can be represented, as shown in Figure 6. Thus, the final version of the designed ILAC was shown in Table 4.

Figure 5.

Simplified final ROM diagram.

Figure 6.

The hierarchical structure of ILAC.

Table 4.

The Designed ILAC.

First-tier criteria	Second-tier criteria
A. Information ability	A1 Identity
	A2 Locate
	A3 Access
	A4 Store
	A5 Manage
	A6 Integrate
	A7 Review
	A8 Evaluate
	A9 Use
	A10 Share
	A11 Communicate
	A12 Create
B. Information awareness	B1 Attitude
	B2 Information requirements awareness
	B3 Participative awareness
	B4 Critical awareness
	B5 Information security awareness
	B6 Life-long learning awareness
C. Knowledge about information	C1 Understanding
D. Ethics of information	D1 Legal use
D. Ethics of information	D2 Social responsibility

Validation

Comparison of the Designed Criteria Contents

A comparison of ILAC designed by the IEA and EBDEA is conducted, as shown in Table 5.

Table 5.

Comparison of ILAC Designed by the IEA and EBD-enabled Approach.

ILAC designed by the IEA		ILAC designed by EBD
First-tier criteria	Second-tier criteria	First-tier criteria	Second-tier criteria
1. Collecting and managing information	Know and understand	1. Information Awareness	Attitude
	Know and understand		Information requirements awareness
	Access and evaluate		Participative awareness
	Access and evaluate		Critical awareness
	Manage		Information security awareness
			Life-long learning awareness
		2. Knowledge about information	Understand
2. Producing and exchanging information	Transform	3. Information ability	Identity
			Locate
			Access
	Create		Store
			Manage
			Integrate
	Share		Review
			Evaluate
			Use
			Sharing
	Use		Communicate
			Create
		4. Ethics of information	Legal use
		4. Ethics of information	Social responsibility

The ILAC designed by IEA includes two first-tier criteria and seven second-tier criteria. While the ILAC developed by the EBDEA includes four first-tier criteria and 21 s-tier criteria. It was found that most criteria designed by the IEA were included in the ILAC created by the EBDEA. The EBDEA also leads to new items (e., g. information awareness, and ethics of information) that do not exist in that of the IEA. Thus, we can conclude that the criteria created by the EBDEA are more complete than those designed by the IEA.

The Results of the Expert Questionnaire

A questionnaire with 21 items was used to assess experts’ satisfaction with the ILAC designed by the EBDEA. The results are shown in Table 6.

Table 6.

Descriptive Statistics of the Questionnaire.

Criteria	Mean	SD
A1 Attitude	4.50	0.72
A2 Information requirements awareness	4.65	0.47
A3 Participative awareness	4.55	0.65
A4 Critical awareness	4.80	0.39
A5 Information security awareness	4.85	0.47
A6 Life-long learning awareness	4.70	0.70
B1 Understanding	4.05	1.13
C1 Identity	4.75	0.42
C2 Locate	4.45	0.79
C3 Access	4.60	0.72
C4 Store	4.45	0.72
C5 Manage	4.60	0.57
C6 Integrate	4.60	0.72
C7 Review	4.35	0.83
C8 Evaluate	4.80	0.39
C9 Use	4.85	0.35
C10 Share	4.70	0.45
C11 communicate	4.75	0.42
C12 Create	4.80	0.39
D1 Legal use	4.85	0.47
D2 Social Responsibility	4.90	0.29

The results showed that social responsibility (M = 4.90) was the highest-scoring criterion while understanding (M = 4.05) was the lowest-scoring criterion. Experts’ satisfaction with information security awareness showed an overall mean of 4.85, with the same score between criteria: Use (M = 4.85, SD = 0.35) and Legal use (M = 4.85, SD = 0.47). Critical awareness (M = 4.80, SD = 0.39), Evaluate (M = 4.80, SD = 0.39), and Create (M = 4.80, SD = 0.39) were with the same mean. Overall, the mean scores of each criterion were greater than 4.0.

As we know, the ILAC designed by the IEA has been widely recognized in education around the world, with more than 20 countries using this ILAC to assess students’ information literacy, such as Belgium, the Czech Republic, Denmark, Finland, Germany, Norway, Slovenia, and Spain. Therefore, we expected the ILAC developed by the EBDEA to be as similar in content dimensions as possible to the ILAC developed by the IEA. Although the satisfaction of experts with the ILAC is not the same, the ILAC that we developed using the EBDEA has been recognized by more than 20 experts professors, and doctoral students related to information literacy research. It also verifies the effectiveness of ILAC developed by the EBDEA.

Discussion

Comparison of Results from the IEA and the EBDEA

According to the comparison, it was found that the ILAC designed by EBD-enabled is more comprehensive than those designed by the IEA. Specifically, according to the first-tier criteria, the ILAC designed by the IEA is derived from the perspective of information skills and information knowledge. In contrast, the ILAC designed by the EBDEA encompasses four distinct aspects: (1) information awareness, (2) information knowledge, (3) information skills, and (4) ethics of information. Notably, information awareness and ethics of information are new additions in the EBDEA-designed ILAC, absent in the IEA's framework. This discrepancy may be attributed to the premise that information awareness and ethics are particularly critical in the information age. Awareness, as defined by Dourish and Bellotti (1992), involves an understanding of others’ activities and the context of those activities. As a subset of awareness, information awareness is a student's sensitivity, judgment, and insight into information literacy education, which is an essential skill and a key component of literacy education (Arnborg et al., 2000; Jacxsens et al., 2024). This finding aligns with prior studies (Lei, 2015), highlighting the importance of information awareness in assessing students’ information literacy. The ethics of information, defined by Moor (2010), involves analyzing the nature and societal impacts of information, along with formulating and justifying policies for ethical information use. The ethics of information plays an important role in sustaining a country's social order and cultivating skilled professionals (Luciano, 2010). Thus, incorporating ethics of information into the ILAC for K-12 students is essential, a finding supported by previous studies (ACRL, 2000; Lei, 2015).

In terms of second-tier criteria, the results show that almost all of the IEA's assessment criteria are incorporated into the EBDEA-designed ILAC, with the exception of information transformation. Transformation involves changing information into a different form, a criterion implicitly addressed in the ‘integrate’ aspect of the EBDEA-designed ILAC. During integration, information must first be transformed into a uniform format. The similarities and differences in criteria will be further detailed in subsequent sections.

Regarding information skills, the EBDEA introduced new criteria, such as ‘identity, locate, integrate and review’. Identification is crucial for accessing precise information to address problems (Xu et al., 2021). It includes identifying a search topic or question, defining it with simple terminology based on information requirements, and identifying key concepts and terms to formulate focused questions (Gerck, 1998). This information skill offers K-12 students clear guidance in finding pertinent information and solving problems. Locating is essential for retrieving vital information (Cooke, 2002), determining the nature and scope of the required information, devising strategies for locating information, and selecting appropriate media and information sources (Aldahdouh, 2021). However, this skill is often overlooked by researchers.

Integration is another important information skill, requiring both cognitive and technical capabilities from an information literate student (Markauskaite, 2007; Mbandje et al., 2023). Cognitive capabilities enable a student to summarize, compare, and contrast information (Tang et al., 2021), while technical capabilities enable manipulation of this information using various tools (Ahmadi et al., 2007; Zahay & Handfield, 2004). Essentially, K-12 students with advanced information literacy can obtain more comprehensive information to resolve issues by synthesizing various sources. Additionally, they can merge their knowledge to reorganize, process, and integrate old and new information for practical application.

Reviewing information is another important step for K-12 students to acquire accurate knowledge (Lewis, 2000; Rahmaningrum & Oktaviana, 2020), which is often overlooked by other information literacy assessments. When obtained information does not entirely solve a problem, learners should merge it with the information search strategy (Bachelor, 2010), and then re-evaluate the nature and extent of the required information by reviewing the initial information needs and clarifying, revising, or refining the questions.

In summary, the EBDEA-designed ILAC is deemed more comprehensive based on this comparison. Moreover, it is apparent that the criteria satisfy the demands of modern society for evaluating and enhancing information literacy levels.

The Efficiency of the Designed ILAC

Efficiency refers to the ability to reduce the amount of used resources such as materials, energy, effort, money, and time in accomplishing a task or producing a desired outcome (Coelli et al., 2005; Tamburino & Bravo, 2024). This study demonstrates that the process of designing initial criteria to evaluate the information literacy of K-12 students using the EBDEA is more efficient than using the Delphi method.

This efficiency is primarily due to the EBDEA's capacity to facilitate quick and precise information collection from experts. Consequently, experts can leverage their experience to provide more effective evaluations and feedback. Following feedback receipt, the researcher panel can promptly and accurately draft the initial ILAC, thereby reducing the development time. For instance, the ICILS 2013 Technical Report (Fraillon et al., 2013) indicates that designing assessment criteria took approximately one month. In contrast, our team completed the criteria design in just two weeks, half the time spent using the Delphi method.

Furthermore, the recursive logic of the EBD methodology can also expedite the development of the ILAC. The EBDEA provides a structured, step-by-step procedure guiding designers throughout the environment change process (Sun et al., 2011; Zeng, 2011). If the initial criteria do not meet design objectives, researchers can revert to the question-asking stage and revise questions until experts are satisfied with the ILAC. This flexible yet structured process of criterion modification is advantageous for efficient assessment criteria design. Similar results were noted in Pan's research (2013) on developing key performance indicators, reinforcing the EBDEA's utility and efficiency in criterion design. Conversely, encountering issues in the initial ILAC design phase using the Delphi method would require substantial time for reflection, questionnaire revision, and expert selection. Moreover, reaching a consensus on final information literacy criteria demands a significant investment of time from experts.

Additionally, the EBDEA relies less on expert input. Traditional methods often derive initial ILAC from the brainstorming of 15 to 50 experts, which can be susceptible to bias. In contrast, the EBDEA's approach to developing ILAC is grounded in environment analysis, rendering it more objective, comprehensive, and holistic. In our study, only two postgraduate students participated in designing the assessment criteria, indicating that an individual proficient in the EBDEA and literature search techniques can effectively use this methodology to develop ILAC for K-12 students. For instance, Pan (2013), a master's student at Concordia University, developed key performance indicators for a department utilizing the EBD Methodology. In conclusion, anyone with mastery of the EBDEA and basic literature retrieval skills can apply the EDB method to develop and design assessment criteria or other products. In summary, the EBDEA not only facilitates the design of a more comprehensive ILAC but also requires less time and fewer human resources.

However, it should be noted that there are some difficulties and challenges when using the EBD method in designing the ILAC. On one hand, a thorough understanding of the EBD method is necessary before application, along with substantial time dedicated to reading literature related to information literacy. Only with a complete grasp of the EBD method and a whole picture of information literacy can subsequent efforts be meaningful. On the other hand, when collecting documents to answer generated questions, one must ensure the documents are current complete. It is believed that appropriate and scientific responses to these questions are integral to subsequent work.

Conclusion

In this paper, the EBDEA was used to develop the initial draft of ILAC aimed at helping researchers collect information effectively from experts. A case is carried out to develop criteria for the effective assessment of information literacy of K-12 students in China by the EBDEA. The initial draft of ILAC was constructed following the procedure of environment analysis - the first activity of the EBD methodology. The effectiveness of the EBDEA was validated by comparing the ILAC designed by the IEA and the EBDEA and expert evaluation. The results show that the EBDEA not only can design comprehensive criteria and develop new items, but also requires less time and fewer human resources.

While this research has significant implications, it also has limitations. Only the initial ILAC for K-12 students was designed in this study, rather than a complete ILAC. Although this work provided a starting point for designing ILAC, it may not encompass all assessment processes. Future studies should continue to utilize the Delphi method to develop a complete ILAC. Further, the design of assessment tools for information literacy will be encouraged to evaluate the information literacy levels of students.

Footnotes

Acknowledgments

The authors would like to thank Dr. Jaemun Sim for his suggestions about the structure of the paper.

ORCID iDs

Yinghui Shi

Shimeng Wang

Jiami Yang

Yi Dou

Yong Zeng

Ethical Approval

No human ethics approval was required for this paper.

Authors’ Contributions

YS and YZ proposed the idea, developed the framework, supervised the project, and wrote the paper; SW conducted the case study, implemented the research, and wrote the paper; JY developed the framework and wrote the paper; YD conducted the case study and wrote the draft. All authors read and approved the manuscript.

Funding

The work was supported by the Key Project of National Education Scientific “13th Five-Year Plan” in 2020, Research on the Connotation, Standard and Evaluation System of Student Information Literacy (Grant number: ACA200008).

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Code Availability

Not applicable.

Data Availability Statement

Data sharing not applicable to this paper as no datasets were generated or analysed during the current study.

Author Biographies

Yinghui Shi is an associate professor at the National Engineering Research Centre for E-Learning at Central China Normal University. His main areas of interest are information literacy, interactive media, ICT in education, and e-learning. E-mail: yhshi@ccnu.edu.cn.

Shimeng Wang received her M.S. degree in National Engineering Research Centre for E-Learning at Central China Normal University. Her research interests include information literacy and information technology diffusion/integration. E-mail: smwang@mails.ccnu.edu.cn.

Jiami Yang is a Canada NSERC Vanier Scholar and a recipient of the Dominic D’Alessandro Fellowship. She is currently pursuing her PhD degree in the Concordia Institute for Information Systems Engineering at Concordia University. Her research interests include design science, design methodology, and design-driven behavior change in areas such as management, sustainability, health, and education. Her work specifically focuses on investigating the impact of mental stress on human behaviors. E-mail: jiami.yang@mail.concordia.ca.

Yi Dou received her M.S. degree in Concordia Institute for Information Systems Engineering from Concordia University, Quebec, Canada. She worked as a research assistant at Concordia University and Central China Normal University in 2018 and 2019, respectively. Her research interests include creative design theory & methodology and cognitive science. E-mail: douyi0821@sina.com.

Yong Zeng is a Professor in Information Systems Engineering at Concordia University, Montreal, and an Adjunct Professor in the Department of Community Health Sciences at the University of Calgary. He is the President of the Society for Design and Process Science. He was NSERC Chair in aerospace design engineering (2015–2019) and Canada Research Chair in design science (2004–2014). Zeng's research focuses on creative design by developing and employing mathematical and neurocognitive approaches. He has proposed Environment-Based Design (EBD), addressing the recursive nature of design and the role of mental stress in designer creativity. He applies the EBD to algorithm design, the aerospace industry, medical devices, human resource management, municipality, teaching and learning, and health. E-mail: yong.zeng@concordia.ca.

Appendix A

A list of more generated questions and answers

No.	Questions	Answers
Information
1	What are creation and receipt?	The sources of information, which are divided into internal origination and external origination.
2	Why is it necessary to create and receive information?	To access the information source.
3	How does a user create and receive information?	The attitudes towards information, awareness of information requirements, participative awareness of information, the information requirement awareness, the ability to create and identify information, and obey the ethics and laws surrounding that information.
4	What is storage?	The places where any form of information is stored.
5	Why store information?	To find the places for storing information.
6	How is information stored?	The attitudes towards information and the ability to store information.
7	What are distribution and transmittal?	Processes involved in getting the information to locations where it can be accessed and used.
8	Why is information distributed and transmitted?	To find the locations where the information can be assessed and used.
9	How should the information be distributed and transmitted?	The attitudes towards information, the ability to locate, share and communicate information, participative awareness of information, the ethics and laws regarding the information.
10	What are access and use?	Converting the data from one form to another.
11	Why access and use information?	To convert the data to the form that people require.
12	How can a user access and use information?	The attitudes towards information, the ability to access, understand, integrate, manage, review, evaluate, and use information, critical awareness of information, and the ethics and laws regarding the information.
13	What is maintenance?	The management of information, including processes such as information filing, archiving, retrieval and transfers, as well as changing the classification of information as its value, relevance, or validity changes.
14	Why should the information be maintained?	To make information convenient for use.
15	How should the information be maintained?	The attitudes towards information, information security awareness, and the ability to manage information.
16	What are disposal and destruction?	Handling information that is rarely accessed, or information that is required to be retained in specific formats for specific time periods.
17	Why should the information be disposed of or destroyed?	To destroy worthless or unrequired information by appropriately secure means.
18	How should the information be disposed of and destroyed?	By identifying and evaluating information according to the ethics, and the law of information.
Information literacy
1	Why do users need information literacy?	To formulate a problem or decision more accurately and completely.
2	How can one realize the need for information literacy?	Create an awareness of the information required, and the ability to identify information.
3	Why identify and define information?	To meet a need, solve a problem, or make a decision.
4	How can users identify and define information?	The ability to identify and define information.
5	Why is it important to determine information?	To determine whether the needed information exists or not.
6	How can users determine information?	Understanding information knowledge, developing a critical awareness of information, and the ability to identify information.
7	Why is it necessary to find information?	To find the needed information.
8	How do users find information?	The ability to locate and access information.
9	Why is it necessary to create the information?	To create unavailable information.
10	How do users create the information?	The attitudes towards information, participative awareness of information, an awareness of information requirements, the ability to create and identify information; and obeying the ethics and laws surrounding the information.
11	Why is it necessary to understand the information?	To ensure that the information found is what you needed.
12	How can users understand the information?	The ability to understand, review and evaluate information.
13	Why is it necessary to organize the information?	To make sense of information.
14	How do users organize the information?	By understanding where the information can be used.
15	Why is it necessary to analyze, interpret and evaluate the information?	To make a judgment as to its reliability, credibility, and authenticity.
16	How do users analyze, interpret and evaluate the information?	Critical awareness of information, the ability to analyze and evaluate information.
17	Why is it necessary to communicate and present the information?	To communicate and share information with others in appropriate and usable formats and mediums
18	How do users communicate and present information?	The shared awareness of information, participative awareness of information, the ability to communicate information, and the ethics and laws surrounding the information.
19	Why do users utilize information?	To solve a problem, make a decision or meet a need.
20	How do users utilize information?	The attitudes towards information, the ability to evaluate and use information, critical awareness of information, and the ethics and laws surrounding the information.
21	Why is it necessary to preserve, store, record, and archive information?	To keep information for future use.
22	How do users preserve, store, record, and archive information?	The objective and fair attitudes towards information and the ability to store information.
23	Why do users dispose of information?	Information is no longer needed.
24	How do users dispose of information?	The critical awareness of information, the ability to identify and evaluate information, and the ethics and laws surrounding the information.
25	Why is it necessary to safeguard information?	To preserve information that should be protected.
26	How do users safeguard information?	Users utilize awareness of information security, the ability to manage and evaluate information, information, and the ethics and laws surrounding the information.
Criteria
1	What is design?	The design process from the generation of requirements to the acceptance of the relevant solution.
2	Why is it necessary to design criteria?	To collect, analyze, and satisfy the criteria requirements.
3	How do users design criteria?	Utilize the abilities to collect, analyze information.
4	What is formation of criteria？	The implementation process of the design solution into criteria.
5	Why is it necessary to form criteria?	To develop the criteria.
6	How do users form criteria?	Information requirement awareness, information knowledge, the ability to create and identify the information, and obey the ethics and laws surrounding the information.
7	What is the use?	The process of the product being used by the end-user.
8	Why can users use criteria?	To generate new requirements.
9	How can users use criteria?	The ability to use the information and obey the ethics and laws surrounding the information.
10	What is maintenance?	The adaptation made to help the product to suit the new environment.
11	Why is it necessary to maintain criteria?	To optimize criteria.
12	How can users maintain criteria?	The ability to use the information and obey the ethics and laws surrounding the information.
13	What is disposal?	To get rid of the outdated criteria.
14	Why is it necessary to dispose criteria?	To destroy worthless or unrequired criteria by appropriately secure means.
15	How Can Users Dispose Criteria?	The Critical Awareness Of Information, The Ability To Identify And Evaluate Information, And The Ethics And Laws Surrounding The Information.

Appendix B

The questionnaire about the importance of ILAC

Dear experts:

Thank you for taking the questionnaire survey on the importance of ILAC. The purpose of this questionnaire is to select ILAC by the Delphi method. As our valued participant, your opinion is extremely important for the final results.

The survey consists of 2 pages and should take no longer than 10 min. Please read each question carefully and tick a box to indicate your answer. If you have any queries about the questionnaire, please do not hesitate to contact me on email. Once you have completed the questionnaire, please send me an email back.

Thank you for your time and cooperation.

Please rate your satisfaction level with five-point Likert scale for the following items and tick one box only.

Satisfaction level		Very satisfied	Fairly satisfied	Neutral	Fairly dissatisfied	Very dissatisfied
Content		Very satisfied	Fairly satisfied	Neutral	Fairly dissatisfied	Very dissatisfied
First-tier criteria	1 Information Awareness
	2 Knowledge about information
	3 Information skills
	4 Ethics of information
Second-tier criteria	1.1 Attitude
	1.2 Information requirements awareness
	1.3 Participative awareness
	1.4 Critical awareness
	1.5 Information security awareness
	1.6 Life-long learning awareness
	2.1 Understand
	3.1 Identity
	3.2 Locate
	3.3 Access
	3.4 Store
	3.5 Manage
	3.6 Integrate
	3.7 Review
	3.8 Evaluate
	3.9 Use
	3.10 Sharing
	3.11 Communicate
	3.12 Create
	4.1 Legal use
	4.2 Social responsibility

References

ACRL. (2000). Information literacy competency standards for higher education. Teacher Librarian, 9(4), 63–67. http://doi.org/10.1300/J107v09n04_09

Ahmadi

Soltani

Gheitasi

(2007). An ICT technical reference model for Iran Universities. International Conference on Information Technology. IEEE. Las Vegas, NV. http://doi.org/10.1109/ITNG.2007.33

Aldahdouh

(2021). Information search behavior in fragile and conflict-affected learning contexts. The Internet and Higher Education, 50 (100808), 1–12. https://doi.org/10.1016/j.iheduc.2021.100808

Aliyyah

R. R.

Rasmitadila

Humaira

M. A.

Mujahidin

Rachmadtullah

(2020). Are the assessment criteria and the role of educational stakeholders able to make outstanding teacher? International Journal of Psychosocial Rehabilitation, 24(6), 8946–8957. https://doi.org/10.37200/IJPR/V2416/PR260902

Arnborg

Brynielsson

Artman

Wallenius

(2000). Information awareness in command and control: Precision, quality, utility. Proceedings of the Third International Conference on Information Fusion, 2, 25–32. https://doi.org/10.1109/ific.2000.859871

Bachelor

(2010, May). Hypertext in the classroom: Changing definitions of literature and literacy [Master’s thesis]. https://www.learntechlib.org/p/122911/

Bachen

Branch

Doyle

Serviss

Voss

(2018). Teaching teachers to teach writing, critical thinking, and information literacy: A case study of a faculty learning community as a campus-wide pedagogical intervention strategy. Staff Publications, Research, and Presentations, 203, 1–56. https://scholarcommons.scu.edu/cgi/viewcontent.cgi?article=1202&context=library.

Bernard

(2007). Information lifecycle security risk assessment: A tool for closing security gaps. Computers & Security, 26(1), 26–30. https://doi.org/10.1016/j.cose.2006.12.005

Cheligeer

Yang

Bayatpour

Miklin

Dufresne

Lin

Bhuiyan

Zeng

(2023). A hybrid semantic networks construction framework for engineering design. Journal of Mechanical Design, 145(4), 1–31. https://doi.org/10.1115/1.4056076

10.

Chen

Yang

Yao

Yan

Bhuiyan

Zhou

Liu

Zeng

(2022). Data quality criteria for urban waste management policy-making using environment-based design. IFAC-PapersOnLine, 55(10), 1435–1440.

11.

Coelli

T. J.

Rao

D. S. P.

O'Donnell

C. J.

Battese

G. E.

(2005). An introduction to efficiency and productivity analysis (2nd ed.). Springer.

12.

Cooke

(2002). A guide to finding quality information on the internet. Program Electronic Library & Information Systems, 33(4), 386–386. https://doi.org/10.1108/EUM0000000006922

13.

DeLuca

LaPointe-McEwan

Luhanga

(2016). Teacher assessment literacy: A review of international standards and measures. Educational Assessment, Evaluation and Accountability, 28(3), 251–272. https://doi.org/10.1007/s11092-015-9233-6

14.

Dourish

Bellotti

(1992). Awareness and coordination in shared workspaces. In Proceedings of the 1992 ACM conference on computer-supported cooperative work (pp. 107–114). Association for Computing Machinery. https://doi.org/10.1145/143457.143468

15.

Doyle

G. J.

Furlong

K. E.

Secco

(2016). Information literacy in a digital era: Understanding the impact of mobile information for undergraduate nursing students. In Studies in Health Technology and Informatics (Vol. 225, pp. 297–301). https://doi.org/10.3233/978-1-61499-658-3-297

16.

Yang

Chen

Yao

Yan

Nadia

Zhou

Liu

Zeng

(2023). Sustainable policy design—how policy impacts household waste management: A case-study from Shanghai. In Wang

L. L.

Zhai

Z. J.

Ouf

Sun

Wang

(Eds.), Proceedings of the 5th international conference on building energy and environment (pp. 1529–1539). Springer Nature Singapore. https://doi.org/10.1007/978-981-19-9822-5_158

17.

Fraillon

Schulz

Ainley

(2013). International computer and information literacy study: Assessment framework. International Association for the Evaluation of Educational Achievement, 33(4), 95–103. https://doi.org/10.1016/j.autcon.2012.09.012

18.

Gerck

(1998). What is identification, that we can identify it? Part I. Gerck Research. https://www.researchgate.net/publication/290284114

19.

Henkel

Grafmüller

Gros

(2018). Comparing information literacy levels of Canadian and German university students. In G. Chowdhury, J. McLeod, V. Gillet, P. Willett (Eds.), Transforming Digital Worlds (pp. 464–475). Springer, Cham. https://doi.org/10.1007/978-3-319-78105-1_51

20.

Horton

F. W.

(2007). Understanding information literacy: A primer. http://hdl.voced.edu.au/10707/299728

21.

Jacxsens

Pennings

Lentacker

Stoop

Provoost

(2024). Identifying essential information for a valid informed consent (ic) of egg donors: An international delphi study. Human Reproduction, 39(Supplement_1), deae108.143. https://doi.org/10.1093/humrep/deae108.143

22.

Joseph

John

Kurian

S. S.

Tom

(2018). An assessment of information literacy competence of postgraduate students. International Journal of Library & Information Science, 7(2), 45–54. https://doi.org/10.34218/IJLIS.7.2.2018.006

23.

Jun

Han

Kim

Lee

(2014). Assessing the computational literacy of elementary students on a national level in Korea. Educational Assessment, Evaluation and Accountability, 26(4), 319–332. https://doi.org/10.1007/s11092-013-9185-7

24.

Khodyakov

Grant

Kroger

Gadwah-Meaden

Motala

Larkin

(2023). Disciplinary trends in the use of the Delphi method: A bibliometric analysis. PLOS ONE, 18(8), e0289009. https://doi.org/10.1371/journal.pone.0289009

25.

Lei

(2015). A review of the research and practice on the domestic information literacy education in the last two decades. Library Journal, 34(3), 16–22. https://doi.org/10.13663/j.cnki.lj.2015.03.003

26.

Lewis

(2000). Comments - response to Steve Petrina’s book review and John Ritz’s comment. Journal of Industrial Teacher Education, 37(2), 1–9.

27.

Linstone

H. A.

Turoff

(2011). Delphi: A brief look backward and forward. Technological Forecasting and Social Change, 78(9), 1712–1719. https://doi.org/10.1016/j.techfore.2010.09.011

28.

Luciano

(2010). The Cambridge handbook of information and computer ethics. Cambridge University Press.

29.

Markauskaite

(2007). Exploring the structure of trainee teachers’ ICT literacy: The main components of, and relationships between, general cognitive and technical capabilities. Educational Technology Research and Development, 55, 547–572. https://doi.org/10.1007/s11423-007-9043-8

30.

Mbandje

D. C.

Loureiro

M. J.

Lucas

(2023). Digital competence and information literacy: Clarifying concepts based on a literature review. Educational Media International, 60(3–4), 306–316. https://doi.org/10.1080/09523987.2023.2324584

31.

Mohammadi

Yang

Borgianni

Zeng

(2024). Barriers and enablers of TRIZ: A literature analysis using the TASKS framework. Journal of Engineering, Design and Technology, 22(4), 1206–1230. https://doi.org/10.1108/JEDT-01-2022-0066

32.

Moor

J. H.

(2010). What is computer ethics? Metaphilosophy, 16(4), 266–275. https://doi.org/10.1111/j.1467-9973.1985.tb00173.x

33.

Naito

(2019). An information literacy curriculum: A proposal. College & Research Libraries News, 52(5), 293–296. https://doi.org/10.5860/crln.52.5.293

34.

Pan

(2013, September). Developing key performance indicators (KPIs) for a department utilizing environment-based design (EBD) methodology [Master's thesis]. https://spectrum.library.concordia.ca/977812

35.

Rader

(1990). Bibliographic instruction or information literacy. College & Research Libraries News, 51(1), 18–20. https://doi.org/10.5860/crln.51.1.18

36.

Rader

(2019). Conference circuit: Information literacy: The professional issue: Subjects addressed at the Third National Australian Conference. College & Research Libraries News, 59(3), 171–173. https://doi.org/10.5860/crln.59.3.171

37.

Raeis

A. R.

Bahrami

Yousefi

(2013). Relationship between information literacy and creativity: A study of students at the Isfahan University of medical sciences. Materia Socio Medica, 25(1), 28–31. https://doi.org/10.5455/msm.2013.25.28-31

38.

Rahmaningrum

S. A.

Oktaviana

P. P.

(2020). Sentiment classification of hotel service review on traveloka sites using nave bayes classifier (NBC) and binary logistic regression. Journal of Physics Conference Series, 1490(1), 012065. https://doi.org/10.1088/1742-6596/1490/1/012065 .

39.

Razavi

S. R.

Yang

Akgunduz

Bhuiyan

Zeng

(2024). Environment-driven evolution analysis of a product: A case study of braking system evolution. Journal of Integrated Design and Process Science, 27(2), 135–150. https://doi.org/10.3233/JID-230070

40.

Shi

Y. H.

Peng

C. L.

Yang

(2018). Research on the evaluating indicators system of information literacy for K-12 students. China Educational Technology, 45(8), 73–77. https://link-cnki-net-s.web.bisu.edu.cn/urlid/11.3792.g4.20180730.1339.020

41.

Skedsmo

Huber

S. G.

(2021). Data, indicators, and feedback in education. Educational Assessment, Evaluation and Accountability, 33, 251–254. https://doi.org/10.1007/s11092-021-09362-y

42.

Sun

Zeng

Zhou

(2011). Environment-based design (EBD) approach to developing quality management systems: A case study. Journal of Integrated Design and Process Science, 15(2), 53–70. https://doi.org/10.3233/JID-2011-15204

43.

Tachie-Donkor

Ezema

I. J.

(2023). Effect of information literacy skills on university students’ information seeking behaviour and lifelong learning. Heliyon, 9(8), e18427. https://doi.org/10.1016/j.heliyon.2023.e18427

44.

Tamburino

Bravo

(2024). Ecological efficiency: The ability to achieve human well-being while limiting environmental impact. Environmental and Sustainability Indicators, 21(100322). https://doi.org/10.1016/j.indic.2023.100322

45.

Tang

Wang

Lund

Kaiser

M. J.

(2021). Flexibility categorization, sources, capabilities and technologies for energy-flexible and grid-responsive buildings: State-of-the-art and future perspective. Energy, 219(15), 1–43. https://doi.org/10.1016/j.energy.2020.119598

46.

Tewell

(2015). A decade of critical information literacy: A review of the literature. Communications in Information Literacy, 9(1), 24–43. https://doi.org/10.15760/comminfolit.2015.9.1.174

47.

Wang

Cao

Yang

Zeng

(2024). Effective KPI development using environment-based design (EBD) methodology: A case study of airline KPI system. Journal of Integrated Design and Process Science, 28(1), 1–11. https://doi.org/10.1177/10920617241289750

48.

Wang

Zeng

(2009). Asking the right questions to elicit product requirements. International Journal of Computer Integrated Manufacturing, 22(4), 283–298. https://doi.org/10.1080/09511920802232902

49.

Zhang

(2021). A novel inverse procedure for load identification based on improved artificial tree algorithm. Engineering with Computers, 37(1), 663–674. https://doi.org/10.1007/s00366-019-00848-4

50.

Yang

Dou

Zeng

(2023). Environment-based design (EBD): Using only necessary knowledge for designer creativity. Proceedings of the Design Society, 3, 1675–1684. https://doi.org/10.1017/pds.2023.168

51.

Yang

Zeng

Ekwaro-Osire

Nispel

(2022). Environment-Based life cycle decomposition (eLCD): Adaptation of EBD to sustainable design. Journal of Integrated Design and Process Science, 24(2), 5–28. https://doi.org/10.3233/JID200018

52.

Zahay

D. L.

Handfield

R. B.

(2004). The role of learning and technical capabilities in predicting adoption of b2b technologies. Industrial Marketing Management, 33(7), 627–641. https://doi.org/10.1016/j.indmarman.2003.10.004

53.

Zeng

(2004). Environment-based formulation of design problem. Journal of Integrated Design and Process Science, 8(4), 45–63. https://doi.org/10.3233/JID-2004-8404

54.

Zeng

(2008). Recursive object model (ROM)-modelling of linguistic information in engineering design. Computers in Industry, 59(6), 612–625. https://doi.org/10.1016/j.compind.2008.03.002

55.

Zeng

(2011). Environment-based design (EBD). Proceedings of ASME 2011 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, 237–250. https://doi.org/10.1115/DETC2011-48263

56.

Zeng

(2015). Environment-based design (EBD): A methodology for transdisciplinary design⁺. Journal of Integrated Design and Process Science, 19(1), 5–24. https://doi.org/10.3233/jid-2015-0004

57.

Zeng

. (2020). Environment: The first thing to look at in conceptual design. Journal of Integrated Design and Process Science, 24(1), 45–66. https://doi.org/10.3233/JID200005

58.

Zeng

Cheng

(1991). On the logic of design. Design Studies, 12(3), 137–141. https://doi.org/10.1016/0142-694X(91)90022-O

59.

Zeng

(1999). A science-based approach to product design theory part I: Formulation and formalization of design process. Robotics and Computer-Integrated Manufacturing, 15(6), 331–339. https://doi.org/10.1016/S0736-5845(99)00028-9

60.

Zurkowski

(1974). The information service environment relationships and priorities. US National Commission on Libraries and Information Science.