Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability

Abstract

Previous research in cognitive diagnostic assessment (CDA) of L2 reading ability has been frequently conducted using large-scale English proficiency exams (e.g., TOEFL, MELAB). Using CDA, it is possible to analyze individual learners’ strengths and weaknesses in multiple attributes (i.e., knowledge, skill, strategy) measured at the item level. This study explored how a placement test score could be used for diagnosing the L2 reading ability of incoming students to an adult ESL program. Five content experts completed a reading placement test and identified the attributes required for successfully completing each item on the test, while referring to the list of L2 reading attributes. Then, content experts’ codings were analyzed and developed into an item-by-attribute Q-matrix. The Fusion model, a type of cognitive diagnostic model (CDM), was used for refining the Q-matrix and diagnosing 1982 learners’ strengths and weaknesses in L2 reading. Results suggest that 10 major L2 reading attributes were involved in the reading test. In addition, examinees’ strengths and weaknesses were identified for the overall group, three reading proficiency groups (i.e., beginner, intermediate, and advanced), and individual learners. Such information could be provided to ESL program administrator and teachers for enhancing the reading curriculum and developing instructional materials.

Keywords

Cognitive diagnostic assessment ESL placement test Fusion model Q-matrix Second language reading assessment

Cognitive diagnostic assessment (CDA) has recently received much interest in the field of second language assessment since it can be used to provide diagnostic feedback to stakeholders and ultimately promote language learning. A diagnostic assessment could be defined as “a test which is used for the purpose of discovering a learner’s specific strengths or weaknesses” (ALTE, 1998, p. 142). To be effective, diagnostic feedback should contain detailed information on individual examinees’ strengths and weaknesses of a cognitive domain (e.g., reading, mathematics). Moreover, in order to provide high-quality feedback, it is beneficial to identify the various components of a cognitive domain based on both theoretical and empirical evidence. For example, in conducting cognitive diagnostic assessment of second language (L2) reading ability, L2 reading ability and its components could be defined using theories of language ability in general and L2 reading.

Providing stakeholders with detailed feedback on students’ strengths and weaknesses could lead to enhanced learning of English. For example, program administrators and teachers can determine the type of future instruction that is necessary for the students (Hughes, 1989; Lee & Sawaki, 2009a). Specifically, a classroom teacher could use diagnostic information to design remedial instructional materials for an individual learner. At a more global level, program administrators could develop a curriculum that meets a group of learners’ needs.

In CDA, the various components of a cognitive domain are referred to as attributes. Attributes are “[cognitive] procedures, skills, or knowledge a student must possess in order to successfully complete the target task” (Birenbaum, Kelly, & Tatsuoka, 1993, p. 443). That is, attributes are domain-specific skills and knowledge that are needed to demonstrate mastery in a cognitive domain (Chipman, Nichols, & Brennan, 1995; Leighton & Gierl, 2007). CDA is powerful since it allows examining individual learners’ strengths and weaknesses of multiple attributes measured at an item level. These attributes should not be a mere list of skills. Rather, they should be systematically categorized according to a specific cognitive theory. For instance, attributes of L2 reading ability – or L2 reading attributes – consist of knowledge, skills, and strategies, which are involved in comprehending texts (Templin, 2004; Birenbaum et al., 1993). L2 reading attributes include various reading skills and strategies, which will be used interchangeably in this paper. However, in the literature, some claim that skill is a predisposition to perform in a certain way, which is automatized and unconscious in nature (Urquhart & Weir, 1998), whereas strategy is the conscious process involved in reading (Alderson, 2000; Cohen, 2007). Others note the differences between skills and strategies (e.g., Faerch & Kasper, 1983; Purpura, 1999), but state that a strategy could become more automatized and skill-like as readers become more proficient. In the present study, reading attributes will encompass the notion of language knowledge, skills, and strategies involved in reading.

Previous studies in CDA of L2 reading (e.g., Buck, Tatsuoka, & Kostin, 1997; Jang, 2005; Kasai, 1997; Li, 2011; Sawaki, Kim, & Gentile, 2009; Scott, 1998) have used large-scale English proficiency tests (e.g., TOEIC, TOEFL, MELAB) for identifying examinees’ mastery of L2 reading attributes. For instance, in a ground-breaking study of diagnosing L2 reading, Jang (2005) first conducted a verbal protocol analysis to identify the L2 reading attributes used by ESL learners in completing the reading section of the Test of English as a Foreign Language (TOEFL). Nine attributes were identified: (1) deducing word meaning from the context; (2) determining word meaning out of the context; (3) comprehending text through syntactic and semantic links; (4) comprehending text-explicit information; (5) comprehending text-implicit information at the global level; (6) inferring major arguments or a writer’s purpose; (7) comprehending negatively stated information; (8) summarizing major ideas from minor details; and (9) determining contrasting ideas through diagrammatic display. The Fusion model (Hartz, Roussos, & Stout, 2002), a type of cognitive diagnostic model (CDM), was used to evaluate the attributes and to diagnose examinee’s L2 reading ability. Individual diagnostic score reports were provided to ESL learners and teachers, which were found to be useful for learning.

More recently, Sawaki et al. (2009) used expert judgment to identify the attributes involved in the reading and listening sections of the Test of English as a Foreign Language Internet-based Test (TOEFL iBT) as part of an attempt to provide diagnostic score reports to stakeholders. For the reading section, six content experts first identified the L2 reading attributes involved in each test. These attributes were identified based on the three reading constructs of the test (i.e., basic comprehension, inferencing, and reading to learn) and task analysis. Afterwards, the Fusion model analysis was implemented to refine L2 reading attributes in to the following: (1) understanding word meaning; (2) understanding specific information; (3) connecting information; (4) synthesizing and organizing information. However, this study did not further examine the pedagogical impact of CDA.

In fact, few studies of CDA in L2 reading, except for Jang (2005), have explored the type of diagnostic feedback that could be provided to stakeholders or the pedagogical implications it could have. It may have been difficult to do so considering that these studies used large-scale proficiency tests, which were often administered outside of instructional contexts. Rather, prior research has focused mostly on the feasibility of using CDMs in language assessment or identifying L2 reading attributes involved in L2 reading tests. To explore the full capacity of CDA, for example within an ESL program, it is necessary to explore the type of diagnostic feedback that could be provided to program administrators and teachers, and how such information could be potentially used for instructional purposes.

Diagnostic information could be highly useful in the context of the current study, an adult ESL program within a US college. In the program, placement test scores have been used for placing incoming students into classes at different proficiency levels. In the beginning of each semester, ESL teachers are given information on the learners’ total placement scores as a reference. Yet it is often insufficient to identify their students’ strengths and weaknesses in reading. Therefore, some teachers design their own diagnostic tests (e.g., a short essay or a conversation test) or assignments to better gauge their students’ needs, indicating a need for more diagnostic information regarding the students. Using CDA, diagnostic information could be extracted not only at the individual learner level, but also at different language proficiency levels (i.e., beginners, intermediates, advanced). Such information could be provided to ESL program administrators and teachers, and lead to enhanced reading curriculum at both the classroom-level and program-level.

In addition, the identified L2 reading attributes in previous studies vary widely, partly because the attributes were not drawn from a unified reading theory or language ability model. The current study defines reading as the process of constructing meaning while interacting with the text (RAND Reading Study Group, 2002). It is an instance of L2 language use in which language ability is used for understanding the written text (Bachman & Palmer, 1996). Therefore, reading ability could be discussed within the larger framework of a language ability model, which defines language ability as consisting of language knowledge and strategic competence (e.g., Bachman & Palmer, 1996; Purpura, 2004). In other words, reading ability can be examined in terms of both language knowledge and strategic competence, which are necessary when interacting with the text. Using a language ability model, complemented by various reading theories, as a framework for L2 reading research will allow one to define the relationship among attributes very clearly in a systematic fashion.

Therefore, the main purpose of the current study was to diagnose ESL learners’ L2 reading ability based on their placement test scores with the ultimate goal of providing diagnostic feedback to ESL program administrators and teachers. The Fusion model, a type of CDM, was used for cognitive diagnosis of L2 reading ability. CDMs are psychometric models that can be used to evaluate students’ strengths and weaknesses in a cognitive domain (de la Torre, 2009). Specifically, the Fusion model is an attribute-level item-response model (Roussos, DiBello, Stout, Hartz, Henson, & Templin, 2007), used for determining examinee’s mastery (i.e., strengths) and non-mastery (i.e., weakness) of each attribute at the item-level. To fulfill the study purpose, first, the L2 reading attributes were drawn from a general model of language ability (e.g., Bachman & Palmer, 1996; Purpura, 2004) and reading theories (e.g., Cohen & Upton, 2006; Grabe, 2009; Phakiti, 2007; Pressley & Afflerbach, 1995; Weir, Hawkey, Green, & Devi, 2009). Then, information on the identified L2 reading attributes was used to diagnose L2 reading ability of adult ESL learners, enrolled in an ESL program in a US college. Diagnostic information was extracted from their placement test scores not only at the individual learner level, but also at different language proficiency levels (i.e., beginner, intermediate, advanced).

The present study investigated the following research questions:

What are the major L2 reading attributes involved in successfully completing a reading test?

What are the strengths and weaknesses of the examinees’ L2 reading ability for the overall group, three different proficiency groups (i.e., beginners, intermediates, advanced), and individual learners?

Literature review

Q-matrix development

The preliminary step to conducting a cognitive diagnosis of learners’ test performance is to construct a Q-matrix. A Q-matrix is a “cognitive design matrix that explicitly identifies the cognitive specification for each item” (de la Torre, 2009, p. 2). It represents the cognitive components required in answering test items, by having rows represent items, and columns represent attributes or skills (Tatsuoka, 1990). The Q-matrix is important in conducting studies involving cognitive diagnostic models since the diagnostic power of the models are often determined by the theoretical and empirical soundness of the Q-matrix (Lee & Sawaki, 2009a).

The Q-matrix was first proposed by Tatsuoka (1990), who integrated the conceptual framework by Falmagne and Doignon (1988), and Haertel and Wiley (1993). The matrix indicates unobservable knowledge states into observable item-response patterns (Tatsuoka, 1995). The Q-matrix consists of [i × k] matrix of binary information in ones and zeros, where i is the number of items on the test and k represents the number of attributes. For a given element in the Q-matrix in the ith row and the kth column ( $q_{i k}$ ), one means that the item i assesses attribute k, but zero indicates that it does not. Table 1 is an example of a [3 × 4] Q-matrix, According to the matrix, Item 1 (I1) measures Attribute 2 (A2) and Attribute 4 (A4); Item 2 (I2) assesses Attribute 1 (A1) and Attribute 4 (A4); and Item 3 (I3) measures Attribute 1 (A1) and Attribute 3 (A3) (McGlohen, 2004).

Table 1.

A sample Q-matrix.

	A1	A2	A3	A4
I1	0	1	0	1
I2	1	0	0	1
I3	1	0	1	0

The Q-matrix focuses on the conjunctive interaction of attributes on a task. Conjunctive interaction means that the correct response to an item requires the knowledge of all attributes represented in the item. By contrast, compensatory interaction of attributes on a task refers to when a high level of competence in one attribute compensates for a low level competence of another attribute in completing an item (Dibello, Roussos, & Stout, 2007). Owing to the conjunctive nature of the Fusion model, test-takers are assumed to possess the knowledge of all attributes tested by an item to correctly respond to the item (refer to the ‘Fusion model analysis’ section for more details on the model).

Furthermore, the specificity of the attributes defined in the Q-matrix is crucial because the granularity of attributes affect psychometric analysis. The more fine-grained the attributes are, the richer the diagnosis of learners’ strengths and weaknesses will be. However, there is a trade-off between the richness and the stability of diagnosis (Haberman & von Davier, 2006). According to Lee and Sawaki (2009b),

[a]s the level of specification for the Q-matrix becomes more detailed, a much larger number of items are required to represent a universe of the attributes in the test. In addition, it is likely that the more fine grained the attributes are, the more difficult it can become to maintain the consistency of diagnosis across occasions or test forms, potentially contributing to instability and unreliability of examinee classification. (p. 184)

Thus, often in practice, similar attributes are combined to reduce the number of attributes. For instance, Sawaki et al. (2009) combined the two skills of understanding and connecting information within a paragraph and understanding and connecting information across paragraphs as these were considered to involve similar cognitive procedures. Therefore, they were combined them into one attribute called connecting information (p. 198).

Fusion model analysis

The Fusion model (Hartz et al., 2002), a type of cognitive diagnostic model, is used to make inferences about each examinee’s mastery level of each attribute, based on the examinee’s item responses (DiBello & Stout, 2008). Specifically, the Fusion model is a skill-level item-response model (Roussos et al., 2007), which “expresses the probability of an examinee [j] answering an item i correctly as a function of both the examinee characteristics (skills mastery level profile vector) and item characteristics (item parameters that are linked to an items required skills)” (DiBello & Stout, 2008, p. 9). The item-response function is as follows:

P (X_{i j} = 1 | α_{j}, θ_{j}; π_{i}^{*}, r_{i k}^{*}, c_{i}) = π^{*} \prod_{k = 1}^{K} r_{i k}^{* (1 - α_{j k}) q_{i k}} P_{c_{i}} (θ_{j}) .

That is, $P (X_{j i} = 1 | α_{j}, θ_{j}; π_{i}^{*}, r_{i k}^{*}, c_{i})$ is the probability of person j on item i scoring a correct response (X = 1) instead of an incorrect one (X = 0), given person abilities – $α_{j}, θ_{j}$ – and item parameters $π_{i}^{*}, r_{i k}^{*}, c_{i}$ .

In this study, for model simplification, the Reduced Fusion model was used by repressing ability parameter $θ_{j}$ – which represents the ability to apply the attributes not covered in the Q-matrix – and item parameter $c_{i}$ – which is associated with $θ_{j}$ , hence is automatically disregarded. When there are numerous attributes, using the Reduced Fusion model makes it easier to identify the model. The Reduced Fusion model is similar to the typical two-parameter logistic (2PL) model in item-response theory (IRT) (DiBello & Stout, 2008) in that it has an ability parameter $α_{j}$ (in contrast to 2PL ability parameter $θ$ ) and two item parameters $π_{i}^{*}, r_{i k}^{*}$ (in contrast to 2PL item parameters a and b). In detail,

$α_{j}$ indicates examinee j’s mastery or non-mastery of attributes, identified in the Q-matrix. $α_{j}$ is represented by the attribute mastery level profile vector ${\underline{α}}_{j} = (α_{j 1}, \dots, α_{j K})$ . Here, K is the total number of attributes and each element $α_{j}$ is equal to 0 or 1, depending on the examinee’s level of mastery of attribute k ( $α_{k}$ = 1 for mastery; and $α_{k}$ = 0 for non-mastery).

$π_{i}^{*}$ indicates item difficulty. It represents the probability of answering item i correctly when the examinee has mastered all required attributes for the item, expressed as

P (X_{i} = 1 | α_{i k} = 1 f o r a l l a t t r i b u t e s k s u c h t h a t q_{i k} = 1) .

( $q_{i k} = 1$ means that attribute k is required for item i).

A high value above .6 indicates that examinees will have a good chance of correctly responding to the item if they have mastered all the necessary attributes.

$r_{i k}^{*}$ is associated item discrimination. It represents the inverse discrimination of attribute k of item i and is expressed as

\frac{P (a t t r i b u t e k i s a p p l i e d c o r r e c t l y i n i t e m i | α_{i k} = 0)}{P (a t t r i b u t e k i s a p p l i e d c o r r e c t l y i n i t e m i | α_{i k} = 1)}

That is, it is the ratio of (1) the probability of invoking all attributes on item i given an examinee is a non-master of attribute k but a master of all other attributes for the item and (2) the probability of invoking all attributes on item i given an examinee is a master of all attributes for item i. It compares correct item-response probability between the mastery of attribute k and the non-mastery of attribute k. Thus, it is a reverse indicator of an item’s discriminating power for attribute k. For example, a low $r_{i k}^{*}$ below .3 means that the item discriminates well between masters and non-masters of an attributes k (Johnson, 2006), whereas a high value above .9 indicates that the item does not discriminate well (DiBello & Stout, 2008; Jang, 2005).

The Fusion model also computes the examinee population’s probability of mastery of attribute, or $p_{k}$ , where $p_{k} = p (α_{k} = 1)$ . Given that $α_{k} = 1$ is the examinees’ mastery of attribute, $p_{k}$ represents the proportion of masters of attribute k in the examinee population. The range of $p_{k}$ is between zero and one. When $p_{k}$ is small, the examinee population has few masters of an attribute k, indicating that the attribute is difficult to master. By contrast, when the value is large, there are numerous masters of attribute k, indicating that the attribute is easy to master. Therefore, examinee population’s probability mastery of an attribute directly relates to the difficulty of an attribute.

Moreover, the Fusion model produces information regarding each examinee’s probability of mastery for each attribute, or $p_{k} ({\underline{X}}_{j})$ . Thus, when K is the number of attributes and ${\underline{X}}_{j}$ denotes the scored item-response string for examinee j, a vector of K probability values can be expressed in the following:

p ({\underline{X}}_{j}) = (p (α_{j 1} = 1 | {\underline{X}}_{j})), \dots, (p (α_{j K} = 1 | {\underline{X}}_{j}))

In the function, ${\underline{X}}_{j}$ refers to the scored item-response string of examinee j, and $p (α_{j k} = 1 | {\underline{X}}_{j})$ is the probability that examinee j mastered attribute k, based on his or her observed scored item-response data ${\underline{X}}_{j}$ . Thus, examinee j’s posterior probability of mastery of attribute k can be expressed as $p_{k} ({\underline{X}}_{j}) = p (α_{j k} = 1 | {\underline{X}}_{j})$ . That is, $p_{k} ({\underline{X}}_{j})$ refers to the probability of mastering attribute k, given the item-response string for examinee j. $p_{k} ({\underline{X}}_{j})$ ranges between zero and one and can be used to indicate the degree of mastery of attribute k.

Methods

Participants

Coders

Five content experts in ESL identified the L2 reading attributes involved in the reading test. Content experts were recruited as coders since the purpose of the study was to identify the L2 reading attributes required for successfully completing the reading test items. Numerous attributes could be involved in reading, but the focus was on examining the major attributes that were essential for correctly responding to the test items. The coders were graduate students in TESOL or Applied Linguistics at a US college and had considerable knowledge and experience in language testing and L2 reading. They comprised three females who were near-native speakers of English and two males who were native speakers of English. The coders all had experience of teaching ESL, including L2 reading. Two of the coders had experience of teaching in the ESL program concerned.

Examinees of the reading test

Examinees were 1982 new incoming students to the adult ESL program within a US college. Participants were 18 years or older and had diverse linguistic and cultural backgrounds. Some of the students were immigrants residing in the area, whereas others were international students and international corporate executives. Therefore, students were expected to have a wide range of reading proficiency.

Instruments

The list of L2 reading attributes

The list of L2 reading attributes (as presented in Table 2) was developed by drawing upon previous literature on language ability models (e.g., Bachman, 1990; Bachman & Palmer, 1996; Purpura, 2004) and reading theories (e.g., Cohen & Upton, 2006; Phakiti, 2007; Pressley & Afflerbach, 1995; Weir et al., 2009). The list consisted of L2 reading attributes that were presumed to be involved in reading.

Table 2.

Attributes of L2 reading ability.

L2 reading ability
Language knowledge	Grammatical knowledge	Grammatical form	Phonological/graphological, lexical, morphosyntactic, cohesive, information management, and interactional forms
		Semantic meaning	Phonological/graphological meaning
			Lexical meaning
			Morphosyntactic meaning
			Cohesive meaning
			Information management meaning
			Interactional meaning
			Sentence meaning
			Paragraph/text meaning
	Pragmatic knowledge	Pragmatic meanings	Contextual meaning
			Sociolinguistic meaning
			Sociocultural meaning
			Psychological meaning
			Rhetorical meaning
Strategic competence	Metacognitive strategies	Assessing the situation	Planning before reading
		Assessing the situation	Determining reading speed
		Monitoring	Checking comprehension, during reading
		Evaluating	Evaluating the accuracy of comprehension, after reading
	Cognitive strategies	Comprehension	Identifying word meaning
			Finding information
			Identifying key words/key sentence
			Skimming
			Paraphrasing
			Summarizing the main idea
			Connecting sentences (adjoining sentences)
			Integrating important ideas (ideas from different paragraphs)
			Inferencing
			Predicting upcoming information
		Memory	Recognizing previously read information
			Simplifying information to remember
		Retrieval	Recalling the reading purpose/task
			Using prior knowledge relevant to the topic

Considering that reading ability is an instance of language use, L2 reading ability was defined within a broader communicative language ability (CLA) model (e.g., Bachman & Palmer, 1996; Purpura, 2004), in which language ability consists of language knowledge and strategic competence. Thus, L2 reading ability largely consists of language knowledge and strategic competence, necessary when a learners’ language ability interacts with the written text. Language knowledge is further categorized into grammatical knowledge, consisting of grammatical form and semantic meaning, and pragmatic knowledge. Grammatical form refers to “linguistic forms on the subsentential, sentential and suprasentential levels” (Purpura, 2004, p. 61) whereas semantic meaning concerns the “literal meaning expressed by sounds, words, phrases and sentences” (Purpura, 2004, p. 61). On the other hand, pragmatic knowledge is defined in terms of various pragmatic meanings. Pragmatic meaning draws on grammatical form and semantic meaning to convey the implied meaning of utterances, and can simultaneously embody a range of contextual, sociolinguistic (meaning associated with social norms, preferences, and expectations), sociocultural (meaning related to cultural norms, preferences, and expectations), psychological (meaning embodying attitude, affect), and rhetorical (meaning derived on coherence and genre) meanings (Grabowski, 2009; Purpura, 2004).

L2 reading ability also involves strategic competence, comprising both metacognitive strategies and cognitive strategies. Metacognitive strategies are conscious or unconscious reading activities that regulate cognitive strategies, such as assessing the situation, monitoring, and evaluating (Bachman & Palmer, 1996; O’Malley & Chamot, 1990; Phakiti, 2007; Purpura, 1999; Wenden, 1991). Assessing the situation refers to the higher-order executive function of assessing one’s own knowledge, resources, and constraints of the situation before engaging in a task. Monitoring strategies involve determining the effectiveness of one’s task performance during the task, whereas evaluating strategies refer to determining the effectiveness of task after its completion (Purpura, 1999). The specific constituents of these strategies were drawn from various reading literature sources (Phakiti, 2007; Pressley & Afflerbach, 1995; Weir et al., 2009). In contrast to metacognitive strategies, cognitive strategies of L2 reading ability are processes that directly interact with the target language and involve those required for comprehension (understanding the text), memory (transforming information to store in memory), and retrieval (recalling information from memory) (O’Malley & Chamot, 1990; Phakiti, 2007; Purpura, 1999; Wenden, 1991). These were further categorized into sub-strategies by drawing upon previous reading literature (e.g., Cohen & Upton, 2006; Phakiti, 2007; Pressley & Afflerbach, 1995; Weir et al., 2009).

Reading test

The reading test data was from the reading section of a placement test used in the adult ESL program, developed to place incoming students into classes at different proficiency levels. The reading test consisted of four reading passages and 30 multiple-choice items, which were scored dichotomously. As seen in Table 3, the reading passages differed in a number of ways: topic (i.e., complaint letter, Argentine ants, mysteries of smell, demise of Neanderthals), text structure (i.e., causation, compare/contrast, description, problem/solution), length (i.e., 185 to 533 words), and the number of items measured (i.e., six to nine items).

Table 3.

Structure of the reading test.

Task	Topic	Text structure (content structure)	Length	Number of items
Task 1	Complaint letters	Problem/solution	185 words	6
Task 2	Argentine ants	Description	333 words	6
Task 3	Mysteries of smell	Description	404 words	9
Task 4	Demise of Neanderthals	Causation; Compare/contrast	533 words	9

According to test specifications, the 30 reading test items (Items 44–73 in the placement test) measured two different types of meaning as seen in Table 4, either semantic meaning or pragmatic meaning of the text. Semantic meaning refers to the literal meaning expressed by utterances, whereas pragmatic meaning refers to comprehending highly contextualized implied meaning of the text based on the context or the readers’ prior knowledge (Purpura, 2004).

Table 4.

Item types.

Item type/construct	Number of items	Items
Semantic meaning item	22	44, 45, 46, 47, 50, 51, 52, 53, 56, 57, 58, 59, 61, 62, 63, 65, 66, 67, 68, 69, 70, 72
Pragmatic meaning item	8	48, 49, 54, 55, 60, 64, 71, 73

Procedures

To identify the L2 reading attributes required for successfully completing each item on the reading test, individual items needed to be coded into attribute specifications. First, content experts participated in a group training session and familiarized themselves with the list of L2 reading attributes. Rather than recognizing all the language knowledge and strategies possibly involved in answering an item, they were asked to indicate the major attributes they utilized to respond to each item. It should be noted that not all attributes may be measured using one single test, nor suit the purpose of an adult ESL placement test. For example, if the reading passage is available only in the beginning and becomes unavailable while responding to the items, examinees may need to extensively tap into their memory strategy, which was not relevant for the current reading test. Coders practiced coding sample items by first solving each item and then immediately marking the major attributes they used on the blank Q-matrix. The list of L2 reading attributes was used as a reference whenever needed. After the training session, each coder coded the 30 items on the reading test. Once the coders finished coding the items, they individually met with the researcher to report the identified L2 reading attributes. They explained why they chose certain attributes at the item level. These sessions were recorded using a digital recorder.

Coded data were compiled and analyzed to develop an initial Q-matrix. Attributes which three or more coders agreed upon (with 60% or higher coder agreement) were selected as essential for the item and included in the initial Q-matrix. Owing to the large number of coders (i.e., five coders), Fleiss Kappa was calculated to measure the agreement among the coders (Landis & Koch, 1977). To comply with the criterion proposed by Hartz et al. (2002), attributes that were measured by fewer than three items, thereby not providing statistically meaningful information, were either merged with similar attributes or deleted from the Q-matrix (refer to initial Q-matrix in Appendix A).

The initial Q-matrix was further refined through Fusion model analysis in an iterative process. Reading test data were analyzed in conjunction with the Q-matrix using the Arpeggio Suite program, which had implemented the Fusion model. The first step in the Fusion model analysis involved an analysis of Markov Chain Monte Carlo (MCMC) convergence to guarantee that model parameters reached a stable value (Roussos, Templin, & Hensen, 2007). Three parameters were evaluated: (1) examinees’ probability of mastery for each attribute ( $p_{k}$ ); (2) item difficulty (π*); and (3) item discrimination (r*). MCMC procedures create Markov chains of simulated values for parameter estimation. By using the results from the Arpeggio software, the R statistical package was used to produce both the chain plot and the density plot for each parameter. The plots graphically indicated the degree to which the parameters converged to desired values. After checking MCMC convergence, item difficulty (π*) and item discrimination parameters (r*) were evaluated. Non-informative item parameters were dropped from the Q-matrix and Fusion model analysis was conducted again until satisfactory results were obtained.

After refining the Q-matrix, two types of fit statistics were measured to evaluate the fit of the model to the data: (1) FUSIONStats and (2) item mastery statistics (IMStats). FUSIONStats compare the difference between the observed item p-values (the proportion of observed correct items) and the estimated item p-values (the proportion of estimated correct items). A low difference between the two p-values suggests a good fit of the data. Item mastery statistics (IMStats) compare the observed performance of masters versus non-masters at the item level. In addition, the reliability of the Fusion model was examined by analyzing the Correct Classification Rate (CCR), the consistency of classification of examinees into masters versus non-masters of attributes. Finally, examinees’ probability of mastery for each attribute ( $p_{k}$ ) was analyzed to diagnose examinees’ strengths and weaknesses in L2 reading ability at the attribute level. Results were analyzed for the overall group, three language proficiency groups (i.e., beginner, intermediate, and advanced), and individual learners.

Results

Q-matrix development

Content experts’ coding was compiled into a Q-matrix. In the Q-matrix as seen in Appendix A, the rows represent the 30 items from the CEP reading test (Item 44 to Item 73), while the columns indicate the L2 reading attributes. The overall coder agreement obtained using Fleiss Kappa was .38, indicating fair agreement among the coders (Landis & Koch, 1977). Attributes that were measured by fewer than three items were either merged with similar attributes or deleted from the Q-matrix following Hartz et al. (2002). Specifically, the five different types of pragmatic meanings (i.e., contextual, sociolinguistic, sociocultural, psychological, and rhetorical meanings) were measured by only eight items, so they were merged into one pragmatic meaning attribute. Also, the strategy of integrating ideas was measured by only two items, so it was merged into the summarizing strategy, which was conceptually similar. Likewise, predicting strategy was merged with inferencing strategy owing to their similarities.

Therefore, the initial Q-matrix consisted of 10 attributes, including five knowledge-related attributes (i.e., lexical; cohesive; sentence; paragraph/text; and pragmatic meaning) and five reading strategies (i.e., identifying word meaning; finding information; skimming; summarizing; inferencing). Each item appeared to measure a minimum of two to a maximum of four L2 reading attributes. Numerous reading attributes were involved in completing an item due to the complex nature of reading (Alderson, 2000; Urquhart & Weir, 1998). In addition, all items minimally measured one knowledge-related attribute and one reading strategy, indicating that readers used sequences of knowledge-related attributes and strategies when completing reading tasks.

To examine the statistical meaningfulness of the identified attributes in the initial Q-matrix, Fusion model analysis was conducted using the Arpeggio software. First, Markov Chain Monte Carlo (MCMC) method was used with a long Markov chain length of 200,000 with a 10,000 burn-in period to guarantee convergence of parameters. Among the various parameters, the convergence of overall examinees’ probability of mastery for each attribute ( $p_{k}$ ) was first investigated. This was necessary as the $p_{k}$ estimates may affect the remaining parameter estimation (Johnson, 2006). All $p_{k}$ values converged according to chain plots and the density plots. In addition, parameters for item difficulty (π*) and item discrimination (r*) converged to stable values. Findings revealed that two items (i.e., Items 59 and 62) had π* values below .6, suggesting that they were too difficult and may need some modification. In addition, a number of r* values were above .9, indicating that their corresponding items were not discriminating well between the masters and non-masters of the attributes: r*44.1(first attribute of Item 44), r*46.2, r*48.5, r*49.10, r*54.5, r*62.1, r*64.5, r*63.9. These non-informative attributes were further examined to determine whether they were to be kept or dropped from the Q-matrix. Although a parsimonious model with fewer parameters is preferable, it is first necessary to consider both statistical and substantive criteria when dropping parameters from the Q-matrix. After careful consideration, the large r* parameters were dropped from the Q-matrix one at a time. The Fusion model analysis was run again using the reading test data in conjunction with the refined Q-matrix. MCMC convergence indicated that all parameters converged, and item difficulty and discrimination parameters were generally within the accepted range (refer to Appendix B). Dropping the non-informative r* values did not result in a noticeable change in the Q-matrix. The refined Q-matrix still consisted of 10 L2 reading attributes, and each item measured between one and four attributes.

The fit of the Fusion model to the data was checked using FUSIONStats and item mastery statistics (IMStats). FUSIONStats analysis produced the observed item p-values and the estimated item p-values. The absolute difference between each observed item p-value and the estimated item p-value for each item was minimal, and was below the suggested value of .05 for all items (L. Roussos, pers. comm., March 8, 2010). Moreover, the mean absolute difference between the p-values was low at .005. These results, summarized in Table 5, suggested that the Fusion model fit well to the data.

Table 5.

Observed and estimated item p-values.

Item	Observed item p-value	Estimated item p-value	Absolute observed-estimated item p-value
44	0.923	0.926	0.003
45	0.821	0.828	0.006
46	0.923	0.925	0.002
47	0.823	0.830	0.007
48	0.853	0.854	0.001
49	0.862	0.861	0.000
50	0.753	0.759	0.006
51	0.703	0.711	0.008
52	0.669	0.680	0.012
53	0.812	0.817	0.005
54	0.668	0.670	0.003
55	0.553	0.559	0.006
56	0.625	0.634	0.009
57	0.617	0.619	0.002
58	0.756	0.762	0.006
59	0.301	0.308	0.007
60	0.726	0.729	0.003
61	0.518	0.524	0.006
62	0.479	0.484	0.004
63	0.742	0.752	0.010
64	0.644	0.647	0.003
65	0.659	0.662	0.003
66	0.363	0.366	0.003
67	0.498	0.507	0.010
68	0.457	0.463	0.006
69	0.391	0.401	0.011
70	0.734	0.738	0.004
71	0.455	0.459	0.005
72	0.607	0.613	0.007
73	0.705	0.709	0.005
Mean absolute difference:			0.005

In addition, item mastery statistics (IMStats) was used to compare the observed performance of masters versus non-masters at the item level. Three different values were examined to evaluate IMStats: (1) the phat (m), which refers to the probability of correctly responding to an item given mastery of the attributes required for that item (over all items); (2) the phat (nm), indicating the probability of correctly responding to an item given non-mastery of the attributes required for that item (over all items); and (3) the pdiff, indicating the average difference between phat (m) and phat (nm) across items. In the current data, as seen in Table 6, the average phat (m) across all items was .869, indicating that the average probability of getting a correct response to an item by masters of attributes was high at 86.9%. Contrarily, the average phat (nm) was .474, so the average probability of having a correct response to an item by non-masters of attributes was much lower at about 47.4%. Thus, the pdiff was .395, indicating that the masters of attributes on an item outperformed non-masters of attributes on average by 39.5% across all items. This high value indicated a good fit between the estimated model and the observed data, suggesting a strong diagnostic power of the model.

Table 6.

Probability of correctly responding to an item.

Item	phat (m)	phat (nm)	phat (m) – phat (nm)
44	0.989	0.777	0.212
45	0.978	0.662	0.316
46	0.985	0.785	0.200
47	0.991	0.690	0.301
48	0.993	0.717	0.276
49	0.970	0.745	0.225
50	0.963	0.618	0.345
51	0.976	0.527	0.448
52	0.981	0.409	0.573
53	1.000	0.619	0.382
54	0.809	0.531	0.278
55	0.773	0.380	0.393
56	0.883	0.480	0.403
57	0.921	0.432	0.489
58	0.952	0.600	0.352
59	0.459	0.212	0.247
60	0.883	0.603	0.280
61	0.784	0.298	0.486
62	0.584	0.249	0.335
63	1.000	0.481	0.519
64	0.871	0.426	0.445
65	0.879	0.485	0.394
66	0.657	0.120	0.537
67	0.734	0.253	0.481
68	0.730	0.280	0.449
69	0.798	0.143	0.655
70	0.966	0.542	0.424
71	0.696	0.266	0.430
72	0.912	0.410	0.503
73	0.969	0.494	0.476
Average	0.869	0.474	0.395

Figure 1 visually depicts the probability of getting the item correct for both the masters and the non-masters of attributes on each item. Generally, there was a clear pattern of separation between the masters and the non-masters of attributes for each item.

Figure 1.

Item mastery statistics.

Furthermore, the reliability of the Fusion model was examined by evaluating the Correct Classification Rate (CCR) index. CCR refers to the consistency of classification of examinees into masters versus non-masters of attributes if the same test were administered to the same examinee group multiple times (Roussos et al., 2007). The CCR values range between zero and one. In the current data, the CCR was high at .876, suggesting a high reliability of the Fusion model.

Cognitive diagnosis of learners’ L2 reading ability

Examinees’ L2 reading performance was analyzed in terms of their mastery probability of the L2 reading attributes. The results were analyzed for the following: (1) the overall group; (2) three reading proficiency groups (i.e., beginner, intermediate, and advanced); and (3) individual examinees.

Overall group

The attribute mastery of the overall group was investigated by examining the population’s probability of mastery for each attribute ( $p_{k}$ ). As seen in Table 7, for the knowledge-related attributes, the $p_{k}$ values ranged from .519 (knowledge of cohesive meaning) to .615 (knowledge of pragmatic meaning). That is, 51.9% of examinees had mastery of knowledge of cohesive meaning, making it the most difficult knowledge-related attribute. However, 61.5% of examinees had mastery of knowledge of pragmatic meaning, making it the easiest knowledge-related attribute. On the other hand, the $p_{k}$ values of strategies ranged from .537 (summarizing) to .752 (skimming), making summarizing the most difficult and skimming the easiest strategy. The $p_{k}$ values of the 10 L2 reading attribute are graphically depicted in Figure 2.

Table 7.

Overall group’s attribute mastery probability.

L2 Reading attribute		Mastery probability ( $p_{k}$ )
Language knowledge	Lexical meaning	0.589
	Cohesive meaning	0.519
	Sentence meaning	0.547
	Paragraph/text meaning	0.593
	Pragmatic meaning	0.615
Strategy	Identifying word meaning	0.742
	Finding information	0.744
	Skimming	0.752
	Summarizing	0.537
	Inferencing	0.591

Figure 2.

Overall group’s mastery of L2 reading attributes (N = 1982).

Three reading proficiency groups – beginner, intermediate, and advanced groups

To examine the attribute mastery probability of three reading proficiency groups (i.e., beginner, intermediate, and advanced), the entire examinee population was stratified into five different reading proficiency levels according to their total scores. Using comprehensive sampling (Wiersma, 2000), three sub-groups were selected from the examinee population: beginner (bottom 20%), intermediate (middle 20%), and advanced (top 20%) levels. Then, the L2 reading attribute mastery probabilities of the three reading proficiency levels were obtained by averaging each group’s individual examinees’ attribute mastery probability, $p_{k} ({\underline{X}}_{j})$ . A one-way analysis of variance (ANOVA) was conducted to determine that the attribute mastery probabilities of the three reading proficiency levels significantly differed at the p < .05 level.

As presented in Table 8 and Figure 3, the beginners demonstrated poor performance on the reading test with very low attribute mastery probability across all 10 attributes. Their mastery probabilities showed very little variability, with attributes ranging from approximately .05 to .2. The intermediates demonstrated average performance across the attributes. Their mastery probability for knowledge-related attribute ranged from approximately .53 (cohesive meaning) to .70 (pragmatic meaning), while the probability for reading strategies ranged from .56 (summarizing) to .95 (skimming), indicating more variability than beginners. Notably, the intermediates’ mastery probabilities on three strategies – identifying word meaning, finding information, and skimming – were very high at above .9. This suggests that the three strategies were comparatively easier to master than other attributes. Unsurprisingly, the advanced group had very high mastery probabilities across all 10 L2 reading attributes above .95.

Table 8.

L2 reading attribute mastery of reading proficiency groups.

Level	Language knowledge					Strategy
	1	2	3	4	5	6	7	8	9	10
	Lexical meaning	Cohesive meaning	Sentence meaning	Paragraph/text meaning	Pragmatic meaning	Identifying word meaning	Finding information	Skimming	Summarizing	Inferencing
Beginner	0.102	0.047	0.060	0.099	0.109	0.188	0.210	0.165	0.057	0.085
Intermediate	0.664	0.528	0.569	0.656	0.698	0.906	0.894	0.949	0.562	0.642
Advanced	0.965	0.972	0.968	0.970	0.979	0.996	0.994	1.000	0.973	0.984
Overall	0.589	0.519	0.547	0.593	0.615	0.742	0.745	0.753	0.537	0.591

Figure 3.

L2 reading attribute mastery statistics of reading proficiency groups.

Individual examinees

To diagnose individual examinees’ L2 reading ability, the individual examinees’ attribute mastery probabilities, or $p_{k} ({\underline{X}}_{j})$ , were examined. The Fusion model analysis produced $p_{k} ({\underline{X}}_{j})$ values of all 10 L2 reading attributes for all 1,982 examinees. Due to the sheer number of examinees, 10 learners with a total score of 20 were randomly selected for a close examination. Although the 10 examinees had the same total score as presented in Table 9, their mastery probability of each attribute varied considerably as a result of individual differences. Examinees’ mastery probabilities of knowledge-related attributes ranged from approximately .05 (Examinee B’s mastery of cohesive meaning) to .92 (Examinee J’s mastery of pragmatic meaning), while their mastery probabilities of strategies ranged from approximately .15 (Examinee E’s mastery of inferencing) to .99 (Examinee H’s mastery of skimming). On average, the 10 examinees had comparatively low mastery of knowledge cohesive meaning (.489) and knowledge of sentence meaning (.369), suggesting that these were their weaknesses. However, they had relatively high mastery of identifying word meaning (.858), finding information (.826), and skimming (.955), suggesting that these were their strengths.

Table 9.

Individual examinees’ mastery probability of L2 reading attributes.

ID	Language knowledge					Strategy
	1	2	3	4	5	6	7	8	9	10
	Lexical meaning	Cohesive meaning	Sentence meaning	Paragraph/text meaning	Pragmatic meaning	Identifying word meaning	Finding information	Skimming	Summarizing	Inferencing
A	0.499	0.274	0.139	0.763	0.415	0.799	0.586	0.996	0.876	0.340
B	0.814	0.046	0.082	0.320	0.818	0.947	0.940	0.827	0.650	0.849
C	0.364	0.101	0.573	0.608	0.797	0.611	0.926	0.955	0.415	0.918
D	0.837	0.669	0.231	0.592	0.416	0.974	0.946	0.997	0.706	0.377
E	0.219	0.807	0.165	0.712	0.747	0.967	0.710	0.889	0.867	0.149
F	0.636	0.440	0.389	0.889	0.710	0.774	0.781	0.927	0.970	0.432
G	0.166	0.914	0.542	0.591	0.076	0.619	0.955	0.993	0.225	0.811
H	0.564	0.628	0.325	0.713	0.315	0.939	0.935	0.999	0.179	0.778
I	0.688	0.442	0.500	0.855	0.421	0.977	0.514	0.975	0.911	0.870
J	0.748	0.565	0.747	0.846	0.921	0.974	0.970	0.992	0.960	0.407
Average	0.554	0.489	0.369	0.689	0.564	0.858	0.826	0.955	0.676	0.593

Among the 10 examinees, the performance of two – Examinees A and B – was examined in more detail. As seen in Figure 4, Examinee A had a mastery probability over .7 on Attributes 4, 6, 8, and 9. However, his performance on the remaining attributes was less successful, suggesting that these were his weaknesses. On the other hand, as seen in Figure 5, Examinee B had obtained a high mastery probability of over .7 on most attributes except for Attributes 2, 3, 4, and 9. These cases illustrate how two examinees may receive the same total scores, but vary in their strengths and weaknesses in L2 reading ability.

Figure 4.

L2 reading attribute mastery of Participant A.

Figure 5.

L2 reading attribute mastery of Participant B.

Discussion and conclusion

Summary of the findings

The purpose of this study was to diagnose learners’ L2 performance on an L2 reading placement test with the ultimate goal of developing diagnostic feedback for ESL program administrators and teachers. This first required constructing the Q-matrix. Analysis of the Q-matrix identified 10 major L2 reading attributes necessary for completing the reading test, including five knowledge-related attributes (i.e., lexical; cohesive; sentence; paragraph/text; and pragmatic meaning) and five reading strategies (i.e., identifying word meaning; finding information; skimming; summarizing; inferencing). These findings suggest that the list of L2 reading attributes could be used as a framework for CDA research of L2 reading.

Examinees demonstrated different attribute mastery probability among the 10 L2 reading attributes. This not only indicated their strengths and weaknesses, but also suggested that some attributes were more difficult to master compared to others. Among the knowledge-related attributes, cohesive meaning was the most difficult attribute to master, whereas pragmatic meaning was the easiest. Cohesive meanings are mapped onto various cohesive forms through lexical cohesion, reference, and substitution (Halliday & Hasan, 1976), and replace previously mentioned parts in the text (de Beaugrande, 1980; Halliday & Hasan, 1976). Although research suggests that knowledge of cohesive meaning assists with comprehension (Cohen, Glasman, Rosenbaum-Cohan, Ferrara, & Fine, 1979), few studies have examined the relative difficulty of cohesive meaning compared to other types of meanings on L2 reading tests. However, studies on L2 grammar (e.g., Ameriks, 2009; Dakin, 2010; Liao, 2009) have found that cohesive meaning is more difficult than lexical meaning, which concur with the results of the current study. Cohesive devices require an accurate understanding of the relationship between two parts of the text, making it challenging to master (Dakin, 2010).

In contrast to cohesive meanings, pragmatic meaning was the easiest to master. This finding may appear to counter the claims made in prior research, which has generally viewed pragmatic meaning as highly contextualized implied meaning, making it more difficult to understand than literal or intended meanings (e.g., Gray, 1960; Herber, 1978; Purpura, 2004). However, in accordance with the findings of the current study, Liao (2009) also found that pragmatic meaning items had a higher mean in comparison to semantic meaning items on a reading test. She explained that other factors, such as textual features, may have affected the difficulty of these items. In addition, test-takers may have lacked decoding skills, but had sufficient prior knowledge, which could have compensated for their deficiencies in language proficiency (Stanovich, 1980), resulting in a comparatively higher mean for pragmatic meaning items.

Among reading strategies, summarizing and inferencing were found to be more difficult compared to the other three strategies (i.e., identifying word meaning, finding information, skimming). These findings regarding the hierarchy of L2 reading attributes concur with previous research (Grabe & Stoller, 2002; Lumley, 1993). Summarizing may have been comparatively challenging for L2 learners since it requires readers first of all to comprehend the overall text and then extract the gist from it. Understanding the gist involves numerous elements, such as knowledge of grammar, vocabulary, discourse structure, and various cognitive processes (Pressley, 2002). Similarly, inferencing may have been difficult to master owing to its complexity (e.g., Hammadou, 1991; Hosenfield, 1977; Long, Seely, Oppy, & Golding, 1996). It requires readers to understand not only the literal meaning of the text, but also its implied meaning. Overall, both summarizing and inferencing strategies were identified to be difficult attributes because they required higher-level processing of information from the text (Grabe, 2009).

Moreover, the three different reading proficiency groups (i.e., beginner, intermediate, advanced) showed distinct attribute mastery probability. Not surprisingly, beginners had low mastery probability across all 10 attributes with little variability. The intermediates had attribute mastery probability similar to the overall group, with high variability in attribute mastery. Notably, their mastery probability on three strategies – identifying word meaning, finding information, and skimming – was very high. This suggests that the three strategies were comparatively easier to master than other attributes. The advanced group had very high mastery probabilities across all 10 L2 reading attributes. Although examinees as a group showed a distinct attribute mastery pattern, individuals within the group still demonstrated high variability.

Study implications and suggestions for the future

Pedagogical implications could be drawn from this study regarding the type of diagnostic feedback that could be provided to various stakeholders. The detailed information regarding the attribute mastery probability of the overall group and the three reading proficiency groups, presented in Appendix C, could be provided as part of a score report to ESL program administrators and teachers. They could refer to the information to further refine the L2 reading curricula relevant for each reading proficiency level. For beginners and intermediates, it is important to design a reading curriculum that reflects their disproportionate mastery pattern of L2 reading attributes. Typically, L2 reading attributes with lower mastery probabilities (e.g., cohesive meaning, summarizing, inferencing) would require more instruction in comparison to attributes with higher mastery probabilities (e.g., finding information, skimming). Advanced learners had high mastery probabilities across the 10 attributes, which could be attributed to learners performing exceptionally well on the reading test, making it difficult to identify their weaknesses. To prevent such a ceiling effect, future studies need to include more difficult items to the test.

Similarly, ESL instructors could be provided with diagnostic information regarding individual learners’ strengths and weaknesses for instructional purposes. They could develop lesson plans tailored to individual learners’ specific language weaknesses or needs. For instance, if an individual learner lacked knowledge of cohesive meaning and knowledge of sentence meaning (refer to Figure 4), the teacher should focus on instructing these attributes. This could potentially lead to improvements in the learners’ L2 reading ability.

Although this study provides pedagogical implications for enhancing ESL instruction, some limitations exist. It used an ex post facto design (Wiersma, 2000), in which an existing reading test was retrofitted for diagnostic purposes, whereas ideally it should have been designed specifically as a diagnostic test (Alderson, 2005). In a truly diagnostic test, the item-by-attribute relationship would have been pre-determined and, as a result, ease the process of developing a Q-matrix. Retrofitting a CDM to an existing test could be limited by design in extracting rich diagnostic information. Thus, in this study, attributes that were measured by less than three items, and thereby not adding meaningful information to the matrix, were either merged with other attributes or deleted from the Q-matrix following the suggestions by Hartz et al. (2002). However, using a diagnostic test with a pre-determined Q-matrix may prevent such loss of attribute data, therefore ultimately lead to better CDA of leaners’ language performance.

In the future, CDA research should focus on enhancing methods for developing the Q-matrix. There is as yet no standardized method of Q-matrix development. In this study, the Q-matrix was developed by having content experts code the reading test while referring to the list of L2 attributes. This was based on the assumption that content experts would know the most efficient way to reach the key to the item, whereas ESL learners may not. For instance, a beginning learner may refer to the wrong cues within a text to respond to an item, but may still correctly respond to the item. However, combining learners’ verbal protocol with expert judgment may have provided more elaborate account of the major reading attributes involved in the test. Also, in the current study, coders individually coded the items and did not have an opportunity to convene to come to an agreement, which has led to a relatively low coder agreement. Having additional group meetings may have led to a higher coder agreement and better overall Q-matrix development.

In addition, considering the practical implications of diagnostic assessment, there is a dire need for more CDA research in L2 reading. Few studies have examined the effect of providing diagnostic feedback to stakeholders. Jang (2005) found that diagnostic feedback to learners and teachers contributed to enhancing learners’ L2 reading ability. However, in the study, students’ improvements could not be directly connected to the diagnostic information because they did not notice their weak attributes. She also found that teachers did not necessarily utilize diagnostic information for instructional purposes. However, only three teachers were interviewed, making it difficult to generalize the findings. Therefore, future studies need to address the washback effect of providing diagnostic feedback to different stakeholders. In the current research context of an ESL program, subsequent studies need to investigate program administrators’ and teachers’ perception of the score reports and actual application of diagnostic information for instructional purposes. Also, it would be useful to identify the kinds of diagnostic information that is most informative and beneficial for pedagogical purposes.

Footnotes

Appendix A.

Appendix B.

Appendix C:

Acknowledgements

I would like to thank James Purpura and Michael Kieffer for their comments on earlier drafts, as well as the anonymous reviewers of Language Testing for their valuable feedback and suggestions.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

References

Alderson

J. C.

(2000). Assessing reading. Cambridge: Cambridge University Press.

Alderson

J. C.

(2005). Diagnosing foreign language proficiency. New York: Continuum.

ALTE. (1998). Multilingual glossary of language testing terms. Cambridge: Cambridge University Press.

Ameriks

Y. S.

(2009). Investigating validity across two test forms of the examination for the certificate of proficiency in English (ECPE): A multi-group structural equation modeling approach. Available from ProQuest Dissertations and Theses database. (AAT 3348566)

Bachman

L. F.

(1990). Fundamental considerations in language testing. Oxford: Oxford University Press.

Bachman

L. F.

Palmer

A. S.

(1996). Language testing in practice. Oxford: London.

Birenbaum

Kelly

A. E.

Tatsuoka

K. K.

(1993). Diagnosing knowledge states in algebra using the rule-space model. Journal of Research in Mathematics Education, 24, 442–459.

Buck

Tatsuoka

Kostin

(1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47(3), 423–266.

Celce-Murcia

Larsen-Freeman

(1999). The grammar book: An ESL/EFL teacher’s course. Boston, MA: Heinle and Heinle.

10.

Chipman

S. F.

Nichols

P. D.

Brennan

R. L.

(1995). Introduction. In Chipman

S. F.

Nichols

P. D.

Brennan

R. L.

(Eds.), Cognitively diagnostic assessment (pp. 1–18). Hillsdale, NJ: Lawrence Erlbaum.

11.

Cohen

A. D.

(2007). Coming to terms with language learner strategies: Surveying the experts. In Cohen

A. D.

Macaro

(Eds.), Language learner strategies: 30 years of research and practice (pp. 29–45). Oxford, UK: Oxford University Press.

12.

Cohen

A. D.

Glasman

Rosenbaum-Cohan

P. R.

Ferrara

Fine

(1979). Discourse analysis and the use of student informants. TESOL Quarterly, 13, 551–564.

13.

Cohen

A. D.

Upton

T. A.

(2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph No. MS-33). Princeton, NJ: ETS.

14.

Dakin

J. W.

(2010). Investigating the measurement of grammatical knowledge and civics content knowledge in the context of an English-for-specific-purposes program designed for adult learners with low English proficiency. Available from ProQuest Dissertations and Theses database. (AAT 3436226)

15.

de Beaugrande

(1980). Text, discourse and process: Toward a multidisciplinary science of texts. London: Longman.

16.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130.

17.

DiBello

L. V.

Roussos

L. A.

Stout

(2007). Review of cognitively diagnostic assessment and a summary of psychometric models. Rao

C. R.

Sinharay

(Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 970–1030). Amsterdam: North-Holland Publications.

18.

DiBello

L. V.

Stout

(2008). Arpeggio documentation and analyst manual. St. Paul: Applied Informative Assessment Research Enterprises (AIARE).

19.

Faerch

Kasper

(1983). Strategies in interlanguage communicatio n. London: Longman.

20.

Falmagne

J. C.

Doignon

J. P.

(1988). A class of stochastic procedures for assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41, 1–23.

21.

Grabe

(2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.

22.

Grabe

Stoller

F. L.

(2002). Teaching and researching: reading. Harlow: Longman.

23.

Grabowski

K. C.

(2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking. Available from ProQuest Dissertations and Theses database. (AAT 3368256)

24.

Gray

W. S.

(1960). The major aspects of reading In Robinson

(Ed.), Sequential development of reading abilities (pp. 8–24). Chicago, IL: Chicago University Press.

25.

Haberman

von Davier

(2006). Some notes on models for cognitively based skills diagnosis. In Rao

Sinharay

(Eds.), Handbook of statistics (pp. 1031–1038). Amsterdam: Elsevier.

26.

Haertel

E. H.

Wiley

D. E.

(1993). Representation of ability structures: Implications for testing. In Frederiksen

Mislevy

Bejar

(Eds.), Test theory for a new generation of tests (pp. 359–384). Hillsdale, NJ: Lawrence Erlbaum.

27.

Halliday

M. A. K.

Hasan

(1976). Cohesion in English. London: Longman.

28.

Hammadou

(1991). Interrelationships among prior knowledge, inference, and language proficiency in foreign language reading. The Modern Language Journal, 75, 27–38.

29.

Hartz

S. M.

Roussos

Stout

(2002). Skills diagnosis: Theory and practice. Unpublished manuscript. University of Illinois at Urbana-Champaign.

30.

Herber

(1978). Teaching reading in the content areas. Englewood Cliffs, NJ: Prentice Hall.

31.

Hosenfield

(1977). Learning about learning: Discovering our students’ strategies. Foreign Language Annals, 9, 117–129.

32.

Hughes

(1989). Testing for language teachers. Cambridge: Cambridge University Press.

33.

Jang

E. E.

(2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Available from ProQuest Dissertations and Theses database. (AAT 3182288)

34.

Johnson

J. F.

(2006). Diagnosing skill mastery in the national assessment of educational progress: Applications of the Fusion Model. Available from ProQuest Dissertations and Theses database. (AAT 3223309)

35.

Kasai

(1997). Application of the rule space model to the reading comprehension section of the test of English as a foreign language (TOEFL). Available from ProQuest Dissertations and Theses database. (AAT 9737153)

36.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

37.

Lee

Y.-W.

Sawaki

(2009a). Cognitive diagnosis and Q-matrices in language assessment. Language Assessment Quarterly, 6, 169–171.

38.

Lee

Y.-W.

Sawaki

(2009b). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6, 172–189.

39.

Leighton

J. P.

Gierl

M. J.

(2007). Cognitive diagnostic assessment for education: Theory and applications. New York: Cambridge University Press.

40.

(2011). A cognitive diagnostic analysis of the MELAB reading test. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 9, 17–46.

41.

Liao

Y.-F.

(2009). A construct validation study of the GEPT reading and listening sections: Re-examining the models of L2 reading and listening abilities and their relations to lexico-grammatical knowledge. Available from ProQuest Dissertations and Theses database. (AAT 3348356)

42.

Long

D. L.

Seely

M. R.

Oppy

B. J.

Golding

J. M.

(1996). The role of inferential processing in reading ability. In Britton

B. K.

Graesser

A. C.

(Eds.), Models of understanding text (pp. 189–214). Hillsdale, NJ: Lawrence Erlbaum.

43.

Lumley

(1993). The notion of subskills in reading comprehension test: An EAP example. Language Testing, 10(3), 211–234.

44.

McGlohen

M. K.

(2004). The application of cognitive diagnosis and computerized adaptive testing. Available from ProQuest Dissertations and Theses database. (AAT 3145337)

45.

O’Malley

M. J.

Chamot

A. U.

(1990). Learning strategies in second language acquisition. Cambridge: Cambridge University Press.

46.

Oxford

Cho

Leung

Kim

H.-J.

(2004). Effect of the presence and difficulty of task on strategy use: An exploratory study. International Review of Applied Linguistics, 42, 1–27.

47.

Phakiti

(2007). Strategy competence and EFL reading test performance. New York: Peter Lang.

48.

Pressley

(2002). Comprehension strategy instruction: A turn-of-the-century status report. In Block

Pressley

(Eds.), Comprehension instruction: Research-based best practices (pp. 11–27). New York: Guilford Press.

49.

Pressley

Afflerbach

(1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum.

50.

Purpura

J. E.

(1999). Learner strategy use and performance on language tests: A structural equation modeling approach. Cambridge: Cambridge University Press.

51.

Purpura

J. E.

(2004). Assessing grammar. Cambridge: Cambridge University Press.

52.

Roussos

L. A.

DiBello

L.V.

Stout

W. F.

Hartz

S. M.

Henson

R. A.

Templin

J. H.

(2007). The fusion model skills diagnostic system. In Leighton

Gierl

(Eds.), Cognitive diagnostic assessment for education: Theory and applications. New York: Cambridge University Press.

53.

Roussos

Templin

Henson

(2007). Skills diagnosis using IRT-based latent class models. Journal of Educational Measurement, 44, 293–311.

54.

Sawaki

Kim

H.-J.

Gentile

(2009). Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6, 190–209.

55.

Scott

H. S.

(1998). Cognitive diagnostic perspectives of second language reading test. Available from ProQuest Dissertations and Theses database. (AAT 39912372)

56.

Stanovich

K. E.

(1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16(1), 32–71.

57.

Tatsuoka

K. K.

(1990). Toward an integration of item response theory and cognitive analysis. In Frederiksen

Glaser

Lesgold

Shafto

M. C.

(Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 543–588). Hillsdale, NJ: Lawrence Erlbaum.

58.

Templin

J. L.

(2004). Generalized linear mixed proficiency models for cognitive diagnosis. Available from ProQuest Dissertations and Theses database. (AAT 3160960)

59.

Urquhart

Weir

C. J.

(1998). Reading in a second language: Process, product and practice. New York: Longman.

60.

Weir

C. J.

Hawkey

Green

Devi

(2009). The cognitive processes underlying the academic reading construct as measured by IELTS (IELTS Research Reports No. 9, pp. 157–189). Cambridge: IELTS.

61.

Wiersma

(2000). Research methods in education (7th ed.). Boston, MA: Allyn and Bacon.

62.

Wenden

(1991). Learner strategies for learner autonomy. Englewood Cliffs, NJ: Prentice-Hall.