Abstract
Critical language awareness (CLA) is increasingly identified as a central component of the Spanish heritage language (SHL) classroom (Leeman, 2005; Martínez, 2003; among others). As a minority language, SHL is subject to sociopolitical, cultural, and economic forces that devalue its status. It is devalued in the eyes of the public, as a legitimate U.S. language, and as something worthy of being maintained. It is essential that students receive instruction not only in the heritage language, but also on the contextual factors that affect the Spanish-English sociopolitical relationship in the United States. Such instruction will enable learners to begin resisting heritage language loss by questioning language ideologies that promote English monolingualism and standard, monolingual Spanish as the ideal norm. The aim of this study was to develop a questionnaire with adequate psychometric properties to measure CLA in the SHL context. The respondents were 301 students enrolled in SHL courses in four U.S. universities. The questionnaire results were submitted to a series of statistical analysis to investigate how well the instrument meets the criteria of reliability and validity specified in this study. The final 19-item instrument had adequate psychometric properties and detected change in the CLA of students in a class where CLA was taught.
Keywords
The goal of maintaining and promoting heritage languages is more critical than ever in educational environments in the United States. A recent report from the Commission on Language Learning (2017) identified the education of heritage language (HL) learners as one of the five key goals for language instruction in the United States in the twenty-first century, in recognition of the prospective benefits not only for heritage learners themselves, but also for building the multilingual capacity of the nation as a whole. At the same time, however, because heritage languages are minority languages in the United States, their maintenance is compromised by dominant societal language ideologies that favor English monolingualism over multilingualism, monolingual varieties over bilingual ones, and educated varieties over vernacular ones. These ideologies have been historically reproduced through education and other mainstream avenues and, as a consequence, students of these languages may enter a HL course with deep-seated beliefs about the inferiority of their language, their linguistic variety, and their level of proficiency (Beaudrie & Ducar, 2005; Bernal-Henríquez & Hernández-Chávez, 2003; Martínez, 2003).
In recent years, HL educators have responded with a call to reconfigure the content of instruction so as to make students’ varieties more central and to develop students’ critical language awareness (CLA) (Beaudrie, 2015; Leeman, 2005; Leeman & Serafini, 2016; Martínez, 2003; among others). CLA is an approach originally developed in the 1990s by researchers in the United Kingdom, who argued that it was essential to include discussions of power issues as part of language and literacy instruction. Clark, Fairclough, Ivanič and Martin-Jones (1990, 1991), in a two-part article, presented CLA as an improvement over language awareness (LA) programs. The authors described CLA as seeking to provide “operational and descriptive knowledge of the linguistic practices of their world, but also a critical awareness of how these practices are shaped by, and shape, social relationships of power” (Clark et al., 1990, p. 52). Whereas LA programs in the European context had sought to promote learners’ social integration into the dominant social order by teaching the properties of language and language use, the goal of CLA was to achieve social emancipation (Clark et al., 1990). Most importantly, the primary objective of CLA is to "empower learners by providing them with a critical analytical framework to help them reflect on their own language experiences and practices, the language practices of others in the institutions of which they are a part and in the wider society within which they live" (Clark and Ivanič, 1997, p. 217). In addition, the CLA approach promotes equality both inside and outside the classroom, and nurtures students’ agency to make their own linguistic choices (Leeman, 2014). After Fairclough (1989) first developed CLA in Britain, it quickly expanded across the globe (Granville, 2003).
Recently, several instructional proposals to develop students’ CLA in the classroom have begun to emerge. However, one of the key gaps that remains unexplored is how to measure students’ CLA reliably in order to determine the effectiveness of such instructional programs. This study contributes to the current literature by describing the design and validation of an instrument to measure CLA in the HL context.
Literature review
Academic discussions around the need to include critical approaches in the classroom have a relatively long history in the English language context (Godley & Minnici, 2008). The continued devaluation of African American Vernacular English (AAVE) in particular has attracted the attention of many researchers and educators who seek to reclaim the value of this variety and its speakers. Wolfram (1998) highlighted the groundbreaking impact of the Oakland Ebonics controversy 1 on discussions of dialect and bidialectal appreciation. Wolfram also advocated that educators must have sociolinguistic awareness and, in particular, be informed about the language patterns their students use. As Ali (2005) argued in his review of the history of Black language and education in the United States, CLA can empower students to challenge the attitudes that mark their way of speaking as deficient. As many linguists have previously pointed out, the varieties spoken by disempowered groups have historically been disparaged and institutionally disenfranchised (e.g., Native Americans, Latinos, African Americans, and Hawai’ians). These minority speakers have often faced physical or psychological punishment for using their home language or variety. For instance, AAVE and other nonstandard varieties of English have been targeted for eradication and replacement with prestige varieties. The study of instructional practices to promote CLA has, however, received only minimal attention from educational researchers (Godley & Minnici, 2008). Crucially for the present study, no researchers have developed reliable, well-validated measures to gauge the instructional effectiveness of the CLA components of their curricula.
Reaser (2006) conducted one of the first studies that attempted to measure the effectiveness of an instructional module on language awareness with 129 students enrolled in ninth-grade, honors-level, Language Arts classes in a high school in North Carolina. The researcher developed a language attitude survey and administered it before and after the implementation of a 450-minute curriculum about dialect diversity. The survey was constructed to ask participants to respond to, for example, the kinds of statements about language that non-linguists commonly make. The results showed that students were more sensitive to language variation after the instruction and that, for most of the survey items, they provided more tolerant or more informed responses. The 20-item survey was not submitted to any type of statistical analysis to measure its psychometric properties, however.
Similarly, for a first-year English composition course in the United States, Murphy (2012) designed and implemented an experimental dialect awareness unit adapted from a range of sources, including Reaser and Wolfram (2007), Do You Speak American? (2005), Groome and Gibbs (2004), among others, together with instructional goals taken from Gooden (2007). Murphy sought to determine whether the explicit teaching of dialect awareness can “provide a successful counter to pervasive negative attitudes towards stigmatized dialects” (p. 1). This educational intervention introduced students to the concepts of language change, language variation, stereotypes, dialect discrimination, dominant language ideologies, and code-switching. Both control and experimental groups responded to an identical pre- and post-questionnaire that consisted of Likert items to measure whether their language awareness and tolerance for linguistic diversity had increased in each of the courses. Statistical analysis was used to determine whether students’ attitudes toward stigmatized dialects changed. Additional open-ended responses provided qualitative data about changing attitudes. The results showed marked increases in the experimental group regarding tolerance and knowledge of nonstandard dialects. Unfortunately, the instrument used was not adequately validated before use.
Several classroom studies have been conducted using qualitative assessment rather than quantitative measures to determine the effectiveness of CLA instruction. Godley and Minnici (2008) conducted one such study. Thirty-one tenth graders, who self-identified as AAVE speakers, partook in a weeklong unit where they discussed language variety and dialects. In particular, the students focused on identifying and critiquing dominant language ideologies through dialogues in which they were recognized as experts on AAVE. Data collection consisted of video- and audio-taping, observations and field notes, and written student reflections. The researchers concluded that the weeklong unit exposed students to the stigmatization of AAVE and especially why this stigmatization must be challenged. Similar classroom studies using qualitative methods include Janks (1999), who evaluated student journals for CLA; Ali (2011), who had students analyze an article from a well-known newspaper; Huang (2013), who explored CLA development through writing; and Granville (2003), who conducted an action research project.
In the context of HL in the United States, primarily for SHL, several projects have been designed to expose HL learners to CLA and to begin deconstructing ideologies that promote either English monolingualism or standard, monolingual Spanish in order to resist and reverse HL shift. Bernal-Henríquez and Hernández-Chávez (2003) highlighted the widespread negative ideology about heritage learners’ Spanish bilingual varieties. The authors explained that non-prestigious language features of heritage learners are often perceived as “barbarisms” (see also Villa, 1996, 2002). In the 1970s, textbooks for bilinguals were filled with grammar drills and focused exclusively on the teaching of a prestige variety through mechanical contrastive exercises (Valdés, 1995).
One of the earliest proposals was introduced by Martínez (2003), who argued for introducing CLA (which he called Classroom Based Dialect Awareness) to HL students at beginning levels, rather than only during advanced courses. In this seminal article, he also presented the idea of utilizing activities and games such as “euphemisms” or “nicknames” to promote CLA and, in particular, to “highlight the relationship between language, power, and social groups” (Martínez, 2003, p. 49). For the framework underlying these activities, he expanded on the three original principles proposed by Wolfram (1998): (1) dialects are natural; (2) dialects are regular; and (3) variation occurs on different levels. The latter added a social dimension with three additional elements related to dialect function, distribution, and evolution. Because many students bring to the classroom understandings and experiences of language subordination and variation, activities centered on CLA empower them to make their own choices about language use in their individual environments. Martínez’s article, though important, lacked any insights into how to measure students’ CLA in the classroom.
Leeman, Rabin, and Román-Mendoza (2011) contributed another proposal to develop students’ CLA. They designed a critical service-learning opportunity where students volunteered in local primary schools in an effort to build HL students’ sense of agency and promote language activism. Following a qualitative design, the researchers interviewed participants and found that 78% of the students claimed that this service-learning opportunity made them aware of their agency to promote HL maintenance and of their potential for activism for social change. In particular, students became aware of language ideologies that hinder Spanish maintenance and the reasons why people become language activists. Similar studies involving students in community service-learning opportunities followed (i.e., Lowther-Pereira, 2015; Martínez, 2010; Martínez & Schwartz, 2012; Moreno & MacGregor, 2016; Pascual y Cabo, Prada, & Lowther-Pereira, 2017).
Parra (2013) described an advanced language course, “Spanish and the Community,” that was designed to promote translingual and transcultural competence, critical thinking skills, and CLA. Addressing topics such as immigration, identity, and language practices, students explored a variety of genres grounded in sociocultural contexts (such as readings, film, music, and art) and highlighting a multiliteracy approach to language development. A unique aspect of her approach was the incorporation of art into course assignments as a tool to explore these connections. To foster their connections to the community, students participated in a service-learning component designed to deepen their understanding of and connection to the Latino community. Students successfully developed their translinguistic and transcultural competence, as reflected in a final written essay and oral presentation; CLA development was not formally evaluated. Subsequent articles presented similar approaches with courses designed to provide students with sociolinguistic awareness that develops their understanding of English–Spanish language use in the United States (see Velásquez, 2013; Shin & Hudgens Henderson, 2017). Leeman and Serafini (2016) presented probably the most complete proposal for how to incorporate CLA into the HL classroom to develop students’ “understanding of the social, cultural, and political aspects of language, language variation, and multilingualism” (p. 71). Following an in-depth explanation of the fundamental sociolinguistic concepts necessary to implement a CLA approach, they provided concrete classroom activities and assignments that developed students’ awareness of multilingualism, language variation, multilingual discourse and code-switching, and language ideologies.
Holguín Mendoza’s (2017) study was the only one in the Spanish heritage context that examined the impact of implementing a CLA approach in Spanish HL courses using a quantitative approach. She utilized a linguistic attitudinal survey containing 30 sentences that incorporated stigmatized words of Mexican Spanish used in Spanish spoken in the United States. To complete the survey, students marked whether they considered a sentence correct or incorrect, and whether they would or would not use these specific sentences. The survey was presented at the beginning and at the end of the term. The results showed that students as a group increased their sociolinguistic awareness but, as the author pointed out, this type of attitudinal survey may not be the best tool to fully evaluate progress in CLA after an instructional intervention.
As this review shows, several recent studies have contributed to defining the construct of CLA by describing instructional programs to teach CLA in various contexts, but few studies have sought to study empirically the results of teaching CLA in either the English or the HL context. None of these studies have sought to develop a psychometrically sound instrument to measure CLA. Empirical research is needed first to examine how to assess CLA reliably, and then to measure the effectiveness of CLA instruction in HL contexts. Although a few qualitative or quantitative measures have been used in the past, the present study is the first of its kind in that it is an attempt to design, validate, and implement a questionnaire that has sound psychometric properties to measure this construct in the heritage context.
Study
The purpose of the present study was to design and validate a questionnaire to measure CLA in the HL context. We followed a systematic, seven-step design and validation process for the development of the instrument in order to ensure that the validity and reliability criteria set for this study were met to produce a psychometrically sound instrument. This process follows recommended practices in social sciences (Artino, La Rochelle, Dezee, & Gehlbach, 2014; Dörnyei, 2007). A summary of the steps followed for the development and validation process is presented in Table 1, and each of the seven steps are discussed in detail below.
Seven-step questionnaire development and validation process.
Questionnaire construction
Step 1: Construct definition
After a review of the relevant literature in dialect awareness and CLA, succinctly summarized above, the research team generated the key thematic areas to be included within the general construct of CLA in the HL context. The topics for the first theme, found in Table 2, address the important sociolinguistic topics of language variation and linguistic diversity as well as the intrinsic validity of all ways of speaking. The foundation of this first area is an understanding that language is inherently and naturally variable, prone to linguistic change and contact over time (Bills & Vigil, 2008; Otheguy, 1993; Otheguy & García, 1993; Otheguy & Zentella, 2012; Poplack, 1980; Silva-Corvalán, 1994; Zentella, 1982). It also challenges preconceptions students may have about the concepts of dialect and language and the superiority of one over the other (Bills, 2005; Galindo, 1995, 1996; Hidalgo, 1986; Rivera-Mills, 2000). Lastly, prior research in language ideologies specific to SHL pedagogy (Leeman, 2012; Lowther Pereira, 2010; Villa, 2002) have indicated the predominance and emphasis of the standard in university-level language education (Beaudrie, 2015) and a privileging of European varieties of Spanish (Leeman, 2010, 2012). Students have a tendency to internalize the disparaging attitudes of educators, language programs, and society toward non-dominant dialects, such as U.S. Spanish, who may view them as inappropriate for class use or instruction, and as a chaotic corruption of both English and Spanish (Bernal-Henríquez & Hernández-Chávez, 2003; Zentella, 2008).
Thematic areas for CLA.
The second set of themes in Table 2 speaks to the hegemonic role of English in the United States and, consequently, the pressures Spanish faces from a dominant U.S. culture that promotes monolingualism as the gold standard. In the first topic, “language and power,” the historical privileging of English over other immigrant languages (e.g. Spanish) is included (Schmid, 2000), in order to identify the ideological power instilled in the majority vs. minority language contrast. In particular, attention is given to the Hispanophobia prevalent in society that promotes an anti-bilingual sentiment against the U.S. Spanish-speaking communities (Zentella, 1997). Moreover, the “privileged position of English” topic covers the real-world consequences of not adhering to English proficiency as a core component of American national identity (Pavlenko, 2002, 2003). The final topic, “dominant language ideologies,” points to the institutional ideology that excludes minority languages, such as educational and public institutions (Ruiz, 1988; Wright, 2005).
The third set of items, also in Table 2, focuses on Spanish in the United States in an effort to debunk common linguistic beliefs held about U.S. Spanish and the common phenomenon associated with the umbrella term: Spanglish. A common attitude held by Spanish heritage students views the so-called mixing of English and Spanish as a deficit due to the failure to keep both languages separate. To the contrary, the first topic, “characteristics of in the U.S.,” draws from seminal works in recent decades (Poplack, 1980; Zentella, 1997, among others) that indicate that code-switching is rule-governed, serves discourse functions, and is a marker of group membership. In particular, the second topic, “phenomena of languages in contact,” focuses on the linguistic features of Spanglish and illustrates the common situation of languages in contact from around the world. Lastly, the topic, “code-switching,” focuses on an important area, as it is “the most salient, and this is frequently perceived by both insiders and outsiders as indicative of disfluency or an inability to speak only one language at a time” (Carvalho, 2012, p. 139).
Finally, the fourth theme relates to language maintenance and policies. Under the topic “HL maintenance/loss and the benefits of bilingualism,” attention is given to the combination of dominant societal pressures on Spanish as a minority language and the internal communal deficit views of U.S. Spanish that contribute to its eventual generational shift (Mejías & Anderson, 1988). The “language policies in the U.S.” topic covers the language shift toward Spanish that is a well-documented phenomenon (Bills, 2005; Bills, Hernández-Chávez & Hudson, 1995; Jenkins, 2010; MacGregor-Mendoza, 2005; Villa & Rivera-Mills, 2009) upheld by societal support for language policies that restrict the teaching of minority languages in public schools and its use in public spaces (Cashman, 2006; Wiley, 2005; Wright, 2005). The topic “developing student agency in HL maintenance” explores some preventative measures essential to mitigate language loss, while promoting active language use and advocacy of the heritage language. The four main themes selected as part of what CLA entails based on prior research are summarized in Table 2.
Step 2: Item development
Existing questionnaires were consulted, but none was fully appropriate for the heritage context (Hudgnes-Henderson, 2016; Murphy, 2012; Reaser, 2006). Therefore, the research team adapted some items from existing questionnaires, but generated the majority of the items from scratch. Items were carefully constructed to be succinct and avoid double-barreled items (Dörnyei & Taguchi, 2010), as well as to measure individuals’ attitudes and intended behaviors in relation to the topics in Table 2. The items were constructed using a Likert-type scale (6 = strongly agree or very likely to 1 = strongly disagree or very unlikely). The original item pool consisted of 48 items (12 per thematic area), which were developed and categorized into four thematic sections with three topics per theme, as indicated in Table 2:
Language variation and linguistic diversity (Example of ‘standard vs. non-standard forms’ topic) Attitude: People from Spain speak the purest form of Spanish. Behavior: If I were an English teacher, I would correct a student who says “I ain’t got none” in a formal context such as school.
Linguistic prejudice and language ideologies (Example of ‘dominant language ideologies’ topic) Attitude: In my opinion, families can speak Spanish among themselves at home but when in public they should speak English to each other. Behavior: I would not use Spanish in public, just at home or more private places.
Spanish in the United States, bilingualism, and code-switching (Example of ‘phenomena of languages in contact’ topic. Attitude: I don’t like it when people say “parquear” instead of “estacionar.” Behavior: I would support U.S. Spanish to be taught in Spanish classes in the United States.
Language policies and language maintenance/shift (Example of ‘language policies’ and ‘HL maintenance/loss and the benefits of bilingualism’ topic).
Attitude: In my opinion, primary schools should not allow students to use Spanish among themselves.
Behavior: I would speak to Hispanic students at my university about the benefits of keeping Spanish alive in the United States.
Step 3: Consultation with two groups of naive and experienced test-takers and content experts to investigate content validity
To explore the content validity of the questionnaire, an electronic version the original questionnaire was circulated among a group of six Ph.D. students in Spanish linguistics, a group of six bilingual young adults who served as naïve informants, and two questionnaire design experts to ensure clarity and elicit their technical input on the item quality. Their feedback was solicited in the form of short-answer questions through email. In addition, cognitive interviews were conducted with the first two groups (12 participants) to identify areas where interpretation of the questions was ambiguous (Collins, 2003). A revised version of the questionnaire, with ambiguous or confusing wording of items modified, was then distributed to four content experts (as recommended by Henk and McKenna, 2004), namely, tenured professors at U.S. universities with expertise in Spanish sociolinguistics and who had experience teaching about CLA at the graduate level, to review the questionnaire items and confirm that the items were appropriate to measure CLA in the HL context.
They were provided with a word version of the questionnaire and instructed to use track changes to give feedback and make any changes they considered necessary. Specifically, they were asked to answer the following questions about the items in each thematic area: (1) Do you feel the statements accurately address the topics in the thematic areas? (2) Would you add or take out any of the 48 statements? (3) Do you have any suggestions in relation to the thematic areas or the questionnaire? Based on their comments, 16 items were reworded or eliminated to produce a total of 32 items for the pilot phase. Most of the items eliminated were deemed redundant by the experts. Other items were reworded for clarity or accuracy. For example, the item: “I believe people who speak Spanglish should switch to monolingual Spanish” was changed to “I believe people who speak Spanglish should switch to just Spanish or English” to make it less technical. The item “I don’t like it when people codeswitch because it is not correct” was eliminated because it was considered repetitive. The content experts were also asked if the questionnaire was an appropriate tool to measure CLA and they all provided a positive response. The 32 items (16 for attitudes and 16 for behaviors) were distributed in the four themes as follows: eight items on language variation and diversity; eight items on language ideologies and linguistic prejudice; eight items on Spanish in the United States and bilingualism; and eight items on language maintenance.
Step 4: Pilot test with first group of students
The CLA questionnaire was piloted with 33 young Latino adults. These students were completing or had completed a Bachelor of Arts at a university in California. Most of their 33 young Latinos resided in California and had lived there most of the lives. Most of the participants were females (24, or 73%) and there were nine (27%) males. None of them had taken a HL course at the university level, which makes them comparable to the group of young Latino students who will be taking this questionnaire at the beginning of a heritage course.
The item scores were submitted to two statistical methods to evaluate internal consistency reliability: Cronbach’s alpha and corrected item-total correlations. Examination of individual items by alpha if item deleted indicated that deleting eight items increased Cronbach alpha levels. A Cronbach’s alpha of 0.77 was achieved for the scale scores after these deletions. By these means the questionnaire was reduced to 24 items: seven items on language variation and diversity; four items on language ideologies and linguistic prejudice; 8 items on Spanish in the United States and bilingualism; and five items on language maintenance.
Step 5: Full test
The 24-item CLA questionnaire was administered to 305 students enrolled in SHL courses at four large, public universities in the U.S. Southwest. The questionnaires were administered at the beginning of the semester during class time and took approximately 20 minutes to complete. We excluded all responses with missing data on any of the 24 questionnaire items, resulting in a final sample of 301 students. The participants ranged in age from 18 to 66, with a mean age of 21.60. The majority of students were female (196, or 65.1%); 101 (33.5%) were male, and four respondents did not identify their gender as female or male. Most were born in the United States (262, or 88.2%) and only 35 (11.8%) were born in foreign countries. Regarding their dominant language (i.e., Spanish, English, or other), 247 students (84.3%) reported being English dominant, 26 (8.9%) were Spanish dominant, and 20 (6.8%) reported either equal fluency or another dominant language. Respondents were also asked to rate their level of proficiency from 1 (excellent) to 6 (very poor). The mean scores were as follows: speaking Spanish: 2.89 (good); writing Spanish: 3.33 (good); speaking English: 1.75 (excellent–very good); writing English: 1.89 (excellent–very good).
Step 6: Statistical analysis
The 24-item CLA questionnaire results were submitted to statistical analyses to estimate their reliability and investigate construct validity using SPSS for Mac 25.0. Each item was scored individually, and those phrased negatively were reverse coded. Reliability, or how well the items fit together statistically (Haladyna 1999), was tested using Cronbach’s alpha to assess internal consistency. Construct validity was investigated through factor analysis. Factor analysis is a statistical method designed to measure “the number of distinct constructs needed to account for the pattern of correlations among a set of measures” (Fabrigar & Wegener, 2012, p. 3). Each construct is hypothesized based on the content of the items that load on the corresponding factor. Items that do not load on any of the factors are deleted because they do not contribute to the intended test constructs (Munro, 2005). For this study, the extraction method used was a principal component analysis (PCA). In PCA, all the variance of a factor (total variance) is analyzed and is assumed to be perfectly reliable and without error (Bryman & Cramer, 2005).
Construct validity
The 24 items in the scale were included in a PCA with Varimax rotation. Listwise deletion of missing values, eigenvalues, and factor loading scores of .4 were used to sort items into components (Stevens, 2002). The Kaiser–Meyer–Olkin (KMO) and Bartlett’s test was applied to ensure the sample size was adequate to undertake the factor analysis. A KMO value close to 1 is good and 0.6 is acceptable (Field, 2005). The obtained KMO value of .84 is therefore considered very good. We used three criteria to establish the number of components to rotate: Kaiser’s rule of keeping components with eigenvalues greater than 1, the scree plot of eigenvalues, and the interpretability of the component solution (Tabachnick & Fidell, 2001). Based on these criteria, three components were extracted that explained 37.33% of the variance. Two items were deleted because they did not load onto any of the components (items 15 and 24), and one item was dropped because it did not fit conceptually into the factor that it loaded (item 19). In the final run, the three-component model explained 40.5% of the item variance (Factor 1: 24.05%, Factor 2: 9.38%, component 3: 7.09%).
The final version of the questionnaire contained 21 items (12 attitudinal; nine behavioral). This model is interpreted as follows: Component 1: language variation, language prejudice, and English hegemony (nine items); Component 2: bilingualism, Spanglish, and code-switching (eight items); and Component 3: Language maintenance (four items). Table 3 presents the component loadings for the 24 initial questionnaire items and their weights.
Component loadings for PCA.
Internal reliability results
Cronbach’s alpha and corrected item-total correlations were calculated to estimate the internal consistency reliability of each of the three subcomponents of the questionnaire and to identify any items that did not contribute to reliability. Table 4 reports the corrected-item total correlations and Cronbach’s alphas if each item were deleted for factors 1, 2, and 3 for the 21 remaining items. With regard to item-to-total correlation, which tests for scale homogeneity, we followed the rule that items should be discarded if their item-to-total correlation is less than 0.3 and they are not essential for content coverage (Ferketich, 1991). Items 14 and 19 were deleted for not meeting the 0.3 standard, resulting in a Cronbach’s alpha of 0.79 for factor 1, 0.77 for factor 2, and 0.58 for factor 3. For Cronbach’s alpha, 0.7 is the minimum score considered to be reliable for the purpose of this questionnaire (Bland & Altman, 1997). The final questionnaire with 19 items shows an overall scale reliability of 0.71, after averaging the three subscale reliability scores, which is an adequate score (see Appendix A for the final version of the questionnaire).
Results of item analysis.
Step 7: Questionnaire administration to measure CLA before and after CLA instruction
In order to measure the instrument’s ability to detect change in CLA, the final version of the questionnaire with 19 items was utilized in a subsequent study that sought to design, implement, and test a curriculum to teach CLA to a group of 19 fluent SHL learners over a period of 14 weeks using a pretest/posttest design. This group of students can be considered to have the profile identified in the SHL literature as the most typical, and therefore representative of the intended users of the scale (Beaudrie, 2011, 2012; Carreira, 2014, 2017). The change in the students’ levels of CLA, which ranged from 1 to 6, is reported separately for each factor in the scale. Pretest and posttest measures are presented in Table 5.
Pretest and posttest measures on CLA questionnaire ( = 19).
Paired-sample t-tests performed on the pretest and posttest scores showed significant post-instruction changes for factor 1 [t (18) = −2.38, p < .05], factor 2 [t (18) = −2.18, p < .05], and factor 3 [t (18) = −3.25, p < .05]. Further, Cohen’s effect size, which indicates the relative size of the effect of an intervention, for factor 1, (d = .41), factor 2 (d = .43), and factor 3 (d = .61), suggests small to medium-sized effects (Plonsky & Oswald, 2014). As a classroom-based study, smaller effects are expected than in laboratory studies (Plonsky & Oswald, 2014). It is also important to note that this group of learners showed a high level of CLA (4.6 out of 6) in the pretest. In addition, the results indicate that the instrument was appropriate to capture changes in CLA as students’ scores were significantly higher at the end of the instructional intervention.
Discussion
This study is the first study to attempt to design a psychometrically sound tool to measure CLA within the context of HL instruction. The questionnaire respondents during the development phase were 301 students enrolled in SHL courses in four large universities in the U.S. Southwest. After eliminating some items, the results indicate that the questionnaire meets adequate validity and reliability criteria for use in both research and educational contexts. Content validity analysis showed the items were relevant to the construct under investigation, and the internal reliability results were adequate. The PCA assessed the theoretical construct of the questionnaire and confirmed that three components were addressed in the scale. Principal component analyses resulted in the deletion of additional items that did not fit well into one of the subscales. The resulting average Cronbach’s alpha across the three subscales of 0.71 is adequate, especially for a newly developed questionnaire with a small number of items, and is acceptable for the intended low-stakes purpose.
The final questionnaire contains 12 attitude items and seven behavior items relevant to three themes: (1) language variation, linguistic prejudice, and English and hegemonic language ideologies; (2) bilingualism, Spanglish, and code-switching; and (3) language maintenance. Originally, we designed items in four thematic areas but areas one (language variation and linguistic diversity) and two (English hegemony, language ideologies, and linguistic prejudice) were related conceptually, which resulted in their unification after the statistical analysis.
These findings suggest that becoming critically language aware entails learners becoming cognizant of the naturalness of language variation and its loading of social, political, and economic power structures, while accepting the intrinsic value of their own bilingual varieties and all others. From this awareness, past researchers and pedagogues have discussed the importance of student agency in recognizing inequity and, more importantly, resisting dominant language ideologies by making their own decisions about language use and bilingualism (Clark, Fairclough, Ivanič and Martin-Jones, 1990, 1991; Clark & Ivanič, 1997; Holguín Mendoza. 2017; Leeman, 2018; Leeman & Serafini, 2016; Martínez, 2003; Parra, 2016. Finally, becoming critically aware involves positive behaviors toward maintenance and advocacy toward HLs. However, some of the items in this last thematic area did not endure the rigorous validation process and, therefore, the final questionnaire does not contain an appropriate number of items in this area, which results in an inadequate reliability score. In the future, researchers should continue to design a larger set of items for this section, possibly expanding the topics originally included to add items that are focused on issues of social justice and equity related to language.
In general, behavioral items did not perform as well as attitudinal items, so future research needs to continue to study the fluid structural and internal relationships between attitudes and behavior in the CLA construct. Future research should also contribute to furthering the evaluation of the tool, as well as explore the inclusion of additional possible topics to expand the questionnaire. Additionally, alternative ways to measure CLA should continue to be developed and evaluated in order to test successfully this important construct in HL research and education.
Conclusion
The results indicate that the CLA questionnaire needs further revision in order to meet the construct validity and reliability criteria investigated in this study, and that it possibly could be used to assess CLA in Spanish heritage learners enrolled in a university-level Spanish heritage course. Until now, the field of HL education has lacked studies on the effectiveness of CLA instructional programs, and it is our hope that this questionnaire eventually makes a significant contribution to begin to address this gap. However, one limitation of the study is that the sample population was limited to students enrolled in university-level SHL courses in the Southwest. Future studies should evaluate the generalizability of this questionnaire by expanding its use to other settings, other HLs, and other sample populations. This will also allow further measurement of the effectiveness of the instrument in capturing and comparing potential varying levels of CLA in diverse groups, who may be at different stages of CLA. A second limitation was that the Likert-scale used in this research had only 6 points. Research in educational measurement has shown that larger scales, with 7 to 11 points, may better capture questionnaire-takers’ underlying response variance on individual items and increase subscale reliability. Further revisions in such directions may prove useful.
Footnotes
Appendix A
Final statements used in the Likert-scale questionnaire annotated for theme and topic (summarized in Table 2). All items are scored 1–6, with 6 indicating strongly agree/very likely or, in case of “reversed scoring,” (indicated with *) strongly agree/very likely is scored as 1 and strongly disagree/very unlikely is 6.
Acknowledgements
We would like to express our great appreciation to all the Spanish heritage students, Spanish linguistic graduate students, and sociolinguistic experts who participated in several steps of the design, piloting, and testing of the instrument in this study. We are particularly grateful to the editors and reviewers of this journal for their valuable and insightful feedback on earlier versions of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
