The effect of cognitive vocabulary learning approaches on Chinese learners’ compound word attainment,retention,and learning motivation

Abstract

Research has supported the benefit of cognitive vocabulary learning approaches, such as dual-coding (i.e. visual-verbal approach), on Chinese language learners’ vocabulary attainment. However, few studies have systematically examined how to maximize Chinese word retention. Therefore, the goal of this study was to investigate the effects of two types of visual-verbal approaches, i.e. pictographic-verbal coding approach and pictorial-verbal coding approach, on Chinese language learners’ word attainment. The visual support in our proposed pictographic-verbal coding approach uses pictograms, while pictures are used as the visual support in the pictorial-verbal coding approach. Further, we tested whether repetition would impact learners’ Chinese word retention. One-hundred fourth and fifth grade English-speaking Chinese language learners participated in the study. A series of two-way analysis of variance with repeated measures was conducted. Results showed that the pictographic-verbal coding approach assisted learners in acquiring as well as retrieving more Chinese words compared to the pictorial-verbal coding approach. We also found that learners taught by the pictographic-verbal coding approach demonstrated higher learning motivation. Instructional implications were discussed.

Keywords

Chinese as a foreign/second language learning motivation vocabulary attainment vocabulary learning approach vocabulary retention

I Introduction

In the USA, more people are learning Chinese as a foreign/second Language than ever before. According to Modern Language Association (2015), foreign language enrollment in Chinese increased by 19% between 2006 and 2013, while enrollment in all other languages decreased. Among the foreign languages (e.g. Spanish, French, German, Italian, and Chinese), the Chinese language is considered as the most difficult language to acquire (O’Connell & Norwood, 2007); because English and other European languages use an alphabetic writing system whereas the Chinese language uses a logographic writing system that will take English-speaking students a longer period of time to acquire. According to the Steering Committee for the Test of Proficiency-Huayu (2016) that developed the Test of Chinese as a Foreign Language to measure Chinese language learners’ levels of Chinese proficiency, to reach the beginner level (Level 1), learners are expected to have a vocabulary size of at least 500 Chinese words and spend 240 to 480 hours learning Chinese in non-Chinese speaking settings. To reach the most advanced level (Level 6), learners are required to have a vocabulary size of at least 8,000 Chinese words with more than 3,840 hours spent learning Chinese.

A limited vocabulary size hinders successful development of learners’ language ability (Bromley, 2007) and comprehension skills (Chou, 2011). For Chinese language learners whose native language is English, Chinese character recognition is always considered one of their biggest challenges due to the irregular mapping between grapheme and phoneme correspondences in Chinese characters (Shen & Xu, 2015; Xu & Padilla, 2013). These learners, with limited vocabulary knowledge at beginning level, have been reported to feel depressed as they are not able to comprehend what they read (Li & Tong, 2018). This negative perspective towards Chinese language learning is detrimental to their learning performance (Tseng & Schmitt, 2008). Therefore, the purpose of this study was to discover the effects of two cognitive vocabulary learning approaches, i.e. the pictographic-verbal coding approach and the pictorial-verbal coding approach, on novice Chinese language learners’ Chinese word attainment and retention, as well as on their learning motivation. These two approaches, developed based on dual-coding theory, both provide visual and verbal support. However, the visual support which our proposed pictographic-verbal coding approach utilizes is pictograms, while the visual support in the pictorial-verbal coding approach is through pictures related to the meaning of target words. An example of Chinese pictograms can be found in literature review.

II Theoretical framework

The framework of this study is derived from dual-coding theory (Paivio, 1971) and information processing theory (Lachman, Lachman, & Butterfield, 1979). Dual-coding theory focuses on the concept of mental representations; it highlights the importance of visual and verbal representations on vocabulary learning (Paivio, 1971, 1986). Cognitive psychologists such as Lachman, Lachman, and Butterfield (1979) defined information processing theory as individuals’ memory activity which will be initiated when learning stimuli are provided. When the stimuli along with new knowledge become meaningful to the individuals, this new knowledge will be transferred from sensory memory to short-term memory. To help individuals transfer the knowledge from short-term memory to long-term memory, some activities such as elaborative rehearsal need to be practiced. According to Ebbinghaus (1964), forgetting behavior occurs when the action of learning stops; more specifically, the amount of knowledge that can be recalled decreases as time increases. For example, twenty minutes after learning, 58.2% of learned knowledge is what learners can typically recall; after an hour, 44.2% is likely to be recalled; after eight to nine hours, 35.8% of knowledge can be recalled; after one day, 33.7% can be recalled; after two days, 27.8% can be recalled; after six days, 25.4% can be recalled. Following Ebbinghaus’ interpretation of memory, we postulate that the knowledge which learners can retrieve within 24 hours after learning is considered the short-term memory; the knowledge they can retrieve after 24 hours is considered the long-term memory.

III Literature review

1 The nature of Chinese character creation/formation

Chinese is a logographic language as most Chinese characters are derived from visual symbols (Liu, Perfetti, & Wang, 2006). Although Chinese is termed as logographic orthography, Chinese characters present both semantics and phonology, and are considered to be morpho-phonological (Leong, 1997), logographic-phonetic (DeFrancis, 1989) or morpheme-syllable (Hoosain, 1991). Out of all Chinese words, 80% to 90% are phono-semantic compounds (Tong & McBride-Chang, 2010); less than 1% are phonetic loan characters and derivative cognates (Yeung et al., 2012). Even though most Chinese words are phono-semantic compounds, they still contain a feature of pictograms. For example, 桃 ‘tao2’ (peach) is a phono-semantic word that originated from the appearance of tree . This word contains two basic words: 木 ‘mu4’ (tree/wood) as a semantic hint and 兆 ‘zhao4’ (omen) as a phonological hint. The formation of the word ‘桃’ () peach is: 木 () tree + 兆 () omen . Both and are pictograms. 木 is the semantic radical of 桃, with a meaningful association of ‘peaches are growing on trees’. With semantic radical knowledge, native Chinese speakers are able to quickly and accurately guess the meaning of these characters (Shu & Anderson, 1997).

2 Learning Chinese as a native language

Native Chinese-speaking students usually apply their radical knowledge and the basic rules of Chinese character formation to assist with Chinese character recognition (Shu & Anderson, 1997; Tong & Yip, 2015). At the early stage of learning Chinese characters, these students are taught to develop their radical awareness (National Academy for Educational Research, 2018). Such awareness of radical forms is critical for young native Chinese-speaking students, because it can facilitate predicting unique variances in Chinese character recognition and reduce students’ cognitive load (Ho, Ng, & Ng, 2003; McBride-Chang, Chow, Zhong, Burgess, & Hayward, 2005). In addition, more radical knowledge can assist native Chinese-speaking students in acquiring Chinese words, which will later influence their reading performance (Liu, 2009). When developing these students’ radical knowledge to help memorize Chinese characters, adequate pictures and semantic support are usually provided (Hsu, 2012). However, to improve the students’ memorization of Chinese words, teachers are encouraged to use relevant and meaningful pictures associated with the words (Hsu, 2012) as well as contextual information (Chen, 1998; Ou & Wang, 2004). Ou and Wang (2004) argued that contextual information can lead native Chinese-speaking students to a higher level of familiarity with learned words. In the process of Chinese word learning, it was found that repetitive practices can enhance the students’ memorization of learned words; with multiple practices, the learned words can be transferred from short-term to long-term memory (Chou, 2012).

3 Recommended practice for teaching Chinese vocabulary words to Chinese language learners

Studies have discussed the beneficial effects of radical knowledge on Chinese character recognition for Chinese language learners whose native language is not Chinese (e.g. Wang, Perfetti, & Liu, 2003; Tong & Yip, 2005). Radicals act as codes to help Chinese language learners encode and decode new characters (Tong & Yip, 2015). According to Wang et al. (2003) and Liu et al. (2006), radicals contain meaningful pieces of semantic and pictographic information. To deliver a pedagogical practice, Chinese language teachers are encouraged to first introduce a radical (e.g. 水 or氵 ‘water’), followed by new characters containing this radical (e.g. 海 ‘sea’ and 湖 ‘lake’). In this way, learners can better understand and memorize these characters. Even though a sufficient body of work has reported the benefit of using radicals in Chinese character recognition (Wong, 2017; Xu, Chang & Perfetti, 2014), little research has provided solutions on how radicals can be applied to assist Chinese language learners in understanding and memorizing Chinese compound words.

Compounding is a feature of Chinese morphology (Ceccagno & Basciano, 2007). According to Chen, Hao, Geva, Zhu, and Shu (2009), ‘more than 75% of words in Chinese are compound words’ (p. 616). In the Chinese language, compound words have a very direct meaning. For example, 海灘 ‘hai3 tan1’ is beach and 海洋 ‘hai3 yang2’ is ocean. Therefore, learning compound words is a critical element in Chinese as a foreign/second language learning. Studies (e.g. Sham, 2002; Shen, 2010) suggested using a visual-verbal approach, which was derived from the dual-coding theory introduced by Paivio (1971), as it pertains to Chinese vocabulary word learning. Shen (2010) found that Chinese language learners memorized more words when instructed with a visual-verbal coding approach. Shen (2010) further explained that visual and verbal representations act as two codes, which provide a better chance for memorizing new words. This visual-verbal coding approach has been integrated into technology in Chinese characters learning (e.g. Chuang & Ku, 2011; Low, Hew, & Wong, 2014). Chuang and Ku (2011) and Low et al. (2014) found that the visual-verbal coding approach significantly impacted students’ attitudes in L2 Chinese word learning. Wang and Blackwell (2015) and Li and Tong (2018) further examined the effects of the visual-verbal coding approach presented by multimedia on Chinese vocabulary learning. This approach was detected to have a beneficial effect on Chinese word attainment. Although research has supported the benefits of visual-verbal coding approach on Chinese language learners’ vocabulary attainment, few studies have discussed the alignment of the visual-verbal coding approach with the nature of Chinese word formation (i.e. radicals and pictograms), as well as its effect on word attainment and retention among novice learners in elementary schools where Chinese is taught as a foreign language. Given the nature of Chinese word formation, such alignment is critical in teaching Chinese as a foreign/second language.

Therefore, in this study, we proposed a pictographic-verbal coding approach and its associated teaching procedure. In this type of visual-verbal approach, pictograms are used as visual support, and English translation of the target Chinese words is used as verbal support. The teaching procedure of the pictographic-verbal coding approach contains four steps: (1) a radical (e.g. 水 and氵 ‘water’) with its pictogram (e.g. ); (2) a single character (e.g. 海 ‘sea’) containing the radical with its pictogram (e.g. ); (3) a compound word containing the character (e.g. 海洋 ‘ocean’) with its pictograms (e.g. ); and (4) a story or a context with all compound words. A more detailed explanation of this teaching procedure is provided in the method section. With the use of the pictographic-verbal coding approach, we hypothesize that Chinese language learners will gain insight into target radical forms, and thus more effectively learn characters with corresponding radical forms. More importantly, with these radical forms and characters, learners will more easily learn and be able to identify the target compound words. In addition, when learners hear a story containing the target compound words that is the contextual information, these target compound words will be meaningfully stored in their short-term memory. Additionally, we compared the proposed approach with the pictorial-verbal coding approach on Chinese language learners’ Chinese word attainment.

Further, regarding an optimal way for students to transfer their learned vocabulary from short-term memory to long-term memory, we postulate that through repetitive practices of the pictographic-verbal coding approach on vocabulary review, Chinese language learners should have a deeper understanding of learned words, which may benefit their word retention, or long-term memory. According to Lachman, Lachman, and Butterfield’s information processing theory (1979), rehearsal aids in the retention of knowledge from short-term memory to long-term memory. This has been supported by Karpicke and Roediger (2008) that memory rehearsal acts as a critical role in foreign language word retention. However, we found a lack of research regarding repetitive practices of vocabulary teaching/learning approaches to review learned Chinese compound words. Therefore, in our study, we attempted to discover if different phases of repetition practice would influence learners’ word retention.

4 Quality learning materials and word learning motivation

By understanding this recommended practice of teaching, quality cognitive learning materials with a cognitive process demand can be developed. Cognitive processing requires elaboration towards new information (Takač, 2008). In the process of elaboration, students are required to use mental simulation to enhance new information acquisition, understanding, interpretation, and memorization (Nyberg et al., 2001). Further, visual and semantic representations can enhance students’ mental simulation, which later positively impacts their word learning because these representations help decode Chinese words in a positive way (Ju & Jackson, 1995; Shen, 2010). Given these results specific to the importance of Chinese word creation/formation and practices of teaching Chinese words, we believe that quality learning materials should include both visual (i.e. pictograms) and semantic support.

According to Tseng and Schmitt (2008), quality learning materials can decrease students’ learning anxiety and enhance their involvement in word learning. Similarly, Hui, Hu, Clark, Tam, and Milton (2008) mentioned that Chinese language learners’ motivation can be elevated through quality learning materials containing visual and semantic support, implying that learning through the pictographic-verbal coding approach and its associated quality learning materials will enhance novice Chinese language learners’ motivation for learning Chinese words.

Therefore, in this study we attempt to address the following three research questions:

Is there a difference between the pictographic-verbal coding approach (a treatment condition) and the pictorial-verbal coding approach (a control condition) as measured by novice Chinese language learners’ Chinese word attainment?

Do different phases of repetition practice influence learners’ word attainment?

Is there a difference between the pictographic-verbal coding approach and the pictorial-verbal coding approach as measured by learners’ motivation?

IV Methods

1 Participants

Through a convenience sampling approach, we recruited English-speaking fourth (n = 50) and fifth (n = 50) graders from a local elementary school in Central Texas, USA. These students were enrolled in a Chinese enrichment program and their participation was completely voluntary. None of them had any exposure to Chinese language or culture prior to participating in this study. These students in their respective grades were randomly assigned to either a treatment group or a control group. The treatment group (25 fourth and 25 fifth graders) used the pictographic-verbal coding approach, and the control group (25 fourth and 25 fifth graders) used the pictorial-verbal coding approach. Among the total of 100 participating students, 58 were female and 42 were male.

The Chinese enrichment program has been in place at the elementary school for eight years with strong support and commitment from school administrators and community. The purpose of this enrichment program is to spark English-speaking students’ awareness of, and interest in, Chinese language and culture. The curriculum for the program was developed by the researchers in this study. The curriculum was based on World-Readiness Standards for Learning Languages (National Standards Collaborative Board, 2015) and was aligned with Texas framework for languages other than English (Texas Education Agency, 1997). This program is open to fourth and fifth grade students only. The Chinese teacher in this study, who was also one of the researchers and who holds a doctorate in educational psychology with an emphasis in bilingual education, is a native speaker of Chinese with six years of Chinese language teaching experience and over ten years of experience teaching English as a second language.

2 Target Chinese radicals and words

In this study, 80 target Chinese compound words (nouns, verbs, and adjectives) were chosen from the probe test for each grade, with a total of 160 target words for both grades. The criteria for selecting these target words were as follows: familiarity with the corresponding English translation, and unfamiliarity with the Chinese words. A group of 20 words was selected by topic for each grade: fruit, weather, lake/ocean, and pet animal for the fourth graders; fruit/vegetable, transportation, mountain/forest, and wild animal for the fifth graders. An example of a partial word list for fourth and fifth graders is provided in Appendix 1. The fourth graders were introduced to a total of 14 radicals; the total of radicals for the fifth graders was 17. To ensure that these radicals and compound words had never been introduced to these students before, we gave them a probe test using Vocabulary Knowledge Scale (Paribakht & Wesche, 1992). For each word item, students were asked to choose the best one out of five descriptive answer categories. For example, Category One is ‘I don’t remember having seen this word before.’ Category Five is ‘I can use this word in a sentence: ________________.’ Their responses were rated by two Chinese graduate students majoring in education from a university located within proximity to the research setting. These two raters coded each word based on frequency in categories one or two (unfamiliarity with the Chinese word), and in categories four or five (familiarity with the Chinese word). Further, the raters also evaluated students’ responses in categories three, four or five to identify if the responses clearly showed the students’ understanding of the target words. The inter-rater reliability (kappa) was reached at .96.

3 Procedure of learning target Chinese words

Vocabulary instruction time was set to 40 minutes for each approach (i.e. pictographic-verbal and pictorial-verbal coding approaches). In both approaches, traditional Chinese words and Pinyin, a phonological coding system to facilitate pronunciation, were used in the instruction. During each 40-minute session, 5 minutes were spent on introducing what the students were going to learn and needed to do; 25 minutes were spent on teaching radical forms and vocabulary; and 10 minutes were spent on reviewing, with contextual information provided. Both groups were taught by the same teacher. A detailed description of the two instructional approaches follows:

a Procedure for the pictographic-verbal coding approach

[During a 5-minute introduction]

Step 1: The teacher introduces the students to what topic of Chinese words they are going to learn.

Step 2: The teacher gives five blank sheets of paper to the students for taking notes.

[During a 25-minute instruction]

Step 1: A radical form (e.g. 木 ‘wood/tree’) is introduced with its pictogram (e.g. ) (see Appendix 2).

Step 2: A character (e.g. 果 ‘fruit’) which contains the radical of 木 (wood/tree) is introduced with its pictogram (e.g. ) (see Appendix 2).

Step 3: Compound words (e.g. 水果 ‘fruit’) which contain the character are introduced with their pictograms (e.g. ) (see Appendix 2).

[During a 10-minute review]

Step 1: A story or its context (e.g. 市場 ‘market’) coupled with the taught words along with their pictograms is introduced. An example of a story is provided in Appendix 3. The word 市場 ‘market’ is not included on the written vocabulary test.

Step 2: The teacher reads the story.

b Procedure for the pictorial-verbal coding approach

[During a 5-minute introduction]

Step 1: The teacher introduces the students to what topic of Chinese words they are going to learn.

Step 2: The teacher gives five blank sheets of paper to the students for taking notes.

[During a 25-minute instruction]

Step 1: A radical (e.g. 木 ‘wood/tree’) is introduced with its picture.

Step 2: A character (e.g. 果 ‘fruit’) is introduced with its picture (e.g. the picture of fruit) (see Appendix 4).

Step 3: Compound words and their pictures are introduced (e.g. 水果 ‘fruit’ and the picture of fruit) (see Appendix 4).

[During a 10-minute review]

Step 1: A story or its context (e.g. 市場 ‘market’) coupled with the taught words along with their pictograms is introduced. An example of a story is provided in Appendix 3.

Step 2: The teacher reads the story.

4 Repetitive practice of the approaches to review learned target words

To help strengthen students’ retention, learned words were first introduced and then reviewed for different lengths of time in each of four sessions (see Table 1). In the first session, the participating fourth and fifth graders were presented new Chinese words in Week 1, and Weeks 2 and 3 (two-week repetition) were for reviewing these words. Fourth graders learned words on the topic of fruit, and words on the topics of fruit and vegetable were presented to the fifth graders.

Table 1.

Learning and testing schedules.

Session	Week introduced	Week of pre-test	Week of immediatepost-test	Week of one-week delayed test	Durations of repetition	Week of final post-test
Session 1	Week 1	Week 1	Week 1	Week 2	Weeks 2 to 3	Week 4
Session 2	Week 4	Week 4	Week 4	Week 5	Weeks 5 to 7	Week 8
Session 3	Week 8	Week 8	Week 8	Week 9	Weeks 9 to 12	Week 13
Session 4	Week 13	Week 13	Week 13	Week 14	Weeks 14 to 18	Week 19

In the second session, Week 4 was for Chinese word learning, and Weeks 5, 6, and 7 (three-week repetition) were for word review. Fourth graders learned words on the topics of weather and color, and words on the topic of transportation were presented to the fifth graders. In the third session, Week 8 was for Chinese word learning, and Weeks 9, 10, 11, and 12 (four-week repetition) were for word review. Fourth graders learned words on the topics of lake and ocean, and words on the topics of mountain and forest were presented to the fifth graders. In the fourth session, Week 13 was for Chinese word learning, and Weeks 14, 15, 16, 17, and 18 (five-week repetition) were for word review. Fourth graders learned words on the topic of pet animal, and words on the topic of wild animal for the fifth graders.

A pre-test was given to the students before each topic was introduced (i.e. beginning of Weeks 1, 4, 8 and 13); an immediate vocabulary post-test was administered to the students right after the target words were taught (i.e. end of Weeks 1, 4, 8 and 13); a one-week delayed test was then given to the students (i.e. end of Weeks 2, 5, 9, and 14); and a final post-test was conducted (i.e. beginning of Weeks 4, 8, 13 and 19) before new words were taught. To prevent students from preparing for all post-tests, they were requested to keep their learning materials and notes in the classroom.

5 Instrument

For the purpose of this study, we administered two instruments: vocabulary test and learning motivation survey. The written vocabulary test was developed by the researchers in this study in a multiple-choice format that contained 20 items for measuring students’ recognition and memorization of the 20 target words in each session.

To measure students’ learning motivation, we adapted the Instructional Materials Motivation Survey (IMMS; Keller, 1993), which was developed to ‘measure learners’ reactions to self-directed instructional materials’ (Keller, 2009, p. 277). The original IMMS contains 36 items to address four types of motivation (i.e. attention, relevance, confidence, and satisfaction). Each item includes five choices (not true; slightly true; moderately true; mostly true; and completely true). For example, under the confidence subscale, an item reads, ‘when I first looked at this lesson, I had the impression that it would be easy for me.’ In the adaptation, some items were re-worded so as to be more relevant to this study, which also facilitated participants’ understanding. For example, under the confidence subscale, the original item described above was revised to ‘when I first looked at the topic of today and the materials, I had a feeling that it would be easy for me to learn.’ The students scored each item ranging from 0 for not true to 4 for completely true. The reliability coefficients (Cronbach α) of the four subscales of the adapted IMMS based on the sample of this study are .97 (attention; 12 items), .95 (relevance; 9 items), .95 (confidence; 9 items), and .96 (satisfaction; 6 items), with an overall reliability of .96.

To better understand if students in each group used their designated approach to retrieve words on tests, we adapted a section of Cognitive Strategy Use from the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & De Groot, 1990). The original section contains 13 items on a 7-point Likert scale. The adapted version of this section consisted of seven items (α = 0.97). Both the adapted IMMS and the adapted Cognitive Strategy Use used the same 5-point Likert scale (0 as not true to 4 as completely true).

The vocabulary test was individually administered at both the beginning and end of each 40-minute lesson. The adapted IMMS was individually administered on Week 1. The adapted Cognitive Strategy Use was individually administered right after every immediate post-test, one-week delayed test and final post-test.

V Results

To address research question 1, a 2 (approaches) X 3 (pre-test, immediate post-test, one-week delayed tests) repeated measures analysis of variance (RM-ANOVA) was conducted. To address research question 2, a RM-ANOVA was conducted with approach as a between-participants variable and duration of repetition (2, 3, 4 and 5 weeks) as a within-participants variable. To address research question 3, an independent-sample t-test was conducted.

Also, because all students were given four pre-tests, immediate post-tests, and one-week delayed tests, a series of one-way ANOVAs was conducted to examine whether the two approaches were performed differently within groups. In the treatment group (the pictographic-verbal coding approach), for pre-tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.455, p = .714; for immediate post-tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.258, p = .856; for one-week delayed tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.256, p = .857. In the control group (the pictorial-verbal coding approach), for pre-tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.273, p = .845; for immediate post-tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.339, p = .797; for one-week delayed tests, the one-way ANOVA yielded a non-significant effect, F(3,196) = 0.176, p = .913.

Research question 1

The main effect for approach (pictographic-verbal coding vs. pictorial-verbal coding) was significant, F(1, 98) = 83.646, p < 0.001, η² = 0.46. This result demonstrated a difference between the two approaches. The main effect for time of testing (pre-test, immediate post-test, vs. one-week delayed test) was significant, F(2, 196) = 874.543, p < 0.001, η² = 0.90. A statistically significant interaction of approach with time of testing was also found, F(2, 196) = 79.006, p < 0.001, η² = 0.45. Post-hoc comparisons revealed that for between-participant effects on pre-tests, the performance of students who used the pictographic-verbal coding approach was similar to that of students who used the pictorial-verbal coding approach; the main effect of approach was not significant, F(1, 98) = 0.059, p = 0.81. For between-participant effects on immediate post-tests, students who used the pictographic-verbal coding approach performed better than those who used the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 102.018, p < 0.001, η² = .51. For between-participant effects in one-week delayed tests, students who used the pictographic-verbal coding approach performed better than those who used the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 52.405, p < 0.001, η² = .348. Figure 1 shows mean scores of each test for each approach. The scores refer to the number of correct words. For within-participant effects in the pictographic-verbal coding approach, students’ vocabulary scores on immediate post-test and one-week delayed test were significantly higher than their scores on pre-test; their scores on immediate post-test were higher than their scores on one-week delayed test. For within-participant effects in the pictorial-verbal coding approach, students’ vocabulary scores on immediate post-test and one-week delayed test were significantly higher than their scores on pre-test; their scores on immediate post-test were higher than their scores on one-week delayed test.

Figure 1.

Mean comparison between two approaches on pre-tests, immediate post-tests and one-week delayed tests.

Research question 2

The main effect for approach (pictographic-verbal coding vs. pictorial-verbal coding) was significant, F(1, 98) = 261.895, p < 0.001, η² = 0.728. This result demonstrated a difference between the two approaches. The main effect for duration of repetition (2, 3, 4, and 5 weeks) was significant, F(3, 294) = 72.825, p < 0.001, η² = 0.426. An interaction of approach with duration of repetition was not significant, F(3, 294) = 0.895, p = 0.432. Post-hoc comparisons revealed that for between-participant effects of a two-week repetition on Chinese vocabulary recall, students who used the pictographic-verbal coding approach performed better than those who used the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 119.220, p < 0.001, η² = 0.549. For between-participant effects of a three-week repetition on Chinese vocabulary recall, students who used the pictographic-verbal coding approach performed better than those who used the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 214.306, p < 0.001, η² = 0.686. For between-participant effects of a four-week repetition on Chinese vocabulary recall, the pictographic-verbal coding approach was more effective than the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 153.928, p < 0.001, η² = 0.611. For between-participant effects of a five-week repetition on Chinese vocabulary recall, the pictographic-verbal coding approach was more effective than the pictorial-verbal coding approach; the main effect of approach was significant, F(1, 98) = 143.266, p < 0.001, η² = 0.594. Figure 2 shows mean scores of each repetition for each approach. The scores refer to the number of correct words. For within-participant effects in the pictographic-verbal coding approach, students’ vocabulary scores on the three-week, four-week, and five-week repetitions were significantly higher than their scores on the two-week repetition; their scores on the five-week repetition were higher than their scores on the three-week and four-week repetitions. However, there was no significant difference between the three-week repetition and the four-week repetition. For within-participant effects in the pictorial-verbal coding approach, students’ vocabulary scores on the four-week and five-week repetitions were significantly higher than their scores on the two-week and three-week repetitions; the scores on the five-week repetition were higher than their scores on the four-week repetition. However, there was no significant difference between the two-week repetition and the three-week repetition.

Figure 2.

Mean comparison between two approaches on four durations of repetition.

Research question 3

The average scores of the four factors (attention, confidence, relevance, and satisfaction) on learning motivation are significantly higher for the pictographic-verbal coding approach (M = 3.57) compared to the average scores for the pictorial-verbal coding approach (M = 2.94), t(98) = 6.236, p < 0.001, η² = 0.28) (see Figure 3). For the four factors, students using the pictographic-verbal coding approach scored significantly higher than those using the pictorial-verbal coding approach, t(98) = 5.862, p < 0.001, η² = 0.26, t(98) = 6.722, p < 0.001, η² = 0.32, t(98) = 4.069, p < 0.001, η² = 0.16, and t(98) = 7.186, p < 0.001, η² = 0.35. To compare the two groups of students’ motivation (including attention, confidence, relevance, and satisfaction), Holm-Bonferroni Procedure (Holm, 1979) was adopted. The p value of the four factors was lower than 0.0125, 0.017, 0.025, and 0.05 respectively. Therefore, students using the pictographic-verbal coding approach had higher motivation in learning Chinese vocabulary words compared to students using the pictorial-verbal coding approach.

Figure 3.

Mean comparison between two approaches on the four motivation factors.

Regarding the results of Cognitive Strategy Use, on the four immediate post-tests (Weeks 1, 4, 8 and 13), no difference was found between the pictographic-verbal coding approach and the pictorial-verbal coding approach (see Figure 4), t(98) = 0.082, p = 0.935, t(98) = 0.211, p = 0.833, t(98) = 0.254, p = 0.8, and t(98) = 0.189, p = 0.85. On the four one-week delayed tests (Weeks 2, 5, 9 and 14), no difference was found between the pictographic-verbal coding approach and the pictorial-verbal coding approach (see Figure 5), t(98) = 0.374, p = 0.709, t(98) = 0.521, p = 0.604, t(98) = 0.112, p = 0.911, and t(98) = 0.56, p = 0.577. On the four final post-tests (Weeks 4, 8, 13 and 19), no difference was found between the pictographic-verbal coding approach and the pictorial-verbal coding approach (see Figure 6), t(98) = 0.312, p = 0.756, t(98) = 0.025, p = 0.98, t(98) = 0.083, p = 0.934, and t(98) = 0.793, p = 0.43.

Figure 4.

Mean comparison between two approaches on the four immediate post-tests of cognitive strategy use.

Figure 5.

Mean comparison between two approaches on the four one-week delayed tests of cognitive strategy use.

Figure 6.

Mean comparison between two approaches on the four final post-tests of cognitive strategy use.

To better understand how the four factors (attention, confidence, relevance, and satisfaction) impact students’ perceptions towards the pictographic-verbal coding approach, the following survey items given to students during Week 1 were singled out.

a Attention

94% of the students reported that cognitive vocabulary learning materials were relevant to their needs for vocabulary learning. 96% of the students enjoyed using cognitive vocabulary learning material to learn Chinese words because pictograms are provided. 12% of the students reported that there was too much information on the learning materials, which distracted their attention.

b Confidence

89% of the students reported that as they worked on the lesson, they were confident they could memorize over 75% of learned words. 92% of the students reported that using these learning materials was less difficult for learning new words than they had expected. 12% of the students reported that the materials had too much information and that it was hard to pick out and remember the important points.

c Relevance

96% of the students reported that along with learned words they could relate the story or context to things they have seen, experienced, or thought about in their lives. 91% of the students said that the materials would be useful for them to memorize words. 11% of the students reported that learning words through radical forms and pictograms was more difficult than they had expected.

d Satisfaction

93% of the students reported that it was a pleasure to work on these cognitive vocabulary learning materials. 94% of the students enjoyed using cognitive vocabulary learning materials to learn Chinese words and hoped to use the same type of materials in their future Chinese vocabulary learning. 94 % of the students reported it felt good to successfully complete this lesson.

e Cognitive strategy use

92% of the students reported that on the test they tried to recall what the teacher had said about each target word in class, so they could answer the questions correctly. 93% of the students reported that on the test they used the way they reviewed in class to answer each question. 93% of the students reported that reviews led by the teacher helped them to memorize and recall words.

VI Discussion

The purpose of this study was to discover the effects of two types of visual-verbal cognitive vocabulary learning approaches (pictographic-verbal coding vs. pictorial-verbal coding) on Chinese word attainment and retention for novice Chinese language learners, as well as to measure the effects of these approaches on their learning motivation. As previously stated, the key difference between the two approaches is that the pictographic-verbal coding approach uses pictograms and the pictorial-verbal coding approach uses pictures related to the meaning of target words. Below is the discussion of our findings.

This study highlighted the importance of visual and semantic aspects in learning Chinese characters. Results suggested that the use of pictures along with semantic meaning facilitated Chinese language learners’ memorization of Chinese characters in the early stages of learning the language. This finding is consistent with Shen’s (2010) research that visual (pictorial) and semantic support can assist novice Chinese language learners in differentiating and identifying new Chinese words. Results further revealed that to enhance these learners’ Chinese word memorization, cognitive processing of information is required. This process provides novice Chinese language learners with multiple ways to seek important information for encoding and decoding words based on the characteristics of the words. However, compared with the pictorial-verbal coding approach, the pictographic-verbal coding approach was found to be more effective in providing students with a systematic and meaningful way to process new knowledge and store it in their short-term memory. In addition, students can retrieve words more easily from their short-term memory by using the systematic learning routine of the pictographic-verbal coding approach. Our finding that a systematic learning routine can effectively enhance students’ learning is supported by Karpicke and Aue (2015). Our finding further suggested that to help novice Chinese language learners retrieve more words effectively, the systematic learning routine should closely adhere to the nature of Chinese word creation and the way native Chinese-speaking students learn Chinese words. The aforementioned findings are indicative of two pedagogical practices for Chinese word instruction. Chinese language teachers should introduce how target Chinese words are created, and explain why these words are formed in that way. Then novice Chinese language learners can create a mental linkage, which will have a positive impact on their receptive word recall.

In addition, the results revealed that quality teaching/learning materials matter in the pictographic-verbal coding approach. We recommend that to make quality Chinese vocabulary teaching/learning materials, the nature of Chinese word creation must be considered. Given that Chinese is an orthographic language and most Chinese words are derived from visual symbols (Liu et al., 2006), Chinese vocabulary teaching/learning materials should include more relevant graphics (i.e. pictograms) to help novice Chinese language learners connect the meaning of words. Our findings also showed that learners rely on pictograms and the historical formation of the target words when they try to recall the meaning of those words. We believe that the imagery values of words were increased when learners were presented with pictograms and the history of word formation, which facilitated their word recall. This study presented evidence that learning materials with relevant visual clues and opportunities to explore useful and meaningful information (e.g. a historical or cultural context of word formation and a story or context related to students’ living backgrounds) assisted novice Chinese language learners in recognition, differentiation, and memorization of Chinese words. Because the learning materials contained a story or context relevant to the learners’ own background, it enabled them to feel connected with the materials, which in turn increased their focus on the learning task. When familiar learning contexts are integrated into students’ learning, they will pay more attention to the task, which in turn will positively impact their learning performance (Bergman et al., 2015). Furthermore, we also believe that because students focus more of their attention on word learning, the beneficial effects of the pictographic-verbal coding approach and its associated learning materials on word recall are identified.

Another significant finding of our study is the critical role of repetition on the effects of the pictographic-verbal coding approach on word retention. Results indicated that novice Chinese language learners can transfer more learned words from their short-term to their long-term memory when using the repetitive practices of the pictographic-verbal coding approach and implementing its associated learning materials. More specifically, our participants’ vocabulary retention rates increased on average from 23% after a two-week repetition to 53% after a five-week repetition. It is evident that multiple practices of using the approach to review learned words enhanced the conceptual representations of radicals, pictograms, and semantics. For example, when reviewing the word 草莓 (strawberry) by using the pictographic-verbal coding approach, the radical 艹 (grass) is introduced along with its pictogram. Then, the pictogram and meaning of 草莓 are presented to learners. The pictogram of 草莓 assists learners in understanding the position of the radical and the structures of 草莓. The radical then becomes a lexical entry in the learners’ mental lexicon. In addition, with the multiple practices, this pictographic-verbal coding approach helps learners by providing a more systematic way to retrieve the words from their mental lexicon. Such repetitive practices could enhance students’ self-regulatory capacity for the use of vocabulary learning materials and strategy, which can later influence their involvement and performance (Tseng & Schmitt, 2008; Zimmerman & Schunk, 2001). However, we suggest that in the process of reviewing learned words, the cycle time for repetition should be monitored; if the cycle time is too long students may get bored with using the pictographic-verbal coding approach and its associated learning materials. Once the students have negative feelings, the positive effects of repetition could be minimized. It is recommended that students should review learned words by using the pictographic-verbal coding approach and learning materials on the day of learning or the next day, to help retain what they have learned.

Concerning students’ learning motivation about the pictographic-verbal coding and pictorial-verbal coding approaches and their learning materials, we found that novice Chinese language learners tended to react more positively to the pictographic-verbal coding approach and its associated learning materials that contain Chinese words with corresponding pictograms and contextual information. This approach, along with its quality learning materials, enabled novice Chinese language learners to feel motivated towards their Chinese vocabulary word learning. This in turn generated a higher sense of accomplishment and satisfaction which positively influenced their performance on word attainment and retention. This finding is consistent with previous research about how a learning approach with its associated quality learning materials enhances students’ learning satisfaction and performance (Hui et al., 2007). The results also strengthened the belief that learning materials and their corresponding learning procedures need to be systematic and meaningful to students. This finding is indicative of a pedagogical note that Chinese language teachers should establish a teaching routine by using a cognitive learning approach (e.g. the pictographic-verbal coding approach) to assist novice Chinese language learners in learning and reviewing new words. Once a routine is established, not only will the learners’ anxiety about learning new words be diminished, but their learning motivation will be enhanced. Then learners can concentrate more effectively on their teachers’ instruction, and on their own vocabulary memorization. Regarding the findings of the participating novice Chinese language learners’ Cognitive Strategy Use, we found that these learners used their designated learning approach to answer questions on tests. The results support the effectiveness of both the pictographic-verbal and pictorial-verbal coding approaches on Chinese word attainment and retention. The results also support that the pictographic-verbal coding approach could more effectively benefit novice Chinese language learners on their vocabulary attainment and retention.

VII Limitation

There are two limitations in our study. First, the participants were fourth and fifth grade native English-speaking students learning Chinese as a foreign language; therefore, the results of this study cannot be generalized beyond the characteristics of these participants and the research context. In future studies, we recommend to include learners from earlier grade levels as well as young adults, which can help examine the effects of the pictographic-verbal coding approach on these learners’ Chinese word recognition. Second, the Chinese words taught in this study were traditional characters. Compared to simplified Chinese characters, traditional Chinese characters contain more strokes and visual features (Li, Shu, McBridge-Chang, Liu, & Peng, 2012). Therefore, in future studies we would like to apply the pictographic-verbal coding approach to simplified Chinese word learning; we would like to discover if this approach can assist Chinese language learners in recalling more words than any other traditional teaching approaches.

VIII Conclusions

To conclude, this study profiled beliefs about an effective and optimal way for novice Chinese language learners to learn Chinese vocabulary words. These learners can memorize Chinese words more effectively when they are provided with a quality instructional approach and its teaching/learning materials, along with the nature of Chinese word creation. Word learning process becomes more meaningful and constructive when repetition is implemented (Karpicke & Aue, 2015). Also, repetitive practices can strengthen the benefits of such a quality instructional approach and its materials on word retention. This accounts for the fact that repetition helps transfer knowledge from short-term memory to long-term memory. When learning vocabulary words in such a cognitive, systematic and meaningful way, students’ confidence and persistence will also be enhanced.

Supplemental Material

APPENDIX1 – Supplemental material for The effect of cognitive vocabulary learning approaches on Chinese learners’ compound word attainment, retention, and learning motivation

Supplemental material, APPENDIX1 for The effect of cognitive vocabulary learning approaches on Chinese learners’ compound word attainment, retention, and learning motivation by Jui-Teng Li and Fuhui Tong in Language Teaching Research

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jui-Teng Li

Supplemental material

Supplemental material for this article is available online.

References

Bergman

E.M.

De Bruin

A.B.H.

Vorstenbosch

M.A.T.M.

, et al. (2015). Effects of learning content in context on knowledge acquisition and recall: A pretest–posttest control group design. BMC Medical Education, 15, 1–11.

Bromley

(2007). Nine things every teacher should know about words and vocabulary instruction. Journal of Adolescent & Adult Literacy, 50, 528–536.

Ceccagno

Basciano

(2007). Compound headedness in Chinese: An analysis of neologisms. Morphology, 17, 207–231.

Chen

H.J.

(1998). The ideal and the reality of situated-learning theory. Educational Resources and Research, 25, 47–55.

Chen

Hao

Geva

Zhu

Shu

(2009). The role of compound awareness in Chinese children’s vocabulary acquisition and character reading. Reading and Writing, 22, 615–631.

Chou

P.H.

(2012). Improvement strategy of Chinese character teaching from the learning transfer theory. Journal of Taipei Municipal University of Education, 42, 1–22.

Chou

P.T.

(2011). The effects of vocabulary knowledge and background knowledge on reading comprehension of Taiwanese EFL students. Electronic Journal of Foreign Language Teaching, 8, 108–115.

Chuang

H.Y.

(2011). The effect of computer-based multimedia instruction with Chinese character recognition. Educational Media International, 48, 27–41.

DeFrancis

(1989). Visible speech: The diverse oneness of writing systems. Honolulu, HI: University of Hawaii Press.

10.

Ebbinghaus

(1964). Memory: A contribution to experimental psychology. New York: Dover. (Original work published 1885.)

11.

C.S.H.

T.T.

W.K.

(2003). A radical approach to reading development in Chinese: The role of semantic radicals and phonetic radicals. Journal of Literacy Research, 35, 849–878.

12.

Holm

(1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

13.

Hoosain

(1991). Psychological implications for linguistic relativity: A case study of Chinese. Hillsdale, NJ: Erlbaum.

14.

Hsu

(2012). Teaching strategies: Word recognition. Taipei: Showwe.

15.

Hui

P.J.

Clark

T.H.K.

Tam

K.Y.

Milton

(2008). Technology-assisted learning: A longitudinal field study of knowledge category, learning effectiveness and satisfaction in language learning. Journal of Computer Assisted Learning, 24, 245–259.

16.

Jackson

N.E.

(1995). Graphic and phonological processing in Chinese character identification. Journal of Reading Behavior, 27, 299–313.

17.

Karpicke

J.D.

Aue

W.R.

(2015). The testing effect is alive and well with complex materials. Educational Psychological Review, 27, 317–326.

18.

Karpicke

J.D.

Roediger

H.L.

(2008). The critical importance of retrieval for learning. Science, 319, 966–968.

19.

Keller

J.M.

(1993). IMMS: Instructional materials motivation survey. Tallahhassee, FL: Florida State University.

20.

Keller

J.M.

(2009). Motivational design for learning and performance: The ARCS model approach. New York: Springer Science Business Media.

21.

Lachman

J.L.

Butterfield

E.C.

(1979). Cognitive psychology and information processing: An introduction. Hillsdale, NJ: Erlbaum.

22.

Leong

C.K.

(1997). Paradigmatic analysis of Chinese word reading: Research findings and classroom practices. In C.K.

Leong

Joshi

R.M.

(Eds.), Cross language studies of learning to reading and spell: Phonological and orthographic processing (pp. 379–417). Dordrecht / Norwell, MA: Kluwer Academic.

23.

Tong

(2018). Multimedia-assisted self-learning materials: Examining the effect of E-flashcards on Chinese as a foreign language vocabulary acquisition among elementary English speakers. Reading and Writing, Advance online publication. doi: https://doi.org/10.1007/s11145-018-9906-x

24.

Shu

McBridge-Chang

Liu

Peng

(2012). Chinese children’s character recognition: Visuo-orthographic, phonological processing and morphological skills. Journal of Research in Reading, 35, 287–307.

25.

Liu

(2009). Action research of the most common Chinese radical teaching among first graders. Unpublished master’s thesis, National Taitung University, Taitung City, Taiwan.

26.

Liu

Perfetti

C.A.

Wang

, (2006). Visual analysis and lexical access of Chinese characters by Chinese as second language readers. Language and Linguistics, 7, 637–657.

27.

Low

J.H.

Hew

S.H.

Wong

C.O.

(2014). Innovative pictogram Chinese characters representation. World Academy of Science, Engineering and Technology, 8, 604–608.

28.

McBride-Chang

Chow

B.W.Y.

Zhong

Burgess

Hayward

W.G.

(2005). Chinese character acquisition and visual skills in two Chinese scripts. Reading and Writing, 18, 99–128.

29.

Modern Language Association (2015). Language enrollment database 1958–2013. New York: Modern Language Association. Available at: https://apps.mla.org/flsurvey_search (accessed February 2019).

30.

National Academy for Educational Research (2018). General guidelines of grades 1-12 curriculum: Chinese Language Arts. Available at: https://www.naer.edu.tw/ezfiles/0/1000/attach/46/pta_18510_4703638_59125.pdf (accessed February 2019).

31.

National Standards Collaborative Board (2015). World-readiness standards for learning languages. 4th edition. Alexandria, VA: ACTFL.

32.

Nyberg

Petersson

K.-M.

Nilsson

L.-G.

, et al. (2001). Reactivation of motor brain areas during explicit memory for actions. NeuroImage, 14, 521–528.

33.

O’Connell

M.E.

Norwood

. (Eds.). (2007). International education and foreign languages: Keys to securing America’s future. Washington, DC: The National Academies Press.

34.

S.H.

Wang

C.C.

(2004). Effects of three vocabulary instruction strategies on vocabulary Learning and reading comprehension for students with reading disabilities. Bulletin of Special Education, 26, 271–292.

35.

Paivio

(1971). Imagery and verbal processes. New York: Holt, Rinehart, and Winston.

36.

Paivio

(1986). Mental representations: A dual coding approach. New York: Oxford University Press.

37.

Paribakht

T. S.

Wesche

M. B.

(1992). A methodology for studying the relationship between comprehension and second language development in a comprehension-based ESL program. Retrieved from ERIC database. (ED342237).

38.

Pintrich

R.R.

De Groot

E.V.

(1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82, 33–40.

39.

Sham

D.P.L.

(2002). A dual coding model of processing Chinese as a second language: A cognitive load approach. Published PhD thesis, University of New South Wales, Australia.

40.

Shen

H.H.

(2010). Imagery and verbal coding approaches in Chinese vocabulary instruction. Language Teaching Research, 14, 485–499.

41.

Shen

H.H.

(2015). Active learning: Qualitative inquiries into vocabulary instruction in Chinese L2 classrooms. Foreign Language Annals, 48, 82–99.

42.

Shu

Anderson

R.C.

(1997). Role of radical awareness in the character and word acquisition of Chinese children. Reading Research Quarterly, 32, 78–89.

43.

Steering Committee for the Test of Proficiency-Huayu (2016). Test of Chinese as a foreign language. Taipai: Steering Committee for the Test of Proficiency-Huayu. Available at: http://www.sc-top.org.tw/download/2016TOCFL_guidelines.pdf (accessed February 2019).

44.

Takač

V.P.

(2008). Vocabulary learning strategies and foreign language acquisition. Buffalo, NY: Multilingual Matters.

45.

Texas Education Agency (1997). A Texas framework for languages other than English. Austin, TX: Texas Education Agency. Available at: http://www.sedl.org/loteced/products/framework1.pdf (accessed February 2019).

46.

Tong

McBride-Chang

(2010). Developmental models of learning to read Chinese words. Developmental Psychology, 46, 1662–1676.

47.

Tong

Yip

J.H.Y.

(2015). Cracking the Chinese character: Radical sensitivity in learners of Chinese as a foreign language and its relationship to Chinese word reading. Reading and Writing, 28, 159–181.

48.

Tseng

W.T.

Schmitt

(2008). Towards a self-regulating model of vocabulary learning: A structural equation modelling approach. Language Learning, 58, 357–400.

49.

Wang

Blackwell

A.A.

(2015). Effects of dual coded multimedia instruction employing image morphing on learning a logographic language. Journal of Educational Multimedia and Hypermedia, 24, 281–313.

50.

Wang

Perfetti

C.A.

Liu

(2003). Alphabetic readers quickly acquire orthographic structure in learning to read Chinese. Scientific Studies of Reading, 7, 193–208.

51.

Wong

Y.K.

(2017). The role of radical awareness in Chinese-as-a-second language learners’ Chinese character reading development. Language Awareness, 26, 211–225.

52.

Padilla

(2013). Using meaningful interpretation and chunking to enhance memory: The case of Chinese character learning. Foreign Language Annals, 46, 402–422.

53.

Chang

Perfetti

C.A.

(2014). The effect of radical-based grouping in character learning in Chinese as a foreign language. The Modern Language Journal, 98, 773–793.

54.

Yeung

P.S.

S.H.

Wong

Y.K.

, et al. (2012). Longitudinal predictors of Chinese word reading and spelling among elementary grade students. Applied Psycholinguistics, 34, 1245–1277.

55.

Zimmerman

Schunk

(Eds.) (2001). Self-regulated learning and academic achievement: Theoretical perspectives. Mahwah, NJ: Erlbaum.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.37 MB