Review article: Recent publications on research methods in second language acquisition

Mackey and Gass (2012)

I begin with a brief synopsis of the Mackey and Gass (henceforth M&G) volume, which is intended as a guide for SLA students who are designing their own research projects. ¹ This volume consists of an introductory chapter by the editors, and 14 content chapters divided into two parts: ‘Data types’ (chapters 2 through 10) and ‘Data coding, analysis, and replication’ (chapters 11 through 15). Given the pedagogical focus of the M&G volume, each chapter provides not only the relevant theoretical background and a description of the methodology, but also a practical step-by-step guide for implementing the methodology (including appropriate statistical tools to use for each methodology), and summaries of studies that have used that methodology. Each chapter ends with a ‘Project ideas and resources’ section (which provide ideas for further reading and/or relevant internet URLs) as well as several study questions on the topic, which are intended both to review the main points of the chapter, and to help students apply what they have learned. It should be noted that there is no answer key, and indeed many of the questions are quite open-ended and do not have a single, simple correct answer. Many of the questions include hypothetical scenarios or data sets, and put the student in the position of a researcher who is either designing a study, or drawing conclusions on the basis of the findings. For example, the chapter on surveys and questionnaires includes a study question which asks the reader to examine an actual questionnaire from a prior study (provided earlier in the chapter) and identify those questions in the questionnaire that measured motivated learning behavior. To take another example, one question in the chapter on statistics asks the reader to interpret the result of a correlational analysis.

Four of the chapters in the ‘Data types’ part of the M&G volume address specific data collection measures. Chapter 2, by Sylviane Granger, discusses corpus data, considering both the types of learner corpora (spoken vs. written language, mono-L1 vs. multi-L1 learner corpora, etc.) and the issues that arise in data compilation, data extraction, data analysis, data interpretation, and the potential application of corpus findings to pedagogical tools, in such areas as courseware and language assessment. Chapter 5, by Zoltán Dörnyei and Kata Csizér, addresses the use of surveys and questionnaires in SLA research, covering such topics as designing questionnaire items, sampling a population, and analyzing the survey results; these topics are covered in more depth in the volume by Dörnyei and Taguchi, reviewed below. Chapters 3 and 7 address the data collection methodologies commonly used in experimental research on IL grammars, which have their roots in experimental research with child and/or adult native speakers. In Chapter 3, I discuss the design and use of formal comprehension and judgment methodologies used in generative SLA research, including acceptability judgment tasks and truth-value judgment tasks; examples of the kinds of morphosyntactic and semantic properties that can be investigated using these methodologies include question formation and aspectual interpretation. Chapter 7, by Kim McDonough and Pavel Trofimovich, focuses on psycholinguistic methodologies, covering both online comprehension tasks (self-paced reading and self-paced listening) and online production tasks (picture-word interference and sentence preamble tasks). Linguistic phenomena that can be investigated using online methodologies include the processing of syntactically ambiguous sentences, and the production of number agreement.

Three other chapters in the ‘Data types’ part of the book are framed more in terms of research design than in terms of data collection methodologies. Chapter 4, by Shawn Loewen and Jenefer Philp, looks at instructed SLA, and covers a range of research methods that can be used in the second language classroom, from non-interventionist observational studies to interventionist quasi-experimental research, in which the researcher manipulates the type of intervention (instruction) that a given classroom receives. Chapter 6, by Patricia A. Duff, talks about case study research, while Chapter 10, by Debra A. Friedman, discusses qualitative research design, covering such methodologies as ethnographies and conversation analysis. The remaining two chapters – Chapter 8, by Charlene Polio, and Chapter 9, by Keiko Koda – focus on particular research areas, explaining how to do research on second language writing and reading, respectively.

Whereas the nine chapters in the first part of the M&G volume address primarily issues in study design and administration (although most of these chapters also consider data analysis at least briefly), the five chapters in the second part address what to do with the data after the data collection is complete. Chapter 11, by Andrea Révész, looks at data coding, with a particular focus on coding transcripts of production data; validity and reliability in coding are discussed. This chapter is mainly concerned with coding quantitative data, whereas Chapter 12, by Melissa Baralt, takes up the coding of qualitative data. Both chapters introduce software that can be used for data coding and analysis. In Chapter 13, Jenifer Larson-Hill takes the reader through the basics of statistical analysis, introducing inferential statistics that are commonly used by SLA researchers: t-tests, ANOVAs, correlations, and the chi-square test. Chapter 14, by Luke Plonsky and Frederick L. Oswald, explores the why’s and how’s of doing a meta-analysis, while Chapter 15, by Rebekha Abbuhl, describes how to replicate research.

Overall, the fifteen chapters of the M&G volume provide quite comprehensive coverage of research methodology in SLA. It is not feasible to cover every aspect of research methodology and design in a single volume, and certain topics are left out, such as data collection techniques used in the area of second language phonology, as well as issues related to measuring second language proficiency (see the discussion of the corresponding chapter in the Blom and Unsworth volume, in the next section). This said, I find that the M&G volume largely achieves its stated goal of guiding SLA students in the design of their research projects.

The M&G volume is a highly useful book with multiple potential uses. Each chapter can be used as a stand-alone resource on a particular subject: for example, an SLA researcher interested in learning how to construct a questionnaire can start by consulting the Dörnyei and Csizér chapter, and then move on to the further readings suggested in this chapter, without necessarily needing to read the rest of the M&G volume. At the same time, the book as a whole, or a subset of the chapters, can – and should – be used as a textbook in a single-semester class that focuses on research methods in SLA, at either graduate or upper-level undergraduate level. Taken together, the different chapters provide a basic coverage of nearly all the major issues in data collection, coding, and analysis in SLA, and introduce a variety of theoretical perspectives. Even though the chapters are all written by different authors, there is continuity among them. For example, Chapter 2, which discusses working with production corpus data (including transcripts of learner speech), can be taught in conjunction with Chapter 11, which discusses transcript coding. Chapters 3 and 7, on offline and online experimental methodologies, respectively, complement each other nicely, as do Chapters 8 and 9, on research in writing and in reading. Chapter 6, on case study research, is related both to Chapter 5 on questionnaire design (since questionnaires are often a part of case study research) and to Chapter 10 on qualitative research (since case study research is often qualitative in nature). Chapter 4, on instructed SLA, can be used in conjunction with almost any other chapter on data collection and design, given that a variety of research methodologies (e.g., questionnaires, judgment tasks) and research areas (e.g., research on writing) can potentially be incorporated into classroom research. In a similar vein, Chapter 13, on statistical analysis, is a relevant follow-up to all the data collection chapters that discuss quantitative research methods. An instructor of a research methods in SLA course could choose to use the M&G volume in its entirety, as the main textbook for the class, or to use a subset of the chapters (in conjunction with other texts), depending on the nature and focus of the course. Some of the chapters (such as those on psycholinguistic methodologies, on statistical analysis, and on replication) would also be relevant reading for a survey course on research methodology in language study more generally, as opposed to SLA specifically.

Blom and Unsworth (2010)

I now give a brief synopsis of relevant chapters in the volume edited by Blom and Unsworth (henceforth B&U). The B&U volume is in many ways similar to the M&G volume: the B&U volume is also intended as a general resource on research methodology, and can be used as a textbook or resource for students; it contains 13 chapters by different authors, covering a variety of data collection measures and research areas. At the same time, the B&U volume differs from the M&G volume in three important ways. First, the B&U volume is on language acquisition in general, and focuses primarily on child first language acquisition, with only some chapters addressing SLA and/or bilingualism. Second, the B&U volume, which grew out of the Experimental Methods in Language Acquisition Research workshop (a yearly event held at Utrecht University), reflects the focus of this workshop and considers only formal, experimental quantitative research. SLA students and investigators whose interests lie more in pedagogical and/or qualitative research areas would not find the B&U volume particularly useful, and would be much better served by the M&G volume. On the other hand, SLA researchers whose research is quantitative and experimental in nature, including (but not limited to) those working in the generative framework, would find many chapters of the B&U volume quite relevant, especially when taken in conjunction with chapters on quantitative research in the M&G volume. The third major difference between the M&G volume and the B&U volume is that the latter is less of a textbook than the former. The chapters in the B&U volume do not contain either a resources section or study questions, features that make the M&G volume particularly appealing for use in the classroom. At the same time, many of the chapters in the B&U volume do contain step-by-step guides for designing and implementing specific tasks, and all of the chapters include a useful and practical list of do’s and don’ts at the end. Thus, while perhaps not as ideally suited for instruction as the M&G volume, the B&U volume is nevertheless an excellent resource both in the classroom and outside of it.

Of the 13 chapters in the B&U volume, two are specifically about SLA. The first of these is Chapter 9, by Jan H. Hulstijn, which talks about how to measure second language proficiency. This chapter discusses the need for proficiency measures, provides an overview of several existing proficiency measures, and includes a step-by-step guide for designing a cloze test (a proficiency measure commonly used in research with adult language learners, see also Tremblay 2011). Chapter 10, by the editors (Sharon Unsworth and Elma Blom), addresses cross-population comparisons among children acquiring a first language, children acquiring a second language, and adults acquiring a second language. Given the growing body of research on such developmental comparisons, this chapter is a very useful guide on which factors need to be controlled for, and how, to ensure validity when doing cross-population comparisons. The above two topics – proficiency and child/adult comparisons – are not covered in the M&G volume, so chapters 9 and 10 of the B&U volume would be a good addition in an SLA research methods course that uses the M&G volume as its primary text.

Although the other 11 chapters in the B&U volume do not specifically focus on SLA, several of them discuss data collection methodologies that are just as relevant to SLA research as to research with first language learners. Chapter 1, by Sonja Eisenbeiss, explores production methodology, including both naturalistic and semi-structured elicited production, with reference to both first and second language learners. This chapter covers some of the same ground as Chapters 2 and 11 in the M&G volume (corpus data, tools for analyzing transcripts), but unlike them, it also discusses controlled experimental techniques such as elicited production and syntactic priming (the latter is discussed in more depth in the McDonough and Trofimovich volume reviewed below). Chapter 2 of the B&U volume, by Cristina Schmitt and Karen Miller, takes the reader through a variety of comprehension techniques, including picture matching and truth-value judgment tasks; although this chapter focuses on child first language acquisition, it is relevant for SLA researchers who study comprehension, since many of the same techniques, with some age-appropriate modifications, can be used in SLA research (as discussed in my chapter on comprehension methodologies in the M&G volume). The same holds for Chapter 7, by Theodoros Marinis, which explores psycholinguistic methodologies such as self-paced reading/listening and word monitoring from the perspective of child first language acquisition, but has relevance for the use of these tasks in SLA research as well (see the M&G chapter by McDonough & Trofimovich on the same topic). Chapter 6 by Julie C. Sedivy presents a nice introduction to the use of the eye-tracking technique to study language processing; once again, although the focus of the chapter is on adult and child native speakers, the issues discussed in reference to the equipment, procedure and analysis are just as applicable to SLA. Finally, Chapter 3, by Antonella Sorace, has particular relevance to SLA in that it describes Magnitude Estimation, a technique used in research on acceptability judgments in place of traditional rating scales. This chapter both considers the rationale for using Magnitude Estimation and explains how to implement this technique, and is particularly useful for SLA researchers who work with acceptability judgment tasks and are interested in different ways of estimating acceptability (but see Sprouse 2011 for a recent critique of the use of Magnitude Estimation in acceptability judgment tasks). The remaining five chapters in the B&U volume do not have as direct a bearing on SLA research, but are highly useful readings for students and researchers in language acquisition more generally.

Dörnyei and Taguchi (2010) and McDonough & Trofimovich (2008)

Unlike the two anthologies reviewed above, the Dörnyei and Taguchi (henceforth D&T) and McDonough and Trofimovich (henceforth M&T) volumes each has as its focus one specific type of data collection methodology – surveys and questionnaires, and priming methods, respectively – and provides a more in-depth look at the use of that methodology in SLA.

The D&T book is a second edition, updated from an earlier 2002 edition by Dörnyei alone. The goal of this book is to help SLA researchers construct valid, reliable surveys and questionnaires. The major change from the first edition is a new chapter which illustrates the concepts of the book through the development of a motivation questionnaire. Other changes, which reflect the changing needs of the field, include a section on translating questionnaires as well as a discussion of how to collect survey data via the internet. The D&T book consists of five chapters, which take the reader step by step through the process of constructing, administering, and analyzing questionnaires. Chapter 1 provides an introduction to questionnaires, and gives examples of how they can be used in both quantitative and qualitative research. Chapter 2 takes the reader through the components of a questionnaire, gives specific instructions for constructing questionnaire items, and investigates such issues as optimal questionnaire length, and questions on sensitive topics. Chapter 3 discusses questionnaire administration, addressing, among other topics, issues of sampling and of confidentiality. In chapter 4, data entry, coding and analysis are explored; although statistical analysis is briefly mentioned, there is no detailed information about how to perform statistical tests, and the reader is instead referred to specific statistical software packages. This chapter also contains advice on reporting questionnaire findings in publications, and provides useful tips on how to present the data using charts and tables. Finally, chapter 5 takes an existing motivation questionnaire (provided in an appendix) from prior work by Taguchi and co-authors, and uses it to illustrate the concepts presented in the earlier chapters, from questionnaire construction, through piloting, reliability analysis of pilot data, and modification into the final form. The book also contains a very useful checklist that summarizes the main points of the first four chapters.

The D&T book is a helpful resource for any SLA researcher working with surveys and questionnaires, and should prove quite useful for researchers coming from a variety of theoretical perspectives. Surveys and questionnaires are very common in both quantitative and qualitative research in SLA, and even SLA studies whose focus is on learners’ grammatical knowledge, rather than on attitude/motivations, typically include at least a short background questionnaire to assess learners’ prior experience with the second language. The book can also be used in the classroom, as part of a course on methodology in SLA. But while it has some properties of a textbook – including boxes that set out important points – it lacks any study questions or practice exercises. In light of this, and given the fact that the book is fairly short, and covers only one type of research methodology, it would not be appropriate as the central textbook of a methods course. It would, however, be quite appropriate when used in conjunction with other readings (such as chapters from the anthologies discussed above).

The M&T book on priming is, like the D&T book, quite short, and similarly focused on a particular type of methodology. The M&T book consists of five chapters, the first of which introduces the basic concept of priming in the context of language use: the phenomenon of prior exposure to particular linguistic stimuli influencing subsequent language processing and/or production. Like other online techniques, the priming methodology came to the field of SLA from psycholinguistic research with native speakers. The first chapter of the M&T book briefly discusses the types of topics that can be studied by means of priming techniques, and summarizes representative priming studies in SLA. The next three chapters then take up three distinct types of priming: auditory priming (chapter 2), semantic priming (chapter 3), and syntactic priming (chapter 4). Auditory priming refers to the phenomenon by which prior experience with a spoken word facilitates the processing of that word; in the case of acquisition, auditory priming can be used to study whether child first language learners and adult second language learners process spoken language in the same way. Semantic priming is observed when the processing of a word or a picture is facilitated by prior exposure to a meaningfully related word or picture (e.g., prior exposure to the word bread would facilitate the processing of the word butter). In SLA, semantic priming can be used to explore how second language learners organize the lexicons of their two languages, and to what extent these are interrelated. Finally, syntactic priming refers to a tendency to reuse the same syntactic structure (rather than an alternative equally acceptable structure) that a speaker had been exposed to a short time earlier; in the case of SLA, some studies have asked whether syntactic priming can encourage learners to use a structure that they did not use otherwise (e.g., the passive voice).

Each of the chapters 2 through 4 explains the particular type of priming, provides an overview of the use of that priming technique in research with native speakers, and discusses the use of the technique in SLA research. The three chapters all follow a similar outline, and cover both theoretical considerations (describing the kinds of research questions that can be studied using the technique) and methodological considerations (describing how to construct the materials, and addressing issues of validity and reliability in priming research). Chapter 5 discusses data analysis, and gives step-by-step instructions on how to run statistical tests (different types of ANOVAs, as well as the Linear Mixed Model) using SPSS software. This chapter also provides pointers on publishing priming research, and provides information about psychology journals that SLA researchers might not otherwise be very familiar with.

The M&T book is a must-read for any SLA researcher planning to conduct research on priming. This book has a narrower intended audience than the D&T book, since priming tasks are less common than questionnaires in SLA research. In introducing the topic of priming to SLA researchers, the M&T book builds an important bridge between SLA and experimental psychology. It is written in clear, understandable language, and does not require the reader to have prior background in either online methodology or statistical analysis. The book is also an excellent textbook, with follow-up questions and activities at the end of each chapter. It would be highly appropriate reading for a course on quantitative methods in SLA, as well as a course on online/psycholinguistic methodologies (not necessarily limited in SLA). Given the short length and specific focus of the book, it should be used in conjunction with other readings covering other types of experimental methodologies.