A Field-Wide Metasynthesis of Pedagogical Research in Technical and Professional Communication

Choosing a Metasynthesis Approach

As a field with a large and growing body of scholarship, TPC is sufficiently established that we can reasonably reflect on the field’s research practices. There are numerous methodological approaches for conducting this sort of “research on research.” For example, the integrative literature review has gained traction within TPC (e.g., Clark, 2016; Lauren & Schreiber, 2018; Melonçon & Warner, 2017), as such reviews provide deep insights into specific topic areas and make calls to advance research in that topic area. While useful at the topic level, the integrative literature review would not effectively address our research objective of understanding field-level research since such research spans multiple topics. Thus, we needed a different approach.

For that approach, we turned to the metasynthesis. A metasynthesis involves a comprehensive approach to reviewing and analyzing published scholarship during a specific point in time. As Walsh and Downe (2005) explained, “Meta-synthesis attempts to integrate results from a number of different but interrelated qualitative studies. The technique has an interpretive, rather than aggregating, intent, in contrast to meta-analysis of quantitative studies” (p. 204). As such, it can reveal the overall perspective of a field on a topic or concept at a key moment in its history and development. For this reason, a metasynthesis of qualitative literature is used frequently in allied health sciences and medicine as the qualitative equivalent to a systematic review (which is based on reviewing quantitative studies such as randomized control trials). In a metasynthesis review, researchers restrict their bodies of literature in an attempt to aggregate information and discover patterns in research that help to characterize not only the kinds of research taking place but also areas left unstudied or understudied. Using a metasynthesis approach, likewise, allows us to examine a body of (inter)related literature bound by a broad research category rather than examining a specific orientation toward research (such as methods, cf., Boettger & Lam, 2013; Melonçon & St.Amant, 2019).

We are also using metasynthesis broadly to encompass a field-wide view of approaches and practices and to move outward beyond a specific topic (e.g., service learning). Accordingly, we assume the role of methodologists because research should “move on to consider . . . the ‘how do I know?’ question” (Charney, 2014, p. 106) as related to TPC’s pedagogical research. This field-based metasynthesis approach to pedagogical research allowed us to consider both specific questions about the research itself and broader questions about what pedagogical research reveals about the state of scholarship in TPC. In the next section, we outline how we gathered and coded data.

Process and Practice for Data Collection

To gather a representative data set on this topic, we examined a 5-year sample of publications (2011–2015) in five major journals in TPC:

IEEE Transactions on Professional Communication (IEEE)

We selected these journals based on past research practices involving surveying published scholarship in the field (see, e.g., Boettger & Lam, 2013; Lowry, Humpherys, Malwitz, & Nix, 2007; Melonçon & St.Amant, 2019; Smith, 2000) and the journals’ longevity and recognition within TPC. For example, JTWC is the oldest academic journal in the field (founded in 1971), while TC is the oldest continuously published journal in the field (founded in 1954 as Technical Writing Review, later to become Technical Communication in 1967). We openly acknowledge that TPC-related pedagogical and programmatic scholarship is published in places outside of the journals selected. While other journals have emerged and provide important research outlets (e.g., Programmatic Perspectives), their relative newness (i.e., most have existed for less than 10 years, some for fewer than 5 years) limits their ability to represent a historically established venue with a standardized and continuous publication schedule and that is widely recognized by members of the field. By selecting these five journals, we provide logical parameters and an appropriate scope that parallel prior work of this nature and that fit with our study’s objectives: to better understand pedagogical and programmatic research practices in TPC. Based on these factors, we believe that the journals selected for this project give us a broad view of pedagogical research in TPC. That said, future studies might use our approach to mine newer publications to test our findings against these other publications.

We selected a 5-year sample as that time frame encompasses sufficient data for patterns to emerge and to provide a sense of rigor to the method and findings. To engage in this kind of research, one needs time because it takes time to compile, code, and analyze the data. Until tools are created that can automate the processes of data extraction, cleaning, and coding, much of this type of work still must be done manually. Moreover, there are no shortcuts to reading the scholarship to make decisions about how to code and analyze it. Thus, to do this type of research well means there will be a delay between the period selected for study and the time to publication.

After reading each of the articles published in the five identified journals during the stipulated time frame, the authors identified a series of basic research questions to guide the next step of the process. The overarching research question we wanted to consider was as follows:

What types of research are scholars doing in relation to pedagogy, including programs, in TPC?

This question was foundational as we used it from the beginning of this project. It also led us to more specific questions that guided our approach to coding and, subsequently, to analyzing the data:

RQ1: Where is pedagogical research being published, and how many articles are being published?

Each of the authors read all of the abstracts of the larger data set (n = 404) to initially give each article a summative word/phrase that captured the salient category of the research. After coding articles based on the abstracts, each of the authors read each article. We then discussed whether an article would stay in this data set based upon the topic of the entry and the approach used to examine that topic. Once we had agreed on the data set (n = 82) and with our focused research questions, we conducted the more in-depth analysis using our coding schema (described in more detail in the text to follow).

We coded articles in three ways. One was to identify the type of pedagogical research (explained in more detail later and in Table 1). A second was to identify the topic(s) of research (not included in this analysis). If the pedagogical piece was empirical, the third approach was used to determine what type of research method was used (thus building on work done by Carliner et al., 2011, and Melonçon & St.Amant, 2019). We did not use a set list or predetermined codes when engaging in these activities. Rather, we let the articles guide how we coded them in an open coding scheme. Our coding scheme for categorizing the research can be replicated in the future or—at the very least—refined for a consistent, reflexive look at pedagogical and programmatic research in TPC from 2011 to 2015. In the next sections of this entry, we confine our analysis and discussion to only these 82 articles (see online Appendix A for titles).

OPEN IN VIEWER

Table 1.

Types of Pedagogical Research in TPC.

Pedagogical research categories in TPC
	Programmatic	Empirical	Teacher reflection	Exposition
Focus	Changes or information that can affect more than one class; change (or information) at the program or curriculum level	Empirical research taking place in the classroom	Based on a classroom experience from a teacher-based point of view	Development and, typically, the promotion of a certain pedagogical or theoretical framework or heuristic as the integral point of the article
Key differentiator	The emphasis is on the critique of program or curriculum or of gathering information useful to programs; benefits are likely changes stated tacitly or explicitly.	The research is directly tied to a class (or classes) and is empirical or systematically textual in nature with the overall goal of extending pedagogical practices outward to other locations.	The research is directly tied to a class (or classes) and is more reflexive in nature with the goal of improving local practices and is not easy to apply outside that one context.	The development of a pedagogical framework or theory or approach is in itself the central point of the article. The prose is less empirical and more narrative or reflective.

Note. TPC = technical and professional communication.

Empirical Classroom-Based

Empirical classroom-based research focuses extensively on what is taking place in the classroom. Such research may be conducted through surveys, interviews, textual analysis, teacher research, or other methods, but the site of research where data are collected is the classroom or an academic learning space. This category is well represented in the study corpus, with 35% (n = 31) of the articles fitting into this classification.

One of the articles that exemplifies classroom-based research is Wang’s (2013) study examining whether teaching practices can lead to improved intercultural competencies among students. Wang’s work represents empirically based classroom research because the study design and implications reflect the type of work that offers a clear contribution to pedagogical literature in TPC. The work can be duplicated and built upon. In addition, Wang’s study:

Tested the effectiveness of a predetermined model;

Several of the classroom-based pieces reviewed for this project similarly reported dynamics in which the primary researcher was also the instructor of a course that served as a basis for research. This approach is not a methodological problem per se. In fact, it can offer a firsthand perspective on classroom activities—one more difficult to achieve in situations where the researcher was less familiar with the pedagogical situation.

The issue with such research, however, is the innate difficulty of extrapolating broader findings from this model, particularly when only one class is studied or the sample provides an incomplete picture of the larger population of interest. Bokor (2011), for example, used the classroom-based approach to examine technical translation practices and concepts of a generalized English that privileges aspects of the Western experience. Arguing for a World Englishes approach to teaching international technical communication, Bokor conducted surveys of 30 students in a technical writing course that he taught and interviewed some of them as well. He then used the data collected from these limited samples to assert broader curricular implications. While this project could constitute an effective pilot study, the limited data collected from one small and specific group leads to questions about how generalizable the research findings are beyond the particular class studied. Therefore, Bokor’s important work would be more impactful if the essay were framed in a way that highlighted the generalizability associated with research involving relatively small sample sizes and that called for broader follow-up study.

The same situation is the case for work that presents a practice, approach, or method for providing instruction (e.g., Tatzl, Hassler, Messnarz, & Flühr, 2012) model for collaborative cross-disciplinary technical writing instruction. While such approaches might work well in one class, without testing them in other classes—be it a similar kind of class or a completely different one—there is no way to determine how effective that approach is beyond the specific class in which it was studied. Again, such research focused on studying the dynamics of a single class could constitute an effective pilot study. Yet, without further research to test ideas and trends, generalizing findings beyond that specific context affects the portability and applicability of the important work these scholar-practitioners do when they measure the successes and failures of specific teaching models in their classes. We suggest that calling these studies “pilot” studies and offering their contributions in the form of methodological suggestions for follow-up studies would strengthen their impact. Such an approach could greatly contribute to the field by providing a body of research-based material from which larger scale research endeavors could be developed.

Methods

To increase analytical precision, and to help ensure we understood the research designs more fully, we added four areas of analysis specific to the classroom-based research category. These areas were as follows:

Whether a control group was included;

These areas commonly applied to many (though not all) of the classroom-based research articles we reviewed. These areas are important not only because of their frequent use in the category but also because of the ways in which they potentially affected results. For example, if research was performed across different sections of the same course, each of the sections was likely to have a different instructor (e.g., Kim, Mohan, & Ramesh, 2014); having a different instructor implies differences (however nuanced) in pedagogy that could have a marked effect on results. In many studies, the researcher who wrote the article was also the classroom teacher (e.g., Albers & Marsella, 2011), presenting a whole different set of circumstances to think about. The ways in which student performance and perception were measured also varied among the studies. Some studies used surveys (e.g., Kumi, Conway, Limayem, & Goyal, 2013); others incorporated interviews (e.g., Yu, 2012, which also incorporated surveys and a written response); still others incorporated materials that were more unique within the corpus (e.g., Mehlenbacher, Autry, & Kelly, 2015, which used a feature specification or report as well as a written memo). In effect, there was no consistent method used to conduct research in such contexts, and the replicability of the work (and testing findings) does not always seem possible.

One of the areas we evaluated was whether a classroom-based study incorporated a control group as part of its design. The use of a control group normally implies that a study includes an experimental or quasi-experimental design. In the former, participants are assigned at random to a control or experimental group. In the latter, randomization of those assignments may be reduced because of the researcher’s need to control group assignment manually, the fact that true randomization may not be possible, or some other factor. In this corpus, six of the articles (20% of the classroom-based research category) incorporated a control group of some kind.

In sum,

37% (n = 11) featured studies performed across sections of the same class;

Of the materials used as a basis of data analysis, some of the most common included

surveys: 53% (n = 16);

Less common yet noted materials included reflections (20%, n = 6), tests (7%, n = 2), and videos (7%, n = 2). Again, such factors represent the wide and varied range of approaches used to engage in this kind of research.

Examining the research study designs of classroom-based research is important in establishing a foundational understanding of how TPC researchers are actually conducting research. Without this sort of baseline knowledge, the field cannot work toward innovating methodologically. Moreover, it cannot accurately test findings or collect the data needed to know if a process or approach has been effectively implemented or has been successful.

Programmatic

The second category of research identified was programmatic. Here, we follow Schreiber and Melonçon (2018), who define programmatic perspective as

the critical review of programs involving careful deliberation on the nature of programs to better understand how and why they exist and work. Effectively engaging in such activities involves understanding that TPC programs are both locally situated and shaped by field-wide trends in academia and industry. (n.p.).

Programmatic research can be empirical or textually based. An example of empirical research using interviews can be found in Taylor’s (2011) study of how students interpret instructor feedback. As an example of a textually based study, we consider Ford and Newmark’s (2011), which involves considerations of how to incorporate an emphasis on research through their examination of students’ work on an undergraduate writing journal.

Programmatic research is also typically data-driven in some way. For example, Melonçon and Henschel’s (2013) study of undergraduate degree programs used textually based, data-driven research to enable the field to see trends in undergraduate degree programs over time. Such research also helps to illustrate the difference between the other types of research under discussion. In this case, the information and resultant conclusions are useful at both the programmatic and classroom level.

Exposition

In exposition research, the development and, typically, promotion of a certain pedagogical or theoretical framework is the integral point of the article. This framework is typically developed on other frameworks, and classroom-based research may or may not be used to bolster the theoretical arguments. An example of an exposition piece is Yu’s (2012) article that advances a definition of intercultural competence for TPC and evaluates nine methods of assessing intercultural competence. The focus of Yu’s article is definitional development and the discussion of relevant assessment methods. Another example is Saidy, Hannah, and Sura’s (2011) theoretical discussion of archival research methods and how such methods can be used in the TPC classroom.

In some cases, scholars will bring examples of their own or others’ teaching to bear in an exposition piece as a way of exemplifying the efficacy of the frameworks they discuss. Bourelle (2014), for example, discussed her own experience teaching an online technical communication course using a “Service eLearning” model to examine challenges and best practices for implementing a service-learning model in online TPC courses. In so doing, she incorporated student reflections as part of assessing the results. In this case, the discussion of the pedagogy and the analysis of student reflections were used as part of developing and promoting the pedagogical framework around which the article centers.

For exposition scholarship, as the examples show, the overall aim is to offer a conceptual or theoretical framework to pedagogical practice that should lend itself to testing of the concept or theory through empirical, classroom-based research. However, clear pathways to doing such important classroom-based research are not always present; therefore, we call for researchers to be more deliberate in setting up detailed frameworks for those who would like to test the concept or theory at hand.

Teacher Reflection

The teacher reflection became a key pedagogical research category that emerged during the authors’ research. It was, in fact, one that required a great deal of refinement as the study developed. The term teacher reflection came to denote an article based on a classroom experience from a teacher-based point of view. The research is directly tied to a class (or classes) and is more reflexive in nature. The goal of this type of research is to improve local practices. Such a focus means the nature of the research is to apply findings to one specific classroom environment (the author’s). This means the related findings are not easily applied outside that localized context, and the aim might be to inspire similar, local instructor metacognition such that new, individual insights are possible. The typical article in this category has three components:

a teaching case;

A teaching case, like a case study, is the review of a self-contained, bounded unit of interaction that exists within certain set parameters (e.g., a point in time with a distinct start and end point) and location/setting (e.g., a classroom). What sets it apart from a “tutorial” or a “pedagogical piece” is that tutorials and pedagogical pieces focus on an assignment or activity that is part of a preset process controlled by the instructor. A teaching case involves a study of the events that took place in a class context over a given period of time (e.g., a semester/term). It is not a controlled activity, but a review of the events after the fact that took place in a given location at a particular point in time.

As such, a teaching case cannot by definition be empirical and is simply an extended example that is not necessarily research. This argument applies because the author/researcher is not preestablishing parameters and conditions to test a fixed relationship. Such testing, moreover, must focus controlling the variables individuals use to perform in a process—one they would not have undertaken unless prompted by the researcher to do so (as opposed to observing and recording original behaviors that naturally emerge within the context of a class). This is not to lessen the importance of reflective writing, of course. Instead, we highlight the importance of framing such pieces as thoughtful contemplations meant to inspire teacher metacognition within their classrooms and programs but should, therefore, not be used to promote specific approaches without a research study.

Wolfe’s (2015) article discussing the importance of rhetorical decision-making in data visualization is an example of teacher reflection. In this article, Wolfe applies Perelman and Olbrechts-Tyteca’s idea of “interpretative level” to the choices students make in creating data visualizations and writing about visualizations in informal and formal assignments. Cárdenas’s (2012) article, in comparison, presents a teaching case of students working with a community health department to effectively inform citizens of the operations of an important Texas law. In so doing, she examines intercultural communication practices and how individuals make appropriate rhetorical choices to reach their audience. In both articles, the teaching case, the teacher reflection, and the implicational component are present, despite variations in the ways the discussions of each are structured. In addition, the context of practice in each article is localized (a good example is the South Texas area where Cárdenas’s students were working and the law applied). These factors mean while the models can be adapted to other settings, the cases themselves are often difficult to replicate outside the settings where they were first used. The value they clearly do have, though, is in offering metacognitive models that instructors might use as inspiration in their own reflective critical praxis. If these essays’ contributions were described in such a way, the impact could be recontextualized.

Take Care When Drawing Research Conclusions

An important part of research study design is writing up the research. In the process of the metasynthesis, we realized two important aspects that we wish to caution researchers about. First, pedagogical researchers should be careful of overstating conclusions based on data from a single study, sample, or research design. Within TPC scholarship, this issue occurs when the researcher overpromise what results imply for other classroom settings based on a single case or small sample size (see, e.g., Lauer & Sanchez, 2011) or for programs based on a single case study (see, e.g., Cárdenas, 2012). While single cases can be useful to provide insights in local situations, and more importantly to guide the field in what questions to ask next or to inspire similar metacognition in readers, researchers need to take care in making claims where data do not support relevance of findings beyond a specific context. Rather than make the case for the applicability of the findings for other settings, why not highlight the need for other practitioners to engage in similar metacognition? If the goal is to, instead, advance a particular approach, a different approach to research design would likely be more appropriate and advantageous.

Second, TPC researchers must, of course, avoid drawing conclusions about workplace practices based on classroom research as these two contexts are, out of necessity, highly dissimilar. We wish to note that empirical classroom-based research can be valuable to understanding the dynamics of certain instructional contexts. TPC researchers, however, must be careful about making larger (i.e., extending beyond the classroom) arguments that flow from the analysis of data, or even from the premise itself. Manion and Selfe (2012), for example, used the results of case studies involving different courses to draw connections between industry practice and the use of wikis and student-involved assessment practices. They did raise valid points (e.g., stressing that the workplace context can help students develop discipline-appropriate writing and design skills). However, claims of the transferability of skills observed in these classroom-based cases to the workplace are uncertain without more detailed follow-up with a large number of participants operating in a workplace setting. While we do not discount the value of this work for pedagogical purposes, these kinds of limitations should be carefully foregrounded in the scholarship.

Enhance Classroom Studies

Our metasynthesis revealed that during the time frame we examined, only seven studies were designed to include some form of control group. Individuals can realize immediate impacts in empirical classroom-based research by using control group and experimental studies. While some may push back against the term control groups, it is difficult to make generalizable claims about improved learning or effective teaching without a baseline against which to compare results. As such, study design that includes control groups will bring more rigor and validity to pedagogical research findings. This sort of research can assist TPC program administrators and faculty with making cases about teaching writing to stakeholders outside of our home departments (e.g., service courses or writing across the curriculum situations).

In the majority of other cases published in this time frame, pedagogical or programmatic researchers attempted to design thoughtful studies. For example, Friess’ (2015) study of heuristic evaluations was used across sections. Lam’s (2013) divided students in the same class into two distinct groups, which is a form of a control. However, the small size of the sample study limits the application of the results. TPC could, thus, benefit from more research conducted across sections of the same course within the same term or, better yet, longitudinally, using the same course over multiple terms. By moving outside of singular classrooms through experimental and control group studies, TPC can build better research that could improve the field’s pedagogical approaches. As field-wide impacts cannot hope to occur with single classroom studies, the field needs to challenge its research practices to generate classroom-based research that is sustainable (i.e., that can stand up to scrutiny) and (re)usable (i.e., that can be implemented in other contexts).

Perform Multi-Institutional Research

In the early days of the field, any classroom-based research was welcomed. This was because TPC did not have any research to rely on for our pedagogical practices or to help make local arguments about curricula. For these reasons, TPC often borrowed from related fields such as composition in following their research study design. TPC, however, is no longer a brand new field. Rather, with more than 50 years of published studies, it is now time to challenge current research practices. It is, perhaps, time to insist upon much more pedagogical research that goes beyond a singular case study. Case studies occurred across all of these heuristics of pedagogy research, yet these types of studies are difficult to port to other locations. This call to action is not meant to diminish the good writing and thinking that goes into case studies in the field; instead, we wish to highlight the importance of enhancing such work with other kinds of broader study.

One way to enhance case study approaches would be to engage in projects including multiple institutions. This research is difficult, complex, and cumbersome, but it is especially needed as multi-institutional research and field-wide research offer TPC the most potential for affecting day-to-day practices (see, e.g., Veltsos & Patriarca, 2017). If TPC were to move to more multi-institutional research studies, our pedagogical questions and answers could offer insights that go beyond local cases. Without these rigorous studies, TPC could lose credibility. That is, if case studies were appropriate in the early days of the field because the field lacked other methods, those studies provided the foundation for a robust field of inquiry that is now ready to take on the challenges of multi-institutional study. If we do not begin to do this work, we risk appearing stuck in the past.

Consider Doing Transfer Research

One of the primary premises of TPC programs is that we are preparing students for future professional lives both as engaged citizens and as professionals. In fact, it is an area in which both academics and members of industry are increasingly interested (see Bridgeford & St.Amant, 2015; St.Amant & Melonçon, 2016b). Unfortunately, we have limited research that helps to prove that TPC is doing the very good work that we all know, anecdotally, it is doing. As the need to convince other stakeholders of the value of our work increases in our programs and in the academy in general, strong data to support our worth are necessary.

One area where we could focus our pedagogical and programmatic research is in transfer studies (Brent, 2011), which are studies that focus on understanding how students transfer knowledge from one context to another. Accordingly, TPC scholars could set up well-designed research studies meant to examine how well students are transferring knowledge from TPC courses (e.g., the service course) back to their own majors (e.g., Ford, 2004) or to examine how well students are transferring the knowledge from our courses to the workplace upon graduation (e.g., Schieber, 2016).