Expediting the Analysis of Qualitative Data in Evaluation

Abstract

Qualitative data offer advantages to evaluators, including rich information about stakeholders’ perspectives and experiences. However, qualitative data analysis is labor-intensive and slow, conflicting with evaluators’ needs to provide punctual feedback to their clients. In this method note, we contribute to the literature on rapid evaluation and assessment methods by proposing procedures that evaluators can use to expedite the coding and analysis of qualitative data and comparing these procedures to other possible methods. Specifically, we outline procedures for the rapid identification of themes from audio recordings, which allow evaluators to code and analyze qualitative data without time-consuming transcription. We illustrate the use and assess the reliability of these procedures using qualitative semi-structured interview data from 18 public school administrators on how they locate information about and decide to use instructional, health, and social skills programming in their districts. Finally, we end with advantages and trade-offs of these procedures as well as recommendations for how to apply them.

Keywords

qualitative data rapid evaluation theme identification audio

Qualitative data have several strengths that enhance their utility for evaluation. For example, qualitative data can provide rich details about program or intervention processes as well as stakeholders’ perspectives, expectations, and experiences (e.g., Patton, 2002). Additionally, qualitative data can be used to triangulate or complement quantitative data in evaluation studies, thus enriching results (e.g., Greene, Caracelli, & Graham, 1989; Reichardt & Rallis, 1994). Despite these benefits, transcribing and analyzing qualitative data are often a slow labor- and resource-intensive process (e.g., Britten, 1995; Halcomb & Davidson, 2006; Tessier, 2012). Thus, there is often a tension between providing rich qualitative results and providing punctual feedback to clients.

Researchers and evaluators are often faced with the need to provide timely and ongoing feedback, leading to a movement to propose rapid evaluation and assessment methods (REAM; see McNall & Foster-Fishman, 2007, for review). REAM often involves the collection of qualitative data, including data collected through semi-structured interviews or focus groups. However, the time constraint necessitated by REAM often precludes the traditional steps of transcribing qualitative data and coding transcripts. Instead, evaluators using REAM typically employ alternate methods of analyzing qualitative data such as the analysis of detailed notes or case summaries derived from interviews (e.g., Bentley et al., 1988; McNall & Foster-Fishman, 2007). To date, there is limited detail on processes by which evaluators can integrate the analysis of qualitative data with REAM.

In this method note, we contribute to the growing body of literature on REAM by proposing procedures that researchers and evaluators can use to expedite the analysis of qualitative semi-structured interview data. We begin with a brief review of the challenges of integrating qualitative data and REAM. We advocate the use of rapid identification of themes from audio recordings (RITA) that allows researchers and evaluators to code and analyze qualitative data without time-consuming transcription. Here, we also compare RITA to other common methods of qualitative data analysis (QDA), highlighting that RITA allows for expeditious identification of themes in qualitative data while minimizing the loss of information that often accompanies alternative analyses of detailed notes or case summaries used in REAM. Next, we delineate specific steps that researchers and evaluators can use to employ RITA. We illustrate the use and assess the reliability of these steps using qualitative semi-structured interview data from 18 public school administrators on how they locate information about and decide to use programming in their districts. Finally, we end with advantages and trade-offs of RITA as well as recommendations for how to apply this procedure.

Challenges and Potential Solutions to Integrating REAM and Qualitative Data

McNall and Foster-Fishman (2007) reviewed several variants of REAM arising from both the evaluation literature (e.g., Jamal & Crisp, 2002; McNall, Welch, Rue, Mildner, & Soto, 2004; Wholey, 1983) and ethnographic action research (e.g., Bentley et al., 1988; Chambers, 1994; Desmond et al., 2005). Despite some differences across these variants, they note that a common issue is “achieving a balance of speed and trustworthiness” (McNall & Foster-Fishman, 2007; p. 152). Moreover, they acknowledge that many of the variants of REAM employ diverse methods that include qualitative data collection in the form of interviews, focus groups, or observations. For example, unstructured and semi-structured interviews are commonly used to collect qualitative data in variants of REAM such as real-time evaluation (Jamal & Crisp, 2002), rapid feedback evaluation (e.g., McNall et al., 2004), rapid ethnographic assessment (e.g., Bentley et al., 1988), and rapid or participatory rural appraisal (e.g., Chambers, 1994).

Despite the ubiquitous use of qualitative data in REAM, there is more limited detail on how researchers and evaluators reconcile the time and resource costs of QDA with the aim to provide prompt results. Although some REAM projects involve traditional procedures of transcribing and coding qualitative interview data (e.g., Desmond et al., 2005), it is more common to skip transcription and directly analyze detailed notes (e.g., Bentley et al., 1988) or case summaries (e.g., McNall & Foster-Fishman, 2007) derived from the interviews. As outlined in the general literature on qualitative methods, there are pros and cons to each of these approaches (see the first two columns of Table 1).

Table 1.

Comparison of Different Approaches to Coding and Analyzing Qualitative Data.

	Traditional coding of verbatim transcripts	Analysis of field notes/case summaries	Coding of linked audio file and transcript	Coding of linked audio file	RITA
Level of detail	High. Allows for detailed coding but may result in some loss of nonverbal data^a	Low. Loss of verbal and nonverbal information based on what researcher chooses to record^a	Very high. Allows for detailed integrated coding of audio file and transcript, preserving verbal and nonverbal information^a	High. Preserves verbal and nonverbal information. Challenging to code verbal information from audio^d	Moderate. Preserves verbal and nonverbal information. Looks for selected themes in time segments, thus losing some nuance
Ability to capture intonation	Low. Intonation is difficult to capture in transcription text^a	Low. Intonation must be conveyed in field note by researcher. Prone to bias^a	High. Intonation is preserved because coding is linked to the audio file^a	High. Intonation is preserved because codes are applied directly to the audio file^a	High. Identification of themes takes place while listening to the audio recording
Coding time	Slow. 360 min for transcription of a 60-min interview plus additional time for coding the transcript^b	Very fast. No transcription. Described as “most economical in terms of time” ^a	Very slow. 360 min for transcription of a 60-min interview plus additional time for coding the transcript and audio file^a,b	Moderate. No transcription but audio files must be uploaded and coded in software or clipped and coded^c,d	Very fast. No transcription. 68 min to code a 60-min interview when using 3-min time segments
Software required?	Yes. Word processing or QDA software for transcription and optionally QDA software to code transcripts	No. Optionally, word processing software can be used to record field notes and QDA software to code them	Yes. QDA or other software that can link audio files and transcripts^a	Yes. Video editing, spreadsheet or QDA software to directly link codes to audio file^c,d	No. Optionally, once data are recorded on coding form, it can be entered into Excel or other QDA software

Sources. ^a Tessier (2012); ^b Britten (1995); ^cCrichton and Childs (2002); ^d Evers (2011).

Note. RITA = rapid identification of themes from audio recordings; QDA = qualitative data analysis.

Verbatim transcription of qualitative interview data can facilitate flexible data analysis and researcher immersion in the qualitative data (Gravois, Rosenfeld, & Greenberg, 1992; Halcomb & Davidson, 2006; Markle, West, & Rich, 2011; Tessier, 2012). While the traditional coding of verbatim transcripts results in a high level of detail, some have noted that verbatim transcription is prone to error (Halcomb & Davidson, 2006; Markle et al., 2011) and that transcription may remove important contextual features (e.g., intonation) that are important to analysis of the qualitative data (e.g., Crichton & Childs, 2005; Tessier, 2012). Furthermore, verbatim transcription also demands extensive amounts of time and resources, which may make it infeasible for most REAM projects. In contrast, analyses of field notes or case summaries are often employed in REAM due to benefits of speed and cost-effectiveness (McNall & Foster-Fishman, 2007). However, as Tessier (2012) notes, field notes are prone to bias and can result in the loss of both verbal and nonverbal information because they are dependent on researcher interpretations.

A “happy medium” between analysis of verbatim transcription and analysis of field notes could be particularly useful for REAM. One such possibility might include analyzing the audio recordings themselves. Recent advances in computer software have led to new opportunities for identifying themes from audio recordings, including the capability to embed codes in or digitally link codes to audio files (e.g., Crichton & Childs, 2005; Evers, 2011; Tessier, 2012). At the most detailed level, this involves the direct coding of both the audio file and verbatim transcript in QDA or other special software (Tessier, 2012). This allows for a very high level of nuance and preserves intonation but is not a prime candidate for REAM, as it still involves time-consuming verbatim transcription (see third column of Table 1). Others have experimented with skipping verbatim transcription and instead directly embedding codes in or digitally linking codes to audio files (e.g., Crichton & Childs, 2005; Evers, 2011; Tessier, 2012). However, this process requires access to costly software and can still be time consuming (see fourth column of Table 1). For example, Crichton and Childs (2005) described how they used video-editing software to clip audio files for coding, a detailed and lengthy process. Prior to recent software developments, Gravois, Rosenfeld, and Greenberg (1992) advocated skipping verbatim transcription and instead identified themes from listening to audio recordings. Gravois et al. demonstrated that techniques for identifying themes directly from audiotapes could be just as reliable as coding from transcripts. Although some have questioned the methodology and research design employed in this study (see Lapadat & Lindsay, 1999), identification of themes from audio recordings at least shows promise for projects that wish to capture nuance and details in the qualitative data while still producing rapid results.

Following in the footsteps of Gravois et al. (1992), we propose a method for the rapid identification of themes from audio recordings (RITA) that can be used in tandem with REAM and other projects requiring rapid results. This approach has the time and economic advantages associated with analysis of field notes, but like analysis of verbatim transcription maintains the benefit of identifying themes with a moderate level of detail based on all of the verbal information in the interview. As described subsequently, RITA involves looking for selected themes within prespecified time segments of an audio recording, thus sacrificing some nuance for expeditious results. Additionally, identifying themes from interview audio recordings preserves subtle changes and variation in participants’ nonverbal information (e.g., intonation and disposition) that is lost in transcription and more difficult to capture in fine-grained detail in field notes (see fifth column of Table 1). Notably, RITA also provides the opportunity to capture the valence of particular themes. Specifically, researchers and evaluators using RITA can optionally record whether themes are mentioned in a positive, negative, or neutral context

Procedures for RITA

Here, we outline five steps of RITA that researchers and evaluators can use to obtain rapid results from qualitative data (see Figure 1). In addition, we illustrate how our research team employed RITA in the analysis of qualitative semi-structured interview data.

Figure 1.

Steps in rapid identification of themes from audio recordings (RITA).

Case Example

Interviews were conducted with 18 public school administrators regarding how they locate information about and decide to use instructional, health, and social skills programming in two Michigan districts. Our aim was to understand how these decision processes varied across different types of school districts. The two school districts examined in this study varied in size, geographic location, and demographic composition. Specifically, one school district was small (i.e., enrollment of 2,500 students), rural, served a primarily White student body and was academically high achieving. The other school district was large (i.e., enrollment of 12,500), urban, served a diverse student body and struggled academically. Among the school administrators participating in interviews, 74% were female and 80% were White non-Hispanic. The length of school administrators’ employment in their current school district ranged between 3 and 41 years (M = 20.4). Interviews lasted between 31 and 83 min (M = 55.8) and were conducted by a trained interviewer at a location of the participant’s choosing. To inform the research design of a grant application for a follow-up study with a fast-approaching deadline, our research team required rapid, high-quality feedback from these extensive and detailed interview data.

Step 1: Specify Key Research/Evaluation Foci

The abundant amount of data provided by qualitative methods like semi-structured interviewing is both a strength and a challenge for researchers and evaluators interested in REAM. To avoid “drowning in the data,” RITA begins with the task of pinpointing and specifying key research or evaluation foci for which the rapid identification of themes will be used to obtain preliminary answers. It is important to note that the research or evaluation foci specified should not entail all aspects of the qualitative data that can be analyzed but instead should reflect only those core aspects for which it is important to have rapid feedback.

In our illustrative example, we were interested in including preliminary data in our grant application on two major foci of the program decision-making processes of school administrators: (1) the sources that school administrators use to access information about school programming and (2) school administrators’ attitudes about and use of research evidence in their decision making about school programming. Although we were also interested in many additional questions for our larger study, and although our qualitative data could be used to gain insight into many of these other questions, we maintained a narrow focus on these two foci for our rapid identification of themes.

Step 2: Identify Key Themes and Create a Codebook

After specifying key research or evaluation foci, the second step of RITA entails creating a detailed codebook of key themes. Researchers and evaluators should determine themes to include in the codebook using both deductive and inductive approaches. Some themes may be determined deductively, based on the key research or evaluation foci specified in Step 1, and informed by past theory and empirical research. Additional themes may be determined inductively based on experiences during the collection of interview data, reviewing interviewers’ field notes, listening to several audio-recorded interviews, and in discussions with members of the research or evaluation team. The codebook should include clear definitions of each theme, along with inclusion and exclusion criteria where relevant. In addition to coding for the presence of various themes in the interview data, researchers and evaluators may wish to code for theme valence, that is, whether the theme is mentioned in a positive, neutral, or negative context. This can provide supplementary details for analysis about how participants think about or view a particular theme.

In our study, the first and second authors created an initial list of themes and definitions based on our identified major foci from Step 1, as well as prior empirical research, our own experiences as interviewers on the project, and listening to a subset of the audio-recorded interviews. Our initial list of themes is presented in Table 2.

Table 2.

Initial Themes for RITA.

Initial themes

1. Formal sources of research evidence

2. Informal sources of research evidence

3. Interpreting/using research evidence

4. Attitudes toward research

5. Top-down control

6. Changes in district policy structure

7. Working with researchers

8. Internal research/program creation

Note. RITA = rapid identification of themes from audio recordings.

Step 3: Create a Coding Form

Once researchers and evaluators have constructed a codebook of key themes, the third step of RITA involves creating a coding form. The coding form will allow researchers and evaluators to code for the presence and optionally the valence (i.e., positive, negative, or neutral mentions), of themes while simultaneously listening to the audio recording of an interview. The coding form is a grid with themes listed in the rows and time unit segments listed in the columns. Notably, the columns of time unit segments provide the information necessary to return to the audio files in the future for cross-referencing and verification. Optionally, coders may also wish to briefly transcribe quotations that are particularly good illustrations of themes; however, as a rapid coding procedure, these should be brief and selective. A blank template coding form that can be filled to researchers’ or evaluators’ specifications is provided in Appendix A.

In creating this form, researchers and evaluators must specify the length of the time segments that will serve as the recording unit for coding. Chunking the audio recording into uniform time segments for coding purposes ensures that the same recording unit is used across all coders and interviews to code each theme and facilitates cross-coder comparisons when seeking to establish reliability. It is important to clarify that RITA is designed flexibly to allow researchers and evaluators to select the length of time segments that best fit the goals of their project. In particular, the length of the time segment involves a trade-off between sensitivity and time: Short segments allow coders to capture small yet distinct instances of a theme, but require more time and place a higher cognitive demand on coders, while longer segments allow coding to be performed more rapidly and easily but collapse all instances of a theme occurring within the same segment into a single code.

Thus, the appropriate length of time segments is expected to vary based on many project consideration including the number and complexity of the themes coded, the level of nuance the evaluators aim to achieve, and (most importantly for REAM) the amount of time the evaluators have to analyze the qualitative data. First, researchers and evaluators should consider the number and complexity of the themes that they plan to code with RITA. It may be too cognitively demanding to use short time segments because these require coders to constantly keep their eye on the clock and simultaneously keep track of a large number of themes. Moreover, shorter time segments may not be long enough to fully capture complicated, multifaceted themes. On the other hand, if the number of themes is small and/or the themes are relatively simple, shorter time segments may be warranted. Second, researchers and evaluators should consider the level of nuance that they hope to achieve in their coding. For example, if an evaluator is aiming to make only general recommendations based on a preliminary assessment, longer time segments may suffice in identifying both the range of themes and the prevalence of each theme within the data. Therefore, researchers and evaluators need to determine in advance the level of detail needed to address the research or evaluation foci. Third, it is necessary to consider how much time is available to analyze the data. Notably, shorter time segments will increase the coding time. Establishing a project timeline as well as estimating the length of time necessary to code each interview can help identify how much time to budget for data analysis.

In our study, we initially created a coding form that split each audio interview into 1-minute segments in which coders were asked to record the valence of each theme. However, after testing our coding process (see Step 4 below), we revised the recording unit to 3-minute segments, increasing the efficiency of the coding process. Our final coding form, after Step 4 refinements described subsequently and filled in with codes by a member of our research team, is shown in Figure 2.

Figure 2.

Example of final, filled-in RITA coding form.

Step 4: Test and Refine Codebook/Coding Form Based on a Subset of Interviews

Once a codebook and coding form are created, researchers and evaluators should test and refine these instruments based on a small subset of interviews. It is helpful to have all team members code the same subset of interviews and then meet to compare results and discuss code definitions that are ambiguous or problematic (e.g., too general; applied inconsistently across coders). This is also an opportunity to add additional codes that emerge inductively during the process of coding the subset of interviews and to adjust the length of the time segments.

In our study, all four authors participated in coding a subset of two interviews to test and refine the initial codebook. Each of us recorded our codes on the coding form and met to review and discuss coding discrepancies. In this trial coding, we determined that the initial codes for sources (i.e., formal sources of research evidence and informal sources of resource evidence) were problematic for two reasons. First, we noticed that participants often sought sources of information on programs that would not qualify as research evidence (e.g., a concerned parent). Second, the two categories of sources (formal/informal) were too coarse-grained and did not offer enough detail about the types of sources of information school administrators utilized. Thus, we revised these themes by creating several specific subcategories of sources of information that could be coded. We also added several additional themes that emerged inductively. For example, team members noted that participants often mention costs associated with using research evidence, so this was added as an explicit theme to be coded. Table 3 shows our final, refined codebook.

Table 3.

Final, Refined Codebook With Definitions.

Sources for information about programs—Any reference to looking for or obtaining information about school-based programs from the following sources. If the interviewee identifies a specific source by name, briefly note the name in the coding sheet cell.

Federal/State Government (e.g., Department of education)

Intermediate School District (ISD)/Professional Organization (e.g., via a teacher in-service/training)

Other school (e.g., “I heard about this being used in district X”)

Past personal experience (e.g., “I used to do this when I worked at district Y”)

Parent, neighbor, etc. (e.g., “A parent told me about … ”)

Internet (e.g., “We googled ‘bullying’ and found … ”)

Ivory tower (Academic personnel and documents [e.g., faculty and journal articles])

Internal (e.g., collection/analysis of data by the school districts itself)

Other (record all other sources and unspecified sources here)

Using research evidence

Any references to the use, evaluation, and/or interpretation of research evidence. This code also includes comments about what makes research evidence useful or not useful and comments about barriers/successes in implementation of research evidence

Attitudes toward research

Any comments that convey an attitude (positive or negative) about research or research evidence, whether that attitude is held by the interviewee or is attributed to others

Top-down control

Any reference to top-down or formal bureaucratic control. Top-down control may be exercised inside the school (e.g., a principal controlling teachers) inside the district office (e.g., a superintendent controlling teachers), outside the district office (e.g., political pressure from the school board), or outside the district (e.g., local/state/federal government)

Bottom-up control

Any reference to bottom-up control or delegation of control to lower levels. Bottom-up control may be exercised inside the school (e.g., teachers guiding a principal’s program selection) or inside the district (e.g., principals guiding a superintendent’s policy making). Bottom-up control also includes efforts to empower those in lower levels and to obtain their buy-in

Changes in district policy/structure

Any reference to changes in district policy and/or structure. This may include:

Changes in staffing or in staff responsibilities

Changes in school structure (e.g., moving grade levels and school closures)

Changes in focus areas (e.g., from reading to math and from obesity to drug use)

Costs

Any reference to the costs, either economic or time, associated with the acquisition, use, and implementation of research evidence. This includes cost as a barrier or facilitator to the adoption of evidence-based programs

As mentioned earlier, while testing our codebook, all team members struggled with the initial 1-minute time segments employed as the recording unit in the coding form. The number and complexity of the themes in our codebook played a large role in this struggle. First, our initial codebook contained several different themes to code. We found that 1-minute time intervals coupled with the relatively large number of themes hampered the natural flow of coding crucial for rapid assessment. We found ourselves constantly watching the clock for the end of each time segment, a cognitively taxing process. Second, many of the themes in our codebook were designed to capture relatively complex processes (e.g., interpreting/using research evidence and changes in district policy/structure). In our data, we found that 1-minute segments were often too brief to fully capture respondents’ complete ideas and to determine whether or not these more complex themes should be coded as present. This is a common issue in the coding of qualitative data with Singleton and Straits (2010) noting that smaller recording units “may not be sufficient to extract the meaning of the message” (p. 422). Specifically, because our respondents tended to give lengthy answers to interview questions, 1-minute time segments often ended right in the middle of a respondent’s conversation turn. Thus, the mention of themes often overlapped across multiple 1-minute time segments, leading to confusion and inconsistencies in the coding process.

The team subsequently tested 3-minute time segments, and after reviewing and discussing coding forms, decided that the 3-minute segments led to fewer mentions of themes that overlapped time segments and were less cognitively taxing. Although the choice to use 3-minute segments instead of 1-minute segments as recording units may have sacrificed some nuance, the team agreed that the gains in coder consistency and efficiency offset this concern. Analysis of the codes (described in Step 6 subsequently) demonstrates that 3-minute coding segments could still be used to identify the occurrence of themes within and across interviews.

Step 5: Coding

After refining the codebook and coding form, the research or evaluation team can begin the process of coding all interviews. To increase the validity and reliability of the coding process, all coders should be trained to ensure that they understand the codebook definitions and the procedures for coding by time segments. The presence of themes can be coded in each time segment using check marks, while valence can be optionally coded with + (positive), − (negative), and 0 (neutral) symbols, respectively. Because the time segment is the recording unit, coders are only asked to determine whether or not a given theme occurs in a given segment and not to determine how often it occurs. A subset of interviews should be coded by at least two coders to ensure reliability.

In our study, the third and fourth authors coded the interviews and employed +, −, and 0 symbols to code positive/negative/neutral mentions of themes in 3-minute time segments (see Figure 2). As noted previously, by separately coding positive/negative/neutral mentions, we were able to capture the context in which the theme was mentioned. Example quotes for positive/negative/neutral mentions of “Attitudes toward Research” are provided below:

Positive: “I like that stuff, my undergrad degree was in psychology, um, and so psychometrics and stuff is really neat to me.”

Negative: “You’re not walking the same hallway I’m walking, ya know. So I think the best thing is spending the time, you just gotta really spend the time, have the interaction with the teacher, with the kids to really get any true research. You go in and do a questionnaire or a one shot, maybe three times a year, to get something. I just don't know how good that is.”

Neutral: “Well, I think it's been a transition to, to, you know for 10 or 15 years probably to starting to really think about [research]”.

On average, rapid coding procedures were very efficient, requiring an average of only 13% more time to code than the length of the audio recording itself. Thus, for example, a 60-minute interview could be coded in about 68 min. Moreover, comparison of a subset of three interviews coded by both the third and fourth authors revealed that coding was reliable. We measured reliability using the Jaccard coefficient, which captures the probability that if one coder identified an instance of a given theme with a given valence, the other coder did so as well. The mean within-time segment Jaccard coefficients for the three interviews were .73, .76, and .79, which are all higher than the minimum of .70 recommended for exploratory qualitative pilot studies (Lombard, Snyder-Duch, & Braken, 2002). Overall, our team was able to complete the coding for all 18 interviews, totaling 17 hours of audio recording, within a 2-week period.

Step 6: Analysis of Codes

Once coding is complete, codes can be organized and analyzed to provide additional information about the prevalence of themes within and across interviews and, if positive, negative, and neutral mentions are coded, the context in which the theme was mentioned. Whether examining themes within or across interviews, the first step is to tabulate the total number of occurrences of each code, or each valence of a code, in an interview. This simply involves counting the number of check marks in each row on the coding form or if valence was recorded, counting the number of +s, the number of −s, and the number of 0s in each row on the coding form.

In some cases, examining the frequency of mentions of themes (i.e., row totals) within interview will be useful. For example, the researcher or evaluator may want to know which theme an interviewee mentioned most often or least often (i.e., which row has the highest or lowest total) or the relative frequency of one theme compared to another (i.e., the ratio of two row totals). It may be helpful to know how evenly the interviewee covered all the themes of interest, which may provide an indicator of the interviewee’s breadth (i.e., the ratio of the lowest to the highest row total). Finally, if valence is recorded, the ratio of positive valence codes to negative valence codes may provide clues about the overall tone of a given interview.

In other cases, examining the frequency of mentions of themes across interviews will offer insights into relationships between themes and interviewee characteristics. When comparisons are performed across interviews, it is important to adjust for the fact that, in a longer interview, the interviewee has more opportunities to mention theme. Accordingly, rather than focusing on the number of times a given interviewee mentioned Theme A (e.g., Tom mentioned Theme A 10 times), one should examine the percentage of an interviewee’s mention of any themes that were coded as instances of Theme A (e.g., Of all the times Tom mentioned a theme, 25% of the time he was mentioning Theme A). From these percentages, a researcher or evaluator may examine, for example, whether there is a relationship between the interviewee’s gender and the frequency with which they mentioned a particular theme or mentioned a theme with a particular valence.

In our study, RITA revealed two findings that we were able to incorporate into the research design of a grant application for a follow-up study and to report back to stakeholders in our research site. First, for each interviewee, we computed the ratio of times they mentioned seeking information from external (i.e., Themes 1–7) versus internal (i.e., Theme 8) sources and found subtle but systematic variation across the two school districts. Although school administrators in both districts relied on a mix of internal and external sources of information, the smaller, higher performing, more homogeneous district was more likely to seek information from external sources. Second, we counted the total number of occurrences of each of the “Sources for Information about Programs” themes. We found that our school administrators relied heavily on what are often called “intermediaries” but that this label masks significant variation in who these intermediaries are. That is, although the sources identified in Themes 1–6 might all be grouped under the broad umbrella of intermediary sources, multiple school administrators specifically mentioned each type of source. To be sure, these findings are largely descriptive and will require further, more detailed analysis of respondents’ comments in interviews to interpret. However, using RITA allowed us to obtain these preliminary results just 3 weeks after completing the last interview.

Conclusion

RITA holds several advantages for coding qualitative data and can be integrated with REAM approaches to yield prompt, yet detailed results. Speed is a major advantage of RITA with coding time taking an average of only 13% longer than the length of the interview,¹ compared to transcription of an audio interview which Britten (1995) estimates requires 600% to 700% longer than the length of the interview. Because the identification of themes takes place simultaneously while listening to audio recordings, RITA also retains nonverbal information like intonation and disposition, which is often lost in transcripts (Crichton & Childs, 2005; Tessier, 2012). To be sure, detailed field notes can capture some nonverbal information but, unlike RITA, they rely heavily on what the interviewer chooses to record. Finally, RITA does not require specialized software or the editing of audio files often associated with directly embedding codes in or linking codes to audio files (e.g., Crichton & Childs, 2005; Evers, 2011; Tessier, 2012).

Despite these advantages, there are certain trade-offs to RITA. Researchers and evaluators must decide how to chunk audio recordings into time segments that can serve as the recording unit for coding. Shorter time segments can capture more detailed information but require more time, while longer time segments increase coding efficiency but may lead to more coarse-grained coding of each interview. Second, researchers and evaluators must decide how many themes to code in RITA. Coding more themes decreases coder efficiency but provides more nuanced information about the qualitative data. On the other hand, coding fewer themes increases coder efficiency and speeds analysis and the utilization of qualitative data but provides less detail. Researchers and evaluators must consider the context of their own projects and weigh these different trade-offs when using RITA.

RITA may be particularly useful for REAM (McNall & Foster-Fishman, 2007) and for developmental evaluations that meet demands for evaluators to provide near-real time data that can inform the ongoing planning of a program or innovation (Patton, 2011). When using RITA, researchers and evaluators should consider employing criteria and techniques to ensure the quality of the resulting QDA (e.g., Guba & Lincoln, 1989; McNall & Foster-Fishman, 2007; Patton, 1999). For example, McNall and Foster-Fishman (2007) provide a nice case example of how Guba and Lincoln’s (1989) adequacy criteria can be applied within REAM. Following their lead, strategies like ongoing peer debriefing, discussion of possible coder biases, and keeping a written audit trail of coding decisions may be particularly helpful in verifying the trustworthiness of qualitative findings obtained using RITA.

However, RITA should not be viewed as a replacement for traditional methods of coding qualitative data from transcripts. More traditional methods of coding qualitative data offer a more holistic phenomenological understanding of participants’ lived experiences than RITA. Instead, RITA is a supplementary approach that can be used to provide rapid feedback during the course of an evaluation or research study. Given that RITA is beneficial for identifying themes within a tight time frame while still preserving information about participants’ perspectives, we believe this method holds promise for researchers and evaluators engaged in REAM and other situations that require the expeditious coding and analysis of qualitative data.

Footnotes

Appendix A

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by an Officer’s Research Award from the William T. Grant Foundation (#182241).

Note

References

Bentley

M. E.

Pelto

G. H.

Straus

W. L.

Schumann

D. A.

Adegbola

de la Pena

… Huffman

S. L.

(1988). Rapid ethnographic assessment: Applications is a diarrhea management program. Social Science and Medicine, 27, 107–116.

Britten

(1995). Qualitative interviews in medical research. British Medical Journal, 311, 251–253.

Chambers

(1994). The origins and practice of participatory rural appraisal. World Development, 22, 953–969.

Crichton

Childs

(2005). Clipping and coding audio files: A research method to enable participant voice. International Journal of Qualitative Methods, 4, 40–49.

Desmond

Allen

C. F.

Clift

Butolwa

Mzugu

Plummer

M. L.

… Ross

D. A.

(2005). A typology of groups at risk of HIV/STI in a gold mining town in North-western Tanzania. Social Science and Medicine, 60, 1739–1749. doi:10.1016/j.socscimed.2004.08.027

Evers

J. C

. (2011). From the past into the future. How technological developments change our ways of data collection, transcription, and analysis. Qualitative Sozialforschung/Forum: Qualitative Social Research, 12, 1–31.

Gravois

Rosenfield

Greenberg

(1992). Establishing reliability for coding implementation concerns of school-based teams from audiotapes. Evaluation Review, 16, 562–569. doi:10.1177/0193841X9201600507

Greene

J. C.

Caracelli

V. J.

Graham

W. F.

(1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274. doi: 10.3102/01623737011003255

Guba

Lincoln

(1989). Fourth generation evaluation. Newbury Park, CA: Sage.

10.

Halcomb

E. J.

Davidson

P. M.

(2006). Is verbatim transcription of interview data always necessary? Applied Nursing Research, 19, 38–42. doi:10.1016/j.apnr.2005.06.001

11.

Jamal

Crisp

(2002). Real-time humanitarian evaluations: Some frequently asked questions. Geneva, Switzerland: Evaluation and Policy Analysis Unit, U.N. High Commission for Refugees.

12.

Lapadat

J. C.

Lindsay

A. C.

(1999). Transcription in research and practice: From standardization of technique to interpretative positionings. Qualitative Inquiry, 5, 64–86. doi:10.1177/107780049900500104

13.

Lombard

Snyder-Duch

Bracken

C. C.

(2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587–604.

14.

Markle

D. T.

West

R. E.

Rich

P. J

. (2011). Beyond transcription: Technology, change, and refinement of method. Qualitative Sozialforschung/Forum: Qualitative Social Research, 12, 1–21 Art 21.

15.

McNall

M. A.

Foster-Fishman

P. G.

(2007). Methods of rapid evaluation, assessment, and appraisal. American Journal of Evaluation, 28, 151–168. doi: 10.1177/1098214007300895

16.

McNall

M. A.

Welch

V. E.

Ruh

K. L.

Mildner

C. A.

Soto

(2004). The use of rapid-feedback evaluation methods to improve the retention rates of an HIV/AIDS healthcare intervention. Evaluation and Program Planning, 27, 287–294. doi:10.1016/j.evalprogplan.2004.04.003

17.

Patton

M. Q.

(1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34, 1189–1208.

18.

Patton

M. Q.

(2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

19.

Patton

M. Q.

(2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.

20.

Reichardt

C. S.

Rallis

S. F

. (Eds.). (1994). The qualitative-quantitative debate: New perspectives. New Directions for Program Evaluation San Francisco: Jossey-Bass.

21.

Singleton

R. A.

Straits

B. C.

(2010). Approaches to social research (5th ed.). New York, NY: Oxford Press.

22.

Tessier

(2012). From field notes, to transcripts, to tape recordings: Evolution or combination? International Journal of Qualitative Methods, 11, 446–460.

23.

Wholey

(1983). Evaluation and effective public management. Boston, MA: Little, Brown.