Pretesting Qualitative Data Collection Procedures to Facilitate Methodological Adherence and Team Building in Nigeria

Pretesting Qualitative Data Collection Instruments

The practice of pretesting is highly regarded as an effective technique for improving validity in qualitative data collection procedures and the interpretation of findings (Bowden, Fox-Rushby, Nyandieka, & Wanjau, 2002; Brown, Lindenberger, & Bryant, 2008; Collins, 2003; Drennan, 2003; Foddy, 1998). Supported by the very nature of qualitative research as an iterative rather than a linear process, the pretest interaction to self-correct between design and implementation ensures the best opportunity for attaining reliability and rigor in qualitative inquiry and analysis (Morse, Barrett, Mayan, Olson, & Spiers, 2002). By definition, pretesting involves simulating the formal data collection process on a small scale to identify practical problems with regard to data collection instruments, sessions, and methodology. The value of pretesting can lead to detecting errors in cross-cultural language relevance and word ambiguity, as well as discovering possible flaws in survey measurement variables. Pretesting can also provide advance warning about how or why a main research project can fail by indicating where research protocols are not followed or not feasible. A typical pretest in qualitative research involves administering the interview to a group of individuals that have similar characteristics to the target study population, and in a manner that replicates how the data collection session will be introduced and what type of study materials will be administered (consent forms, demographic questionnaires, interviews, etc.) as part of the process. Pretesting provides an opportunity to make revisions to study materials and data collection procedures to ensure that appropriate questions are being asked and that questions do not make respondents uncomfortable and/or confused because they combine two or more important issues in a single question. It is vital that pretests be conducted systematically and include practice for all personnel who will be engaged in data collection procedures for the eventual main study.

Depending on the type of qualitative interview employed in a study, there are different roles and tasks to be performed. For example, focus group discussions may require a moderator and a notetaker, given the often loose and free-flowing dialogue that is challenging to direct and fully observe. In contrast, an individual in-depth interview places a greater burden on the participants to explain themselves to only one interviewer. In either case, the validity of the qualitative data rests solely on the ability of the moderator/interviewer to produce focused amounts of data on precisely the topic of interest within a reasonably tolerated period of discussion time. If problems arise in the pretest interview, it is expected that similar challenges will arise in the administration of interviews during the formal study. Projects that neglect pretesting run the risk of later collecting invalid and incomplete data. But, completing a pretest successfully is not a guarantee of the success of the formal data collection for the study. Although qualitative pretest findings may offer some indication of the response patterns anticipated in the final data collection phase, they cannot guarantee this given the lack of theoretical saturation and variation of data collected in a pretest in comparison to recruitment projections for the design of a main study (Bowen, 2008).

The specific areas assessed during the pretests are outlined by the following categories listed below. Some criteria were added to the final selection as a result of the outcomes and transition from Pretest 1 to Pretest 2. The following are the main selection criteria used for review to assess the rigor and relevance of our instruments and procedures:

Evaluating Language Competency and Content Validity of Data Collection Materials

Communication competence in the translation between two languages involves both linguistic discourse, as well as sociolinguistic competence (Lindlof & Taylor, 2010). Sociolinguistic competence includes the way language is used in context, including cultural norms and expectation of words or phrases, which can greatly affect how accurately research participants are interpreting and responding to interview questions. It has been stated many times that a study is only as good as the data that is collected. This is especially critical in qualitative research where the aim is to capture an in-depth comprehension of participants' beliefs, emotions, perceptions, and experiences.

One of the first practical considerations needing attention was the cultural relevance and translation accuracy of our data collection materials, all of which were originally created by one of our co-principal investigators for a U.S. study of Spanish-speaking elderly stroke patients. Our Nigerian researchers reviewed all individual interviews with various health-care professionals and hospital administrators for English clarity, which is also the primary language of Nigeria. They also translated focus group scripts into Yoruba to support the common language spoken at home by the majority of our patients and caregivers. Even when diverse cultures share a common language, the local mind-set of each culture enables different ways of thinking and different ways of naming and interpreting objects, causes, and events in their environment. Thus, the first pretest was designed to determine if respondents interpreted the research questions with the same connotative meaning as was intended in the original research. Equally important was the need to resolve how best to stage the wording of a question if the range of responses from pretest participants was highly variable and seemed to suggest a lack of conceptual equivalence.

A preliminary site visit was scheduled at the conclusion of the first pretest trial to provide for a qualitative team review of pretest data collection efforts and to begin integrating team relationships at the research site. The visit was also an excellent opportunity to familiarize the visiting U.S. investigator with first-hand exposure to local Nigerian culture, the context of clinic and hospital facilities, and locations where study activities would take place. Reviewing the pretest interviews additionally presented a means for the U.S. investigator to assess skill levels or experience gaps in the local Nigerian team interviewers who had carried out the pretest interviews. The site visit also presented the opportunity for determining ways to unify and strengthen the knowledge of our entire qualitative team before official data collection began.

Confirmation of our initial concerns relating to the cultural relevance of our interview topics emerged in the responses of health-care professionals and hospital administrators in which they were asked their opinion about utilizing a nurse practitioner for stroke care coordination. Although Nurse Practitioners (NP) have been delivering primary care for nearly fifty years in the United States (Landau, 2011), in Africa the NP movement is barely gaining ground. As it happened, the question has little social or medical relevance to the current Nigerian nursing profession. Rather than delete the question, our team determined to reconstruct the wording to focus on current expectations of nurses in Nigeria and the extent to which healthcare professionals and hospital administrators would be in support of general nurses taking on greater responsibility in stroke care coordination. This new question was presented in the second pretest trial, and substantive feedback was obtained to endorse the new version of the question for use in the Phase 1 interviews.

Another question, directed to all interview participants asked about the importance of a “group clinic” for stroke survivors. In this instance again, the westernized concept of the function of a group clinic is not operationalized in our Nigerian study locations. In responding to the question, caregivers and patients confused the notion of group clinic with the concept of support groups. During both the first and second pretest trials, the patients and caregivers were unanimous in their interest in support groups, but continued to misinterpret the activity as one that would bring them increased contact and communication with their physicians. Much of the persistent misunderstanding of this question was also intensified by the confusion of our Nigerian interviewers over the semantic differences between these terms. In this case, both participants and our Nigerian researchers were drawing upon a tacit knowledge of culture in Nigeria and these items did not fit in their frame of reference. Once we identified the challenge, we directly sought counsel with our co-principal investigators to explore the intended meaning of the terms, as well as the relevance of the question to our research goals. This incident also encouraged the qualitative team leaders to update training objectives and consider additional and more didactic means of maximizing skills and knowledge for interviewers prior to conducting the main study interviews.

Although there are straightforward qualitative techniques available to assist investigators in evaluating the content validity of interviews, it was not feasible for the research team to explore these methods during pretest trials, or integrate them during the main Phase 1 interviews. The major concern involved the burden of time already reflected in completing the entire set of data collection materials during the pretest trials. One technique is known as concurrent think-aloud interviews in which respondents literally think aloud when answering questions and responses are probed. Although the process is fairly simple, it allows interviewers to clarify meaning, decode idiomatic words, make personal connections, question the respondent for greater detail, and summarize what has been said. The other approach is retrospective think-aloud interviews where respondents are asked how they arrived at their answers (Campenelli, 1997; Gambier & Doorslaer, 2010; Young, 2005). An important measure that was integrated for all interviews involved the inclusion of summary notes by the moderator for individual interviews and a note-taking partner for the focus group discussion moderator. The summaries and notes were designed with the intention to document some of the nonverbal behavior of participants that could provide important information about any embarrassment or discomfort experienced with the content or specific wording of questions in the interview script. We anticipate the use of these techniques to have broader implications for the actual Phase 1 interviews.

Estimating Time Length of Full Interview Delivery and Marking Periods of Respondent Fatigue

Respondent fatigue is a well-acknowledged phenomenon in qualitative research that occurs when interview participants either become tired during the task, or when the interview process is too lengthy and the quality of data that interviewees provide begins to deteriorate (Rubin & Rubin, 2011). Expected problems also include those relating to the physical and mental health status of elderly participants or participants with a physical or speech disability. In addition, this phenomenon may occur simply because participants' attention and motivation drop, typically toward the later closing sections of an interview.

The interview environment can also exacerbate problems if it is impossible to secure a room that enhances adequate rapport and continuity as the interview is being conducted. If there is too much background noise or personal distraction, or in some cases if respondents are placed in an area that is crowded, cramped, or windowless, it is probable that participants (particularly in a focus group) may feel threatened or uneasy about “opening up” in front of strangers in the room. In such a case when respondents become tired or disengaged, they will more frequently begin to answer questions with responses such as “I don't know.” Again, in the case of focus group discussions, respondents who become disengaged are more likely to choose answers similar or identical to the respondent before them, or potentially give up answering the questions altogether. Therefore, when respondents demonstrate fatigue, distress, or disconnection while completing an interview, research investigators must look for ways to either shorten the duration of the interview or provide for a respite in between the presentation of other data collection materials.

An important component of our pretest trials was to determine not only the total length of time needed to conduct the individual and focus group interviews but also concerns over the numerous confidential matters covered in the interview that could potentially affect complications of emotion, fatigue, and rapport. With regard to hospital administrators and health-care professionals, our pretest challenge was to keep their office interview sessions limited in time, so as not to interfere with other “on-call” hospital or clinic obligations, yet to aim for determining a standardized length of time that met all criteria of the data collection materials in order to secure a maximum standard response from each participant. Attention was given to the sequence of administering each of the data collection items, but most frequently the approach remained routine in moving from informed consent to demographic survey, followed by the main interview script. Careful observation and documentation of the onset of fatigue, distress, or disengagement from the interview, and the total time required to complete the sections of the data collection materials were of great learning value to our qualitative research team in determining the final strategy for data collection procedures, which were practiced again by interviewers during a second session of group and didactic training just prior to initiating the Phase 1 interviews. In the second training session, led by the U.S. qualitative expert, a new technique was introduced to the interviewers to “tag” sections of the interview script that were similar if not nearly identical in topic to later questions listed in the guide. The interviewers were then coached to actively listen for potentially unplanned but valuable information provided early in the interview that actually applied to additional inquiries found midway or toward the end of interview. Devising this “cue system” within the interview script greatly enhanced the Nigerian interviewers' familiarity with the content of the document and strengthened their confidence and understanding of when and how to probe for greater details that could avoid redundancy in repeating questions that, in essence, had already been asked and answered in previous sections.

Concerns over respondent fatigue and interview locations for patients and caregivers resulted in somewhat different considerations and strategies for improvement. All pretest focus group interviews were held at a hospital on clinic day, which carries a certain amount of distress for patients and caregivers alike. In scheduling the pretest interviews, the strategy was to recruit around patient medical appointments, which in Nigeria are commonly scheduled on a walk-in basis as patients can only sign in for services once they arrive at the hospital. This practice proved challenging as patients and caregivers were reluctant to give up their spot on the wait list if focus groups were scheduled too early in the day. When interviews were scheduled later in the afternoon, the patients and caregivers were often too weary to stay a longer time in order to participate. Our research team concluded that the best solution in these circumstances would be to offer patient and caregiver interviews simultaneously so that neither group was detained longer than the other, and patients were not left without supervision and social support such as if interviews were scheduled back-to-back. To assess respondent fatigue or anxiety during the focus group interview process we assigned and instructed several of our interviewer team to serve as focus group notetakers. The notetakers were trained to carefully assess for the causes and consequences of any visible respondent fatigue or emotion and document the observed events and when they appeared in the interview process. A unique aspect of the role of both the notetaker and moderator was that all focus group interviews were conducted in Yoruba and, therefore, with communication needs met, our team felt confident we were likely to accurately capture the implied meanings, signals, and other cultural determinants that might explain problems or improvements needed in rapport. As such, few issues occurred during either pretest trials. We noted that patients were often more eager and cooperative for discussion when compared to their caregivers, who were frequently fatigued or emotionally drained from the onset of the interview.

Maximizing Methodological Skills and Achieving Proficiency Standards for Qualitative Data Collection

The importance of qualitative research training cannot be overstated in team-based projects, particularly in research collaborations that unite developed and developing countries. One can expect that the greater the cultural difference, the greater the potential for poor communication and misunderstanding among team members. Thus, it is imperative not only to initiate team training from the very beginning of a project but also to revisit ways to maximize team skills through well-designed practice-based training sessions and scheduled debriefings at regular intervals throughout the entire data gathering process. The motivation for team training should also reflect enhancing team dynamics as an essential element in building team capacity. Participation in training, moreover, should be considered a requirement for all team members regardless of an individual's level of expertise or role in the study, and every effort should be made to allocate funding for training needs in the project budget. The entire team's involvement in training ensures that all members possess equivalent knowledge of the study objectives, as well as expectations in performance of the methodologies designed to conduct the research.

Following our first pretest trial, we began a review of our study protocols and our team members' strengths and qualifications related to the goals of the project. A number of procedural mistakes were made during the first pretest, despite the assumption that our interviewers had become well familiarized with the data collection materials, as well as the expectations for producing “standardized” verbatim research transcripts. To lay the groundwork for correcting our mistakes and preparing for a second pretest trial, the first site visit to Nigeria was scheduled so our team members and U.S. and Nigerian qualitative experts could meet. This visit was productive in quickly dispensing with awkward interactions of working on a team with people who had never met, and resolving potential differences in understanding about the standard conduct of research and methodological practice that typically exist between developed and developing countries. The site visit provided a positive face-to-face, two-way learning exchange supported through candid communication. As a result, many recommendations evolved from the week- long meetings, which strengthened both the preparation for the second pretrial and future data collection in the main study.

The first revisions proposed by our team were logistical and focused on identifying three additional interviewers to make up a team of six staff and to organize a formal training prior to when the second pretest interviews commenced. For the training, a series of handouts and instructional materials were designed to provide tips and directions for improving technique and accuracy in all data gathering procedures. The training involved only the Nigerian members of our team, and our resident qualitative research investigator in Nigeria provided the instruction. The training agenda was developed to guide interviewers through a thorough review and hands-on rehearsal of the interview script to ensure that everyone understood the purpose of each question, as well as the presentation of the intervention patient report card, which had gone missing in the first pretest trial. The instructional materials for the training were created in a stepwise sequence according to data collection procedures, with the goal of being easy to follow and refer back to for any future questions during the main study. Surprisingly, the fact that some members had more experience and proficiency than others made for continued challenges and mishaps even after the training, as noted in the second pretest trial where the interview styles varied widely in data collection, data processing timelines were significantly delayed, and many items such as focus group notebooks were never properly word processed as a final step for eventual analysis.

To resolve the continued procedural issues an additional 2-day training workshop was scheduled for the Nigerian team to include a second visit from the U.S. investigator so that the entire qualitative team was present. The workshop included both group and didactic training sessions led by the U.S. qualitative expert, and the original training handouts and materials were updated, put in manual form, and submitted as part of the archived study documentation. Aside from training, the workshop prioritized getting to a better place in understanding the needs and feelings of our interview team. Research investigators encouraged the interviewers to be a voice in the shared resolution of our data collection hurdles. Several of the interviewers, who considered themselves to have more experience, expressed resentment and resistance to the stipulated training procedures and requirements, not understanding that the approach for our current project dictated implementing more rigorous global standards of performance. These issues required tactful negotiation and presentation of the reasons for raising the bar to the highest of international collaborative standards, based on the expectations of our global consortium funders. To further encourage the objectives, goals, and outcomes of the second training session, the U.S. qualitative expert traveled with the entire interview team through the completion of the Phase 1 qualitative interviews to ensure proper coaching, supervision, and positive support.

Assessing the Feasibility and Fidelity of Translation and Transcription Protocols in Preparation of the Interview Text for Qualitative Analysis

Study protocols for the translation of interview recordings from Yoruba to English (for patient and caregiver interviews) were designed with great caution to avoid several key challenges that can be present when translating semantic content and structure across major language groups. Problematic issues that researchers encounter may include words that have no equivalent translation, or words that have semantic equivalence but represent slightly different epistemological concepts. These conditions create the possibility for a different response than the intended meaning, thus creating an unnecessarily complicated or awkward text. This situation can influence translators to either leave their translation ambiguous or decide on a single interpretation and translate all interviews accordingly. To correct for this confusion, it is advisable to include written notes and memos of the translation process. In this way, the tangible elements of notes and memos become a potential audit trail and “code-able” document of the formal study to record decisions made by different translators in choosing particular phrases and the reasoning behind them. It is inevitable that the process of translation adds some level of interpretation to the final transcript, which makes a “memo log” all the more necessary to capture meaning-based translations to assist in data analysis (Esposito, 2001; Temple, 1997; Temple & Young, 2004).

The goal of achieving semantic equivalence in translation is far more challenging when one is working with phrases and words that describe attitudes, emotions, and opinions because these ideas are more abstract in context. A unique example of semantic misunderstanding was noted during the first pretest interviews. The phrase “excessive thinking” emerged several times in focus group interviews as a perceived cause for why strokes occur. During a team review of Pretest 1 transcripts, we determined that many patients and caregivers used similar expressions involving closely related terms such as stress, trauma, and anxiety to describe their experiences and explanatory models for the cause of stroke disease. In an effort to conceptualize points of difference, our team explored the translated concept of “excessive thinking” and dissected all possible differences of understanding that might be represented by the phrase. In English we might interpret the description closely to the emotion of brooding or fretting. In contrast, considering the Yoruba worldview, the phrase was interpreted as obsessive, compulsive worry, which many of the respondents more explicitly considered to be the “devils work.”

Although our translation protocols originally planned for the careful execution of back translated copies to compare the original Yoruba version with the translated Yoruba–English–Yoruba transcript, repeated efforts at back translation during the pretest trials proved to be quite difficult. Noted variance based on dialect differences between our bilingual interviewers, not to mention the exceedingly time consuming task it created compared to all other procedures in data collection, forced us to rethink our attempts to produce lexical equivalence between the source, target, and back translation. Instead, at the suggestion of our Nigerian research team, we elected to eliminate our focus on bidirectional translation equivalence and make use of our bilingual interviewers not only as translators, but also as cultural informants to increase our understanding of the potential lexical variation used by Yoruba speakers. Using documented memo logs of the original translation process of Yoruba focus group interviews, our revised translation and transcription protocols now make use of cross proofreading of scripts and memo logs by all members of our bilingual interview team to enrich and document all potential differences in interpretations. This protocol will be used with our formal Phase 1 data collection procedures.