Effective questionnaire design: How to use cognitive interviews to refine questionnaire items

Abbreviations

NICU

Neonatal intensive care unit

1 Background

Questionnaires are a relatively low-cost and efficient data collection method and are thus attractive for health research and educational projects. However, writing new questionnaire items can be tricky, and a questionnaire of poor quality can preclude meaningful results. Cognitive interviewing is a formal technique to evaluate whether questionnaire items are understandable to respondents and will be useful for making conclusions [1]. It is a cost-effective approach for improving questionnaire-based research and provides evidence for the content validity of items. Prior research across various disciplines has demonstrated how cognitive interviews can be used to detect issues related to clarity, comprehension, ambiguity, cognitive recall burden, timeframe, missing answer categories, inaccurate instructions, and relevance of questionnaire items [2–4]. Herein we offer an overview of cognitive interviewing methodology, drawing on our experience using cognitive interviews during the development of a questionnaire designed to assess knowledge of concepts related to preterm birth in parents of premature infants.

2 Methods

We used cognitive interviews to test and improve a knowledge questionnaire meant to assess participants’ understanding of concepts related to preterm birth. In preparation for the study, the objectives were outlined and clarified among team members: to test a questionnaire of concepts related to preterm birth, with the goal of removing or improving problematic items to ensure understandability of the questionnaire and enhance its validity. Established methods and best practices were followed to draft each item of the initial preterm birth questionnaire [5]. We developed a cognitive interview guide consisting of the questionnaire items to be tested followed by probing questions [1]. Probing questions are designed to assess the participants’ understanding of each item, the mental process used to answer, how they mapped their answer to one of the response options provided, and any judgments or decisions related to responding (e.g. social desirability). Scripted probes for each item ensured standardization of analyses across interviews. Common probes included, “Can you rephrase the question in your own words?” or “How did you decide on that answer?” The research team took care in designing and updating the interview guide to ensure that probes on earlier items did not contaminate participant interpretation of later items and ensured that sufficient time was allocated within the interview for the items of greatest concern to be probed adequately. Probing may be conducted using either the concurrent or retrospective technique [1]. The concurrent technique involves the interviewer using targeted probes immediately after the participant responds to each questionnaire item; whereas, with the retrospective technique, the interviewer asks the respondent probing questions after the full questionnaire has been completed. The concurrent technique was used for this study, probing the participant on how they understood and answered each questionnaire item immediately after they provided their answer. Prior to beginning interviews, approval was obtained from an institutional review board. Informed consent was obtained from participants, and it was emphasized with each participant that they could refuse to answer any question or stop participation at any time.

In preparation for the cognitive interviews, team members participated in training, which included a review of literature related to cognitive interviewing and an overview of best practices in the field, such as those outlined in Willis’ Cognitive interviewing a ‘how-to’ guide [6], and Boeije & Willis’ cognitive interviewing reporting framework (CIRF) [7].

Prior to beginning the study, interviewers practiced role playing cognitive interviews, and held detailed discussion with the project investigators to understand the intent of each questionnaire item. Novice team members were observed by experienced team members and received feedback to improve their technique.

Participants were recruited from a NICU (six participants) and a high-risk OB clinic (four participants) to obtain a sample similar to our population of interest. However, to obtain a sample naïve to the topic of the questionnaire items and responses (knowledge of prematurity), we excluded parents of infants who had been admitted for issues of prematurity. We aimed to reflect the diversity among our target population by interviewing mothers and support-persons, aiming for racial diversity and a range of literacy levels. Because as few as four interviews may be sufficient to identify problematic questions [8] our goal was for each item to be reviewed by at least five parents, at least two of whom had less than a high school degree or less than a ninth grade reading level. This purposeful sampling promotes a heterogenous sample and helps achieve a type of sensitivity and specificity [8].

Participants provided written consent and received an incentive of $50. Each interview lasted approximately one hour and was conducted in person in a quiet and comfortable private space. To encourage uninhibited feedback, interviewers distanced themselves from the investigators by using phrases, as suggested by Willis, such as: “I didn’t write these questions, so don’t worry about hurting my feelings if you criticize them. It’s my job to find out what’s wrong with them” [6]. For each interview, one team member led the interview while an additional team member took notes. Four members of the team participated in the interviewing and note take process, one who only interviewed, one who only took notes, and two who did at different times did both. Interviews were recorded but not transcribed. Participants were provided a paper copy of the questionnaire and directed to silently read each item, mark their answer, and then read the item aloud along with their chosen response. The interviewer asked probing questions to elucidate the participant’s comprehension of each item before moving on to the next. The interview notes were compiled in a spreadsheet organized by item, which served as the primary source of data.

The full research team- representing expertise in neonatology, nursing, survey research methods, sociology, and psychology [9], met after each set of five interviews was conducted and took a reparative approach for analysis of the results. Careful inspection of each item and response allowed identification of “dominant trends” (problems that repeatedly emerged), as well as “discoveries” (problems that may have emerged only in one interview but still posed a threat to valid data) [8]. The team collectively decided the best way to improve flawed items to reduce response error. Substantially revised items were tested in additional interviews.

3 Results

During the cognitive interviews, issues emerged that led to the revision of certain items. For the sake of brevity, results from every individual questionnaire item are not systematically described here, but a selection of questionnaire items that were revealed to be not-well understood or otherwise problematic are described in Table 1. For example, multiple participants interpreted the phrase “at risk” as a fact or expected outcome, rather than a potential outcome. We clarified questions that used the phrase “at risk” by adding a concrete comparison. For example, the questionnaire item: “A baby born before 25 weeks of pregnancy is at risk of having problems learning due to prematurity. True or false?” was revised to: “Compared to a baby born after 37 weeks, is a baby born before 25 weeks of pregnancy more likely to have problems learning?”

While many of our original items were true/false statements, through the cognitive interviews we discovered that some participants found this presentation unhelpful, even confusing. Thus, the team revised some questionnaire items to a traditional question format with a yes/no answer, instead presenting a “true or false” statement that the participant was asked to evaluate.

During the cognitive interviews, there were multiple instances when the participant gave the right answer, but probing revealed it was for the wrong reason. For example, when presented with the true or false statement: “Most women who go into preterm labor will deliver within the next week”, the correct answer would be false (only one in ten delivers within a week). However, one participant interpreted this to mean preterm labor could last a week and responded: “I hope not. I’d say false. I’d hope it would be within the next day or two.” While she provided the correct answer (false), probing revealed that meaningful understanding of the topic had not been achieved. To clarify, the question was revised: “Most women who go into preterm labor will deliver within the next day or two”. The answer remains false; however, the confusion that the preterm labor lasts for one week was resolved.

The overarching goal in our research is for parents to have critical information about preterm birth in order to participate in shared decision making, so understanding their underlying thought process for selecting a response was essential. For example, learning that some participants may have selected the right answer for the wrong reason allowed the research team to recognize the participant may not have fully understood the concept, thus undermining their ability to leverage the information when making healthcare decisions for their child. Cognitive interviews highlighted problems with our questionnaire items and allowed us to revise or eliminate unclear items and gain confidence in using them as an outcome measure. The main limitation of our study was that participants came from a single institution and may not represent the whole population, though the sample size was appropriate for a qualitative study, [6] and it is generally assumed that even a modest sample size will expose the most critical problems [8].

For studies collecting data through questionnaires, the validity of the questionnaire items is crucial. We recommend that researchers (including student researchers) developing new questionnaire items utilize available resources [1, 8] to design and complete cognitive interviews, thus improving their items and increasing confidence in study conclusions.

Disclosures

None.

Conflict of interest

The authors have no conflicts of interests to disclose.

Funding

Funding for this study came from R21 HD092664 from the National Institute of Child Health and Human Development (NICHD).

Role of funder

The NICHD had no role in the design and conduct of the study.

Human research statement/IRB statement

Approval for this study was obtained from the IRB at the Medical College of Wisconsin, Milwaukee, WI, USA.