An Innovative Approach to Pretest Questionnaire: The Analysis of Respondents’ Comments in the Flexible Interview

Standardized Versus Flexible Interview

In the behaviorist tradition, the comparability of answers is the highest priority to achieve through the standardization of the interview situation. According to the behaviorists, answers are comparable when all questions are read to all respondents in the same form and order. Interviewer has to read question as it appears in questionnaire and handle the interaction with respondent neutrally, without suggesting one specific answer. Interviewer has to respect the order of questions strictly. Neither questions nor answers can be commented by interviewer, and the tone of voice has to be formal and professional. If respondent asks for help to understand question correctly, interviewer has to reread it as it is written, without adding comments or information not provided by researcher. If respondent asks interviewer “What do you think about it?,” interviewer has to stress the different roles they have: interviewer asks questions, respondent answers ¹ (Cannell and Kahn 1957).

In other words, interviewer has to apply the instructions provided by researcher during the training because the comparability of answers is guaranteed by the invariance of administered stimuli. The lack of uniform stimuli administered by interviewer (in reading the questions, in handling the interaction with interviewees, and in recording the responses) risks to bias the data quality ² : If a question is read unlike researcher expects, this question can be interpreted by respondent unlike researcher intends. To avoid it, researcher standardizes interviewer’s behavior in order to standardize question meanings transmitted to all respondents (Dykema, Lepkowsky, Blixt 1997; Fowler and Mangione 1990).

In such a picture, “the stimulus–response mechanical model” is the best theoretical reference to apply to the standardized interview because it responds to the behaviorists’ needs to control interviewer’s behavior and minimize the interaction with respondent (Bradburn and Sudman 1979). The basic assumption of the model is as follows: one stimulus generates one—only one—response; if the form of stimulus changes, response changes. In the standardized interview, one question produces one—only one—answer; if the formulation of question changes, answer changes. To guarantee the link between “preformulated” question (stimulus) and “true” answer (response), interviewer has to read question as it is formulated by researcher. In fact, using the behaviorist terminology, an answer is “true” when interviewer reads question as it is formulated by researcher: This behavior assures that respondent interprets question as researcher intends. An answer is “false,” by contrast, when interviewer reads question as it is not formulated by researcher: this behavior prevents respondent from interpreting question as researcher wants. According this point of view, the change in question wording implies the change in question meanings transmitted to respondents (Biemer et al. 2004). In order to avoid that the strength of the connection between the “preformulated” stimulus and “true” response is weakened, the behaviorists reduce the interaction between interviewer and interviewee to a stimulus–response mechanical sequence, when interviewer’s and respondent’s behaviors do not deviate from the ideal sequence (Sykes and Collins 1992).

The phrase “ideal sequence” was born from this strict perspective of the interviewer–respondent interaction. There is only one acceptable sequence: Interviewer reads question verbatim, respondent gives a valid answer, and interviewer records the right answer and goes to the next question. Any behavioral deviation ³ is an indicator of measurement error, which must be studied and, if possible, erased before the survey starts (Sykes and Morton-Williams 1987). ⁴

The apparent confidence in the possibility of administering questions standardized in form and meaning comes from the assumption that the joint between the sphere of concepts (meanings) and the sphere of language (words) is hard. Following this thesis, one word has only one meaning, and this connection is eternal and remains valid for all and forever; it cannot be questioned. But this connection has already been questioned (Tyler 1969). Stating that “a word is a carrier of only one meaning,” and that “the link between language and sense is firm,” is equivalent to regress decades in the epistemological theories (Marradi 2007:36).

Nowadays, the flexible nature, never firm, between the sphere of concepts and language is widely assumed: the same term can be interpreted by people in various ways (Eco 2008; Montesperelli 2014; Sartori 1984). As Peirce wrote, “there is not any guarantee that two people, born and grown in the same linguistic community, connect one word with only one meaning similarly, and vice versa” (1931:126). The cuttings of the same conceptual slice of reality can be different among people, and there is not any necessary relationship between what we think (the meanings that researcher wants to transmit) and words (questions) that we use to express it (Violi 1997). Furthermore, the meaning of a term varies, depending on both the communicative needs of the speaker (Akmajian et al. 1990) and interpretative demands of the recipient (Schober and Conrad 1998), who work interactively to encode and decode the message. In all forms of communication, meanings are fluid and redefined whenever social actors want, as an ongoing interactive process between the speakers and the listeners (Schutz 1932).

Many scholars, who explicitly criticize the dogma of the invariance of administered stimuli, are very sensitive to the relational character of the interview, to the point that their scientific contribution have led to the development of many other types of nonstandardized interview: the hermeneutic interview (Montesperelli 2001), the biographical interview (Bichi 2002), the cognitive interview (Willis 2005), the focus group interview (Kamberelis 2013)…to name just the very well-known.

In this article, I focus the attention on “the conversational interview,” conceived by the scholars who highlight the resemblances between interview and ordinary conversation (Houtkoop-Steenstra 2000; Maynard et al. 2002; Ongena and Dijkstra 2007; Schober and Conrad 2002).

Despite undeniable reciprocal differences (Suchman and Jordan 1990), standardized interview and conversation have a common element: “the interactional substrate” which people use to repair local misunderstandings, in order to build their mutual comprehension. From this point of view, the interviewer’s and interviewee’s deviations from the question–answer mechanical sequence are not sources of error but resources, coming from the ordinary conversation, to use for a collaborative construction of the meanings exchanged in interview (Schaeffer and Maynard 2002; Schober 1999).

A lot of empirical studies show that, despite the constraints of standardization, interviewer and interviewee tend to deviate from the mechanical course of interaction and to resort to the ordinary conversation resources, to shape their mutual comprehension. In the fieldwork, it is quite frequent that interviewers feel like caged by standardization because they cannot say whatever they want to solve local interactional problems. For example, when respondent asks interviewer for clarifying the sense of a question or the meaning of a term, interviewer is allowed only to repeat the question without altering its formulation, and, if the request persists, to put the interpretation in the respondent’s hands, using the phrase: “Whatever it means to you.” But this behavior is senseless both in the ordinary conversation and in the standardized interview because it is not instrumental in shaping the mutual comprehension (Maynard 2006).

Consequently, the directive probing techniques should be revalued. When interviewer reads a question altering the original formulation, he or she does it because respondent has previously misunderstood the question as formulated by researcher. If interviewer reads only part of response alternatives, this happens because interviewer has already taken into account respondent’s answers and comments made during the interview. When respondent’s answer does not match with one of provided response alternatives, interviewer may suggest one answer which represents the interviewee’s state on the property. When the first answer unveils an interviewee’s incomprehension about the response instructions, interviewer can suggest “a candidate answer” to show how to answer (Houtkoop-Steenstra 2000).

As well as the ordinary conversation, the standardized interview remains a linguistic and interactional event, when misinterpretations and misunderstandings cannot be dealt by researcher with the strict rules of standardization, which forces interviewer in repeating the same words across the respondents (Schaeffer 1991).

It appears, therefore, necessary to abandon an outdated model, founded on the invariance of behaviors, in order to adopt a new model based on the invariance of meanings, to achieve a real conceptual alignment between interviewer and interviewee. In this new model, no course of interaction is better than others: The only desirable one is when interviewer and interviewee comprehend each other.

This new theoretical orientation has led to the development of the “collaborative model” (Schober and Conrad 2002) or the “common ground model” (Schaeffer and Maynard 1996). The collaborative model assumes that a linguistic expression is considered “complete” when it is grounded, that is actually comprehended by the listener as it is intended by the speaker. The common ground is an area of shared meanings, without which speaker and listener misunderstand each other. In ordinary conversation, the speaker and the listener actively cooperate to achieve the mutual understanding. This happens also in the standardized interview, when the communication between interviewer and interviewee is not fast and the question–answer sequence does not end in two rounds (interviewer asks question and interviewee answers), but it requires several ones.

On the basis of this model, the principles of standardization should be revised because they are able to guarantee interviewers’ uniform behaviors but not to transmit uniform meanings to all respondents. To achieve a substantial standardization, interviewer should feel free to rely on the resources which he or she uses in conversation with friends or strangers at bus stop. It is a collaborative perspective, when researcher assigns to interviewer the task to capture the signals of interviewee’s misunderstanding in order to tackle and solve them.

This new interviewer—the negotiator of meanings—is the gatekeeper of question meaning, whose goal is to transmit uniform meanings to all respondents. The interviewer is the key to lead the transition from a formal standardization to a substantial standardization. Researcher should not be interested in how interviewer reads a question or how and when she or he probes; what really matter is the interviewer’s ability to collect the information sought by researcher: adapting the question formulation to the interviewee’s interpretative needs, evaluating the response accuracy with respect to the property operationalized in the question, and managing the interaction with respondent to clarify the meanings.

The transition from a formal standardization to a substantial standardization is based on a new type of interview, “the flexible interview,” also named, “the conversational interview.”

The flexible interview is a sort of soft standardized interview designed to assure that all respondents understand questions as intended by researcher; it is not wording, therefore, but meaning that is standardized. Thanks to the freedom allowed to the flexible interviewer, phenomena of misinterpretation and misunderstanding can be handled. The flexible interviewer mediates between the question meaning, as it is intended by researcher, and the interpretative needs of respondents. To settle the interactional problems which cannot be predicted upstream by researcher and to convey a full congruity of meanings, interviewer is free to say whatever is necessary to help respondents to interpret the questions correctly, deviating from the question–answer mechanical sequence and adopting the behaviors appropriated for the interview situation (Gobo and Mauceri 2014).

Schober and Conrad (1997, 1998) tested the flexible interview. In the studies conducted in laboratory, they compared the quality of data collected with the standardized interview and with the flexible interview. The correspondence between the respondent’s conceptual schemes, used to formulate the answer, and the researcher’s conceptual schemes, used to formulate the question, has been evaluated: They are equivalent when the respondent understands the question as intended by researcher; in case of question misunderstanding, they are not equivalent.

The results of these studies provide empirical evidences that support the effectiveness of the flexible interview: The greater the complexity of question meaning is (complex scenario), the more crucial the role played by the flexible interviewer is in assisting the interviewee in interpreting the question meaning. Basically, if the scenario is simple, the accuracy of the answers given by the respondent interviewed in standardized mode is good. When the question is easy to interpret, an active role of the interviewer is not necessary. But if the scenario is complex, it is likely that the respondent misinterprets the sense of the question. In this case, a passive and standardized role of the interviewer is harmful to data quality because the interviewer cannot assist the respondent properly. When the interviewers are allowed to behave in any way they want to clarify the terms misunderstood by the respondents, they collect excellent data (the negotiation of meanings starts either when the respondent requests it or when the interviewer senses verbal and nonverbal signals of the respondent’s misunderstanding).

In other words, when interviewer and interviewee collaborate in the meaning construction, the accuracy of collected information is much higher than when they do not do it. The scholars, who strongly support for formal standardization, point out that it cannot be taken for granted that interviewer recognizes nonverbal misunderstanding signals, when respondent does not request the help of interviewer explicitly. But the empirical studies on the flexible interview show that this is not a problem because it depends on the nature of the scenario. In front of a complex scenario, respondent tends to ask interviewer to clarify the question meaning; instead, a simple scenario does not stimulate the respondent to seek the interviewer’s support because the question meaning is easy to decode (Schober, Conrad, and Fricker 2004).

The Usefulness of the Flexible Interview in the Questionnaire Pretest Stage—The Study

The flexible interview is the future of social research because it is a crucial tool to tackle the biases afflict the standardized interview. The flexible interview is usually used at the data gathering stage of a survey. It is my opinion that the flexible interview should be applied also at the stage of questionnaire pretesting. ⁵ This strategy ensures that the quality of the collected data increases at the stage of both questionnaire pretesting and data gathering.

In the data gathering stage, the data quality increases because the flexible interviewers are asked to realize when respondents misunderstand question or part of it and say whatever necessary to help them in understanding the question as researcher expects, in order to correct the collected information before it is coded, inevitably, biased.

Also in the questionnaire pretest phase, the data quality enhances. In fact, the flexible interview requires highly motivated interviewers, as well paid as they are trained, with interactional and methodological skills uncommon among standardized interviewers; the conduction of a flexible interview is a hard work. The use of the flexible interview, also at the pretest stage, would help the flexible interviewers to work at the data collection phase as best as they can, thanks to a questionnaire pretested in line with the “collaborative/common ground model” principles. The questions that caused misunderstanding in many respondents in the pretesting phase have already been reworded.

In this article, I present a study designed to test whether the flexible interview, used to pretest questionnaire, gives better results than conventional pretesting or not. So I compared these two pretest techniques, empirically.

Survey researchers have always shown a remarkable confidence in conventional pretesting, which is a sort of dress rehearsal when interviewers administer a questionnaire as they would do during a survey proper. In fact, the main characteristic of conventional pretesting is to reproduce the same conditions which usually occur in a standardized interview: Interviewer conducts interview as if she or he is administering questionnaire during the data collection phase. It is common opinion that a small number of conventional interviews often reveals numerous problems, such as questions that contain unwarranted suppositions, awkward wordings, or missing response categories (Presser et al. 2004).

The reliability of pretested questionnaire is examined through debriefings in which interviewers relate their experience with the questionnaire and offer their opinion about the questionnaire problems (Sudman and Bradburn 1982). At the end of pretest phase, interviewers report the questions difficult to read and hard to be understood by respondents correctly. In addition, interviewers suggest how to reword the reported questions, which prevented interviewer and interviewee from having a fluid and quick interaction without misunderstandings. This approach requires full cooperation of the interviewers, who are aware of the interview purpose.

The two conventional pretest assumptions are as follows: well-trained interviewers are able to recognize problems with questions and respondents should not be informed of the pretest purpose (Converse and Presser 1986). This emphasis on interviewer perceptions is nicely illustrated by Sudman and Bradburn (1982:49): “A careful pilot test conducted by sensitive interviewers is the most direct way of discovering these problem words.” The researcher, who adopts the conventional pretesting, is very confident that trained interviewers are able to detect the main biases afflict the pretested questionnaire. This confidence is based on the widespread opinion that all biases, especially the respondents’ misunderstandings and misinterpretations due to badly formulated questions, can be discovered from interviewer observing respondents’ verbal behaviors.

A pretested question should be reformulated when respondents show frequent signals of misunderstanding, deviating from the question–answer mechanical sequence. ⁶ It is taken for granted, in fact, that a poorly formulated question alters the linear flow of the question–answer sequence and that the respondents’ verbal deviations reveal question misinterpretations. Conversely, a pretested question has not to be revised when respondents follow the question–answer mechanical sequence ⁷ because a well-done question guarantees a fluid communication between interviewer and respondent, and this should be the evidence of a good mutual comprehension.

But can I consider observable verbal behaviors as good indicators of nonobservable, or difficulty observable, interpretative mechanisms?

In my opinion, I cannot do it. Interviewer cannot recognize problems which are not apparent from observing respondents’ verbal behaviors. In fact, respondents can misunderstand or misinterpret a closed question without giving any signal. It is quite common that respondents misinterpret the question, even if they follow the standardization rules; as it is frequent that respondents comprehend questions correctly, despite they break the standardization rules and deviate from the paradigmatic sequence. Basically, “good” behaviors are not indicators of “correct” interpretations, as much as “bad” behaviors are not indicators of “mistaken” interpretations.

To exceed the limits of conventional pretesting, the solutions are as follows: respondent should be aware of the pretest purpose, interviewer should be trained to ask “Why?,” and respondent should feel free to talk about question meanings. These strategies can be adopted to avoid that respondents misunderstand closed questions without showing any misunderstanding signal and also to distinguish between “useful” misunderstanding signals, which are good indicators of real interpretative distortions, and “useless” misunderstanding signals, which are not good indicators of real interpretative distortions (they are just part of a lively interaction that depart from the constraints of standardization).

The Research Design

I designed a nonprobabilistic sample of Roman residents over 18 years old; my methodological purpose did not require a statistically representative sample. The sample was made up of 100 subjects, differentiated by gender, age, and education (Table 1).

OPEN IN VIEWER

Table 1.

The Quotas of Nonprobabilistic Sample of 100 Subjects.

Males 50						Females 50
Nongraduated 25			Graduated 25			Nongraduated 25			Graduated 25
18–30 years	31–55 years	Over 55	18–30 years	31–55 years	Over 55	18–30 years	31–55 years	Over 55	18–30 years	31–55 years	Over 55
8	8	8	8	8	8	8	8	8	8	8	8

Note: Each interviewer interviewed 24 subjects. So the total number of the interviews is 96. The remaining four interviews were considered “wild cards”: each of the four interviewers was left free to choose, at own discretion, the profile of own 25th interviewee.

Four professional interviewers interviewed all respondents; each interviewer interviewed 25 subjects. All sampled subjects were face-to-face interviewed: 50 respondents were interviewed by two standardized interviewers (who spent many years working for important research institutes specialized in face-to-face interviews) and 50 respondents by two flexible interviewers (who gave evidence of being experienced interviewers in academic research). I personally trained both standardized and flexible interviewers.

The training of the selected standardized interviewers lasted for two sessions (two days), plus a third session when a real interview was simulated. My purpose was to train the standardized interviewers to follow the rules of standardization: to read questions as written; to score responses without subjective interpretations; not to make directive probing, in case of inadequate response; and to put the question interpretation into the hands of interviewee, using the phrase “Whatever it means to you,” when respondent keeps requesting interviewer’s support in question comprehension (Fowler and Mangione, 1990).

The training of the selected flexible interviewers lasted for four sessions (four days). Despite the flexible interviewers were used to a sizable flexibility in conducting interviews (given their professional background), the training required some additional meetings, during which I and the flexible interviewers simulated several flexible interviews.

Initially, the instructions received by the flexible interviewers consisted of reading question without changing the wording (the objective was to pretest the question formulation). After the question was read as written, the interviewers were trained to encourage the respondents to release free comments to explain the motivations of the given answers. All respondents were informed that the interviewers were ready and willing to listen to their free comments.

The contribution of the flexible interviewers has been essential in pretesting the questionnaire because the collection of the respondents’ comments, spontaneous or requested, has been the key for my data analysis strategy. ⁸

Furthermore, the flexible interviewers had another important task to accomplish. After they administered the question and received the answer commented by the respondent, they had to compare these three informations: question–answer–comment. When the flexible interviewer realized that interviewee was misinterpreting the question (the comment revealed an incongruence between question and answer), the interviewer had to initiate whatever action necessary to clarify the question meaning as intended by me, and also, if necessary, to amend the question formulation, in order to help the respondent to understand the question correctly.

The pretested questionnaire was constructed to study the social representation of extra-European immigrants among Roman respondents. Thanks to the analysis of scientific literature on this issue, the following semantics aspects of this multidimensional concept were identified: conflict, deviance, change, intrusiveness, identity, and reciprocity. This conceptual grid helped me to choose the questions to include in the questionnaire. In this activity, I applied two criteria: constructing a highly structured questionnaire to pretest and finding questions (the operational definitions and formulations) often used to study the attitude toward stranger immigrants in national surveys.

This questionnaire has been pretested both with a conventional method and with a nonconventional method. To pretest the questionnaire conventionally, I organized a debriefing with my two standardized interviewers. I asked them for reporting an overview of the questionnaire pretesting experience, to identify the problems faced in the fieldwork. To maximize the usefulness of the meeting, I asked them three questions:

Did you find any difficulty in reading the questions exactly as I worded?

They identified the worst questions, which prevented them from administering questions without respondents’ misunderstandings. I asked the interviewers to evaluate how these questions worked on the field and to suggest me how to revise them. Thanks to this procedure, the two standardized interviewers contributed to the questionnaire evaluation, equally and deeply.

To pretest questionnaire nonconventionally, I analyzed the content of the respondents’ comments released in the flexible interviews. To evaluate the correspondence between the researcher’s conceptual schemes (used to formulate question) and the respondent’s conceptual schemes (used to interpret the question and formulate the answer), ⁹ I expressed a judgment of question reliability at the end of each question–answer–comment sequence. The question reliability judgment ¹⁰ is positive when the question is understood by the interviewee like I intend; it is negative when the sequence reveals a respondent’s misunderstanding. This judgment is the outcome of the comparison between the answer, the comment made by the interviewee, and the property that question operationalizes. When these three elements are equivalent, the judgment of reliability is positive because the question collects the information for which it is designed for; otherwise, the judgment is negative because a misunderstanding between the interviewee and the interviewer occurs. ¹¹ The aim is to identify the respondents’ misunderstandings, locate the questions which caused these interpretative distortions, and, finally, correct them.

The formulation of the question reliability judgment has been more easy when the flexible interviewer played an active role in the interaction with respondent, catching the inconsistency of the answer; informing the respondent, at that moment, of the distance between the answer and the original sense of question and helping her or him to understand the question as expected. This interaction between interviewer and respondent offered me unreplaceable ideas on how to rephrase the badly formulated question, in order to prevent this distortion from occurring again in course of the survey.

The flexible interviewer, who does not merely restrict himself or herself to reap the interviewee’s comments, leaving the researcher alone in comprehending the causes of any eventual bias occurred during the interview, is the real mediator of the question meanings, contributing to the process of questionnaire pretesting both as expert with own ability to identify problems of wording and interpretation and as an active subject of the relational and interactional process.

The Results of the Analysis

In my opinion, the essential indicator for evaluating a pretest technique is the capability to identify problems that can generate the survey error. For this reason, the comparison between the conventional and the unconventional pretest technique, proposed in his paper, is based on the following indicators: (1) the types of the identified problems, (2) the total amount of the identified problems, and (3) the number of biased questions located in the questionnaire.

First of all, I report the types of problems identified as problematic by the two pretest techniques. The pretested questionnaire comprised 46 closed questions; all questions were pretested.

The conventional pretest procedure identified three types of problems:

the question syntax is formulated incorrectly (the terms used in the question text/response instructions/response alternatives are unclear or ambiguous and not suitable to communicate the meanings as intended by the researcher),

The unconventional pretest procedure identified six types of problems:

the question syntax is formulated incorrectly,

What about the total amount of the problems identified in the questionnaire? The conventional pretest procedure, based on interviewers’ observation, identified 14 problems which afflict 14 different questions, each of these questions is biased by just one problem: 9 questions by an incorrect syntax of the question text, 3 questions by an excessive number of response alternatives, and 2 questions by an incomplete list of response alternatives. On the contrary, the unconventional pretest procedure, based on the analysis of the respondents’ comments, identified 42 problems which afflict 27 questions, 5 of the 27 biased questions are twisted by two or three or four problems, simultaneously ¹² : 24 questions by incorrect syntax of the question text text/response instructions/response alternatives, 7 questions by an incorrect classification of the response alternatives, 5 curvilinear questions, 3 double-barreled questions, 2 questions by an excessive number of response alternatives, and 1 question whose operationalized property is also indicator of another aspect of the general concept to measure.

The total number of the problems discovered by the unconventional procedure (42) is higher than the number of the problems found by the conventional procedure (14).

As well as the number of the types and the number of the problems and also the number of the problematic questions identified by the unconventional procedure (27) are higher than the number of the problematic questions located by the conventional procedure (14).

To summarize, the analysis of respondents’ comments is a pretest technique more efficient than the conventional pretesting, for the following reasons:

the number of the types of problems identified by the unconventional pretest procedure is higher (the twice),

Furthermore, even when the two pretest techniques detected the same type of problem, the number of biased questions located by the unconventional technique is higher than that one determined by the conventional pretest technique. For instance, both of the pretest techniques found that the syntax of some questions is formulated incorrectly (the terms used in the question text/response instructions/response alternatives are unclear and ambiguous, preventing interviewers from communicating the meanings as intended by the researcher and respondents from understanding the question as expected). But, while the interviewers’ observation located 9 questions biased by a wrongly formulated syntax, the respondents’ comments analysis located 24 questions that are affected by the same type of problem. As another instance, both of the pretest techniques identified that some lists of response alternatives are not well done because they are incomplete and/or there is no mutual exclusiveness among the alternatives. But, while the interviewers’ observation located five questions biased by a bad list of response alternatives, the respondents’ comments analysis located nine questions biased by the same type of problem.

Moreover, even when the same question is considered as problematic by both of the pretest techniques, the unconventional technique has the great capability to identify several types of problems afflict this question. The conventional technique has never identified more than one problem for each question: Each of 14 biased questions, located by the conventional pretesting, seems to be afflicted by only one problem; otherwise, 5 of the 27 wrongly formulated questions, found by the unconventional technique, are afflicted by two or three or also four different types of problems simultaneously.

For instance, the following question “We should enable immigrants, who have a job, to stay in Italy. Please, outline your degree of agreement or disagreement: strongly agree, agree, neutral, disagree, strongly disagree” has been considered as problematic by both of the pretest techniques. The conventional pretest technique identified only one problem: the list of response alternatives is excessively long. The standardized interviewers report that many respondents give an invalid answer, showing a general position of agreement/disagreement (i.e., “I agree” or “I disagree”), without specifying their agreement or disagreement intensity. In these cases, the interviewers reread all response alternatives, by a nondirective probing, or they read two response alternatives which appear to be close to the real state of the interviewee, by a directive probing. It is my opinion that these respondents give an invalid answer because they are focused on interpreting the question/item text correctly, forgetting to give the due attention to the response alternatives interpretation; in addition, the response alternatives number is so long as to confuse the respondents in the comprehension of the cognitive task requested by the researcher. To help respondent in understanding the item text and the response instructions and the response alternatives, I propose to abandon a so long and complex Likert-type scale and adopt a more streamlined version, easier to remember for the interviewee, such as follows: “I strongly agree,” “I somewhat agree,” “I slightly agree,” “I disagree,” and “I do not know.”

On the contrary, the unconventional pretest technique identified three problems plague this question: the list of response alternatives is excessively long, the question syntax is formulated incorrectly, and the item is curvilinear. The first problem has been found by both of the pretest techniques. The second problem concerns how the question syntax is formulated. Many respondents said “Well, it must be a regular job! If it is undeclared…no! Otherwise, there is exploitation on the one hand and tax evasion on the other hand,” and others asked the interviewer “Do you mean also work in black? or “Is the work regular or undeclared?” The respondents often modified part of the question content because, according to them, to stay in Italy, the immigrants should live legally and have a regular employment. “Having a job is not enough; the employment must be regular!” The respondents show frequent interpretative difficulties in understanding that the researcher is not interested in the nature of the employment that the immigrants should have to remain in Italy. To avoid this misinterpretation, it is necessary to insert the terms “regular job” or “legal employment,” amending the question text as follows: “We should enable immigrants, who have a legal employment, to stay in Italy.”

The third problem refers to the fact that the question is curvilinear. It is a common opinion that the respondents, who agree on this phrase, have an open and positive attitude toward immigrants. But, thanks to the analysis of the respondents’ comments, I noticed that this is not always true. It sometimes happens that two people, who say they agree on this item, have antithetical motivations. For example, a respondent answered, “I strongly agree because this would be the perfect way to select immigrants instrumental or useless to our economy,” and another one said, “I agree because, if they work, the economy goes on and increases, then I see no reason to send them away, indeed!” These two respondents show signals of racial prejudice because they see immigrants as an instrument in the Italian entrepreneurs hands, “tools to make money.” Other respondents, who agreed on this item, have an opposite motivation: “I totally agree because they are human beings, and, if they have a job, they should not be driven out.” These respondents have no racial prejudices toward immigrants, and their attitude is open to immigrants considered as human beings who have the right to remain in the place where they emigrated. The respondents with no racial prejudice give the same answer as the people with a high racial prejudice, but the reasons are antithetic. If the researcher proves that the question is curvilinear, it should be removed from the questionnaire.

Conclusions

In comparison with the conventional procedure, the unconventional procedure has proved to be a more efficient pretest technique to identify potential problems for questionnaire, which generate the survey error. In fact, the unconventional procedure not only diagnoses the same problems which the conventional procedure detects, but it also identifies other problems which the conventional procedure does not detect, as this study shows. In my opinion, this methodological mechanism is due to the nature of the techniques analyzed in this study.

The conventional pretesting is based on the interviewer’s observation: interviewer is asked to be, at the same time, a good interviewer and a meticulous observer. On one hand, researcher requests interviewer to have a standardized behavior, following the instructions received in the training: to read each question as worded, to record the appropriate response choice, and to use a nondirective probing. On the other hand, interviewer is asked to observe the respondent’s behaviors during the question comprehension and the answer formulation process, in order to grasp verbal and nonverbal misunderstanding signals and to determine whether the respondent is having difficulty with the question or not.

Reading, again and again, the reports of my interviewers, it is clear that it is not easy for an interviewer to perform both of these duties: Interviewer should realize whether respondents understand questions correctly, without inquiring their question comprehension process.

Rec sic stantibus, conventional pretesting, fails to detect the problems the questionnaire poses for respondents, who are unaware of the pretest purpose and are not asked to contribute to the development of the pretested questionnaire. Conventional pretesting is suitable for detecting the problems the questionnaire poses for interviewers, who are very good at finding problems related to their responsibilities, such as typographical errors, faulty instructions, poorly worded questions, incorrect skip instructions, inadequate space to record answers, and so forth. Furthermore, when interviewers find the problems the questionnaire poses for respondents, the interviewer’s observation remains a superficial pretest technique because it relies on observable misunderstanding signals.

To better identify the problems questionnaire poses for respondent, also in case respondent does not show any misunderstanding signal, I recommend to involve interviewee in the questionnaire pretesting. The unconventional pretest technique, which I propose in this article, is based on the analysis of the respondent’s comments and, in wider terms, of the interaction with interviewer in the flexible interview. The contribution of respondent is precious: He or she comments on the given answers and criticizes the questions meaning/formulation. The assignment of flexible interviewer is to collect respondent’s comments—spontaneous and requested—and to have an active role when the respondent shows to misunderstand the question, to help him or her to understand it correctly.

Thanks to the comment released by respondent, researcher has three crucial information to compare: question, answer, and comment. For each question–answer–comment sequence, researcher expresses a question reliability judgment ¹³ : positive, when these three elements are congruent because none misinterpretation occurred; negative, when the comment reveals an inconsistency between question and answer because respondent misunderstood question. The questions with many negative judgments have to be reformulated because the comments revealed numerous respondents’ misunderstandings.

How should these questions be reworded? The respondents’ comments analysis is necessary to locate the wrongly formulated questions which cause many respondents’ misunderstanding, but it gives few information on how to reword them. From the analysis of how the interviewer helps the respondent to give an accurate answer based on the correct comprehension of question, researcher takes the cues on how the badly formulated question should be rewritten. In fact, the fieldwork of a flexible interviewer, left free to behave however he or she wants to repair the local misunderstandings, is a precious tool to comprehend how the questions, which cause many relational problems with respondent, should be modified.

In conclusion, the strength of the respondents’ comments analysis is to discover problems the questionnaire poses for respondents, even when the respondents do not give any misunderstanding signal, and to find a solution to these problems. This technique locates the questions harmful to data quality and also suggests how to reword them, offering both the irreplaceable respondent’s point of view on the questionnaire and the crucial contribute of the flexible interviewer in finding a solution to the respondent’s interpretative problems.