Abstract
If you want to know something, don’t ask. Throughout the many chapters of George Beam’s critical text, the author works to highlight the innumerable limitations associated with survey methods. The Problem, as the author coins it, is that “answers to questions are not reliable.” Often laced with cutting humor, Beam provides an easy to read and user-friendly framework for understanding the flaws associated with survey research. Importantly, Beam follows the harsh criticism with explicit recommendations for enhancing the rigor of social science research. His suggestions are provided not to disregard social scientists’ efforts but, to curb academia’s addiction to asking. I admit I was concerned to open this book as a great deal of my research experience is founded upon survey methods—however, Beam provides an important perspective, one that can only help social work researchers to advance our research designs and therefore, support our individual clients and their families, change communities, and shape society.
Part 1 of the text aptly entitled “Ubiquity of Survey Research” includes three chapters that, when unified, set the stage for The Problem. Chapter 1 (Everyone Asks) begins with the acknowledgment of the irreparable flaws of survey research by first illustrating the ever-growing list of askers. Everyone is guilty of contributing to The Problem including businesses, corporations, consulting firms, interest groups, governments, political parties, unions, churches, schools, volunteer groups, media organizations, research centers, and of course, academics. Chapter 2 further builds this case by providing a detailed narrative cataloguing a mere sample of those whom are asked. In “Everyone is Asked,” Beam compiles a wide variety of askees including parents of all sorts (e.g., adoptive, foster, single, noncustodial), Central Intelligence Agency officials, leaders and politicians, shoppers and climatologists, wives and political activists, and prisoners. It is clear that no one is left unasked. To wrap up, Beam adds that there is no boundary regarding about what is asked in Chapter 3, “Ask About Everything.” Questions are asked, and therefore unreliable information is obtained, about coitus, crime and punishment, shovel pricing, genital grooming, diet and calcium intake, elections, homosexuality, gun sales, anger, health insurance, and the list goes on. These first few chapters of the text point to a disconcerting theme: Everyone is asked by everyone about everything—and this contributes to The Problem.
While the first few chapters of the text create a foundation, Part 2 “The Problem,” provides a concise summary of how concerns with survey research have been previously identified, explained, documented, and acknowledged. Despite this, of course, Beam indicates reliance on such methods remain. Chapter 4 opens by identifying and defining the two types of answers that exist: Category 1 answers are those responses that reflect what is really happening, whereas Category 2 answers do not reflect what is really happening. The concern, Beam notes, is that with only answers to questions, we are incapable of teasing the two types of answers apart. If askers do not have a means to verify that answers are correct or accurate, they are left with unreliable information. They are left with The Problem. Chapter 5 continues by showcasing how this pervasive Problem has been fully acknowledged and documented in the literature. Beam notes that previous research has been published, where authors acknowledge the fact that their data often lack sufficient corroboration or have described answers as “wobbly.” Some have said because words are used to construct questions, and since no single way to construct a question exists, answers can be considered “disputable.” That scientists are so clearly aware of concerns with survey research, yet continue down the path to The Problem is concerning.
Parts 3–5 seek to highlight the trifecta of Problem causes: instruments, settings, and askers. Part 3 includes Chapters 6 and 7, Shaky Instruments and Shaky Instrument (continued). Both chapters are presented to elucidate the numerous instrumentation and measurement concerns associated with survey research. Beam notes that questions about both objective and subjective phenomenon and those who produce oral or written responses are riddled with holes. For instance, he argues that respondents are queried using words (symbols) and then respond using words (symbols)—responses cannot actually capture the reality queried (non-symbolic). Chapter 7 continues by showcasing the effects of formatting such as physical features, question structure, patterns of questions, and other unknowns and their contribution to The Problem. Moreover, these effects and others, along with those noted in Chapter 6 (and subsequent chapters), can become entangled. It becomes impossible, therefore, to know what specific elements have which specific effects on which specific answers.
Part 4 includes Chapters 8 and 9, Consequential Settings and Consequential Settings (continued), and focuses on the effects of societal and immediate settings on answers. Societal settings include an individual’s socialization, position within society (e.g., gender, socioeconomic [SES] status, leadership roles), and experience with law. Each of these impact respondent’s answers. Immediate settings, in contrast, are those physical locations, where survey research takes place (e.g., doctor’s offices, schools, war). Each of these setting effects (viz. societal and immediate) can comingle with one another and with instrumentation and asker effects. For these reasons, again Beam states, “if you want to find out what’s really going on, don’t ask.”
To expand considerations associated with instrumentation and settings, Chapter 10 through 13 (constituting Part 5) describe varying askers: stimulating askers, addicted askers, and alchemic askers. Beam also identifies many of the means by which answers are acquired and sometimes altered in Chapter 12, “Promise anonymity or confidentiality, probe, prompt, use props, con, and tinker for more.” Askers are stimuli, just as instruments and settings, and therefore—illicit responses from those being asked. Beam recounts several attributes of askers that contribute to The Problem: our judgmental processes (i.e., expertise), our experiences and competencies, our ethnicity, our SES features, our gender, and finally, our age. Each with their own independent, interactive, and intertwined (and often times inconsistent) effects, instrumentation, setting, and asker characteristics ultimately impact answers and leave researchers with no new knowledge but with The Problem.
Part 6 concludes with two chapters whose efforts are to provide explicit recommendations to alleviate the numerous problems with survey research: “Proper Methods and Research Designs.” Chapter 14 outlines opportunities for direct observation, the role of experimentation, and the use of multiple, nonasking sources in social science research. Chapter 15 provides similar recommendations related to the use of models, document analysis, and comparison designs. Both chapters unify as proverbial “plugs” for the overwhelming holes noted throughout the text. Importantly, Beam notes that many of these methods would be initially cost prohibitive and would require a social science discipline shift—one that would enhance funding pipelines and opportunities. No doubt a large feat.
Although it is possible that social work scholars would find such a text a scathing attack, it is an essential perspective for assuring our growth as researchers, educators, and practitioners. Just as we have endured previous criticism that our field did not possess the critical components of a profession, Beam’s words can be viewed as an impetus for growth. Social workers embrace their colloquial title of “change agents,” and this is an opportunity for evolution. Beam’s text is an important tool that can be used in our efforts to assure our development and subsequently, that of our clients, communities, and academic and societal culture.