Abstract
Our understanding of programs is enhanced when trained, skilled, and observant evaluators go into the field – the real world where programs are conducted – paying attention to what’s going on, systematically documenting what they see, and reporting what they learn. The article opens by presenting and illustrating twelve reasons for building fieldwork into evaluation designs. While in-depth fieldwork is the evaluation ideal, the evaluation reality is that brief site visits are the more common design. The article identifies systematic weaknesses in how site visits are commissioned and conducted, and proposes standards for improving quality and use.
The Value of Evaluation Fieldwork
Our understanding of programs is enhanced when trained, skilled, and observant evaluators go into the field—the real world where programs are conducted—paying attention to what’s going on, systematically documenting what they see, and reporting what they learn. A site visit to an employment program revealed that the location of the program in a run-down, unsafe area was affecting how participants who came from elsewhere in the city felt about the program—and contributed to a high dropout rate. Talking to participants on site, we discovered that free bus passes were used by different staff in different ways. Some used them as rewards, some gave passes to all of their clients without question, and some insisted on evidence of need. In a visit to a rural African school that was supposed to serve both boys and girls, the first thing that stood out was that there were no girls. Routine monitoring reports submitted to headquarters showed roughly equal numbers of boys and girls. So where were the girls? They were on a waiting list. The school lacked space to serve everyone, so they enrolled everyone, boys and girls, and reported enrollment data to headquarters and the government, but only boys were allowed to actually attend. Observing a leadership program revealed that participants had self-organized into two groups, designated the “turtles” and the “truckers.” Participants in those groups were having significantly different experiences and outcomes.
Exhibit 1 offers a dozen reasons for building fieldwork into evaluation designs. The challenge is that fieldwork adds to the cost of evaluations and, when done, is too often of poor quality. This article argues that learning in the field (Rossman & Rallis, 2012) is essential and that, being essential, must be done systematically and rigorously. Thus, after reviewing the need for more and higher quality fieldwork, I propose standards for site visits.
Twelve Contributions of Fieldwork to Evaluations: Examples and Explanations.
In-Depth Fieldwork
In my 45 years of evaluation practice, the most rewarding evaluation experiences have involved participant observation in which I engaged fully in the program as an evaluator and a participant. This was the core design for evaluating several different leadership programs. Beyond participant observation, ongoing regular engagement with a program as it unfolds is essential for developmental evaluation (Patton, 2011).
The evaluation of the implementation of the Paris Declaration on International Development Aid provides a striking illustration of the importance of fieldwork. The evaluation involved 22 country case studies with national evaluation teams doing the case studies in each country. An example of a Paris Declaration principle is alignment between donor country interests and recipient country priorities. The evaluation documented that differences in country context (political, economic, and historical development patterns) led to wide variations in implementation and outcomes. Capturing, interpreting, and understanding these contextual effects and variations required in-depth knowledge of each country obtained through fieldwork. Even though evaluators lived in the countries on which they did the fieldwork, the systematic engagement in the field through observation, interviewing, and document analysis revealed things that they did not and could not have learned through questionnaires and secondary data analysis alone. The most important insights, most significant findings, and most useful results came from fieldwork (Patton & Dablestein, 2012). The same was true for the meta-evaluation of the Paris Declaration Evaluation I conducted. The meta-evaluation included questionnaires, document analysis, and extensive review of secondary data, but the most important insights came from direct observation of the evaluation team in action (Patton, 2012b).
While in-depth fieldwork is the evaluation ideal, the evaluation reality is that brief site visits are the more common design. Having hopefully established the value of fieldwork, the remainder of this article analyzes the realities of site visits, identifies systematic weaknesses in how they are commissioned and conducted, and proposes standards for improving quality and use.
What follows is part analysis and part rant. The analysis leads me to conclude that the evaluation profession has a problem, which leads to the rant. It is fair to consider the ranting aspects the ravings and ruminations of a curmudgeonly old man who has devoted thousands of hours and miles to evaluation and developed a passion for the profession at its best, and it is not fair to dismiss the rant conclusions for that same reason. I do not deny that the rant is biased. On the contrary, I assert that it is accurately, validly, reliably, meaningfully, appropriately, and transparently biased. Enjoy this oxymoronic irony as you journey with me to visit the sites of my concerns and distress. But first, analysis.
Site Visits
Site visits are a commonly employed, but little discussed, evaluation procedure.
Do site visit practices constitute a set of methods or a methodology? This is the question that framed a review of the state of the art regarding site visits undertaken by Lawrenz et al. (2003) in this journal over a decade ago. They concluded that “site visits are much more than a collection of methods, that site visits constitute a methodology in their own right, distinct from other evaluative methodologies” (p. 342). They examined how site visit methodology is based on Ontological beliefs about the nature of reality and epistemological beliefs about whether and how valid knowledge can be achieved. Ontologically, in order to conduct site visits the evaluator must assume that there is a reality that can be seen or sensed and described. Epistemologically, site visits are based in the belief that site visitors are legitimate, sensing instruments and that they can obtain valid information through first-hand encounters with the object being evaluated. (p. 343)
They opened by addressing the issue of definition: A comprehensive definition of a site visit is not readily available. It appears to be something all evaluators know but that few write about. We propose the following definition. An evaluative site visit occurs when persons with specific expertise and preparation go to a site for a limited period of time and gather information about an evaluation object either through their own experience or through the reported experiences of others in order to prepare testimony addressing the purpose of the site visit. Our extensive search of the literature revealed that little is published about the philosophies underlying site visits, or about who to involve, how to ensure appropriateness, how to use the information generated, or how to implement site visits successfully. (pp. 341–342)
Their analysis discussed three major concerns about site visits: reliability, validity, and usability of the information. In addressing these issues, they noted that site visits can be viewed from at least three different perspectives: the site visitors, the people at the sites being visited, and those organizing or commissioning the visit. They questioned reliance on site visits for high stakes summative evaluations because of the time limitations and methodological challenges of conducting high quality site visits that yield credible data. They also raised questions about the expertise, competence, training, and preparation of those who conduct site visits. Because site visits “rely heavily on the expertise of the visitors … this reliance indicates an epistemological belief that site visitors possess legitimate ways of knowing. However, this belief in legitimacy should be supported by having much more rigor and transparency in the selection and perhaps the training of high-quality visitors” (pp. 350–351). They concluded their review with a recommendation: Given their frequency and consequences, the evaluation literature should provide more focused attention to site visits (p. 351).
This current article, more than a decade later, picks up where they left off, because the focused attention they called for has not occurred. Indeed, the important issues they raised are, in my judgment, more urgent than ever because the use of site visits as a primary method has become more widespread, the stakes have grown higher, and the quality is as problematic as ever, if not more so.
What are Site Visits?
A “site” is the location of an organization, program, project, community activity, meeting, or event. In short, a site is where something is going on that is of interest and importance to a funding or oversight agency (like a philanthropic foundation, an international agency, or a government administrative unit) or relevant for the inquiry of a researcher or student doing fieldwork. The “visit” involves a person or team going to the site to observe what is happening, interview key people, review relevant documents, and write a report. Here are examples, both international and domestic. The World Bank is funding a 5-year mother and infant nutrition project in Bangladesh, a well-digging project in Burkina Faso, or an agricultural development project in the Andes. Halfway through the project, a three-person team is selected by the World Bank to visit the project for a few days to assess progress. A philanthropic foundation is funding an adult literacy program in a rural community and sends a program officer out for a day to observe some classes and talk with participants. The U.S. Department of Transportation and several state agencies are funding driver education programs aimed at reducing use of cellular phones while driving, a team of national and state staff go for 2-day site visits to observe four such programs in different states.
In essence, evaluation site visits involve fieldwork on location to report independently on how a project is being implemented and what it is accomplishing.
Midterm and end-of-project evaluation teams may well be the most common form of formal evaluation activity in the world. There are thousands of such reviews commissioned by scores of funding agencies every year. Many are well done by competent site visitors (evaluators, researchers, and program experts) who produce credible reports that are helpful both to people at the site and to the agency that has commissioned the site visit. So what’s the problem? Many site visits are organized poorly, staffed inadequately, and conducted badly (poor quality fieldwork) leading to useless reports. How and why does this occur? Here’s the tip of the iceberg.
Site Visit Problems
Evaluation site visits are too often disgraceful. Over the years, I’ve heard more complaints from program staff and evaluators about wasteful and useless site visits than any other aspect of evaluation. What kinds of complaints? Little things like that site visits are ineptly conceived, planned, staffed, timed, and conducted. The stories I hear lead me to believe that site visit quality is too often poor in more developed countries and too often outrageous in the developing world. The problem strikes me as huge. Quality standards, now nonexistent, would be a start. I’ll propose such standards at the end of this article. First, let me review some of the specific challenges and problems.
Recruitment and Competence Problems
Contracting processes to hire independent international site visitors are often mired in bureaucratic solicitation procedures that create delays in getting teams assembled and into the field. Once funds are finally released, the procurement time lines are unmanageably short. I regularly see the solicitations seeking evaluators to conduct site visits, many such requests every month. I receive phone calls and e-mails asking if I’m available. But who’s available to pick up and leave on 2-weeks’ notice to spend a month doing site visits in remote areas of Eritrea? Or India? Or Ecuador? Or North Dakota? Except in unusually fortunate cases, the least qualified evaluators are the ones who are available on short notice, including novices and those who don’t have much work because they simply aren’t very good. International evaluation colleague Ricardo Wilson-Grau has dubbed the contract solicitation process “tendering madness” and he has his own ruminations about it which includes this astute observation, with which I heartily agree: Tendered evaluations commit the mortal sin of focusing on the formal written scope of work rather than on the flesh and blood. Even the most sophisticated check-lists for selecting the best tenders, and which include of course the evaluators’ qualifications, do not emphasize what matters most to the success of an evaluation. (the evaluator)
Another conversation enlarges on why this problem is so pervasive. I posed the question to another international development aid site visit contractor with 25 years’ experience. He told me “Truthfully, at the end of the day the main people available are retired, alcoholic, burned out, cynical-to-the core former aid workers and foreign service officers who want to escape their spouses and pick up a few bucks while drinking their way through the assignment.” Exaggeration? Hyperbole? Rant? As I probed his perspective, he related stories based on his years of administering, organizing, managing, and supervising a large number and variety of site visits. It became clear that he knew firsthand that of which he spoke for he was, himself, a near-retirement, alcoholic, burned-out, cynical to the core aid administrator. He confessed that he didn’t really care about the quality of the work of people hired as site visitors. “No one pays any attention to site visit reports anyway,” he explained. “It’s all part of the game, just window-dressing to give the appearance of some kind of accountability.” So he hired people like himself who followed regulations, did the paperwork right, didn’t make waves, and made good drinking buddies.
I asked another highly experienced international colleague if he thought drunks conducting site visits was a widespread problem. He launched into his own war stories, including an especially embarrassing incident in an indigenous community that struggled with alcoholism where he was the last to recognize that a person assigned to his team, feigning illness, was actually drunk. The local program people had to point out the problem to him. Then, he added, “And don’t forget to mention the sexual predators and harassers.”
Projects and communities in developing countries are especially vulnerable to poor quality site visits, but stories of incompetence and ineptitude abound in North America as well. I have found more than once that project resistance to evaluation is based in bad experiences with site visits. I was on the receiving end of such an experience early in my career when I directed the Evaluation Methodology Training program at the University of Minnesota. I recounted that experience in an oral history interview some years ago (Oral History Team, 2007), but its relevance to my views here make it worth summarizing again.
On Being Site Visited
The Evaluation Methodology Training program at the University of Minnesota was funded by the National Institute of Mental Health (NIMH, 1973–1978). I was a postdoctoral fellow in its first year and became director in the second year. We chose evaluation use to be the core integrating theme of the training program (a decision that led us to conduct research on use that resulted in utilization-focused evaluation, Patton, 1978, 2008, 2012a). The evaluation methodology training program was designed and implemented as interdisciplinary with faculty from nine departments or schools and staff from agricultural extension. We placed graduates in local and state government units doing useful evaluations and staffed evaluations in not-for-profit agencies and philanthropic foundations in Minneapolis and Saint Paul. Indeed, our primary criterion of success was the placing of graduates in evaluation positions (many of them newly created as we generated supply) and supporting them in conducting utilization-focused evaluations that were actually useful and used. But we also had other criteria and another side to the question of whether the program was successful. That’s when I experienced being site visited.
When the program came up for renewal by NIMH, the external review team consisted of two social science deans from major universities, the head of a well-respected sociology department, and a senior program manager from NIMH. None of them, by the way, had any expertise in evaluation. Their criteria did not match ours: Were we were teaching what they considered a national curriculum (based on comparison to the other four funded training programs)? Were we placing graduates of the program in tenure-track positions in major universities and national research organizations? Were we teaching experimental designs, sophisticated statistics, and cost-benefit analysis as the core methods of the program? They considered the program’s emphasis on mixed methods a weakness. They considered our qualitative study of use in federal agencies to be no more than a weak seminar project and not serious scholarship. The fact (which they did not dispute) that we had established ourselves as the premier training ground for local evaluators was judged a negative because it gave the program a professional rather than scholarly flavor. They viewed evaluation as an emerging subdiscipline of social science and wanted the program to produce evaluation research scholars. As a federally funded program, the site visitors thought we should be preparing scholars to conduct evaluation research on federal programs. That’s what they considered the purpose of the program. Our purpose for the program was different: Our focus was on training professional evaluation practitioners who could conduct useful evaluations locally. The visitors didn’t think that local evaluation was important. All in all, the official site visit evaluation was extremely negative, and the program was not refunded.
By the way, it was during the site visit that I heard these NIMH criteria for the first time. As program director for 4 years, I had no idea how the program was going to be evaluated or even that it was going to be evaluated. In reality, the funder had not communicated with us, the grant recipient, about evaluation criteria, and we had never even received any feedback on our annual reports. I was completely blindsided. Since I was very proud of the program, I had looked forward to the opportunity to show off what we had accomplished. I could not have been more surprised and disappointed. I learned a lot from that experience, especially what it’s like to be evaluated by independent, external evaluators who bring and impose their own criteria, and what they infer to be the funder’s criteria and have no interest in what has emerged at the ground level in practice.
Now, from a utilization perspective, the summative evaluation by the external evaluation team was certainly used. Indeed, it was the primary basis for the nonrenewal decision. The stakes were high, I was a novice and naive and, unbeknownst to me, it turned out I wasn’t viewed as a stakeholder. My criteria of success didn’t matter nor did the criteria of the program’s faculty advisory committee, and I was in no way involved in negotiating the external evaluation focus, design, questions, or use. The experience has no doubt influenced the intensity of my views about site visits and the standards that I’ll propose in this article.
Poor Planning and Preparation
During my subsequent 45 years conducting evaluations, I’ve learned that my early site visit experience was not unique. From program directors and staff, I have heard complaints about site visits being planned and imposed with no input from them as to timing, purpose, or process. The logistics are a nightmare. The site visit teams are ill prepared; haven’t done their homework; take up lots of staff time, expect VIP treatment; and are condescending, arrogant, rude, and too often incompetent, corrupt, or both. Junior or novice evaluators are often assigned tasks that should be done by senior evaluators (who may be off carousing instead of working). Domestically, philanthropic foundations and governmental agencies too often send program officers and administrative staff into the field to conduct site visits with little or no training, a vague scope of work, and little advance notice to sites and programs being visited. Those complaints are somewhat abstract, so let me illustrate with some concrete examples.
Examples of the Underbelly of Site Visits
A session at the joint American Evaluation Association [AEA]/Canadian Evaluation Association International Evaluation Conference in Toronto (2005) on evaluation site visits featured many horror stories documenting that external site visit evaluators hired by international donors often lack minimum evaluation competence and expertise to carry out their assignments. Alexey Kuzmin (2005) did his doctoral dissertation on evaluation capacity building. In the course of his fieldwork, he turned up a number of instances of unscrupulous evaluators engaged in unethical practices while conducting site visits. In one case, an American evaluation site visitor to a small program arrived late 1 day, left early the next, never visited the program, and refused to take the documents offered by the program. Sometime later, the program director received an urgent demand from the funder for required documentation (the very documentation that the evaluator was supposed to have taken, analyzed, and presented to the funder), which it took considerable expense to get to the funder by express courier. Subsequently, the program was denied additional funding for lack of an evaluation having been completed.
In another instance, an external evaluator arrived, one without any substantive expertise in the program’s area of focus. He spent a couple of days with the program, then disappeared leaving an unpaid hotel bill and expenses that the program had to cover because it had made arrangements for his visit. A couple of months later, the program director received a draft evaluation report for her comments, but the deadline for sending comments had already passed. The evaluation contained numerous errors and biased conclusions based on the evaluators’ own prejudices. The program director spent a considerable amount of time writing a response to the evaluation, but the evaluation, with its errors and negative judgments, had already been posted on a prominent agency web site. The program had to resort to expensive legal action to have the evaluation removed.
A third example concerns a very affable evaluator of a program undergoing an external midterm review. The American evaluation team leader was a charming man who gave a great deal of attention to the program director. In a private conversation with her, he talked in-depth about an organizational development model he had recently implemented in another country. It gradually became clear that the evaluator was offering positive evaluation findings if the director hired him as a consultant to return and implement his model in her agency. He said, “I really want to say good things about your program. I want to support it, but I need to hear some things from you before I do that.” The program director, feeling quite vulnerable, contacted a lawyer for advice about how to protect herself and her agency, advice she heeded, but found the whole experience traumatic.
A more recent example involved an evaluator who refused the full services of a translator during a 3-day site visit. His scope of work included interviewing program participants. He said that he would use the translator to ask questions in the participants’ African language, but he didn’t want the responses translated. He explained, “I’m an expert at reading body language. I’ve been all over the world. I can tell by watching people whether they are having good program experiences. Translations can’t be trusted. I trust what I see.” The program director was flabbergasted but afraid of alienating him, so she deferred.
In a similar instance, the program director contacted the donor agency that had sent the site visit evaluator and got an appropriate response. The site visit was terminated. But the program director felt great trepidation about complaining to the donor agency about an inept evaluator fearing that it would be interpreted as resistance to being accountable. The power dynamics are palpable.
Not long ago, a major philanthropic foundation in Minnesota hired a prestigious East Coast evaluation firm to do site visit fieldwork of a community development initiative in rural Minnesota. The urban born and raised site visitors had never before been on a farm and were palpably ignorant of farm life and work. The farmers interviewed were tolerant and amused, but the site visit team could not be considered culturally competent by AEA standards. Their report was rich with their own personal learnings about farm life but failed to generate programmatic insights.
Student Site Visits
Site visit problems are not always (or even usually) as dramatic as those described earlier. Nor are site visits just done by professionals. Students often conduct site visits to local agencies as part of a course, an academic paper assignment, or thesis work. Seldom do the professors who assign such site visits prepare or train students. They just send them off to get some experience in the real world. I’ve heard story after story from program directors about students who were attentive, engaged, sensitive, and a delight to involve—and I’ve heard stories about students who were insensitive, inappropriate, and clueless.
An evaluation colleague told me about observing a graduate student do a site visit while she was also on site for an evaluation she was conducting. I remember the student sitting in one of the classrooms for quite a long time, more than a couple of hours. I circled back to this room and asked her how it was going. I was especially interested in what she was learning about the culture of the school, which was the focus of her observations. I asked, so what are you learning? She yawned, looked bored, and said, “Not much,” and added that she would probably leave soon. I noticed that she had only a half-page of notes. The next time I looked for her, she was gone. Imagine my surprise when I found out that she had written up her observations as a chapter in a book on school innovation. The methods description in that chapter did not correspond to the degree and nature of fieldwork I knew had actually occurred.
Anecdotal Evidence
These anecdotes are sufficient, I hope, to illustrate the nature and variety of site visit problems. I have many more I could share, including many positive ones. They are illuminative and illustrative not representative or generalizable. I hope that what I’ve reported here are outlier anecdotes to illustrate the nature of the problem, not the scope of the problem, which is hard to determine. In discussing these examples with a former graduate student, Donna Podems, who is a full-time independent evaluation consultant working globally, commented:
I have been doing site or field visits for 20 years. It makes up a bulk of the work I do. I have had a range of experiences with site visits, some great, some awful, but the majority of my site visits are good. Not always great. Rarely bad. But good.
That strikes me as fair and balanced picture.
Stan Capela is the vice president for Quality Management and Corporate Compliance Officer for HeartShare Human Services based in New York City. Stan has been on and led a number of accreditation site visits for social service agencies. Accreditation visits involve substantial training, rigorous protocols, and diligent oversight. They are standardized with explicit criteria and methodological procedures. Such site visits are quite different from the more typical unique site visits constructed ad hoc by evaluation teams. Many of those, probably most, are thoughtfully designed and appropriately implemented. But the patterns of ineptness and ethical lackadaisicalness in the negative anecdotes I’ve shared are sufficient, in my opinion, to raise concern. Certainly, I know of agencies that work hard to ensure quality site visits and evaluators who bring knowledge, skill, and competence to the work of site visits. In the hope of further enhancing sensitivity to the nature of the problem, let me review a famous historical example examined in the fieldwork chapter of my qualitative methods book (Patton, 2015).
Site Visit Ethics: Tainted History
Potemkin Village Site Visits
The phrase Potemkin village refers to a fake village, built only to deceive and falsely impress. According to legend, in 1787 Grigory Potemkin, Governor of Crimea erected fake settlements along the banks of the Dnieper River in order to fool Russian Empress Catherine II and her entourage, including several foreign ambassadors, during their visit to the region. Potemkin wanted to show that the region, which had been devastated by war, had been rebuilt and was prospering under his governance. He constructed portable village facades to give the appearance of development. As the Empress’ barge passed by, Potemkin had his men dress up as peasants and look like they lived and worked in the flourishing village. Once the barge floated down river, the fake peasants would take down the phony facades and rebuilt at a new site during the night, repeating the deception at various spots along the route.
The phrase Potemkin village has since come to be used generally to designate and denigrate any fake creation built to deceive people into thinking that conditions are better than they really are. For observers engaged in fieldwork, the story raises the question of whether what is being observed is real and typical. For program evaluation site visits in particular, which are often short in duration and planned by the people whose program is being evaluated, the potential for making things appear rosier than they are a serious validity concern. Observers need to be able to exercise some control over where they go, what they see, and who they talk to in order to ensure that they are not being manipulated into observing only what those in authority want them to observe.
True or False
From a research and evaluation perspective, the Potemkin village story also symbolizes the problem of finding out what the truth is. Historians debate whether the Potemkin village deception actually happened and, if it did, whether the Empress was in on the plot as a way of impressing foreign ambassadors accompanying her. This last possibility is boosted by the fact that Potemkin and the Empress were lovers though, it must be acknowledged, it is not unprecedented for lovers to deceive each other. Whether the original story is true or false, the story has endured and serves as a cautionary tale for observers to look behind the facades offered by those in positions of authority who want to make a positive impression. Cultivate multiple sources. Examine a situation from diverse perspectives. Dig beneath the surface to find out what’s really going on. Triangulate.
Modern Potemkin Examples
Wikipedia (2015) has a Potemkin village entry that includes more recent examples, noting that the phrase Potemkin Village “is now used, typically in politics and economics, to describe any construction (literal or figurative) built solely to deceive others into thinking that some situation is better than it really is.” Among a long list are these modern examples (with supporting documentation): In 1982, New York City Mayor Ed Koch covered the windows of abandoned buildings in the Bronx with decals of plants and Venetian blinds to hide the blight. In 1998, the multinational energy company Enron built and maintained a fake trading floor on the sixth story of its downtown Houston headquarters to impress Wall Street analysts attending Enron’s annual shareholders meeting and included rehearsals conducted by Enron senior executives. In 2010, 22 vacant houses in a blighted part of Cleveland, Ohio, were disguised with fake doors and windows painted on the plywood panels used to close them up, so the houses looked occupied. Similar deceptions have been undertaken in Chicago and in Detroit during those cities participation in the baseball World Series. In preparation for hosting the July 2013 G8 summit in Enniskillen, Northern Ireland, large photographs were put up in the windows of closed shops in the town so as to give the appearance of thriving businesses for visitors driving past them.
The Infamous Theresienstadt Site Visit
A classic example of purposeful deception occurred near the end of World War II as rumors circulated about the Nazi extermination of Jews. The Nazis wanted to refute those rumors, so they took one camp, called Theresienstadt, in what was then Czechoslovakia, and turned it, if ever so briefly, into a model town. They shot a movie there to prove how well they treated the Jews and invited the Red Cross to inspect it. This was 1944. In preparation for the Red Cross inspection, a beautification plan was implemented by the Nazi administration. They painted buildings, planted flowers, opened stores, and put up a bandstand in the town square. They built a kindergarten in a small park, near a children’s home. They opened a coffee shop. They turned the concentration camp into a pleasant little town. And they made it a lot less crowded. Just before the Red Cross delegation arrived, the Nazis shipped 7,500 people off to the Auschwitz death camp, reducing crowding and creating more open spaces.
On June 23, 1944, the Red Cross delegates came for a one-day inspection. The Red Cross delegates went exactly where the Nazis took them—and only where they took them. They didn’t question a single prisoner. The Swiss head of the delegation took pictures showing how happy the children looked—and included them in his report. The final report of the Red Cross delegation reported that Theresienstadt looked like a normal provincial town where the “elegantly dressed women all had silk stockings scarves and stylish handbags.” The delegates also wrote that Theresienstadt was a final destination camp and that people who come there were not sent elsewhere—because that’s what the inspection team was told. In fact, by the time of the visit, some 68,000 people had already been shipped from there to the death camps.
Back to the Present
Think Theresienstadt is just yesterday’s news? I recently had a lengthy exchange with an evaluator who was disturbed by having just come from a midterm site visit of a large and important but controversial project funded by an international agency in an Asian developing country. All sites visited were selected by the host government. Only project participants selected by the host government were interviewed and always with project and/or government people present. Most requested documents and data were “not available.” The team had to produce and hand in their report in the country before they departed. It is not clear what happened to the report. On the midterm review team of four people, only one person had any evaluation background, knowledge, or training—the person who was telling me about the experience—and that person was the only one who saw any problem with the way the review was organized and the only one who complained, to no avail. I can’t reveal details of the project without breaking confidentiality, but I can tell you that the project involved issues of life and death for the intended beneficiaries and huge potential embarrassment to the host country. Every aspect of the site visit was managed and manipulated to produce findings that would portray the government’s policies in a positive light. The evaluator complained to me about the corruption of the process but admitted having signed off on the report because “otherwise I wouldn’t have been paid and my expenses wouldn’t have been reimbursed.” (At the end of this rumination, I’ll suggest some guidelines to help avoid or at least ameliorate this kind of situation.)
The Perspective From the Site
Programs on the receiving end of being visited for evaluation recognize the high stakes and act accordingly. A colleague summed up the situation succinctly: The site does not have direct power over a site visit as the visit is almost always externally mandated and contracted. The information collected will potentially be used to make decisions about the site or intervention. So a site visit is usually like a high stakes evaluation without any real accountability, affecting continuation, expansion, contraction, or even termination. So sites try to protect themselves, which is the absolutely rational thing to do. The site prepares for the visit by sweeping up messes, cleaning, priming community storytellers, arranging with counterparts or counterpart-proxies to drop by for support, filing data in surprisingly obscure places, and even going so far as to arrange a preemptive invitation by a well-known, high status expert on the pretext of providing advice to the project, but making sure the site visitor knows. The truth is that everyone is gaming the system on a site visit. But sometimes, sometimes, an external visitor is competent, attentive, and sincere and appears to have something to contribute and wants to do so. Learning can occur. Mutual understanding can emerge. Good things can happen. One such visitor came to a long-term project where I was M&E team leader and got the keys to the kingdom, listened well, offered insightful feedback, and left it a better place. He is still remembered.
Reframing Site Visits
A highly experienced evaluation colleague reports she has been “wrestling with the site visit ball-and-chain for some time.” While the term site visit can be understood to mean something very formal, “a very planned dog-and-pony show by program staff to demonstrate the greatness of a project, we’re trying very hard in a large, multi-site evaluation to avoid those situations by reframing site visits as case study data collection opportunities.” The evaluation team was working to avoid site visits becoming “program song and dance routines, but we’re running up against a program culture of expectations that are hard to change.” She went on to describe arriving at a site to conduct a planned 2-hour interview with the leadership of a project only to be confronted with a formal PowerPoint presentation with a mere half hour after the presentation allotted to ask questions. They were able to negotiate more time for an interview, but the program staff still felt compelled to make a formal presentation, thus, illustrating the high-stakes politics and pressures that can surround site visits.
Site visits may exacerbate normative differences between the culture of evaluation (emphasis on inquiry) and the culture of a program (emphasis on service and action). Such cultural differences can give rise to both ethical dilemmas (Whose interests are primary?) and overt conflict (Who decides?). Rallis (2014) has reported and analyzed a case study of such a conflict between cultures (organizational and evaluation) that presented a number of challenges including this classic at the heart of site visit reporting: “Finding balance between perspectives” Participants’ voice or researchers’ truth?” (p. 233). Rallis shares reflective questions which her evaluation team asked of themselves in this particular case. These are also generic questions that can and should be asked in any evaluation site visit?
Burden versus benefit
Do the participants in site visits carry a burden balanced with benefit? If and when we witness instances of harm and disrespect, in what ways do our actions facilitate or mitigate the situation? What actions do we, as evaluators, take to support or inhibit the harm-care balance or to facilitate respectful and ethical interactions? Did our actions serve to prevent harm to participants? Did our relationships enact care and respect? Did our efforts support fairness and reciprocity, consistently ensuring that those who bore the burden of our evaluation activities also benefited?
Public good
The AEA Guiding Principles for Evaluators remind us to go beyond analysis of particular stakeholder interests to consider the welfare of society as a whole. Did our decisions serve to protect or enhance the public good? (Adapted from Rallis, 2014, p. 243)
Whose story is told?
A school leader read an evaluation report and asked: “Whose story is this anyway? This is not what’s happening at [our school]! The question of whose story is told is one any ethical site visitor should ask. (adapted from Rallis, 2014, p. 246)
Diverse Purposes of Site Visits
The preceding examples and years of experience lead me to believe it is time for the evaluation profession to adopt minimum standards for site visits. Before doing so, and as context for doing so, it is important to acknowledge the wide range of types and purposes of site visits. Rigorous accreditation site visits are different from accountability-focused site visits. Research-focused site visits are different from program evaluation site visits (Lawrenz, Thao, & Johnson, 2012). Spending multiple days at a site is different from the infamous classroom 15-min walk-through that is all too common in schools. In developing a typology of site visits, Haynes and Murphy (In press) have noted the wide range of site visit purposes that lead to variations in what questions frame the visits, when site visits happen in the development of the program or initiative, what is at stake, and how the visit is designed, for example, the degree to which the protocol is standardized versus contextually specific. They distinguish three distinct purposes: learning with, learning about, and accountability.
Learning with. Site visits with the focus on “learning with” seek to build the capacity of those at the site. Purposed include process use, technical assistance, and informing program or initiative development. These site visits are driven, at least in part, by the learning needs of on-site staff. The stakes are often low in that the a site visit is judged as successful if something was learned that supports improved program quality, deepened understanding about what it takes to do the work, improved capacity, or program development. These may be conducted internally or in partnership with an external evaluator with a high degree of emphasis placed on the experience of the internal partners.
Learning about. Site visits that focus on “learning about” are trying to reveal some aspect of the initiative and include purposes such as a needs assessments or formative evaluation. These types of site visits are often driven by a funding or reporting requirement and the stakes can range from low to high depending on how the site visit findings are used. These site visits are judged as successful if a specific learning need was met. For example, how do midterm outcomes compare to what was stated in the logic model or theory of change? These may be conducted internally, often for reporting to an external audience, or by an external evaluator. How high stakes they are depends on how the information is used either formally or, in some cases, informally.
Accountability. Accountability site visits such as audits, licensing and accreditation site visits are often driven by the needs of an external authority and have very high stakes attached to the results. The loss of accreditation can mean the end of a health care, educational, or social service institution. These site visits are often conducted by trained professional who is following a highly proscribed protocol and inter-rater reliability is of utmost importance.
Variation is the Rule
Beyond the three purposes identified above, site visits can vary in multiple ways. I identify several dimensions for describing variations in site visits: degree of standardization from standardized to customized protocol, level of consequence from low stakes to high, personnel from individual to team, time line from to rapid turnaround/appraisal to long term, training from none to specialized for all visitors, scope from narrow, focused to comprehensive review, reporting from informal feedback to formal report: priority purpose from accountability to learning, magnitude from one site to many, and nature of site engagement from entirely externally directed to collaborative/participatory. Standards must be appropriate for and relevant to this diversity.
Site Visit Standards
Most site visits may well be conducted appropriately, professionally, and effectively. Certainly many are. Some of my most memorable experiences have been conducting site visits with engaged and insightful colleagues. Going into the field to see what is happening is essential. On the other hand, some proportion of site visits is conducted insensitively, incompetently, inappropriately, and unproductively. How many is not and cannot be known. My sense is that the numbers are sufficiently large and the poor practices are sufficiently widespread to constitute a significant problem. As a starting point toward reducing the problem, then, I propose 10 standards for the conduct of site visits (Patton, 2015, pp. 352–353).
Draft Standards for Site Visits
Competence. Ensure that site visit team members have skills and experience in qualitative observation and interviewing. Subject matter expertise, like an economics doctorate (to pick a random example), does not suffice. Nor does simply being available for the assignment.
Knowledge. Where the purpose of the site visit is evaluative, ensure that at least one team member, preferably the team leader, has evaluation knowledge and credentials. This standard deserves emphasis: Subject matter expertise does not suffice. Nor does simply being available for the assignment. Evaluation is a profession grounded in a body of knowledge, standards, specific ethical principles, and skills, including cultural competence. However, radical this suggestion may seem, teams conducting evaluation site visits should include a professional evaluator.
Preparation. Site visitors should know something about the site being visited. Whether this knowledge is procured through background materials, briefings, prior experience, or some combination thereof, one of the most common complaints from people at the site is that site visitors display a blinding ignorance of place, context, and what goes on at the site. Those who commission, fund, and otherwise engage site visitors have an obligation to ensure some reasonable level of preparation.
Site participation. People at the site should be engaged in planning and preparation for the site visit to minimize disruption to program activities and services to program beneficiaries. Logistics, costs, scheduling, access to documents, and the nature of the inquiry should be planned and transparent with reasonable consideration for how the inquiry may be intrusive.
Do no harm. Site visits are not benign. The stakes can be high. People can be put at risk. Programs can be put at risk. Good intentions, naivety, and general cluelessness are not excuses. Apologies after the fact are insufficient. Be alert to what can go wrong. Take responsibility. Be accountable. Commit as a team that you will first and foremost, do no harm. (This goes for those who commission evaluations as well as those who conduct them.)
Credible fieldwork. While those at the site should be involved and informed, they should not control the information collection in ways that undermine, significantly limit, or corrupt the inquiry. The evaluators should determine the sample of activities observed and people interviewed, insofar as possible, and should be able to arrange confidential interviews to enhance data quality. Moreover, I would propose abandoning the term “site visit” in favor on fieldwork. A visit sounds like a friendly get-together, spend some time together eating and drinking, share some stories, and hopefully a good time is had by all. But we’re not visiting. We’re supposed to be doing evaluation fieldwork—work in the field.
Neutrality. An evaluator conducting fieldwork should not have a preformed position on the intervention or the intervention model. Openness to what the data reveal is essential but cannot be assumed.
Debriefing and feedback. Before departing from the site visit (fieldwork), some appropriate form of debrief and feedback should be provided to key people at the site. In addition to providing highlights of the findings, insofar as possible and appropriate, those at the site should know when they will receive a report either oral or written or both. If no reporting will be shared, why not? What follow-up will occur?
Site review. Those at the site should have an opportunity to respond in a timely way to site visitors’ reports, to correct errors, and provide an alternative perspective on findings and judgments, if necessary. Site visit reports, often done in haste, may be infused with site visitors’ biases and preconceptions. The fact that they’re independent and external doesn’t make them neutral or accurate. Triangulation and a balance of perspectives should be the rule.
Follow-up. The agency commissioning the site visit should do some minimal follow-up (a brief survey or phone call) to assess the quality of the site visit from the perspective of the locals on site. And the site visit team should know in advance that such a follow-up assessment will take place. A philanthropic foundation in Minnesota routinely has its own program officers’ conduct due diligence site visits. The Executive Director sends out a short follow-up survey that comes directly to her—and that she reads and responds to. The results have turned up a number of problems that have informed staff development on how to conduct a site visit appropriately.
Rethinking Site Visits as Evaluation Fieldwork
These proposed standards accept site visits as an ongoing method for conducting external evaluations. But the notion that a valid, meaningful, and useful evaluation can result from a short visit by an external team with limited knowledge of context and little time to collect data is hugely problematic. The widespread use of site visits as a central form of accountability evaluation needs to be rethought, and the method is inherently flawed. So while my preferred solution is eliminating ritualistic site visits in favor of more planned fieldwork, the proposed standards are aimed at some modest improvements and preventing the worst abuses.
But 10 standards may be too many. In that regard, one principle may well suffice, the standard offered by Goodyear, Jewiss, Usinger, and Barela (2014, p. 275) as the final sentence in their important book on Qualitative Inquiry in the Practice of Evaluation, The responsibility of a qualitative evaluator is to leave the setting enriched for having conducted the evaluation.
Conclusion
This article opened by identifying the importance to evaluations of in-depth fieldwork. Exhibit 1 offered and explained 12 specific contributions of in-depth fieldwork. Although in-depth fieldwork is the evaluation ideal, the evaluation reality is that brief site visits are the more common design. The article then turned to analyzing the realities of site visits, identified systematic weaknesses in how they are commissioned and conducted, and concluded by proposing 10 standards for improving quality and use.
In introducing the site visit discussion, I warned that it consisted of part analysis and part rant. The analysis has led me to conclude that the evaluation profession has a problem, which led to the rant and the proposed standards. The analysis and rant reflect my own experiences, information sources, concerns, and values. Therefore, I invite evaluators to conduct your own analyses, reach your own conclusions, and share your own recommendations for improving our professional practice.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.