Abstract
In the spring semester of 1998, Elliot Eisner and I taught a course, Theory of Educational Evaluation. Elliot taught it from California, I from Illinois. We notified the students of our intent with the following briefing: This semester, together, we are going to try to increase our understanding of how the concept of quality is represented in program evaluation. We will find that the term, quality, is rarely used in the theoretical literature; yet the concept of quality is implied and inferred in most writings and dialogues. Quality is a matter not only for philosophers and aestheticians; it is a commonplace in life. When we seek quality, we seek merit, what is good. And lack of quality, poor quality, shortcoming, evil. As we find in Noah Webster's dictionary and Michael Scriven's Thesaurus, evaluation is the finding and expression of merit, that is, of quality. The seeking of quality is fundamental to any and all kinds of evaluation. But, for a moment, for a single study, or for grand policy, it is regularly difficult to reach a good definition of quality. Robert Pirsig wrote Zen and the Art of Motorcycle Maintenance to reflect upon the elusiveness of quality. He spoke of the elegance of the technology of keeping a bike running smoothly, relating performance to precision of measurement and diagnostics. And he spoke of a different knowing of merit: the holistic, the Eastern, the existential, the intuitive—not as raw and undeveloped, but as beyond measurement and defying present capabilities of specification. He found this dualistic definition of quality in ordinary personal experience. Certainly, we find it in our research. In evaluation, for seeking the merit of an evaluand, we have both qualitative and quantitative methods. Because a robust definition of quality is so rare, we often substitute, for quality, other more formulaic concepts, such as goal attainment or efficiency. We let them stand for quality. In the end, do these surrogate concepts better serve the client or are we drifting into disregard for what is most important?
When we talked, Elliot and I, about teaching this course, he urged that we work from realization of two meanings of the term, quality. He reminded me that quality refers not only to goodness but also to texture and to composition. Quality of a sunset can mean its beauty, its thrill, and its capacity for inspiration. But its quality also is its composition: pinnacles of cloud, pinks and the occasional green, radiating beams of light, and reflections on the water. And so with educational programs. What is the texture: the striving, the context, the sense of episode, challenge, and multiple realities? We spoke of the countenance of evaluation, drawing from two grand data banks, the observational and the judgmental: descriptive data on the left and judgmental on the right.
Technically, we can evaluate merely by registering a judgment of quality. Judgment can occur without formal analysis, without description, without reference to standards—but if it is to be an informing judgment, a judgment useful to others, it should acquaint them afresh with the evaluand. It is important to clarify for others not only how good it is but just what it is that we are calling good.
When we are doing program evaluations for other people, we have the responsibility of representing quality to them. Both the qualities and the quality. Usually, we describe first. Somehow, we should convey—whether we deeply understand it or not—the composition, the entity, and the being of the program. If we describe things well enough, we can help the reader toward complex understandings well beyond our own. With whatever media we can, we need to represent the composition, the qualities, of the program.
But the culminating representation of quality is that which conveys the merit, the well-being, the worth, the integrity, and the goodness of the program. The judgment can be normative in that it compares the program to others. Or the judgment can be ipsative, comparing the observed program with its conceptualized self, using a goal, an ideal, a dread, or comparing with what this same program was in other times or under other conditions. Goodness or quality can be defined in almost an infinity of ways, but always—to be evaluative—an abiding interest in quality is essential. The evaluator is often the one who knows best how to represent it.
Elliot urged educational researchers in general to examine more carefully how they represent the complex phenomena of our discipline, particularly teaching and learning, but extending to every intricate responsibility and policy. Ultimately, we wanted to apply these concerns about representation to that extraordinary complex representation, the representation of quality.
So how is the evaluator to best understand and represent that quality? According to Elliot, the evaluator should be a connoisseur. But what is a connoisseur? A person with deep specialized knowledge of a field. And Eisner saw connoisseurship as the art of appreciation—a special awareness, enriched by acuity and experience. It requires acquaintance with what others have experienced and expressed and placing the beholder’s experience within the experience of others. It is not just the holding of knowledge, but of crafting the beholding of the object, situation, or performance into an appreciation of the various perceptions of its value. Connoisseurship evaluation is a formalization of this beholding of an evaluand, a professional approach to providing practitioners, administrators, and the public with comprehension of the quality of the evaluand.
Thirteen doctoral students and eight auditors gathered in Urbana for weekly discussions with Elliot and myself. We met once at the Art Institute in Chicago. We talked about education more than art but perhaps still more about the art of evaluation. We all expected him to champion elegance and fastidiousness in teaching, and he did but moved quickly on to the quality and qualities of opportunity to learn.
The most difficult deliberations came with the function of criticism, which Elliot repeatedly ranked as important as the evaluator’s gaining knowledge and experience. He was not so much saying that we should challenge program advocacies; in fact, he was not dealing much with what to do with our findings of quality. But Elliot insisted that professional evaluation and connoisseurship greatly depended on the disciplined search for understanding and misunderstanding the program’s rationale and happenings, the teasing out of relationships and implications that others were not seeing.
Elliot Eisner will be remembered as a connoisseur of art education. He should also be remembered as one of the very few who saw formalized educational evaluation not so much as a social technology but as a central function of the humanities.