Legitimate judgment in art,the scientific world reversed? Maintaining critical distance in evaluation

Abstract

This article considers affinities between artistic and scientific evaluations. Objectivity has been widely studied, as it is thought the foundation for legitimate judgments of truth. Yet we know comparatively little about subjectivity apart from its characterization as the obstacle to objective knowledge. In this article, I examine how subjectivity operates as an epistemic virtue in artistic evaluation, which is an especially interesting field for study given the accepted relativism of taste. Data are taken from interviews with 30 book reviewers drawn from major American newspapers including The New York Times, The Los Angeles Times, The Washington Post, and others. The data reveal that critics invest in a set of strategies to effectively ‘objectivize’ the subjectivity intrinsic to artistic evaluation, which I refer to collectively as strategies for maintaining critical distance. I argue that the concrete procedures for producing legitimate judgment in the world of art can be usefully compared to the norms for legitimate judgment in science.

Keywords

art critics evaluation/judgment knowledge legitimacy objectivity review science subjectivity

Introduction

[I]n judgments of taste and in judgments of truth, we approach intersubjectivity from different directions, but that is where, in both cases, we tend to wind up. (Shapin, 2012: 7)

Objectivity is often thought the foundation for fair and legitimate judgment in science. Many studies, such as analyses of peer review, have problematized this ideal by pointing to how emotions, interpersonal dynamics, idiosyncratic preference, and other forms of subjectivity come to play roles in evaluative processes (Cole et al., 1978; Langfeldt, 2004; Travis and Collins, 1991).

In the present analysis, I contribute to our understanding of subjectivity as a mode of knowledge making rather than as a mere impediment to objective evaluation. I use the world of artistic judgment as my case study: specifically, how book reviewers evaluate the quality of new fiction. This is a good case because of the generally accepted subjectivity of taste. The Latin proverb ‘De gustibus non est disputandum’ translates as ‘There’s no arguing for taste’. And there is no arguing for taste because it is understood to be subjective, idiosyncratic, and thus inarguable, since it is irrational (Shapin, 2012). This suggests interesting questions for those interested in objectivity and subjectivity in producing judgments: If aesthetic taste is subjective, what distinguishes legitimate and illegitimate aesthetic judgment? What are the customary rules for deploying one’s taste?

The link between subjectivity and legitimacy in evaluation has been discussed by others. Bourdieu (1983, 1989, 1993, 1996) addresses the topic of how artistic and academic value is produced in Homo Academicus and his writing on the field of cultural production. He argues that agents, such as scholars and art critics, are engaged in constant competition with one another to have their own definitions of excellence accepted as ‘commonsense’ in the field. Lamont (2009) goes beyond the idea that evaluative criteria simply reflect competitive dynamics in a field to assess the range of meanings that evaluators attach to evaluative criteria when assessing scientific excellence in the context of peer review panels. Work elsewhere has established how formal evaluative criteria effectively function to transform personal taste into a more general form (Shrum, 1996). But here I contribute insights into the socio-cognitive processes behind the deployment of these discourses – that is, how critics’ perception of literary quality is formed by broader norms concerning legitimate evaluation.

In this study, I am also interested in the social grounds of evaluation and the legitimation of particular evaluative criteria; however, I venture away from thinking about how value is produced to consider how the legitimacy of individual judgments are established. By value, I do not mean the perceived ‘accuracy’ or ‘truth’ of an evaluative judgment. The focus here is on process: How do social actors come to think of their aesthetic appraisals as good ones, and thus valuable, given the subjectivity of taste? This is an important process to understand because the means by which actors generate what they consider legitimate judgments will likely shape the final value that is produced.

To examine these questions, I interviewed 30 book critics who have written fiction reviews for major newspapers, including The New York Times, The Guardian, and The Washington Post. I asked critics to reflect upon their review process, including the practical steps they took to arrive at an assessment of its aesthetic quality. Book reviewers were the first to acknowledge the subjectivity of their literary judgments, yet they were also invested in practices and customary norms that collectively ‘objectivize’ literary judgments, transforming them into reasoned and valid evaluations, while simultaneously acknowledging the subjectivity of taste.

In what follows, I provide a review and comparison of epistemic norms for legitimate evaluation in scientific and artistic fields. Following that I provide a detailed look at the four strategies of critical distance: (1) recognizing conflicts of interest, (2) preserving critical autonomy, (3) evidence-based reviewing, and (4) reflexive reading. To explore why book critics should be so concerned with objectivizing their opinions given the accepted relativism of taste, I examine a negative case wherein a reviewer describes his failure to maintain critical distance. I conclude by considering how the rules and concrete procedures critics describe for producing legitimate judgment in art are similar or different from the case of science.

Epistemic rules for judgments of ‘taste’ and ‘truth’

Evaluation is an important part of knowledge production since it is through evaluative processes that standards of excellence are defined and reproduced (Bourdieu, 1984; Lamont and Huutoniemi, 2012). Just as definitions of value vary across contexts, so do the norms for what constitutes legitimate evaluation (Huutoniemi, 2012). Knorr-Cetina (1999) emphasizes the importance of context in her concept of epistemic cultures: the practical, symbolic, and material settings that structure what types of knowledge are produced and valued. We can similarly think of evaluative cultures, which vary by their methods of comparison (i.e. ranking or rating), their evaluative criteria, their customary rules, and other facets that constrain how evaluators behave and inform the contents of their judgments (Lamont, 2009, 2012).

Customary rules refer to the taken-for-granted norms governing what is considered appropriate behavior for social actors engaged in evaluation (Lamont, 2009). These rules range from substantive issues (i.e. what criteria are ‘irrelevant’) to epistemic issues (i.e. bracketing self-interest) and even norms for interacting with others (i.e. deferring to others’ expertise in peer-evaluation settings). Fairness in evaluation is the ‘collective outcome’ of following these rules (Lamont and Huutoniemi, 2012). In addition, procedural fairness increases the perceived legitimacy of the resulting judgment (Clay-Warner et al., 2005; Hegtvedt and Johnson, 2009).

The customary rules for fair and legitimate judgment in science are tied to norms of objectivity (Merton, 1973). Objectivity is associated with reason and rationality, while subjectivity is typically positioned opposite, as its ‘photographic negative’ (Daston and Galison, 2007: 379) or objectivity’s ‘doppelgänger’ (Shapin, 2012). Gieryn (1983) relays how science has been historically defined as ‘objective knowledge free from emotion, private interest, bias or prejudice’ in direct opposition to the ‘subjective and emotional’ character of religion and other fields (p. 785).¹ The exact rules and procedures for achieving objectivity, however, evolve along with the ideal of objectivity itself.

In their history of objectivity as an epistemic virtue, Daston and Galison (2007) note that in the 19th century, norms of mechanical objectivity prevailed such that researchers endeavored to remove all traces of individual intuition and interpretation, for example, through photography to ‘quiet the observer so nature could be heard’ (p. 785). By the 20th century, however, scientists’ interpretations were reconceived as productive rather than distorting. Researchers’ interpretations and intuitions were seen as productive for knowledge making, though scientists were expected to sort between productive intuitions and ones that were misleading and better off discarded. This skill was honed through years of professional training, which is why Daston and Galison (2007) refer to this epistemic style as ‘trained judgment’.

Why draw upon the world of scientific judgment to understand the role of subjectivity in evaluating art? At first glance, it appears that the two are worlds apart. While customary rules for scientific judgment attempt (albeit unsuccessfully) to excise subjectivity from evaluation, subjectivity in artistic judgment is not seen as something to be bracketed, but as a tool for producing knowledge in the field. And it is generally assumed that differences of artistic opinion cannot be resolved through rational or empirically grounded argument because aesthetic judgment is a matter of taste, which is personal, emotional, and idiosyncratic (Karpik, 2010). The epistemic culture of the artistic world appears to be like that of the scientific world reversed.

I argue that the differences between judgments about fact and fiction are likely overdrawn. Studies of scientific peer review provide many examples of how various forms of subjectivity manifest in the process of adjudicating scientific knowledge, including preconceived views of what constitutes ‘good science’ (Daston and Galison, 2007; Mahoney, 1977; Travis and Collins, 1991), debate about what legitimates criteria and methods for evaluation (Dirk, 1999; Guetzkow et al., 2004; Langfeldt, 2001; Laudel, 2006; Roy, 1985), and interactional dynamics among evaluators (Lamont, 2009; Lamont and Huutoniemi, 2012; Langfeldt, 2004; Olbrecht and Bornmann, 2010). These studies document how actors have been unsuccessful at eradicating subjectivity from knowledge-making processes. Furthermore, these works suggest that emotions, morals, interactional dynamics, and other ‘extra-cognitive’ considerations are intrinsic to the evaluation process, and thus they cannot be seen as extraneous contaminants (Lamont, 2009; Mansilla et al., 2010).

The goal of the present analysis is to move beyond binary thinking about objectivity in aesthetic and scientific judgments. I achieve this by examining the concrete steps critics take when adjudicating matters of artistic quality, finding that some customary rules for producing legitimate judgment in art and science bear significant similarities.

Data and methods

The study is based on in-depth interviews with 30 fiction critics who reviewed for prominent American newspapers. I use the terms ‘critics’ and ‘reviewers’ interchangeably to refer to those people who write reviews for a general newspaper audience.² Participants were selected using the following procedure. I began by generating a list of the names of people who had published a fiction review in one of three newspapers in 2007. Respondents were then randomly selected from list of names and invited to participate in the study.

I do not specify which papers I used to generate the original list of names to preserve the anonymity of my respondents, but the publications were selected based on a combination of criteria including their having (1) among the largest national circulation numbers, (2) comparable targeted audiences, and (3) a reputation for paying attention to books. For example, USA Today and the Wall Street Journal are among the more widely circulated papers but do not have stand-alone book sections. And while I used only three publications to generate my initial population of reviewers, all of my informants have reviewed for multiple publications, among them The New York Times, Los Angeles Times, The Washington Post, Chicago Tribune, The New Yorker, The Guardian, The Times (UK), and other news outlets.

There were 14 women and 16 men in the sample. The majority were ‘occasional’ reviewers, meaning that in addition to writing reviews, they make their living through freelance journalism, teaching English or Creative Writing at the college level, writing books, or more likely a combination of all the above. Only four of my respondents had full-time staff positions with a newspaper as book critics or book section editors at the time of the interview. Discussions with industry informants (publicists and book section editors) suggest that this is a typical sample. Newspapers have greatly diminished the space allotted for book reviewing, with many stand-alone book sections being folded into general ‘Entertainment’ sections due to the changing economics of print publishing. With less space for book reviews, there are fewer review assignments and a lower demand for full-time book critics on staff. The editor for a major east coast newspaper book section estimates that there are probably only 12 full-time critics left in America, and other informants estimate that it cannot be ‘more than a handful’.

Interviews were conducted over the telephone and typically lasted 60–90 minutes, during which time I asked critics to tell me their thoughts about what made for good fiction and how they went about the task of book reviewing. Despite widespread interest in evaluation, little work has investigated evaluation as a phenomenological practice (Boltanski and Thévenot, 2006; Bourdieu, 1977; Chong, 2011; Lamont, 2009; Shapin, 1996, 2012). The interviews provided critics with the opportunity to explore ideas and assumptions about their practice upon which they had not explicitly reflected, since reviewing is a very informally organized professional activity (Pool, 2007). Furthermore, customary rules for evaluation are not often formally articulated but become apparent when criticizing or praising the practices of others (Lamont, 2009). This makes a phenomenological look at evaluation all the more important for understanding the concrete steps actors take when producing what they consider to be legitimate judgments of quality or worth.

Strategies for maintaining critical distance

In Science in Action, Latour (1987) follows the production of scientific ‘facts’ (i.e. black-boxing processes). These include the rallying of ‘allies’, the use of instruments, and trials of strength that test the relation between instruments and the scientists who interpret their data. Scientists are meant to report on whatever facts and data their instruments reveal. But if a critic (or ‘dissenter’) can show that a researcher’s interpretation has been distorted by some kind of subjectivity, then the scientist is revealed as a ‘subjective individual’ rather than an ‘objective representative’ of the empirical world (p. 78). And the scientific claim in question becomes less a ‘fact’.

However, critics do aspire to objectively represent the content and quality of the novels in question, though not in the sense of being ‘right’ in any universal way. Book critics recognize that their reviews reflect their individual assessment of a book and that other readers and reviewers are likely to differ in their opinions. Critics aspire to some form of generality in their artistic judgments: their reviews should reflect their judgment as connoisseurs, not simply reflect their preferences as private consumers. So how do critics accomplish this balance between the particularities of taste and the generality of the judgments conveyed in their reviews?

Book reviewers engage in specific practices to accomplish what I term critical distance, which is an epistemic virtue characterized by critical autonomy and the achievement of an optimal balance of distance and engagement. The customary rules associated with this epistemological style include (1) recognizing conflicts of interest, (2) preserving critical autonomy, (3) reflexive reading, and (4) evidence-based reviewing. I detail the contents of each critical distance strategy, demonstrating how they effectively transform evaluators’ individual tastes into reasoned and legitimate representations of a book’s quality.

Recognizing conflicts of interest: to recuse or not to recuse

The first decision that a book reviewer must make is whether or not to accept a review assignment. Because it is common practice to invite novelists to review the work of other novelists, this can introduce conflicts of interest into reviewing. One critic explains that he avoids putting himself in situations where he reviews a book by someone he knows because ‘you are tempted to skew somehow, you know’? And if you review ‘someone who is a friend, you might artificially inflate your review. Or, somebody who is not a friend, you might artificially depress your review’. Recusing oneself in situations where a preexisting relation precludes one from writing an unprejudiced review is crucial to the perceived fairness of the review process.

Preexisting relations can be personal or professional, though sometimes the two overlap:

It’s hard not to have met someone as a writer if you’re in the literary game over time.

But there’s a difference between having met someone and knowing them, or spending a week at their house on the coast in the summer, or something.

The tightness of literary communities means that the simple fact of personal acquaintance is an impractical criterion for recusing oneself. Instead, it is a question of what degree of familiarity is sufficiently compromising. The above critic draws the line at summering at someone’s home. Another critic was offered this folk wisdom:

Somebody once told me, ‘You can review a book by people you know who are acquaintances, but if you know the names of their children, you shouldn’t review them’.

Critics are very conscious of how habits and bonds of friendship might compromise their ability to be critical as readers and reviewers, so they do their best to avoid such situations by not reviewing work by writers with whom they are too friendly.

Critics are also aware of how even indirect professional ties could bias a review. One reviewer explains:

I know friends that have gotten nasty reviews, for example, from a reviewer who was turned down by their editor, or turned down by their agent.

[T]hey have never met the writer. It’s not like they’re old high school buddies. There is no real personal connection, except they have a little axe to grind … their ego is a little bruised, and … they’re picking up the book saying, ‘What’s so good about this? Why did the agent take this book and not mine? Why did the editor take this book and not mine? Why did this person get this prize and not me?’

The critic describes how bruised egos and a sense of competition between the writer of the book and the writer of the review can bias a critic’s judgment. In this case, the reviewer might be tempted to be unduly ‘nasty’, at which point the review ceases to be a representation of the book’s quality and becomes a reflection of the reviewer’s own career frustrations. Interestingly, critics were less likely to be concerned with the opposite possibility – of reviewers and writers who shared publishers or agents being tempted to positively skew their reviews because of their shared professional association. Critics were more concerned about undue criticism than undue praise. Pool (2007) observes this asymmetry when reflecting on legal cases brought against reviewers that concerned undeserved slander rather than undeserved acclaim.

Even before turning the first page of a book, reviewers stress how important it is to screen for personal and professional conflicts of interest that might preclude them from fairly assessing of a novel. When critics describe this first task of deciding to take a review assignment, more than considerations of time or money, critics frame this as an ethical decision.

Of course, avoiding conflicts of interest is not solely a matter of personal moral conviction. Reviews are not anonymous: they are published with the reviewers’ names printed in the articles’ bylines. And many critics can bring to mind examples where the reviewer ‘had no business’ writing about another author because of some known relationship: ‘I’ve seen many times people reviewing other people who I know damn well are friends of theirs, and I think, no fucking way should they … review this book!’ Such instances undermine not only the validity of the review but also the reputation and credibility of the reviewer when readers sense that a review is being used to advance a personal agenda (‘That person had an axe to grind’).

In summary, critics recognized that social ties – both personal and professional, positive and negative – can hinder their ability to be sufficiently critical. And this would compromise the integrity of their reviews since their judgments would be based on a preexisting relationship rather than a reasoned assessment of the book at hand.

Preserving the purity of one’s opinion: peers and other contaminating influences

Beyond the prejudicing influences of friendship and professional acquaintance, critics were wary of how the opinions of other reviewers could influence their own judgment. It is not uncommon for reviews of the same book to be published around the same time since newspapers report on the same pool of newly published fiction. This creates a situation where reviewers can read what other critics have said about the same book they are reviewing.

Some critics are tempted to read others’ reviews to make sure they ‘got it [the book] right’. This insecurity relates to the radical uncertainty of artistic quality (Karpik, 2010). But respondents were clear that they would only consult others’ reviews after they had submitted their own – never beforehand:

I try not to read other people’s reviews of until I’m done with my own piece … most of us [critics] like to think that we are not easily influenced. And I’ve come to learn that is not true.

[I’ll read others’ reviews] sometimes afterwards, but never before or during; it’s just way too confusing. It’s – I mean, you think that you are good enough and pure enough, but you just aren’t. Their phrases, their opinions and their tones, it just stains you instantly.

Critics felt strongly about preserving the autonomy of their opinions, here described in terms of ‘purity’ and avoiding the potential ‘stain’ of peer opinions (Douglas, 2002 [1966]). This may be related to the fact that reading (and reviewing) is often thought of as a solitary relation between a reader and a book (Nell, 1988). And critics felt that they must protect this personal relation from ‘contaminating’ exterior influences.

Concern about the ‘purity’ of critics’ judgments extends to contempt for the outside influence of commercial interests (Bourdieu, 1993). When I asked one reviewer about whether he consults other critics’ reviews, he replied:

Reviewer:

Oh, no, never. Never, never, never … not only don’t I look at reviews of books I’m reviewing, I don’t even look at the publicity material that comes with the book. I just throw it out as soon as I open the envelope, the book. I never look at that material, no.

Interviewer:

Can I ask you why?

Reviewer:

I don’t want my reading clouded by anything that doesn’t have to do with the book itself. I don’t want my opinion biased … that to me is so extraneous and that’s wrong.

Here again, the critic foregrounds the book and its reader as the only relevant beings for assessing a book’s quality. The publicity material that accompanies the novel is not ‘the book itself’, it is ‘extraneous’ to the reader–book relation, and thus, it is ‘wrong’ to consider in this evaluative context.

What critics are describing here and above are efforts to preserve the legitimacy of their judgment by reducing the epistemic context to reader–book relation. And anything outside of critics’ personal experience of reading a novel was seen as extraneous, irrelevant, and thus corrupting.³ By limiting their evaluative criteria to the logic of artistic appreciation (e.g. the reading experience), critics are less vulnerable to attack for considering irrelevant information (e.g. their judgments are seen as legitimate).⁴ Thus far I have shown how critics made efforts to remove external sources of bias, including conflicts of interests related to friendships or grudges as well as preserving the autonomy of their literary judgments by avoiding outside influences. I now consider how critics manage internal sources of bias in reviewing.

Evidence-based reviewing

When reflecting upon the review process, many critics felt that the most difficult task was not the evaluation itself but conveying to readers why they reached a particular conclusion. When I asked how critics knew whether they liked a book, they could offer only vague responses: ‘I just know if I like it or if I don’t’. ‘Well, to some degree, it’s instinctive’. ‘I don’t think it’s anything more complicated than a gut reaction.’ The apparent unintelligibility of this evaluative moment is not unique to fiction; it has also been observed in other cultural forms including curatorial work (Acord, 2010) and fine cuisine (Leschziner, 2010). What these studies have in common is that the first ‘indicator’ of artistic quality is embodied: an emotional response.

The challenge becomes translating these personal emotional reactions into a reasoned review. As one reviewer explains: ‘When a book is good, a book is good. And I’ll identify it usually in a very sort of visceral, maybe even primitive emotional manner. It just has to hit me that way’. This critic describes her ability to decipher the value of a book in embodied terms (i.e. ‘visceral’ and ‘primitive’ reactions), but goes on to say ‘[I]t’s my job to articulate emotion in a rational format.’ Critics have to move from knowledge that is embodied or instinctive to knowledge built on rational arguments.

Many critics report feeling ‘engaged’ or ‘absorbed’ by successful fiction:

A novel succeeds if it engages me so deeply that I find myself reading without awareness of anything else that’s happening. I’m completely engaged in that creative world …

When you are in the hands of a really good writer, you believe there is something fundamentally real about the world they’re creating. The problem with a writer who isn’t as good is that, within pages, you are seeing what you are reading as a construct.

While critics disagreed on which specific titles best exemplified high-quality literature, they converged around the standard that good fiction allows the reader to be swept away into an author’s fictional landscape. And in the case of poor-quality fiction, bad writing is highly visible and can, because of it is apparent artificiality, obstruct the reader’s attempt to become engaged in the story (‘you are reading a construct’). Thus, a novel’s quality is evidenced by the affective experience of reading.

Critics also made reference to emotions as indicators of a book’s quality when recalling why they gave particular books positive or negative reviews. But these emotional indicators were framed as properties of the novels themselves. Take, for example, one reviewer who recalls a recent novel he reviewed favorably and describes as:

Very sympathetic. Very moving. Very funny. It was a great example of technical skill, [the writer’s] ability to construct stories in these different voices.

The critic describes the merits of this novel in emotional terms (i.e. ‘sympathetic’, ‘moving’, and ‘funny’). And he takes these emotional qualities as objective evidence of the author’s ‘technical skill’. The idea that readers’ emotional responses are conjured by the force of author’s talent exemplifies what Bourdieu (1996) called the ‘charismatic ideology of “creation”’ (pp. 167–169; see also Becker, 1982). Such an enchanted conception of the artist obscures the role that critics (and art dealers, publishers, and other consecrating institutions) play in actively producing the value of cultural goods (Becker, 1982; Bourdieu, 1996).

Sometimes critics addressed their emotional experiences more directly. This was often the case when critics addressed why they gave books bad reviews:

I guess the last book I hated was an historical novel […] You just never knew anything – and after 528 pages! So that was a terrible book and aroused my wrath.

[The book] was extremely annoying to me, and in fact provoked a lot of hostility in me, but I had to really think about why … everything was a hot concept.

These critics explicitly refer to their personal emotional responses to the novels (i.e. ‘hate’ and ‘hostility’). As with the previous example, however, even these internal reactions are understood as the impersonal effects of the books themselves. The critics are external to critics because they are reactions to the technical failings of the object the critics are evaluating. Critics’ emotional responses become properties of the texts as an effect of the writing – and evidence of the novel’s quality.

Reflexive reading: reading like a scientist

Given their emphasis on the embodied nature and private intimacy of the reading experience, how did critics manage to move beyond their idiosyncrasies as readers and offer a general assessment? I found that critics turn their subjective experience of books into an object of scrutiny. This involves taking a reflexive reading approach, as a reviewer describes in the following:

My process was to read this book and to constantly stop and look at the page and think, ‘Was that really good?’ I mean, I just thought I read something that was really good, but … I had to try to interrogate the reading experience.

Here, the reviewer describes a hypothesis that the book is really good. But he then describes distancing himself from his immediate reaction to ‘interrogate’ it from a different perspective. These two perspectives suggest a double-reading strategy. There is what one respondent described as the ‘civilian’ reading, which is reading for the sheer pleasure and enjoyment of experiencing a story. And then there is the ‘critical’ reading, which interrogates a book and ‘tests’ the critic’s hypothesis through analyses of language, structure, and plot to answer the question: ‘Was that really good?’ What is significant about reflexive reading is the a priori presumption that critics can separate their critical and personal sensibilities and that they can somehow ‘check’ the influences of one with the other. The same critic goes on to explain how he vets his initial impressions of the book through tests of formal critical vocabulary and devices: for example, taking a closer look at the book’s ‘perky tone’ and the ‘intelligence of the writing and the structure’.

Another critic describes a similar reflexivity when describing a time he reviewed a book negatively because the author’s voice was ‘irritating’:

[It’s] one of those things where you look at it and you go, ‘Hmm, I’m having a negative reaction to this. Is it my personal idiosyncrasy or is it that this book is not very good?’

The critic describes a moment of critical pause during which he suspends his immediate reading experience, and he investigates where his negative reactions are coming from: the civilian part (i.e. his ‘personal idiosyncrasy’) or the critical part that is simply evaluating a ‘not very good’ writer. By posing the question, the critic assumes that he is able to distinguish where his subjective preference ends and where the intrinsic qualities of the book begins. In effect, these critics engage in ‘trials of strength’ (Latour, 1987) that test to what extent critics are speaking on behalf of the book or their own behalves as private consumers.

It is not so important whether or not critics are actually able to isolate their personal preferences from their professional opinions but that they believe these are ontologically separate realities: the ‘personal idiosyncrasy’ of the reviewer and the actual quality of the book.

A common tool for successfully separating the two was through the use of direct quotations from the novels within the reviews. One critic explains it this way:

I usually like to use quotations because I like to let the book speak for itself … You don’t want it to be just a long string of quotations. You also don’t want to put a quotation in that, though it seems to fit your sort of thesis, the quotation isn’t representative in a way.

You know, every book has at least one bad sentence in it.

Many critics expressed a similar concern with how to accurately and fairly represent a novel; using quotations allows the book to ‘speak for itself’ unmediated by the reviewer. The emphasis on foregrounding the book as an independent entity again reinforces the idea that critics were trying to objectively represent the content and quality of books in their reviews.

So far I have outlined the customary rules for producing fair evaluation in book reviewing. These include specific discursive and practical strategies that critics believe enable them to avoid external sources of bias (i.e. peer influence and conflicts of interest) as well as bracket internal sources of bias (i.e. through reflexive reading and evidence-based reviewing) and thus arrive at a more valid evaluation. But given the accepted relativism of taste, what is the value added of attaining this critical distance? The value placed on critical distance as an epistemic practice is best illustrated by considering a negative case where a critic failed to maintain critical distance.

When critical distance fails

In the following example a reviewer describes a time when he gave a book a positive review for the wrong reasons and conveys critics’ investment in critical distance:

I once reviewed a book [title and author removed] which is about an obsessive love affair, or a kind of failed love affair, but about a romantic obsession let’s say. I read it, frankly, too closely on the heels of having gone through a similar experience myself, which meant that I didn’t have enough critical distance. I didn’t realize it at the time, but … I identified too closely with the character and with the situation, which meant that I misread the book.

I liked the book a lot [and] I have no qualms about having given that book a positive review, but I gave it a positive review for the wrong – I saw the wrong stuff in the book.

The critic describes an instance where he failed to maintain critical distance, resulting in a ‘misreading’ of the book. He is referring here to a failure of judgment. The reviewer allowed his subjective self (in terms of his private romantic history) to intrude on his critical judgment: he ‘identified too closely’. While he has no qualms about having written a favorable review, what was problematic was the basis for this evaluation.

This same critic reflects in hindsight that while an affinity between reviewer and book is okay, the reviewer should be able to ‘step back and really assess the book without it becoming a piece of autobiography’. Good fiction is supposed to make one believe that there is something fundamentally real about the story whether or not it conforms to an individual’s specific experience. So while critics do not dispute the subjectivity of reading, they do believe that there are ‘right’ and ‘wrong’ ways to review a novel.

The difference between right and wrong concerns how critics manage their individual preferences as readers. What critics describe as part of their review process are the ways taste can be ‘reasoned’ and ‘rationalized’. By doing so, critics’ opinions move away from purely subjective responses toward a more intersubjective appraisal of the work. It is this approach toward intersubjectively intelligible judgments that best approximates an ideal of ‘objectivity’ in arts reviewing (Shapin, 2012). While art and science appear epistemologically opposed, with the objectivity of truth at one end and the subjectivity of taste on the other, in practice, intersubjectivity is where ‘we tend to wind up’. It is this intersubjective quality – enabled by critical distance – that is foundational for the perceived legitimacy of artistic (and other) forms of evaluation.

Critical distance is also an effective strategy for drawing boundaries around the critics’ cultural authority by distinguishing their opinions from those of the average reader (Abbott, 1988). The value of critics’ opinions is not evidenced by the valence of their aesthetic judgments alone. Determining artistic (literary) quality is an uncertain affair adjudicated by the application of personal taste. Critics rely on the relative ‘neutrality’ of their recommendations – resulting from an objectivizing review process – to legitimate their professional judgments. In other words, it is not the particular valence but the perceived validity of critics’ artistic judgments that distinguishes their opinions from that of the average reader. This is an especially important style of boundary making (Gieryn, 1983) because, unlike the world of peer review in science, the actual content of the expert knowledge that supposedly sets book reviewers apart from the average reader is less clear. Thus, critical distance enabled critics to acknowledge the subjectivity of taste while simultaneously attesting to the legitimacy of their professional artistic judgments.

Discussion and conclusion

At first, the epistemic logic of aesthetics appears like the scientific world reversed – whereas objectivity and rationality are prized in the case of scientific objects of knowledge, emotions, personal preference, and taste are fundamental ways of relating to artistic objects. However, by focusing on the concrete steps critics take to judge quality, we can identify points of convergence in how the evaluation of fact and fiction is established.

Subjectivity is never completely absent from evaluative processes: whether as an epistemic virtue in artistic appreciation or vice in the case of scientific judgment. But the two converge on a disciplined approach to incorporating subjectivity – including emotions, preferences, and intuitions – into their judgments. For book reviewers, this entailed reflexive reading, which enabled them to separate idiosyncratic responses from reasoned judgments about a book itself (reporting only on the latter). This echoes the disciplined use of subjectivity seen in the trained judgment discussed by Daston and Galison (2007), in which subjective intuitions can be useful, but only if wielded in a cautious and informed way by a trained expert.

Another point of convergence is evaluators’ concern with removing extraneous and potentially corrupting factors from an evaluative situation. For book critics, these factors included friendships or conflicts of interest that would positively or negatively bias their reviews. Critics are expected to avoid taking on reviews in such cases. In the world of scientific review, there are similar concerns about how personal and professional relations might bias the evaluative process. Scholars invited to review journal articles are similarly expected to recuse themselves if there are any relations that would preclude them from impartially adjudicating a work. And nepotism is an ongoing concern (Sandström and Hällsten, 2007; Wennerås and Wold, 1997). There is a shared concern about the biasing influence of personal relations in judgments of artistic and scientific values, though the two worlds may vary in what they consider sufficiently corrupting social ties.⁵

Emotions also matter. Fiction is seen as intrinsically emotional and so reviewers of fiction use their emotions as instruments for detecting a novel’s quality: the better the book, the greater the emotional engagement. The place of emotions in science includes the emotional motivations and rewards for scientific work (Collins, 1998; Hermanowicz, 1998, 2003) and the importance of emotions like trust and commitment as a context for facilitating scientific work (Golden, 1994; Lamont, 2009; Mansilla et al., 2010; Parker and Hackett, 2012). But more could be done to develop how emotions act as a compass for directing scientific judgment, including how work is valued and how scientists decide on different courses of action (Barbalet, 2011; Polanyi, 1974).⁶

Book reviewers were also wary about letting the opinions of others shape their assessments and describe insulating themselves from the judgments of other critics. In contrast, consulting colleagues in scientific work is not only not a source of distortion but can sometimes be a means to intellectual rigor.⁷ And the value of critical distance strategies likely depends on the context of reviewing.

Evaluations may take the form of single-author reviews or can take a more collaborative form such as panel deliberations. But neither form is exclusive to evaluation in art or science. In this article, I study single-author reviews written by literary critics, but the sciences may also use single-author reviews for journal articles and academic books. Collaborative evaluation includes panel deliberations used to make funding decisions in the sciences or for selecting the winner of prizes in literature. In these contexts, to ignore the opinions of other members of the panel would be nonsensical if not detrimental. One fruitful avenue for future study would be to compare how rules for critical distance translate from individual to collaborative forms of evaluation characterized by their own customary rules for fair evaluation. When an evaluator enters a collaborative evaluation scenario, how are the rules for critical distance amended? Do norms surrounding group dynamics (e.g. maintaining collegiality or deferring to others’ expertise) take priority? How do evaluators maintain critical distance from processes such as ‘group think’?

The argument of this article does not hinge on whether critics’ statements accurately reflect their actions. What is significant is that the gestures they describe would enhance the perceived validity of their judgments. But that leaves open the question of how closely critics’ descriptions of their review process approximate their actual practices. Surely, some critics would seize the opportunity to help out a writer-friend or eviscerate a professional rival. And there are those reviewers who consult what other critics are saying so they can either toe the line or be noticeably contrarian.

While subjectivity of taste is an accepted condition of the art world, legitimate artistic judgment is not wholly subjective. It requires evaluators to ‘add dollops of objectivity’ to the process (Shapin, 2012: 7), to push their judgment toward intersubjectivity, and to aspire to some degree of generality. This is accomplished through critical distance practices that aim to remove corrupting influences and regulate the place of subjectivity in knowledge making about artistic quality. In this, we see significant affinities between the customary rules and procedures for rendering legitimate judgments in art and science so often considered worlds apart.

Footnotes

Acknowledgements

I would like to thank Vanina Leschziner, Shyon Baumann, and Michele Lamont for their guidance in developing this work. I am also grateful to the support of members of the School of History, Culture, and Communication at Erasmus University Rotterdam where I completed this manuscript. I also thank Sergio Sismondo, Michael Lynch, and two anonymous reviewers for additional suggestions that improved the piece.

Funding

Social Sciences and Humanities Research Council’s (SSHRC) Canada Graduate Scholarships Award #s: 771-2008-0033 and 767-2008-2373.

Notes

Author biography

Phillipa Chong is a PhD Candidate in the Department of Sociology at the University of Toronto. In her dissertation, Exploring the Logic of Evaluation, she examines the practical accomplishment of artistic value and legitimacy in the literary field. Chong has been a pre-doctoral fellow in Sociology at Harvard University as well as a visiting student research collaborator at Erasmus Universiteit Rotterdam. She has published articles in in Poetics and the forthcoming book, Valorizing dissonance: Cultural perspectives on newness, edited by Antal, B and Hutter, M., and Stark, D.

References

Abbott

(1988) The System of Professions: An essay on the Division of Expert Labor. Chicago, IL: The University of Chicago Press.

Acord

(2010) Beyond the head: The practical work of curating contemporary art. Qualitative Sociology 33(4): 447–467.

Barbalet

(2011) Emotions beyond regulation: Backgrounded emotions in science and trust. Emotion Review 3(1): 36–43.

Becker

(1982) Art Worlds. Berkeley, CA: University of California Press.

Boltanski

Thévenot

(2006) On Justification: Economies of Worth ( Porter

, trans.). Princeton, NJ: Princeton University Press.

Bourdieu

(1977) Outline of a Theory of Practice. Cambridge, MA: Cambridge University Press.

Bourdieu

(1983) The field of cultural production or: The economic world reversed. Poetics 12(4/5): 311–356.

Bourdieu

(1984) Distinction: A Social Critique of the Judgment of Taste ( Nice

, trans.). Cambridge, MA: Harvard University Press.

Bourdieu

(1989) Homo Academicus. Stanford, CA: Stanford University Press.

10.

Bourdieu

(1993) The Field of Cultural Production: Essays on Art and Literature. New York: Columbia University Press.

11.

Bourdieu

(1996) The Rules of Art: Genesis and Structure of the Literary Field ( Manuel

, trans.). Stanford, CA: Stanford University Press.

12.

Chong

(2011) Reading difference: How race and ethnicity function as tools for critical appraisal. Poetics 39(1): 64–84.

13.

Clay-Warner

Hegtvedt

Roman

(2005) Procedural justice distributive justice: How experiences with downsizing condition their impact on organizational commitment. Social Psychology Quarterly 68(1): 89–102.

14.

Cole

Rubin

Cole

(1978) Peer Review in the National Science Foundation: Phase One of a Study. Washington, DC: National Academy of Sciences.

15.

Collins

(1998) The Sociology of Philosophies: A global theory of intellectual change. Cambridge, MA: Harvard University Press.

16.

Daston

Galison

(2007) Objectivity. New York: Zone Books.

17.

Dirk

(1999) A measure of originality: The elements of science. Social Studies of Science 29(5): 765–776.

18.

Douglas

(2002[1996]) Purity and Danger. London, Routledge

19.

Drewry

(1966) Writing Book Reviews. Boston, MA: The Writer.

20.

Gieryn

(1983) Boundary-work and the demarcation of science from non-science: Strains and interests in professional ideologies of scientists. American Sociological Review 48(6): 781–795.

21.

Golden

(1994) On the way to Jupiter: Psychological dimensions of the Galileo mission. Social Studies of Science 24(3): 463–511.

22.

Guetzkow

Lamont

Mallard

(2004) What is originality in the humanities and the social sciences? American Sociological Review 69(2): 190–212.

23.

Hegtvedt

Johnson

(2009) Power and justice: Toward an understanding of legitimacy. American Behavioral Scientist 53(3): 376–399.

24.

Hermanowicz

(1998) The Stars Are Not Enough: Scientists – Their Passions and Professions. Chicago, IL: The University of Chicago Press.

25.

Hermanowicz

C (2003) Scientists and satisfaction. Social Studies of Science 33(1): 45–73.

26.

Hooper

(2010) Writing Reviews for Readers’ Advisory. Chicago, IL: American Library Association.

27.

Huutoniemi

(2012) Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 42(6): 897–921.

28.

Karpik

(2010) Valuing the Unique: The Economics of Singularities. Princeton, NJ: Princeton University Press.

29.

Knorr-Cetina

(1999) Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.

30.

Lamont

(2009) How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.

31.

Lamont

(2012) Toward a comparative sociology of valuation and evaluation. Annual Review of Sociology 38: 201–221.

32.

Lamont

Huutoniemi

(2012) Comparing customary rules of fairness: Evaluative practices in various types of peer review panels. In: Camic

Gross

Lamont

(eds) Social Knowledge in the Making. Chicago, IL: The University of Chicago Press, pp. 209–232.

33.

Langfeldt

(2001) The decision-making constraints and processes of grant peer review and their effects on the review outcome. Social Studies of Science 31(6): 820–841.

34.

Langfeldt

(2004) Expert panels evaluating research: Decision-making and sources of bias. Research Evaluation 13(1): 51–62.

35.

Latour

(1987) Science in Action: How to Follow Scientists and Engineers through Society. Cambridge, MA: Harvard University Press.

36.

Laudel

(2006) Conclave in the Tower of Babel: How peers review interdisciplinary research proposals. Research Evaluation 15(1): 57–68.

37.

Leschziner

(2010) Cooking logics: Cognition and reflexivity in the culinary field. In: Farrer

(ed.) Globalization Food and Social Identities in the Asia Pacific Region. Tokyo, Japan: Sophia University Institute of Comparative Culture.

38.

Mahoney

(1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1(2): 161–175.

39.

Mansilla

Lamont

Sato

(2010) Successful interdisciplinary collaborations: The contributions of shared socio-emotional-cognitive platforms to interdisciplinary synthesis. Paper presented at the American Sociological Association annual meeting. Atlanta and Atlanta Marriott Marquis, Atlanta, GA, Aug 13, 2010.

40.

Merton

(1973) The normative structure of science. In: Merton

The Sociology of Science: Theoretical and Empirical Investigations. Chicago, IL: The University of Chicago Press, pp. 267–278.

41.

Nell

(1988) The psychology of reading for pleasure: Needs and gratifications. Reading Research Quarterly 23(1): 6–50.

42.

Olbrecht

Bornmann

(2010) Panel peer review of grant applications: What do we know from research in social psychology on judgment and decision-making in groups? Research Evaluation 19(4): 293–304.

43.

Parker

Hackett

(2012) Hot spots and hot moments in scientific collaborations and social movements. American Sociological Review 77(1): 21–44.

44.

Polanyi

(1974) Personal Knowledge: Towards a Post-Critical Philosophy. Chicago, IL: The University of Chicago Press.

45.

Pool

(2007) Faint Praise: The Plight of Book Reviewing in America. Columbia, MO: University of Missouri Press.

46.

Roy

(1985) Funding science: The real defects of peer review and an alternative to it. Science, Technology, & Human Values 10(3): 73–81.

47.

Sandström

Hällsten

(2007) Persistent nepotism in peer-review. Scientometrics 74(2): 175–189.

48.

Schudson

(1989) How culture works: Perspectives from media studies on the efficacy of symbols. Theory and Society 18(2): 153–180.

49.

Schudson

(2001) The objectivity norm in American journalism. Journalism 2(2): 149–170.

50.

Shapin

(1996) A Social History of Truth: Civility and Science in Seventeenth-Century England. Chicago, IL: The University of Chicago Press.

51.

Shapin

(2012) The sciences of subjectivity. Social Studies of Science 42(2): 170–184.

52.

Shrum

(1996) Fringe and Fortune: The Role of Critics in High and Popular Art. Princeton, NJ: Princeton University Press.

53.

Travis

JDL

Collins

(1991) New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values 16(3): 322–341.

54.

Van Rees

(1983) How a literary work becomes a masterpiece: On the threefold selection practised by literary criticism. Poetics 12(4–6): 397–417.

55.

Wenneras, Christine, Agnes Wold. 1997. Nepotism and sexism in peer-review. Nature 387: 341–343.