Do you really know your reporters? Evaluation methods of editors-in-chief

Introduction

The New York Times’ 2020 group, established to outline the newsroom’s future strategy and aspirations, wrote in its 2017 report,

when the newspaper business was stable, the newsroom could do without tracking the success of individual elements of the report. […] The excellence of the overall bundle overshadowed specific deficiencies. And it was cumbersome to quantify success. […] But today our business is changing rapidly. (The New York Times, 2017)

One of the most essential ‘individual elements of the report’ is the performance of the individual reporter, which has become especially essential and challenging following recent commercial, technological, occupational, and cultural shifts. Such shifts have occurred due to the following reasons.

First, decisions on hiring, firing, and promotion are taken more often in current news ecosystems, where journalistic labor has turned precarious (Anderson et al., 2012; Shapiro, 2010; Siles and Boczkowski, 2012; Zelizer, 2015). Furthermore, the margins of error in such decisions are narrowing, due to the shrinking labor and resources of news organizations and the increasing competition over crumbling revenues and fragmenting audiences (Kunelius, 2006; Mancini, 2012).

Second, professional evaluations of excellence in journalism may help to restore the deteriorating public trust in the media (PEW, 2016) by developing a ‘common lexicon of evaluation’ (Shapiro, 2010: 145), which may serve news audiences, as well as evaluators such as critics, scholars, award committees, and juries, to assess quality in journalism.

Third, traditional evaluation methods are facing transformative pressures with the emergence of new metrics of audience engagement, such as number of clicks, likes, and followers (Anderson, 2011; Karlsson and Clerwall, 2013). These metrics are added to older key performance indicators (hereinafter KPIs), such as number of items or words produced by a reporter (Picard, 1998).

Fourth, the mixed evidence on whether or not new journalistic skill sets emerge throughout the years (Bakker, 2014; Cleary and Cochie, 2011; Young and Carson, 2016) invites deeper exploration of the ways in which old and new skills are evaluated by news organizations.

To enrich our perspectives on the challenges and opportunities of evaluating creative occupations like journalism, this article combines journalism studies with theories from sociology and economy of culture. Findings are based on a series of in-depth interviews with six present and seven former editors-in-chief (‘executive editors’ in United States) from six news organizations in Israel. These people, whose voices are rarely heard in empirical contexts, offer a uniquely authoritative, panoramic, and longitudinal perspective. The composite perspective of the editors-in-chief reveals a tendency to impressionistic and informal appraisals, exposing blind spots in their field of vision and unawareness to their crucial role as generators of signals of excellence, which are essential in shaping value chains of immaterial products in cultural markets.

Theoretical background

One of the greatest challenges of news organizations in rapidly changing news ecosystems is how to make informed evaluations of the performance of their workforce (Kunelius, 2006; Shapiro, 2010). According to Deuze (2006), such evaluation becomes more challenging as labor becomes more fragmented, precarious, individualized, globalized, and networked, and the traditional distinctions between information and infotainment, news and advertising are increasingly liquefied.

Three strands in journalism studies reflect on journalists’ performance – ‘Good journalism’ studies, new skill set studies – and what we call here prescriptive works. The first strand is interested in the ways journalists and audiences evaluate ‘good journalism’. Using mainly surveys and interviews, these studies map series of parameters associated with ‘high’ journalistic performance, such as being objective, covering newsworthy stories, getting information quickly, providing analyses, and so forth (De Zúñiga and Hinsley, 2013; Kunelius, 2006; Van der Wurff and Schoenbach, 2014).

New skill set studies are interested in detecting emerging journalistic skills, typically analyzing longitudinal samples of their job advertisements (Bakker, 2014; Cleary and Cochie, 2011; Van der Wurff and Schoenbach, 2014; Wenger et al., 2014; Willnat et al., 2013; Young and Carson, 2016). These studies are often stricken by the adherence of news organizations to traditional skills. Young and Carson (2016), for example, found that the Australian media is characterized by ‘general traditionalism’ (p. 15). The only new skills they found were online skills, such as maintaining websites and using social media.

Prescriptive works refer to a small series of publications that tried to promote particular tools for more systematic evaluations of journalists (Beaupre, 1991; Giles, 1987; Picard, 1998). Giles, a former editor-in-chief and an author of a classic manual of newsroom management, developed the most impressive work in this field. In light of his aspiration for ‘objective rating of the work of individual journalists’ (Giles, 1987: 275), his work offers a systematic and formal tool for the evaluation of journalists’ performance. Beyond his slightly naïve ‘objectivism’, however, Giles’ approach is deep, cautious, and creative, highly aware of the nature of newswork and the methodological, professional, and emotional challenges of its evaluation.

The question of how journalists are to be evaluated is deeply associated with broader pressures to reshape media and limit journalists’ professional autonomy (and probably other knowledge institutions, such as academia). More formal and measurable evaluations can help close the gaps between media and audience’s notions of Good journalism and hopefully indicate ways to restore the deteriorating legitimacy of the media as a public institution (Ahva and Heikkilä, 2016); they can help pin down New skills and their impact of the changing boundaries of journalism (Carlson and Lewis, 2015; Waisbord, 2013), and they can echo trends of stability and change in the styles of news management (Andersson and Wiik, 2013; Anu, 2016; Underwood, 1993).

To set the stage for a deeper analysis of journalists’ performance evaluation, we move to discuss three major themes of scholarship: the extent to which evaluation of journalists constitutes an ‘art’ or a ‘science’, the differences between formal and informal evaluations, and the reliance on quantitative measurements and their strengths and weaknesses.

Art or science? Whether evaluation of journalists’ performance constitutes an art or science largely depends on the perceived nature of journalism itself. While a minority of scholars emphasize the mechanistic and repetitive ‘factory-like practices and processes’ (Lowrey, 2006: 492) in journalism, most scholars recognize its more complex and creative nature (Ettema and Glasser, 1998; Lenhart and Fox, 2006; Reese et al., 2007; Singer, 2003).

A combination of the repetitive and the creative is manifested in Cook’s (1998: 75) metaphor, which likens journalism to jazz playing – based on a well-known and fairly constrained repertoire of tunes while still open to improvisations. Here, then, lies the challenge: Can a jazz performance be measured and formally evaluated as prescriptive works suggested, or, as Leo Bogart (2004) argues, is the assessment of journalistic performance ‘as murky as critical judgment of poetry, chamber music or architecture’? (p. 40)

A deep reflection on evaluating artwork is suggested in cultural economy and sociology. Although addressing cultural goods rather than artists, both fields have developed profound understandings of creative labor, relevant to journalism and other producers of ‘experience goods’ (Cameron, 2003; Velthuis, 2011; Wijnberg, 2003), such as books, films, music, and art. The evaluation of experience goods is considered challenging due to their complex, intangible, variegated in style and genre, unique, and experience-based nature (Colbert, 2003), and their being of cultural products is challenging, ‘desired for their ability to generate surprises’ (Hutter, 2011: 201). Hence, unlike scientific products, creative products have a broader ‘locus of legitimate interpretation’ (Collins and Evans, 2007).

Due to these hurdles of evaluation and since every field of cultural production is ‘a universe of belief, which produces the value of the work of art’ (Bourdieu, 1996: 229), cultural markets need ‘surrogate measures of quality’ (Wijnberg, 2003: 81), which include prizes, competitions, and critics’ reviews. Audiences often prefer such value chains over their own experience (Wijnberg, 2003).

Due to their (real or perceived) knowledge, experts enjoy the symbolic capital, allowing them to evaluate ambiguous cultural products. According to Bourdieu (1993), one should include here museum curators, publishers, gallery owners and art critics (and editors-in-chief in our case) who use for their evaluations ‘the whole ensemble of political and administrative authorities competent in matters of art’ (p. 37). According to Colbert (2003), in order to establish and maintain their status as credible evaluators, these experts must demonstrate their ongoing commitment to cultural considerations over market orientation.

Methodology

To empirically explore this question, during 2015–2017, we conducted face-to-face in-depth interviews with six current and seven former editors-in-chief in six Israeli news organizations, which dominate the field in terms of staff size, professionalism, public exposure, and intermedia agenda-setting power. Editors-in-chief bring a uniquely panoramic and authoritative perspective. Their perspective integrates pretty much the different layers of the editorial hierarchy, including the major role that section editors are playing in some of the organizations.

To account for the largest and most diverse employers of journalists and be able to learn about changes in approaches and procedures of evaluation over time, we interviewed all past and present living editors-in-chief of the two oldest and most prominent national dailies: the elite Haaretz ¹ and the popular Yedioth Ahronoth. ² Interviewees include Haaretz’s three living editors-in-chief, who led the newspaper from 1988, and the deputy of one editor-in-chief who passed away, as well as all six editors-in-chief who had led Yedioth Ahronoth since 1989.

Because financial news organizations are considered closer to corporate culture than general-interest newspapers (Doyle, 2006; Manning, 2013; Tambini, 2010) and thus might be more prone to a formal style of manpower evaluations (Hempel and Sue-Chan, 2011; Hutter, 2011; Jordanous, 2012; Mumford, 2000), we also interviewed all then-serving editors-in-chief of the three financial dailies in the state: Calcalist (its first and only editor-in-chief), The Marker (the editor between 2013 and 2017), and Globes ³ (the editor between 1997 and 2017).

Except for Yedioth Ahronoth, which is a pure print organization, all interviewees bring considerable digital perspective, being in charge of their online news sites. Still, to enhance the online perspective, we included in our sample the editor-in-chief of the largest news site in Israel, Ynet (the founder and editor between 2000 and 2013, and again since 2017. The interview was conducted a year before he returned to office).

Interviews last for about 2 hours per editor, long enough to allow interviewees to reflect on their strategies, policies, considerations, observations, and practices. Our semi-structured questionnaire included questions such as the following: What are the characteristics of outstanding reporters? How can an editor detect and assess them? What are the editor’s sources of information about the performance of a particular reporter?. While obviously not immune to biases and efforts to rationalize and valorize their tenure, interviewees gave us a feeling of open discussion concerning their considerations and observations, flavored with examples and anecdotes. In order to detect intricate meanings within the data set, interview materials were analyzed thematically (Guest, 2012).

Findings

Findings show how 13 top editorial leaders, who managed six prominent Israeli print and online news organizations, have evaluated the performance of their reporters over the past 25 years. They are organized around the three leading themes presented in the theoretical background: the extent to which the editors perceive evaluation as an art or/and science, their reliance on formal/informal procedures of evaluation, and their attitude toward quantitative measures of performance.

Art or science?

Both present and former editors-in-chief showed a strong tendency to perceive evaluation of journalists as an art more than a science. Although, as elaborated below, all our interviewees applied quantitative measures at least to some extent, none of them expressed a positivist approach or a deep commitment to the feasibility of scientific measurement of reporters’ performance. At the end of the day, they perceived reporters’ evaluation as being based mainly on ‘vibrant’ impressions, echoing Bogart’s (2004) observation that measuring quality in journalism is ‘as those of any art’ (p. 44).

‘I just know’, said one editor, when asked how he recognized quality in reporting. Others talked about ‘a vibe in the air’ and ‘a knowledge that pervades from the floor’. One mentioned ‘anecdotal, eclectic, not-systematic and coincidental feedbacks’, which arrive at the editor’s desk and shape her or his impressions, while another said that the evaluation is a result of ‘ongoing criticism that you cannot present to the state comptroller, but is still very effective’. In other words, the most consensual factors in reporters’ evaluations, according to the editors-in-chief, were soft and immeasurable traits prone to editors’ subjective and intuitive judgment. From Bourdieu’s (1992) point of view, such perspective preserves the symbolic capital of the evaluating expert (i.e. the editor-in-chief) as the one who possesses such nuanced and untraceable knowledge, while Giles (1987) would probably remind us here of the risks of potential biases that accompany impressionistic assessments.

When we urged editors to unpack the impressionistic process of evaluation, asking them to elaborate specific factors of quality, most editors-in-chief emphasized reliability, not only concerning news audiences but rather mainly vis-à-vis editors. Such understanding echoes findings of ‘good journalism’ studies, according to which accurate reporting is a highly accepted parameter of quality among both journalists and audiences (De Zúñiga and Hinsley, 2013; Kunelius, 2006; Van der Wurff and Schoenbach, 2014). The set of skills described by the former and even current editors did not reveal any expectations for new skills, echoing the ‘general traditionalism’ (Young and Carson, 2016: 15) found in new skill set studies (Bakker, 2014; Cleary and Cochie, 2011; Van der Wurff and Schoenbach, 2014; Young and Carson, 2016).

According to their testimonies, editors-in-chief’s impressions of a reporter’s performance are informed by three major sources of evidence: daily dialogues with section editors, comparative reading, and external complaints.

Addressing the first source, an interviewee said, ‘As an editor-in-chief you are not the conductor of the orchestra, rather the conductor of conductors. You work through editors’. All described a routine of daily editorial meetings as well as countless informal encounters with section editors that involve discussion of reporters’ achievements and blunders, editors’ complaints about particular inaccuracies, and so forth. All editors-in-chief described their evaluation process as the product of a constant, intensive, and non-structured dialogue with mid-level editors. The uniform descriptions of our interviewees, as well as the integrative and dialogic nature of the evaluations, might indicate that intra-sectional evaluations are highly similar to those of the organizational level. This impression, however, needs further exploration, as its domain lies beyond the scope of the current study.

All editors-in-chief ascribe high authority to section editors’ opinions. Some said they never overruled them. All indicated that their editorial department has a nuanced picture of all reporters, based on daily observations and ongoing comparisons of their performance against those of their colleagues both inside and outside the news organization. One editor-in-chief, however, criticized her own overdependence on mid-level editors. This, she maintained, creates blind spots and biases, when section editors misinform editors-in-chief because they are trying to promote their own agendas or to minimize conflicts with reporters.

One interviewee estimated that 70 percent of the editor-in-chief’s evaluation of a reporter is based on section editors. He ascribed another 20 percent to the second source of evidence – the editor-in-chief’s comparative reading of reporters’ output against the competition. ‘This is where a reporter can stand out, sometimes to the editor’s surprise. Her or his ability to promote her or his items in an ocean of voices pops up’. Another editor said that ‘[as an editor-in-chief] you look at an item, see how the reporter analyzed the issue, you know how much editing was, or was not, invested here and you evaluate. It is very hard for a reporter to hide’.

The third source of evidence for the performance of their reporters were complaints about professional or ethical misconduct, which arrived routinely on the editors’ desks from external factors. To evaluate these complaints, editors employed an intricate set of parameters. ‘A lot of complaints about a reporter might indicate that this is either a great reporter or a professional troublemaker’, one explained. A second added,

Spokespersons who call and complain are nothing, part of everyday noise. Libel suits or forceful demands for an apology, however, are like [soccer] red or yellow cards. They may indicate that a ‘foul’ was committed in a procedure or in content.

While a single complaint might evade the editors’ notice, repetitive ones capture their attention, especially when addressing the complaints consumes increasing organizational resources.

Other sources of evidence that could have been expected to play a role in the evaluation process are journalism prizes for excellence and journalistic achievements (one such national prize having been granted annually in Israel since 1956), meta-journalistic discourse fora (mainly one Israeli online site and one weekly TV show dedicated to communication critique), trade unions (there are two rival journalists’ unions in Israel, although the power of the unions has declined through recent decades), and the Israel Press Council (founded in 1963 to cope with ethical issues and conduct ethics tribunals). All these entities, however, were not mentioned by our interviewees, probably attesting to their marginal effect on reporters’ evaluations.

Although the editors were generally satisfied with their methods, some interviewees admitted they have blind spots regarding most reporters, who neither excel nor fail on a daily basis. ‘The real problem are the 60 percent of your reporters, whose name is never mentioned [in conversation with section editors, daily news meetings or by external factors], neither positively nor negatively’, said one editor. Such a huge blind spot toward the majority of reporters is dramatic when margins of error in labor management are narrowing (Kunelius, 2006; Mancini, 2012).

Formal, semi-formal, and informal procedures

Although all editors-in-chief, across time and news organizations, relied similarly, as we saw, on impressionistic evaluations, we found significant differences among outlets concerning the organizational procedures of evaluation and their levels of formality. Three models of evaluation were found: formal, semi-formal, and informal.

The formal model was found in two of the three financials – Calcalist and Globes. Both have conducted annual feedback conversations with their reporters over the last 5 (Calcalist) and 10 (Globes) years. While in Calcalist it is the editor-in-chief who meets the reporters, in Globes it is the section editors. In both cases, the feedback meeting format is similar: the editor opens up with a verbal evaluation of the reporter’s strengths and weaknesses, indicating expectations and future targets, ending up with the reporter’s optional response. The preceding evaluation procedures, however, are remarkably different. Calcalist uses a detailed quantitative questionnaire, which covers 13 performance indicators. ⁴ Senior editors, equipped with reporters’ number of words’ record and front-page stories, fill in the data. In Globes, on the other hand, the human resource manager guides the editors on how to conduct the conversations and supplies data on a reporter’s productivity upon request.

The main benefit of their formal evaluation process, according to both editors-in-chief, was the opportunity to learn, often to their own surprise, about reporters’ aspirations for promotion and job changes, and to highlight unseen aspects in the relationships between editors and reporters. ‘This information helps me in managerial decisions, especially in workers mobility’, said one editor.

As in the cases recorded by Giles (1987) and Beaupre (1991) in the United States, here too the formal model of evaluation was not accepted without resistance. According to one editor, ‘it is very difficult to implement a structured methodology in a field like journalism that has no habits of measurement’. Both editors acknowledged the associations between their use of formal procedures of evaluation and their outlets’ proximity to corporate culture. One explained that ‘financial news outlets need more strict procedures due to the more complex nature of their subject matter’, while the other said, ‘We cannot cover organized corporations, if we ourselves are not’. She was highly aware, however, of the bitter irony: unlike the corporations they cover, higher evaluations of journalists weren’t followed by pay raises.

Quantitative measures of evaluation

Despite the dominance of impressionistic evaluations and informal or semi-formal procedures, only one editor-in-chief expressed an overall rejection of quantitative measurements of news output, calling it ‘a disaster’ and ‘a surrender to the publisher’s pressures’. Even she, however, admitted that such measures provide ‘one more indication’. At the other end of the spectrum, two interviewees (from different outlets and periods) enthusiastically prescribed measurement of KPIs, although neither as exclusive nor as a sufficient measure. Most editors chose a midway position – critical support of quantitative measures to evaluate journalists.

All in all, editors mentioned four main KPIs: number of words (the most discussed parameter), number of exceptional stories, popularity (traffic data), and engagement in social media, all of which were subject to further analysis by the editor-in-chief in order to assess their implications. We open with the newest parameter, engagement in social media, continuing with the remaining KPIs according to the order of their prominence in the interviews.

Discussion and conclusion

Editors-in-chief in leading Israeli news organizations evaluate their reporters daily (according to their performance, as compared to their competitors), occasionally (according to unfolding organizational needs), and, in many cases, also periodically. Despite recent pressures to intensify the evaluation of their rapidly changing workforce, and despite the availability of new metrics, these editors strongly adhere to impressionistic methods of evaluation and are consistently reluctant to adopt formal appraisal procedures and quantitative measurements. The adherence to informal and impressionistic methods reflect editors’ deep conviction that the evaluation of reporters’ performance is, and should be, an art much more than a science, allowing a broad ‘locus of legitimate interpretation’ (Collins and Evans, 2007).

Findings indicate that the formal and prescriptive approach to newsroom evaluation, that was enthusiastically promoted by Giles (1987), and his followers in US newsrooms, during the 1980s and 1990s (Beaupre, 1991; Picard, 1998), was rejected, at least in the studied case. The universality of these findings merits further research in other news cultures. The relative silence of the scholarly literature during the last 30 years about matters of evaluation may indicate, however, that similar evaluation regimes prevail in other Western countries as well and that the prerscriptive legacy had a short lifespan after its original celebration.

Findings also reveal that editors-in-chief are not attentive enough to their own reporters. Their most intensive evaluations, the ones performed on a daily basis, tend to focus on a tiny minority of reporters, those who either excelled or failed on that particular day, compared to their competitors. This means that a vast majority of the reporters, no less than 60 percent according to one interviewee, are constantly overlooked, remaining outside the evaluative daily discourse of the newsroom’s management. The limited contribution of daily and occasional evaluations could also be seen in the surprise exhibited by the few editors who handled formal appraisals and listen to their journalists’ feedback, when they encountered their reporters’ frustrations, satisfactions, and expectations. The existing evaluation regimes increase editors’ dependence on mid-level editors, who rely on the same sources and the same methods of evaluation while having their own blind spots, biases, and agendas.

While, in broader labor markets, consumers and audiences are playing an increasing role in the appraisal of workers’ performance (Adler et al., 2016; Ledford et al., 2016), when trying to develop 360 degree evaluations (Adler et al., 2016), editors consistently ignore audience metrics, preventing them from having a voice in the evaluative discourse. Ignoring audiences is becoming growingly unaffordable in an era when news organizations are desperately searching for new avenues to audiences, seeking to regain public trust and attention and to establish their relevance to readers’ public and private lives (The New York Times, 2017; The Trust Project, n.d.).

Taken together, these findings suggest that news organizations are poorly equipped to make optimal decisions pertaining to the hiring, maintaining, assigning, promoting, and laying-off of the right people.

Editors’ tendency to keep their evaluations impressionistic and informal, as well as their refusal to use audience metrics, can be seen as a strategy to preserve professional autonomy and as an editorial effort to block the ‘Trojan horses’ of managerialism, commercialism, and bureaucratization, thereby maintaining both self-governance and an autonomous definition of what counts as ‘good work’ (Waisbord, 2013; see also Carlson and Lewis, 2015). According to Waisbord (2013: 62), ‘Managerial instruments, such as review performance, […] represent the sacrificing of professional autonomy’, while according to Andersson and Wiik (2013), they represent the ‘conflicting discourses of professionalism and managerialism’ (p. 706).

While this strategy seemed to work in earlier periods, it won’t, however, necessarily protect journalistic professionalism in the longer run, since, according to Waisbord (2013: 60), professionalism in general is ‘under siege’, becoming ‘ambiguous and contested’ (p. 15) due to the growing ‘individualization’ of newswork (Deuze, 2009). In the longer run, ‘professional journalism is better at insulating itself from citizens than from powerful market interests or political sources’ (Waisbord, 2013: 225). Thus, adherence to impressionistic and haphazard evaluation looks, today, more like a sign of stagnation and conservatism, of editors succumbing to the allure of the familiar and the prevailing editorial Doxa, rather than as an enlightened adherence to professionalism. The survival of viable journalism in a fast-changing technological, commercial, and cultural environment calls for less rigidity and faster adaptation, more openness to internal and external signals, less taken-for-grantedness, and a greater awareness of potential blind spots in one’s field of vision.

From the perspective of cultural sociology and economy, editors-in-chief also neglect their responsibility as expert cultural gatekeepers (Bourdieu, 1993, 1996; Cameron, 2003; Hutter, 2011; Velthuis, 2011; Wijnberg, 2003). Due to the difficulties in evaluating cultural products, expert gatekeepers – such as museum curators, book publishers, gallery owners, art critics, and editors-in-chief – are expected to signal their quality standards and, hence, influence the value chains of their products through the application of measures such as prizes, competitions, and reviews. Public-signaling systems of this kind may be especially helpful when news organizations try to regain audiences’ trust in the media and other public institutions. Instead, editors-in-chief preserve their internal symbolic capital vis-à-vis their reporters (and maybe also their publishers), but do not apply that precious capital externally on audiences and cultural markets.

The Israeli case studied here shares a lot of similarities with other Western countries. Despite the small size and centralized character of the Hebrew-speaking media, which might encourage a somewhat less formal and less institutionalized process of performance evaluation, the national Israeli media is quite innovative and highly competitive, with large newsrooms that employ hundreds of journalists, as well as smaller ones that employ dozens. Israeli news outlets enjoy high levels of freedom and established ethical regulations while suffering from crumbling revenue models and declining public trust. Israeli editorial culture is deeply influenced by Anglo-American traditions (with some traces of East European heritage).

Further studies can focus on the ways in which evaluative culture is built up in newsrooms through editorial and commercial routines and practices, journalists’ adjustment to the pressures of metrics, and their expectation for feedback and support from their managers. Studies may focus as well on other journalistic cultures to explore the extent to which the trends described here are indeed applicable elsewhere. Such studies may also address other factors, such as the orientation of the news organization and the type of ownership, which were overlooked here, as well as other aspects that were only mentioned briefly. This may include the involvement of other intra-organizational players, like section editors, HR managers, publishers, and so forth, as well as the impact of external factors, such as journalists’ unions, prize committees, and press councils on the evaluation process.