Scholarly Merits: From Measurement to Judgment

Give Outsiders a Chance

Think of Albert Einstein, who developed his important inventions about special relativity outside of academia when he was working for the Swiss Patent Office in Bern, Switzerland. Or think of Nassim Nicholas Taleb, who introduced, for example, the concept of “antifragility” as a businessman and not while he was working as a philosopher at a university department (Taleb, 2012).

How should we select scholars for an academic track? Should good scientists “marry their intrinsic interests with its extrinsic reward and impact” (Feist, 2016, p. 896)? My recommendation would be to form a selection committee of peers; this committee would discuss the work of a candidate not only from the perspective of measurable indicators (such as the “hip hip h index”; Ruscio, 2016) but also from the perspective of judging wisely whether this person really “burns for science” (i.e., would work on the topic even without being paid for it, talks about the topic in private contexts such as a private dinner), shows at least some productivity, and has clever ideas. The measurable indices have some advantages: They are less susceptible to various kinds of biases, they are quantitative, they cannot easily be faked, and they have proven useful in distinguishing better from worse scientists (see Ruscio, 2016). At the same time, we all know that what is measurable does not describe the world completely.

And because such an evaluation is hard to do (even for a committee of intelligent, knowledgeable scholars), proxies, such as quantity and place of publications, h index, third-party money, academic distinctions, and so forth, surely help to evaluate a candidate’s productivity with respect to science. Many indicators might be used for the evaluation of scholarly merit, and they all contain some information that might help in making hiring or promotion decisions. Only the use of many different criteria will give a fair picture. Today, quantity is predominant over quality—a mono-method bias? Einstein—or Stroop (1935), who produced just one important article—did not produce much in terms of quantity, but one good idea could be enough to change the world or to give food for thought for generations of scientists!

Besides quality and quantity of products, time plays a role in the evaluation process. There is a small window of opportunity (in Germany, between the ages of approximately 35 and 45 years for a first professorship) that closes fast. Young people who start their family life have to work hard and fast to produce the materials required for an evaluation of their applications. Maybe we have to relax the age limit for appointment for a professorship in Germany. The high speed that is expected from “good” candidates stands in contrast with the ideal of a “slow professor” (Berg & Seeber, 2016): Take your time and think deeply. Because an immediate evaluation of the deep quality of the ideas from a candidate is nearly impossible (good ideas sometimes require a lot of time to be understood properly by the peers), ¹ we cannot evaluate fairly and have to accept errors in our hiring decisions. How can we minimize erroneous decisions?

Selection as Problem Solving

For me as a researcher in the field of problem solving, selecting someone for a permanent position, such as a professorship, is a kind of “complex problem solving” (see, e.g., Sternberg & Frensch, 1991): It is a nontransparent situation (the candidate will selectively show his or her best side to the selection committee), it is complex (many different and incomparable criteria are relevant), and it is polytelic (there is often not only one single goal, such as scientific quality, but also secondary goals, such as finding a cooperating colleague). The selection process is normally a group decision and is thus subject to “groupthink” (Janis, 1982). For complex problems, sometimes only “clumsy solutions” seem possible. We have to accept imperfect solutions.

In my own academic career, I have met scientists who served me as role models of good scientific practice. Not all of these people became famous, but they were at least highly respected colleagues in the scientific community. On the other hand, some of the famous people that I met showed what today are called “questionable research practices” (see, e.g., Bakker, van Dijk, & Wicherts, 2012)—these are not people who should serve as role models for younger scientists. We should add ethical behavior (in terms of good scientific practice, according to the standards of the community) to the list of criteria for scholarly merits; if we wait until scientific fraud or other ethics violations have been proven, it is too late. People who see science as an educated way of life (“Wissenschaft als Lebensform”) are different from those who see science primarily as a business that has to be managed. We have to avoid hiring managers; we should search for the intrinsically driven friends of science, those who want to make the world a better place with their work.

Conclusion

Evaluation of scholarly merit is a difficult issue. We should admit this and not avoid the burden of informed judgment by simply using “blind” quantitative measurements. Collini (2012) recommends a shift back from merely measuring to carefully judging scholarly merit—maybe this implicit depreciation of measurable units is not completely fair. The recommendation implies that selection committees have to evaluate applicants with respect to the quality and quantity of their ideas, their impact to the scientific community (to psychology), and the impact to society. But we have to be aware not to romanticize the judgment process.

In the aftermath of World War II, the German philosopher Karl Jaspers in 1946 defined the goal of universities as “bedingungslose Wahrheitsforschung” (search for truth without any conditions), and he stated that only the search for truth can be accepted as the motive for good scientists: “Universities are the place where society and state bring to development the clearest consciousness of the age. At this place, teacher and students come together with only one profession: to grasp the truth. For it is a human claim to conduct an unconditional search for truth” (Jaspers, 1946, p. 9; my own translation). Some pages later, he stated that “real scientists are fighting intensively with each other in search for truth and at the same time they express a deep solidarity with each other” (p. 59; my own translation). This description might help us to find the best people for academia: those who search for truth without any conditions, without side interests, and with a deep solidarity among all searchers for truth.