Abstract
Publications and reports on ranking journals are commonly found in various disciplines, including hospitality and tourism. They rank the journals on a list based on subjective judgment of experts and/or objective third-party indicators. The ranking of a journal represents its quality and prestige in the field. However, there exists no commonly agreed rankings of hospitality and tourism journals due to the different methods adopted. The current commentary critiques the relevance and accuracy of such rankings by revealing the potential methodological biases and limited implications in knowledge development.
Research publications in refereed journals are frequently related to various academic activities, such as recruitment, appraisal, promotions, funding, and tenure, for scholars in different disciplines, including hospitality and tourism. Given the substantial surge of journals in recent years, scholars have attempted to rank hospitality and tourism journals according to different indicators, representing their relative impacts, prestige, and importance. Generally speaking, the higher the rank a journal receives, the better the perceived quality of its research outputs. Some scholars have adopted the method of subjective judgment by industry experts and top scholars in hospitality and tourism (Gursoy & Sandstrom, 2016; McKercher, Law, & Lam, 2006; Pechlaner, Zehrer, Matzler, & Abfalter, 2004), whereas others have advocated the use of objective measures, such as citation counts, third-party reports, and search engine indices (Law & van der Veen, 2008; McKercher, 2012; Murphy & Law, 2008). Clarivate Analytics annually publishes journal citation reports (JCR) and ranks journals into different quartiles based on their impact factors (IFs).
Ranking journals through either objective or subjective methods is debatable. Recently, published studies have proposed more hybrid methods to rank hospitality and tourism journals. Okumus, Zhao, Van Niekerk, and Law (2018) developed a balanced rating index to overcome the limitations of existing rankings by combining subjective and objective dimensions. The index considers five dimensions, namely, article influence, journal reputation, publisher’s reputation, efficiency and effectiveness, and readership. However, this study did not provide any empirical findings or actual rankings. Koseoglu (2018) introduced a novel approach that ranks hospitality and tourism journals based on the journals’ IFs and social network metrics. The approach takes into consideration the development of authorship patterns, collaboration patterns, and journal networks.
Despite the scholarly efforts expanded on the ranking practices, no commonly agreed hospitality and tourism journal ranking exists. With the exception of a few leading journals, such as Annals of Tourism Research, Tourism Management, and Journal of Travel Research, which consistently occupy the top ranks, clear distinctions exist among journals depending on the ranking methods that each study employed (Gursoy & Sandstrom, 2016; Koseoglu, 2018). Apparently, a comprehensive and accurate ranking of journals in hospitality and tourism is yet to be available. Instead of proposing another method to rank journals, the purpose of this commentary is to argue the lack of necessity for such rankings by illuminating their potential errors in methodology, as well as limited implications, if not threat, to hospitality and tourism knowledge development.
A fundamental pitfall of journal ranking, among all, is the limited range of journals included in the ranking practices. More than 300 journals in the tourism, hospitality, and events field are currently available (International Academy for the Study of Tourism, 2016). However, the journal ranking published by Clarivate Analytics evaluates and ranks only 50 of them, which are listed in the Social Sciences Citation Index. Many other reputable and well-established journals, such as International Journal of Hospitality & Tourism Administration and Journal of China Tourism Research, are out of sight. Scholarly works also tend to rank a selective number of journals. For example, Koseoglu (2018) ranked 25 hospitality and tourism journals, whereas Law and van der Veen (2008) ranked only eight hospitality journals. McKercher et al.’s (2006) work may have the widest coverage of 70 journals, but it still does not represent the full picture. Journal ranking is sensitive to the critical set of journals being selected for analysis. The rankings obtained from these reports and studies, therefore, may not be entirely objective. This case is especially true when a proportionate index is used. McKercher (2012) developed an influence ratio for each journal that represents a journal’s share of total citations in a set of selected journals in relation to the share of papers published by that journal. The influence ratio, and therefore the results, may vary significantly depending on the list of journals being selected.
However, ranking journals with different scopes, audience, and specialties is also not ideal as it presents yet another problem of unfairness and irrelevance in comparison. While some of the existing rankings divide journals into two main groups, namely, tourism and hospitality (Gursoy & Sandstrom, 2016; McKercher et al., 2006), other studies sought to combine academic journals from all related disciplines, such as tourism, hospitality, and events (Murphy & Law, 2008; Pechlaner et al., 2004). Clarivate Analytics ranks performance of journals under the broad category of “hospitality, leisure, sports, and tourism.” Journals with a generic focus, such as Tourism Management, Annals of Tourism Research, and Current Issues in Tourism (being ranked in the first quartile), are directly compared with specialized journals, such as Tourism Geographies and Tourism Economics (ranked in the fourth and second quartiles, respectively). This practice may naturally disadvantage specialized journals, which are less popular due to their dedication to more focused topics and a narrower group of audience (Law & van der Veen, 2008). Specifically, specialized journals tend to receive fewer citations than generic journals and are therefore ranked significantly lower on the list. However, does it mean that a generic journal is of better quality than a specialized journal? It seems not. Each journal has its own objectives and target audience and therefore should not be penalized because of this. Consequently, grouping and ranking journals from all hospitality and tourism disciplines can be misleading and problematic.
A journal’s seniority is rarely taken into consideration in journal rankings. A direct comparison between journals of varying length of existence is similar to comparing a junior scholar with a full professor, which is meaningless and unfair (Law, 2017). For example, the Cornell Hospitality Quarterly (previously known as Cornell Hotel and Restaurant Administration Quarterly) was established in 1960, whereas the International Journal of Hospitality & Tourism Administration started publishing only in the 2000s. When industry experts and leading scholars were asked to rank or indicate their perceived quality of each journal during a peer assessment exercise (Gursoy & Sandstrom, 2016; McKercher et al., 2006), they may consciously or unconsciously favor older journals over younger ones given the bias caused by the formers’ longer years of establishment.
Another major concern of ranking hospitality and tourism journals, which is connected to the discussion, is the indicator(s) adopted. Citation-based IFs are one of the most frequently used indicators due to their perceived objectivity. An IF refers to the average number of citations that each “citable” article published in the immediate past 2 years received in a specific year (Clarivate Analytics, 2018). The deficiencies of such a system have been emphasized and reiterated by various scholars (e.g., Law & Li, 2015; Mingers, Macri, & Petrovici, 2012; Poria, Schwartz, & Uysal, 2015). Questioning the accuracy of IFs, Law and Li (2015) have performed an audit on the IFs for four hospitality and tourism journals and found inconsistent numbers of published and expected citable articles. Subsequently, the system enhanced its transparency by disclosing the list of articles contributed to the calculation of IFs. The authors of this article thus conducted a follow-up audit for the journals being ranked in the first quartile by the JCR. The results are shown in Table 1. The sum of all citations based on citable articles is generally smaller than that used for calculation of IFs, which is based on the citations from citing journals. A close look at the counted articles shows that research notes, research papers, and review papers are included as “citable articles,” whereas proceedings, book reviews, and conference reports are excluded. The identified mismatch of total citations can possibly be attributed to the inclusion of citations for “uncounted” publications into the calculation of IFs. This method casts further doubts on the reliability of the IFs used for the generation of journal rankings.
Comparison of Total Journal Citations in 2017
Source: Clarivate Analytics (2018).
Evaluating how “impactful” a journal is on subsequent knowledge production through citation count equates quality to quantity. However, this number game is vulnerable to manipulation by journal editors in various ways. IFs tend to penalize journals that publish a large number of papers (Mingers et al., 2012). Editors can therefore stay highly restrictive on publishing a very small number of “attractive” papers to receive many citations, which will result in a high IF. This method may ensure the quality of the journal itself in terms of ranking based on IF but does not necessarily contribute more knowledge than a journal that publishes a lot of quality papers. Journal editors may also control the types of articles that they publish. For example, literature reviews and theoretical papers tend to receive more citations than empirical studies (Mingers & Xu, 2010), whereas papers covering mainstream topics, such as destination visit intention, are likely to attract more citations than innovative and niche research, such as prosocial behaviors of tourists. Their respective contributions to extending the boundaries of knowledge cannot be evaluated and compared based on a solitary measure of citation counts (Poria et al., 2015). Moreover, the variance of citation distribution among articles in a specific journal is largely ignored in the calculation of IFs. In particular, a large number of citations for a few outstanding articles may still generate a reasonably good IF for the journal and maintain its position on the list, even if its other articles remain uncited (Law, 2012).
Another way for editors to boost IFs is through the technique of self-citations. Editors can prefer papers that include many citations to the journal itself or a review paper that assesses the journal’s own papers (e.g., Law, Leung, Au, & Lee, 2013; Law, Leung, & Cheung, 2012). This method will instantaneously inflate their journals’ IFs. However, review articles performed on a dedicated journal have little, if any, “impact” toward the collective interest of the academia. They rather serve as an overview of a topic’s knowledge structure and development. JCR also reports IFs without self-citations, which are different from the published IFs. Comparing the journals rankings in the first and second quartiles based on the two different IF figures (i.e., with and without self-citations), we found that except for the first three journals (Tourism Management, Journal of Travel Research, and Annals of Tourism Research), as well as the Sport Education and Society, which demonstrate consistent rankings regardless of the IF figures used, all other journals are differently ranked if IFs without self-citations are used instead. Remarkably, the Journal of Hospitality & Tourism Research will move up by four ranks to the eighth place, whereas the Current Issues in Tourism will move up by two ranks to the fourth place. The Journal of Hospitality Marketing & Management will also move up by two ranks to the 11th place and be classified into the first quartile instead of the second as reported. These data show the potential bias in the existing method of ranking journals based on IFs and reconfirm the vulnerability of ranking results to the critical mass of journals selected for comparison.
Ranking journals based on IFs assumes that the more “impactful” a journal is, the better its quality becomes. However, the definition of “impact” is ambiguous and subjective. Essentially, a particular article can be cited for various reasons. For example, the work may be novel, well-known, or a new concept, or the work may be flawed and therefore deserves criticism (Case & Higgins, 2000). Some researchers have even commented that citing a particular piece of work may endorse the authority of their own work. Thus, “IF” may only imply popularity rather than quality, and ranking journals based on IFs is thus of limited, if any, meaningful use.
In summary, existing journal rankings in hospitality and tourism are subject to various concerns. First, none of the existing lists provide a comprehensive representation of all hospitality and tourism journals. The narrow range of journals included in the ranking practice generates variable results. Second, the direct comparison of journals with different scopes, focal points, years of experience, and target audience is problematic. Third, rankings based on seemingly objective indicators, such as IFs and citation counts, are subject to various integrity and validity problems. These concerns make ranking of hospitality and tourism journals a highly complex issue. Instead of achieving the intended purpose of allowing researchers to identify the “best-fit” journals for publication (Clarivate Analytics, 2018), the ranking practice inevitably puts journal editors into the competitive playfield and provides a misleading reference to hospitality and tourism academics. As a result, editors will become more interested in winning the number game rather than really extending the knowledge boundary. They seek to publish more “citable” papers to achieve high rankings. Furthermore, hospitality and tourism academics, who are strongly encouraged to publish their works targeting at leading journals with high reputation and impacts, will become more conservative and confined in reexploring similar research questions rather than excavating more innovative ideas as they fear of not getting their papers published. This mind-set will unfortunately limit, if not destroy, junior scholars’ creativity in their research and strangle their passions and devotion. Considerable expertise will be cultivated in a certain area at the expense of the extinction of specialized scholars in niche topics. The contribution of ranking journals to knowledge development is, therefore, limited, and it may even cause undesirable consequences to the hospitality and tourism academia.
The purpose of this commentary is not to suggest a new method of ranking journals but to disapprove the heavy reliance on such rankings. While more comprehensive and balanced methods came to light recently (Koseoglu, 2018; Okumus et al., 2018) and more ranking practices will likely emerge, the implications of such methods and rankings should be carefully considered by authors and editors before their publications. The Research Excellence Framework (http://www.ref.ac.uk/) in the United Kingdom and Research Assessment Exercise (https://www.ugc.edu.hk/eng/ugc/index.html) in Hong Kong adopt a holistic approach to assess the research performance of individual scholars based on research outputs, research grants, and esteem measures. Each person can submit up to four research items for the evaluation process, which is subject to a peer evaluation process. A similar approach can perhaps be adapted to evaluate the quality of a journal. Instead of generating an overall index or subjective rate, the determination of the quality of a journal should go down to the microlevel, with individual research output as the unit of analysis. Moreover, the hospitality and tourism industry is application oriented. Identifying the actual impacts of these journals on the industry and society would be more meaningful.
Nevertheless, if new ranking methods are to be developed, the following questions should be considered: (1) “Which area of the discipline should be ranked to allow a fair comparison (e.g., hospitality, tourism, events, or sports)?” (2) “Which indicator(s) should be used to rank journals?” (3) “If more than one indicator is proposed, what weight should be assigned to each indicator?” (4) “How long should the time coverage be (i.e., longitudinal or a one-point assessment)?” and (5) “Like any other research studies, what are the theoretical and practical implications of these rankings to the academia?” Most important, we should not overlook the role of academic journals and other forms of publication, which can serve as a platform for the interested readers to collaboratively produce, disseminate, and exchange academic knowledge. To conclude, the essence of knowledge production process should not be distorted by any form of ranking practices and judgments.
