Towards new ways of assessing the impact of local medical journals: A proposal and call for change

Abstract

The relevance of research output to the local community is critical to changing practice. Research relevance has to be determined using measurements that show that the knowledge that arose from that research has made an impact on society. This paper, based on a literature review and preliminary research results, advocates for research impact measurements which take account of local contexts when evaluating the relevance of a journal article or indeed any research output. It concludes that a journal should go beyond traditional measurement metrics of citation analysis and bibliometrics alone as a measure of research impact. Although it is important to standardise measurements, it is also important that local communities should be encouraged to choose measurements of research output that matter to them. The proposed ways of assessing research impact are: (a) change in policies in the health sector, (b) effect on local medical treatment guidelines, (c) effect on case management, (d) use in continuous professional development, and (e) impact on local knowledge production.

Keywords

Assessment bibliometrics citation analysis impact factor knowledge production research research output Zambia

Introduction

African researchers are faced with a myriad of problems when it comes to publishing and to making the impact of their research output felt both within their country and internationally. The problems of inadequate infrastructure and finance, and a lack of publication outlets and post-publication dissemination channels, are just some of the problems faced by African researchers. There is also a lack of knowledge and competencies, such as authorship skills, copyright, where to publish, what choices to make when the manuscript is finally ready for publishing, and which journal to finally submit the manuscript to, whether locally or internationally. These problems were clearly articulated at a meeting organized by the Planet Earth Institute and held in October 2013 at the House of Lords, and chaired by the Rt Hon Lord Paul Boateng. Lord Boateng indicated that “African research should be assessed using a broad range of metrics as recommended by the San Francisco Declaration on Research Assessment” (Schemm, 2013) whilst Dr Alicia Wise stated that “much critical research remains unpublished as ‘gray literature,’ largely inaccessible beyond the walls of an institution. Much African research is also published in local, regional and national journals that are not yet indexed” (Schemm, 2013).

As argued by Smart et al. (2004: 330) “apart from financial problems, there are many infrastructural and cultural factors that affect the dissemination of quality information and have resulted in a poorly developed information economy and a lack of representation within the international research community”. It follows, therefore, that after one has been able to publish an article despite these problems, it is important that we know how that article has been utilized. For the authors, an important question to ask would be: what has been the impact of that one published article on Zambian society? This question cannot be adequately answered by traditional measurement metrics, or more recently by the alterative measurement metrics of altmetrics.

Objectives of the paper

The aim of this paper is to review various ways of measuring the impact of research output in the African context, and to propose alternative measurements, using the Medical Journal of Zambia as an example.

Methodology

The paper adopted a theoretical review of the literature, using the Medical Journal of Zambia as a case study, and combined with the personal reflections of the authors. An earlier version was presented at the International Federation of Library Associations and Institutions (IFLA) Annual Conference held in Kuala Lumpur, Malaysia from the 24th to 30th August 2018.

The Medical Journal of Zambia

The Medical Journal of Zambia (MJZ) is published quarterly by the Medical Association of Zambia (ZMA), a membership body of medical doctors in Zambia, and features articles from different medical specialties as well as those that are at the junction of medicine, health, and the social, economic, environmental, and political spheres. It is a “peer-reviewed quarterly journal intended for the publication of papers from all specialities of medicine (Internal Medicine, Surgery, Paediatrics and, Obstetrics and Gynaecology) and their subspecialties, basic sciences, public health, social medicine and medical politics”. Its mandate is to be the:

dissemination forum for medical and allied research as well as clinical opinion and practice;

source of medical information for evidence-based patient care and public health for its readership;

provide medical researchers, biomedical scientists, clinicians and all allied health worker professionals with a forum to enhance their publication skills; and

source of local continuing education to almost all health care professionals in Zambia (Kachimba, 2010).

The MJZ was first published three years after Zambia gained its independence on behalf of the ZMA in Ndola in 1967, by the Associated Reviews Limited. This also occurred one year after the University of Zambia (UNZA) Medical School was established; the journal has long been associated with the school although it has been, and is still, owned and published by the ZMA. This association with the UNZA Medical School has been established mainly because, for a long time, the Editor-in-Chief of the journal was a faculty member of the UNZA School of Medicine. What that means is that the journal secretariat has been held by the UNZA School of Medicine. However, the UNZA School of Medicine’s site of practice is the University Teaching Hospital, the largest tertiary-level hospital in Zambia. They share the same physical premises. The journal is currently published online only, with the print version ceasing publication in 2010. Among the major international databases MJZ is only indexed in African Journals Online (AJOL) which is a very important African database. Due to this lack of exposure it does not enjoy the benefits of traditional research impact metrics.

Research impact

Research impact is a subject that is of interest to many people in the knowledge production process. This interest applies both at society, institutional, and at a researcher level. People do not produce knowledge just for the sake of producing knowledge; they want to know whether that knowledge has been utilized, implemented by society, and ultimately changed their lives. What then is the meaning of research impact? Research impact may mean different things to different people and audiences. A comprehensive definition of research is offered by the United Kingdoms’ Research Excellence Framework (REF), who define research impact as ”an effect on, or change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (REF 2014, 2014: 26). The Research Councils of the United Kingdom (RCUK) define research impact as “the demonstrable contribution that excellent research makes to society and the economy” (UK Research and Innovation, n.d.), a definition that does not focus only on academic impact but also on societal impact as a whole. The London School of Economics (LSE) Public Policy Group (2011: 5) say research impact is a “recorded or otherwise auditable occasion of influence from academic research on another actor or organisation”. According to Greenhalgh et al. (2016: 1) research impact “occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base”. Dunikowski and Freeman (2016: 266) have argued that:

[the] meaning of impact for funders and policy makers might be conceived of as influence on policy, health service delivery, and population health outcomes. Academic institutions are more interested in how a discipline is represented in the academic milieu. For researchers, impact might mean the influence of their work on other researchers and practitioners in the field. For clinicians, the focus will be on the effect on daily practice.

All the above definitions emphasize the importance of research making an impact in people’s lives, whatever the sphere of activity they may be involved in.

In the health sector, just like any sector of society, knowledge production is critical to the success of patient care. Knowledge production requires a publication outlet in which the knowledge product may be published and made available to the relevant constituency. Once knowledge is produced, it requires utilization. In knowledge utilization, journals play a major role, as they are the publication outlets for most of the knowledge produced by scholars and researchers in the biomedical field. After the knowledge has been produced, it must see the light of day through a journal or other media platform. It is at that point that evaluation becomes relevant, because that knowledge now needs to be evaluated on its impact at all levels of society – researcher level, institutional level and indeed the general society as well. Measuring the impact of research output is pivotal to the exercise of knowledge production. In other development fields, this process is sometimes referred to as monitoring and evaluation. This means, that even in scientific knowledge production, it is imperative that the knowledge that is produced is evaluated on its impact, not only on society in general, but on its relevance to the local community in which that knowledge has been produced. The definitions of research impact cited above bring to the fore the argument that research impact goes beyond the academic.

Research impact evaluation

Moving beyond traditional research impact evaluation

In academia, the benchmarks that have been available to evaluate knowledge production are based on criteria that have been set by the academic community themselves. This evaluation is based on a system of peer review and may not be applicable in all instances of measuring the impact of research. Peer review which, according to Ware (2008: 4), is the “process of subjecting an author’s scholarly manuscript to the scrutiny of others who are experts in the same field, prior to publication in a journal”. In peer review, peers evaluate the likely impact of new knowledge before publication. It is grounded on the assumption that because peers are subject specialists and knowledgeable in that particular scientific field, they are best placed to judge the quality of that knowledge.

Other criteria for evaluating knowledge in academia are the impact factor and citation metrics. According to Cross (2007:1) “journal impact factors give the average number of citations to articles in a particular journal; essentially, the average number of times that articles in a journal are referenced by other articles”. This number is derived from “dividing the number of times a journal is cited by the number of articles that it publishes” (Brown, 2007: 562). The impact factor is used to measure the importance or rank of a journal by calculating the number of times its articles are cited.

In academia, impact factor metrics have been used as one of the ways of measuring research output impact. However, this has not been accepted by everyone and has led some to argue that a “journal’s impact factor says nothing at all about how well read and discussed the journal is outside the core scientific community or whether it influences health policy” (PLOS Editors, 2006: 0707). Alternative journal measurement metrics have been recommended to help highlight that impact achieved by journal articles which is not captured by traditional academic databases. These alternative measurement metrics, called altmetrics, have been enabled by the increasing usage of online platforms which has been made possible by the advent of the Internet. Altmetrics

measures the number of times a research output gets cited, tweeted about, liked, shared, bookmarked, viewed, downloaded, mentioned, added to favourites, reviewed, or discussed. It harvests these numbers from a wide variety of open source web services that count such instances, including open access journal platforms, scholarly citation databases, web-based research sharing services, and social media. (Loria, n. d.)

These article-level metrics offer a “wide range of metrics about the uptake of an individual journal article by the scientific community after publication” (Fenner, 2013: 1).

Further recognition of the inadequacy of the traditional impact factor is demonstrated by the San Francisco Declaration on Research Assessment (n.d.), which recommends that the two-year impact factor should be used with other factors such as the “5-year impact factor, EigenFactor, SCImago, h-index, editorial and publication times, etc. that provide a richer view of journal performance”.

Research impact evaluation: The Zambian context

Since some journals in Africa, just like the MJZ, are not indexed in major databases such as those of the Medline, Scopus, Web of Science, and other internationally-recognized indexing databases, it is sometimes difficult to measure and track its impact by using traditional measurement metrics of citation impact. Making local research output visible has been problematic for both researchers and institutions especially in sub-Saharan countries. This is because of a myriad of factors, such as the difficulty with which researchers in most African countries find themselves in, that is, to break through and have their articles published in ‘high-impact’ international journals. It is important therefore that the MJZ considers other alternative journal measurement metrics in order to be able to indicate its impact.

Studies that make the linkage between research output and impact are required so that the benefits that knowledge production brings to society are well articulated. Journals are produced for the same reasons: that articles published in those journals are used by people to make a positive change in their lives. In this case, the knowledge contained in a journal such as the MJZ should be utilized by medical and health practitioners to improve their clinical practice and ultimately for the continued well-being of Zambian society. When medical and health practitioners improve their clinical practice by utilizing the knowledge obtained in local journals, they improve the services they offer to both patients and the general public.

Although traditional measurement metrics of bibliometrics, citation analysis, and other alternative measurement metrics are important in trying to establish the impact of a journal, it is critical that measurements that are qualitative in nature are also taken into consideration. This is even more so when researchers are involved in tackling local issues that may not be relevant to international audiences. For instance, in Zambia, such research may be on Konzo disease, a neglected disease of limited interest to health professionals in other countries. Konzo disease is a “distinct neurological entity with selective upper motor neuron damage, characterised by an abrupt onset of an irreversible, non-progressive, and symmetrical spastic para/tetraparesis” (Nzwalo and Cliff, 2011). According to Zambian studies, this disease has been diagnosed in people who eat a lot of cassava, especially in the western part of Zambia. An article in the MJZ reporting “Increased sensitization of health workers leading to detection of unintended cases of acute flaccid paralysis: A case of a ‘Konzo’ outbreak in Western Zambia” (Mtonga et al., 2016) would be of particular interest to the Zambian medical community. Statistics on its citation patterns might be useful, but for Zambians, the interest would be whether health workers are able to learn from the research about the management of Konzo disease. Additionally, there might be need to change local treatment guidelines so that any health worker in Zambia who comes across a case of Konzo disease would know how to handle the patient anywhere in Zambia.

How then does one decide how important such a journal is to its local community? Should we use traditional measurements metrics to decide the impact of the journal? Whatever measurements are adopted, they should be tied to the aims and objectives in which that particular knowledge was produced. This paper argues that it is imperative that measuring the impact of a journal goes beyond metrics. In discussing these measurements this paper is cognizant of the manner in which the journal itself sees its impact would be felt by their constituents by focusing on alternative ways of measuring the impact of local research.

It is clear that, from the onset, the MJZ saw itself as going beyond academia; and perhaps the fact that it is published by the ZMA demonstrates that its focus is mainly on impacting health care delivery in the country. A relevant question to ask is that one that has been alluded to by Maeseneer et al. (2007: 402) that; ”journal impact factor has an impact on how research and researchers are valued by their academic peers and on the allocation of research funds, but does it have an impact on what research is ultimately about, improving quality of life for all citizens?”. There are arguments advanced that advocate for measuring a journal’s worth by looking at its contributions to society in general (Maeseneer et al., 2007). How do we arrive at a point where we can say: this is how the local community benefitted from the information coming from a journal article where a journal is geographically located?

Suggested ways of measuring research impact

Although traditional measurement metrics such as citation analysis, and the related field of bibliometrics, are important in trying to establish the impact of a journal, this paper argues that it is essential to go beyond these, particularly in the context of Africa or the global South. In discussing these alternative metrics, this paper puts forward some ways of measuring research impact. In this regard, Kachimba (2010) at the African Journal Partnership Project Annual Review and Planning Meeting held from 8th to 19th May 2010 in Atlanta, Georgia, (United States) looked at the impact of the MJZ through several perspectives, namely:

Impact with respect to its publication scope;

Impact with respect to its readership and circulation;

Impact with respect to calibre of its authors. (Kachimba, 2010)

Following on from the above yardsticks, we argue that the following aspects are important in measuring the impact of the MJZ

Impact on policies in the health sector

Use of the research output contained in the MJZ can change policies in the health care sector. These changes can either be targeted towards the health care provider, officials in the Ministry of Health, health care seekers, patients, or the general population. The information in the MJZ informs the Ministry of Health and other policy makers on the problems and progress that happen in Zambia’s health system through research and thereby help in changing the policy direction of the country. However, these changes are not always definite; as it has often been argued by Cruz Rivera et al. (2017: 16) who quote van Driel et al. (2007) stating that:

whilst it is true that wider dissemination, especially when delivered via world-leading international journals, may well lead eventually to changes in healthcare, this is by no means certain. For instance, case studies evaluated by Flinders University of Australia demonstrated that some research projects with non-peer-reviewed publications led to significant changes in health policy, whilst the studies with peer-reviewed publications did not result in any type of impact.

These findings indicate that sometimes research published in leading international journals may not meet the specific local needs. This is why this paper strongly makes the assertion that the MJZ’s impact be measured according to how its articles influences local policy, i.e. if its contents are being used to change policy direction. Others such as Ravenscroft et al. (2017), claim that Altmetric.com and other altmetrics providers already examine some policy documents as part of their offerings. However, there are some concerns about the coverage, or lack thereof, of policy providers with respect to geographical regions and scientific fields (Ravenscroft et al., 2017: 2). The major criticism of altmetrics is similar to the criticism of the citation analysis and bibliometrics. They are based on metrics that do not respond to the core needs of the local community. This is because most often the “recipients of research can reliably assess the relevance of publications in a practical context” (Bornmann and Haunschild, 2017: 941). It has further been alleged that ”research funders and science politicians are interested in the broad impact of science, i.e. the impact of the sciences beyond the sciences” (Haunschild and Bornmann, 2017: 1210), and that “data can be used for assessing policy impact, but there are clearly still problems with how the software identifies and attributes research papers” (Tattersall and Carroll, 2018: 10). Specifically, there are no measurement metrics that respond to the need of the local community. They further argue that existing analysis of policy documents is based on counting references in these documents to scholarly articles (Tattersall and Carroll, 2018: 10), while this paper argues that counting references in the documents to scholarly articles is not good enough, but utilization of policy contents will show appreciation of its existence.

Impact on local medical treatment guidelines

Local articles should reflect changes in treatment guidelines as adopted by the country’s Ministry of Health. This is helpful as medical practitioners can reflect on the treatment regimens in their practice. A local article discussing the treatment of a particular ailment might be more relevant to the local context than to the international context. Studies such as ‘A Review of the 2010 WHO Adult Antiretroviral Therapy Guidelines: Implications and Realities of these changes for Zambia’ (Patel et al., 2010) can bring significant changes in policy and benefit to the local population. The article speaks more to the local context as it brings out issues that are specifically relevant to Zambia. Consequently, the article can be used to change treatment guidelines to meet the needs of the local people; then its impact should be rated high as it has a substantial impact on the general health of the locals. Weiss (2007) states that measuring the penetration of research into the clinical domain is thus a complex but important first step in assessing the impact of science on patient health. Weiss further contends that “measuring awareness of a particular research finding is insufficient, for one simple reason: there is a tremendous difference between knowing and doing” (Weiss, 2007: 210). Weiss (2007) cites Wilson et al. (2008) who noted that:

the naive assumption that when research evidence is made available it is routinely accessed by practitioners, appraised, and then applied in practice is now largely discredited. While awareness of a practice guideline or a research-based recommendation is important, it is rarely, by itself, sufficient to change practice. (Weiss, 2007: 210)

This then demands a concern about the extent to which results are implemented or/and utilized, as high citation rates, likes, tweets and referencing may not mean usage.

Impact on local case management

Case management may sometimes differ; and, therefore, the relevance of local articles discussing the management of a particular disease in one part of Zambia might be more relevant to another area or town in Zambia than a city in other regions of the world such as Asia, Europe, Latin America, United Kingdom, or the United States of America. The MJZ publishes articles on diverse medical conditions, some which are Zambia specific or rare cases in the tropics. The rare disease called Konzo, identified in the Western province of the country (Mtonga et al., 2016) and reported in the MJZ, was discussed above. This may serve as a notification to the rest of the Zambian medical community to look out for the disease as well as to learn about its management. Such publications may not come out in the traditional measurement metrics and alterative measurement metrics of altmetrics but may be significant in the local journal measuring metrics, as their importance may only be felt by the locals. This is why this calls for a more comprehensive way of measuring a journal’s impact to include utilization of evidence from such publications in policy and in practice.

Impact on continuous professional education

Research output is important in the knowledge management chain. In order for health practitioners to perfect their practice, they require access to knowledge. It is also important for health practitioners to access information that is relevant to their studies, be they ordinary studies, specialist studies, or continuous professional development (CPD). CPD requires access to empirical evidence in order for it to be successful (Collin et al., 2012). CPD is a stage that is necessary to research, as it is a stage where the evidence arising out of knowledge production is to be utilized.

Impact on knowledge production

Medicine is a very dynamic field that requires one to keep up to date with recently-validated information. It is important that Zambia’s local knowledge not only impacts the health care delivery process, but is continuously used in the local knowledge production process. This is only possible if Zambian authors are encouraged to write on Zambian health issues, issues that affect Zambians, and that these articles are published in a local Zambian journal such as the MJZ. Eriksson (2000: 248) notes that there is a “need for learning and knowledge-production using both quantitative and qualitative approaches for developing the evidence base for public health action”. Knowledge production is a continuous cycle that needs both learning and research to feed into the process of knowledge production so as to produce new evidence.

Conclusion

The need for publishing locally by Zambia authors cannot be overemphasized when one looks at the publishing focus of the MJZ. It is important that local authors publish in local journals for their local audience, but this should not mean that 100% of a local journal’s authorship should only be from within the country. There should be a balance between local articles and those coming from international authors; it is critical that there is a cross pollination of ideas. It follows, therefore, that local Zambian authors should also be encouraged to publish in international journals. In this context, Smart et al. (2004: 330) have argued that:

Most African journal editors and African scholars see potential advantages in publishing within their own region. If academic publishing is seen as an exchange of ideas and information among scholars working in the same area rather than as a tool for promotion up the academic ladder, it is clear that a regional pool of ideas governed by the interests of those living there is desirable. If this sounds parochial, consider how surprised Canadian scholars would be to find their publications largely dominated by the interests and theories of Malaysians rather than their own.

Local journal production is important to the success of any local knowledge production process. It is generally agreed that knowledge production underpins any profession and medicine is no exception. However, knowledge once produced needs to be utilized, and in order to know what the impact of the produced knowledge is on society, it needs to be measured. The measurements must be appropriate to the context that it is measuring. Although it is important that the assessment of the impact of knowledge on society is standardized, we argue that it should be a combination of measurements that are cognizant of the local environment. In recognition of the local context, it is significant that the MJZ considers other alternative journal measurements in order to be able to indicate its impact to Zambian society. It is also critical that metrics that look at citation data, page views, and downloads are used in conjunction with qualitative evaluation to indicate the true impact of a journal and its responsibility to its local community. The knowledge contained in a journal such as the MJZ should be utilized by medical and health practitioners to improve their clinical practice and ultimately for the continued well-being of the Zambian society. And by improving their clinical practice, they improve the services they offer to both patients and the public. In the end, a traditional evaluative research conducted over time intervals would be able to indicate the impact and relevance of the MJZ to Zambians. It might be apt to proclaim that “inadequacies of simple bibliometrics and altmetrics illustrate the adage that not everything that counts can be counted” (Dunikowski and Freeman, 2016: 268), but rather quantitative statistics must be combined with qualitative data to measure the true value of a journal to its local community. The San Francisco Declaration recommended that we “challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs” and ”consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice” (San Francisco Declaration on Research Assessment, n.d.).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bornmann

Haunschild

(2017) Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Scientometrics 110(2): 937–943.

Brown

(2007) How impact factors changed medical publishing – and science. BMJ (Clinical research edn.) 334 (7593): 561–564.

Collin

van der Heijden

Lewis

(2012) Continuing professional development. International Journal of Training and Development 16(3): 155–163.

Cross

(2007) Impact factors – Back to basics. Editors’ Bulletin 1(2): 23–30.

Cruz Rivera

, et al. (2017) Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Medicine 14(8): e1002370–e1002370.

Dunikowski

Freeman

(2016) Impact of family medicine research: Bibliometrics and beyond. Canadian Family Physician Medecin de Famille Canadien 62(3): 266–268.

Eriksson

(2000) Review Article: Learning and knowledge-production for public health: A review of approaches to evidence-based public health. Scandinavian Journal of Public Health 28(4): 298–308.

Fenner

(2013) What can article-level metrics do for you? PLoS Biology 11(10): e1001687.

Greenhalgh

Raftery

Hanney

, et al. (2016) Research impact: A narrative review. BMC Medicine 14: 78.

10.

Haunschild

Bornmann

(2017) How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics 110(3): 1209–1216.

11.

Kachimba

(2010) Medical Journal of Zambia. In: AJPP meeting, Atlanta, Georgia, USA, 18–19 May 2010. Available at: karibouconnections.net/ajpp/downloads/1005mtg_mjz.ppt%0A%0A (accessed 23 January 2019).

12.

London School of Economics (LSE) Public Policy Group (2011) Maximizing the Impacts of your Research: A Handbook for Social Scientists. Available at: https://blogs.lse.ac.uk/impactofsocialsciences/files/2018/06/Handbook-PDF-for-the-LSE-impact-blog-April-2011.pdf (accessed 12 April 2019).

13.

Loria

(n. d.) Altmetrics and open access: A measure of public interest. Australia Open Access Strategy Group. Available at: https://aoasg.org.au/altmetrics-and-open-access-a-measure-of-public-interest/ (accessed 24 January 2019).

14.

Maeseneer

van Driel

Maier

(2007) Measuring the impact of family medicine research: Scientific citations or societal impact? Family Practice 24(5): 401–402.

15.

Medical Journal of Zambia (2018) About the Journal. Available at: https://www.mjz.co.zm/index.php/mjz/about (accessed 20 January 2019).

16.

Mtonga

, et al. (2016) Increased sensitization of health workers leading to detection of unintended cases of acute flaccid Paralysis: A case of a ‘Konzo’ outbreak in Western Zambia. Medical Journal of Zambia 43(2): 57–60.

17.

Nzwalo

Cliff

(2011) Konzo: From poverty, cassava, and cyanogen intake to toxico-nutritional neurological disease. PLoS Neglected Tropical Diseases 5(6): e1051.

18.

Patel

Moyo

Bositis

(2010) A review of the 2010 WHO adult antiretroviral therapy guidelines: Implications and realities of these changes for Zambia. Medical Journal of Zambia 37(2): 118–124.

19.

PlOS Editors (2006) The impact factor game. It is time to find a better way to assess the scientific literature. PLoS Medicine 3(6): e291–e291.

20.

Ravenscroft

, et al. (2017) Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements. PloS one 12(3): e0173152–e0173152.

21.

REF (2014) Assessment framework and guidance on submissions. Available at: https://www.ref.ac.uk/2014/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf (accessed 22 April 2019).

22.

San Francisco Declaration on Research Assessment (n.d).San Francisco Declaration on Research Assessment: Putting science into the assessment of research. Available at: https://cetaf.org/sites/default/files/documents/san_francisco_declaration._final_dora.pdf (accessed 20 April 2019).

23.

Schemm

(2013) Roundtable: What are the barriers to scientific independence in Africa? Policymakers, publishers and nonprofits gather in UK Parliament to debate problems and solutions. Available at: https://www-elsevier-com-s.web.bisu.edu.cn/connect/roundtable-what-are-the-barriers-to-scientific-independence-in-africa (accessed 22 April 2019).

24.

Smart

Pearce

Tonukari

(2004) E-publishing in developing economies. Canadian Journal of Communication 29(3): SI.

25.

Tattersall

Carroll

(2018) What can Altmetric.com tell us about policy citations of research? An analysis of Altmetric.com: data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics 2: 9.

26.

UK Research and Innovation (n.d) Excellence with impact. Available at: https://www.ukri.org/innovation/excellence-with-impact/ (accessed 2 May 2019).

27.

Ware

(2008) Peer review: Benefits, perceptions and alternatives. London: Publishing Research Consortium.

28.

Weiss

(2007) Measuring the impact of medical research: Moving from outputs to outcomes. American Journal of Psychiatry 164(2): 206–214.