Advancing a framework for social impact assessment of tourism research

Abstract

This article aims to advance the methodology for assessing the social impact of tourism research. An evaluation framework was designed to measure social impact in three stages—ex-ante, in-itinere and ex-post—and the ex-post evaluation was applied to a tourism research project, the POLITUR project, to test its validity. The collected information originated from interviews and documentary material. The analysis was structured according to six main areas—communication and promotion, policies and regulation, economic benefit, new technological resources, environment and social improvements—and four dimensions—temporal, applied, geographical and sustainability. The results are followed by a discussion of the domains and dimensions of the social impact assessment of tourism research. The need for further improvement in methods for measuring the social impact of tourism research and the importance of research that generates social impact are highlighted.

Keywords

Social impact assessment tourism research social impact of tourism research POLITUR project

Introduction

The social impact of scientific research is a complex phenomenon (Lauronen, 2020; Lima and Wood, 2014) because it is diverse in nature (Vanclay, 2002) and generates changes that affect people’s quality of life (Burdge et al., 1995). As social impact has become a priority in the academia (Van Den Akker et al., 2017), there is an increasing need to demonstrate how research results benefit society at large (Lima and Wood, 20014). However, the evaluation of social impact, understood as “the process of identifying the future consequences of a current or proposed actions, which are related to individuals, organisations and social macro-systems” (Becker, 2001), is still under development. Evaluation is, in this context, a tool for understanding the societal consequences of scientific research and aids decision-making by managers, funders and society as a whole (Ahmadvand and Karami, 2017; Vanclay et al., 2015). Since funding agencies seek to ensure that funded projects clearly identify societal benefits (Holbrook and Frodeman, 2011) but lack valid measurement systems, it is the researcher’s responsibility to demonstrate the social impact of their work (Sordé, et al., 2020) and make it visible (Lauronen, 2020), rather than assume that this communication and impact occur by default (Wilsdon et al., 2015). To this end, social impact can be conceived as a goal to be taken into account at every step of the research process, from beginning to end (Spaapen and Van Drooge, 2011).

The development of tourism activity causes positive and negative impacts that affect various areas (Andereck, 1995; Archer et al., 2005; Milano et al., 2019; Seetanah, 2011; Yen and Kerstetter, 2008), including, as is well known, the social sphere (Andereck and Nyaupane, 2011; Andereck et al., 2005; Brougham and Butler, 1981; Kousis, 1989). However, the social impact of tourism and the social impact of tourism research are very different terms that refer to different social, economic, institutional, territorial and environmental dynamics. The social impact of tourism refers to the impacts that tourism activity has generated in society (Deery et al., 2012), but the social impact of tourism research refers to the effects that tourism research can have when it is published and transferred to society and the benefits it generates in the population (Flecha, 2018). The social impact of tourism has been extensively studied by researchers (Deery et al., 2012), the literature on the social impact of tourism research is very limited if not practically non-existent. For this reason, this article aims to advance the methodology for assessing the social impact of research in a specific thematic area, namely tourism. Additionally, in order to test the validity of the proposed evaluation framework, the results of a tourism research project were used as a case study. The case study is considered the best option for showing the validity of the social impact framework (De Jong et al., 2014; Tahamtan and Bornmann, 2020) because it allows researchers to systematise complex information (Bornmann et al., 2016). Analyses such as the one proposed in this study contribute to the concept of social impact of research, particularly in the field of tourism, and to the development of an evaluation method that can be widely used by researchers.

To the authors’ knowledge, this is the first analysis that seeks to develop and apply a methodology for assessing the social impact of tourism research. This type of study is common in other scientific areas such as the medical sciences, and there have been attempts to design frameworks for analysis such as the Payback Framework, the Research Institute Framework and the Societal Impact Framework (Van der Weijden et al., 2012). Thus, the paper is also an example of how publicly funded tourism research may contribute to improving people’s lives.

The article is structured as follows. The next section proposes a conceptual framework for the evaluation of the social impact of research. This is followed by the section explaining the assessment methodology used in this article. This is followed by the section that extracts the results of the analysis. This is followed by a discussion of the method in the light of the results obtained, and the last section presents the conclusions of the research.

Conceptual framework

This article proposes a methodology for assessing the social impact of tourism research and applies it to a selected research project. The design of this methodology is a complex task because of the interdisciplinary nature of social impact (Hill, 2016). Social impact assessment must consider all contributions to the social impact of a research effort (De Jong et al., 2014). Most evaluation systems focus on capturing direct and immediate impacts but fail to capture indirect and long-term impacts (De Silva and Vance, 2017). This methodology seeks to address this problem. Previous analysis on research’s social impact identifies three key moments for assessing the social impact of a research project: ex-ante, in-itinere and ex-post (Flecha, 2018; Kvam, 2018; Redondo-Sama et al., 2020). Additionally, several authors have proposed a number of indicators for this evaluation (Chams et al., 2020; Corsi et al., 2019; Cunha et al., 2012; McCombes et al., 2015; Smith, 2001; Van Den Besselaar et al., 2018).

In this paper, different tools to collect information are applied depending on the moment of evaluation, and four dimensions are considered in order to classify the resulting indicators: applied dimension, temporal dimension, geographical dimension and sustainability dimension. The social impact of research can be of a direct or indirect nature (Alla et al., 2017), which is what it is referred to in this study as the applied dimension. This is determined by whether the impact has been produced by the project’s research or by some action related externally to the project. The temporal dimension groups the impact into short-term or long-term, depending on when it occurs. In general, it takes a long time for an impact to occur, so it may be the case that at the time of the evaluation, some impacts do not yet exist (Bornmann, 2012; Sivertsen and Meijer, 2020). The geographical dimension categorizes the impact as either local, when it occurs in the same place where the research was conducted, or extended, when it occurs in a place other than where the research was conducted. The sustainability dimension relates to the impact of a research project on the achievement of the Sustainable Development Goals (SDGs). These goals have similar principles to those found in social impact assessment, based on social protection aspects such as the establishment of basic infrastructure for water, housing, energy, schools and economic development through job creation or support for small businesses (Aucamp and Lombard, 2018). There are tools such as the Social Impact Open Repository (SIOR) that show the social impact of research results based on the outputs created to address the SDGs (SIOR, 2017). The proposed framework is illustrated in Figure 1.

Figure 1.

Framework for assessing the social impact of a research project. Own elaboration.

An ex-ante evaluation can be carried out when knowledge from the project has not yet been generated; it determines the potential social impact (Redondo-Sama et al., 2020; Olmos-Peñuela et al., 2014). The ex-ante assessment could be considered a simulation of what would happen if the project were to be developed. A project evaluation that identifies social risks and benefits is considered necessary to manage such risks and benefits throughout the project (Kvam, 2018). Research agencies may request this assessment for project funding, as is the case, for instance, with the National Science Foundation (NSF) in the United States (Boshoff and De Jong, 2020). Such requests increase the need to achieve minimum societal goals (Cunha et al., 2012). For this assessment, some scholars propose the analysis of stakeholders and their involvement from the beginning of the project (Bornmann, 2013; Eschenbach, 2017). Determining stakeholder interests serves to frame the research around social issues (Lauronen, 2020) and make the design of the thematic areas and indicators relevant to that project. This assessment focuses on direct impacts that can be achieved if the project is implemented. Relevant social data are extracted to serve as a future prediction of how the research will influence society (Kvam, 2018). This prediction will provide answers to what kind of change is expected with the implementation of the project, where this change will occur, what effects it will have and whether this effect is important for society (Helming et al., 2011). The assessment of the project’s area of influence categorises the impact as local according to the geographical dimension. As it is a forecast, the time dimension can be either short-term or long-term. Identifying which of the SDGs will be achieved answers the sustainability dimension (Pejić et al., 2020).

Once the project is underway, an in-itinere evaluation can be carried out to mitigate possible deviations and errors (Macombe and Loeillet, 2017; Vanclay et al., 2015) and to incorporate or remove indicators from thematic areas. At this point, an analysis of productive interactions can be proposed. Contact with stakeholders causes them to engage with the research. This is what is called “productive interaction” (Molas-Gallart and Tang, 2011), which can be interpreted as the social impact of new actions taken by stakeholders as a result of interaction with the research (Molas-Gallart and Tang, 2011). The use of productive interactions also allows researchers to understand that every step is valuable for the achievement of social impact, from the first contact to the conclusion of the research (Spaapen and Van Drooge, 2011). These interactions are considered a precondition for successful research (Molas-Gallart and Tang, 2011). Examples of productive interactions include participation in advisory boards or policy development committees, collaborative research, informal meetings, media interviews, presentations, demonstration, advice to organisations or expert testimony before a legislative body (Ozanne et al., 2017). In-itinere impacts can be direct or indirect (De Jong et al., 2014). When a research project concludes and researchers transfer knowledge to other projects (Molas-Gallart and Tang, 2011), the impact is considered indirect. Productive interactions are a good way to clarify how knowledge contributes to the observed impact. The framing method, interviews and analysis of documentary material can be used to identify productive interactions (Esko and Tuunainen, 2019; Esko and Miettinen, 2019). Another evaluation tool is the communicative methodology, used by Redondo-Sama et al. (2020) to evaluate the in-itinere social impact of psychological projects (Tellado et al., 2020). This methodology incorporates arguments from social actors and developing actions that promote positive social policies (Gómez et al., 2013). Different techniques can be used to extract information, including interviews and observations (Redondo-Sama et al., 2020) such as communicative daily life stories (Gómez et al., 2019; Tellado et al., 2020), communicative focus groups and communicative observation (Gómez et al., 2019). In this in-itinere evaluation, the geographical dimension could be local or extended, but as the project is still under development, the evaluation only considers short-term impacts. The sustainability dimension will be used to check whether any SDGs have been achieved or whether the research needs to be redirected to achieve them.

In the ex-post evaluation, it is possible to detect impacts that add indicators to the thematic areas proposed in the initial phase of the project (Li et al., 2014; Vanclay et al., 2015). The analysis of productive interactions and the communicative methodology can also be applied in this evaluation, which helps capture not only the direct impact but also the indirect impact. A final report about the social impact is proposed, identifying the status of the research and the knowledge that has been developed in a language accessible to all stakeholders (Bornmann and Marx, 2014). The information is made available to society and can be used by public officials for policy reporting (De Jong et al., 2014). Complementarily, an analysis of alternative metrics used to measure the social impact of social media research may be considered (Bornmann, 2015; Bornmann et al., 2016; Bornmann and Haunschild, 2018; Bornmann et al., 2019; Cho, 2017; Dardas et al., 2019; De Silva and Vance, 2017; Garcovich and Adobes-Martin, 2020; Kale et al., 2017; Kolahi and Khazaei, 2018; Pejić et al., 2020; Pulido et al., 2020; Sedighi, 2020; Tonetti, 2019; Viana-Lora and Nel-lo-Andreu, 2020). Interestingly, research projects are increasingly using social media as a tool for dissemination of results (Garcovich and Adobes-Martin, 2020; Pulido et al., 2020; Viana-Lora and Nel-lo-Andreu, 2020) beyond the scientific field (Bornmann, 2015). This can be a growing metric, but it should not be considered the only tool because data may vary depending on the source from which they are extracted (Cho, 2017). For such reasons, digital data has not become widely accepted by academics (Jamal and Alimohammadi, 2015). The geographical dimension of ex-post impacts is both local and extended thanks to the global scope of the Internet. The ex-post evaluation can be carried out in the short or long term (Flecha, 2018). If the analysis is carried out right at the end of the project, short-term impacts will be mostly detected, but if the analysis is carried out at a later time, long-term impacts can be captured as well. For example, in the medical sciences, up to 17 years have elapsed between medical trials and the demonstration of benefit to society (Ozanne et al., 2017). The SDGs ultimately achieved by the project add value to the sustainability dimension.

Methodology for analysing the social impact of applied research

This article applies the framework presented in the previous section, focusing on an ex-post evaluation of a tourism research project. The aim is to report the impact achieved by a project and to validate the methodology. This section introduces the sources of information used, the analysis techniques deployed and the impact domains defined to evaluate the case study and to design the analysis procedure.

Sources of information, analysis techniques and areas of impact

There is no single way to measure the social impact of a research project (Bornman, 2012). Data collection (Chams et al., 2020; Cunha et al., 2012; De Jong et al., 2014; Hanna et al., 2020; Hill, 2016) increasingly tends to incorporate social media-based metrics (Cho, 2017; Dardas et al., 2019; Garcovich and Adobes Martin, 2020; Sedighi, 2020; Viana Lora and Nel-lo Andreu, 2020), productive interactions (De Jong et al., 2014; Eschenbach, 2017; Esko and Tuunainen, 2019; Molas-Gallart and Tang, 2011), interview results (Duque et al., 2020; Esko and Miettinen, 2019; Tellado et al., 2020) and evaluation reports (Bornmann and Marx, 2014). Data processing techniques may be qualitative, quantitative or mixed, and those that best suit the nature of the project should be selected. In view of the experience gained so far (Chams et al., 2020; Corsi et al., 2019; Cunha et al., 2012; Deery et al., 2012; McCombes et al., 2015; Smith, 2001; Van Den Besselaar et al., 2018), the information can be categorised into six main areas: communication and promotion, policy and regulation, economic benefits, new technological resources, environment and social improvement.

Characterisation of the case study

For the case study, a basic research project in the field of tourism entitled “Analysis of the role of territorial policies in the management of tourist destinations in the age of mobility (POLITUR)”, with reference CSO2017-82,156-R, was selected. It was a project funded by the Spanish Ministry of Science, Innovation and Universities in the 2017 call for R&D&I projects aimed at societal challenges. The duration of the project was 3 years. Its main objectives were to identify, understand and review the role of place-related issues regarding destinations’ tourism policies as a necessary and fundamental driver for improving prosperity and sustainability (environmental, economic and cultural); to discuss, evaluate and propose prescriptive systems to guide decision-making mechanisms of the actors (both public and private) involved in the destination’s governance; and to promote the implementation of effective mechanisms for regulating the activity regarding the tourist use of space. In this context, POLITUR also included, as an innovative issue, the specific objective to develop a system of indicators to evaluate the validity of the solutions that it might propose. This was motivated by the relevance of social impact for the Rovira i Virgili University (URV), the university to which the project was attached, which has a Social Council responsible for bringing the concerns and needs of society to the University and for projecting the potential of the activities carried out by the University to society.

Designing the procedure for analysing the social impact of research

Qualitative techniques were selected to assess the social impact of the POLITUR project. Qualitative data collection tools have been established as the most effective way to assessing social impact (De Silva and Vance, 2017), as they help to better understand the multidimensional and contextual nature of impacts (Reale et al., 2018) and make visible how research is at the service of citizens (Sordé, et al., 2020). From this perspective, it is important to determine what has been done with the research results, who has used them and the reason for their use, rather than to highlight the number of times that the research has been used (Bornmann, 2014). Furthermore, this impact could be interpreted from a multitude of perspectives (Lauronen, 2020). The complexity of assessment is accentuated in the social sciences by the fact that the results are less tangible compared to other fields of study (Lima and Wood, 2014; Olmos-Peñuela et al., 2014). As shown in Figure 2, the proposed procedure was structured in five phases: information collection, transcription and ordering of the information, codification of the information, data analysis and integration of the information.

Figure 2.

Phases implemented. Own elaboration.

Firstly, the information was obtained. The data collection technique consisted of semi-structured interviews and analysis of documentary material. The first set of data was collected from semi-structured personal interviews with the members of the research team, with the goal of detecting the social impact of their research. The interviews took place at the end of 2020, coinciding with the end of the 3-year project. All 18 project researchers were interviewed using a combination of face-to-face, telephone and online communication. The interviews consisted of two parts: a structured part in which the researchers answered eight key questions, and an unstructured part that sought to find out, within the narrative of the interviewees, which actions have had a social impact. The observation of the social aspects related to the project was important. The second set of data was documentary material, comprising published scientific articles, articles and dissemination material, theses completed within the project and publications in social media. Secondly, the information collected from the interviews was transcribed and sorted, and the documentary information was organised. Thirdly, the information was coded in order to group it into comprehensive ideas. In this particular case, and due to the uniqueness of the project, the information was grouped into six main areas: communication and promotion, policy and regulation, economic benefits, new technological resources, environment and social improvements. This information allowed for the development of a set of indicators in each thematic area. Fourthly, the information was analysed to detect impacts based on the four dimensions of action: applied dimension (direct or indirect impacts), temporal dimension (short- or long-term impacts), geographical dimension (impacts on a local or extended scale) and sustainability dimension (linkage between the impact and the SDGs). Finally, all the information was integrated and the results were extracted, as shown in the next section.

Results

The results of the analysis are summarised in Figure 3. A number of indicators classified within the six thematic areas were developed. Each indicator was analysed to highlight the impact on each of the dimensions.

Figure 3.

Project impacts by thematic area. Own elaboration.

For ease of analysis, the following section has been divided into two main sections: thematic areas and dimensions.

Thematic areas

Communication and promotion

The scientific production derived from the project has been published in high-impact journals and presented in national and international workshops and research conferences. Nevertheless, this dissemination does not generate social impact as such. It is the dissemination of results through media at the local, national and international level that causes benefits in society as their coverage via newspapers and news programs allows the information to be transmitted to a wider audience. In the POLITUR project, researchers also communicated with the general public via talks and activities aimed at disseminating the results of the project. The COVID-19 pandemic made it necessary to adapt this communication to the online modality, and the information has also been shared on social media through social networks such as Twitter and the project Web site (http://politurproject.org/). This Web site contains all the information on the work team, the publications and the related conferences. The social impact because of social media communication and dissemination is direct and in the short term (Pejic Bach et al., 2020). In this vein, a survey conducted by Álvarez-Bornstein and Montesi (2019) about Twitter users who follow researchers found that users obtained ideas for their work and implemented new scientific and technical advances in their professional areas, allowing them to interact with researchers to resolve doubts and ask questions.

Policy and regulation

During the project’s development, the members of the POLITUR team maintained important relations with destination managers and organisations with influence on tourism policy and regulation, thus facilitating the reception of proposals and initiatives. The researchers interviewed public and private institutions throughout the project with the aim of gathering the opinion of stakeholders on the subject matter. Through surveys, they also incorporated the views of tourism business associations, tourists and residents into the project. Thus, POLITUR had the opportunity to improve the awareness of the actors interviewed, all of whom had the power to influence public and private decision-making. In particular, the project has had an indirect long-term social impact by contributing to the design of spatial tourism behavior management policies at the local level and by providing managers tools for better management and planning through the development of governance procedures related to the coordination and design of common agendas between stakeholders.

Economic benefit

POLITUR generated a direct social impact in the short term by creating two new jobs: a trainee researcher and a technician. Thus, it directly incorporated graduate students into the labour market. Additionally, beyond the execution of the project itself, POLITUR favoured the creation of value from knowledge-developing contributions to increase the economic benefit of tourism in the areas under study. This is the case, for example, with recommendations related to the design of onshore itineraries for cruise tourists, widening the range of activities in which they could participate, directing them to places with commercial and recreational activities and increasing the supply of complementary leisure activities. These contributions were communicated to the tourism management officials in the tourist destination so that they could incorporate them into their planning policies, generating an indirect social impact in the long term by increasing the economic benefit of the destination.

New technological resources

The results of POLITUR highlighted technology’s capacity for the intelligent management of destinations. The project analysed the benefits of using different digital sources, the interaction between mobile populations in destinations and the management opportunities derived from the digital footprint of visitors. An example of this relates to the direct social impact derived from the exploration of potential mobility patterns through the recording and tracking of people in video sequences. This allowed the POLITUR team to redirect tourist flows to avoid overcrowding and to improve the quality of life of residents in the long term. In addition, project researchers also participated in other related projects with the aim to deploy strategic and digital tools directly related to the short-term management of visitors’ journey to and within the destination.

Environment

The project researchers transferred knowledge to the managers of tourist destinations in order to reduce the environmental impact caused by tourist activity. In this vein, they suggested measures to adapt buildings to the future climate scenario, to raise environmental awareness among tourism stakeholders, to facilitate the improvement of energy efficiency and consumption and to help to increase the commitment to the protection of beaches to combat erosion. There were also indirect social impacts through researchers’ involvement in other related projects aiming to provide guides for sustainable practices and climate services for better governance of tourist destinations.

Social improvements

POLITUR made it possible to analyse the factors that produce social exclusion among the most vulnerable groups at destinations as a result of the tourism specialization of certain spaces, generating a direct and long-term social impact. Research initiatives related to POLITUR that have focused on solving social problems and have been awarded at the national and European level have aimed to develop innovative solutions to the social conflicts and externalities produced by tourism-related mobility in cities. Thus, this project provoked an indirect social impact resulting from the development of other projects aiming to design alternative policies for more socially inclusive places.

Study dimensions

Applied dimension

As of the writing of this paper, most of the reported impacts of the POLITUR project have been direct. In fact, direct impacts are most common when any project is just finished. However, it should be noted that indirect impacts may be also important after the end of the project and that, to be adequately identified, they need particularly designed methods to assess them (Silva and Vance, 2017). For the moment, most of the identified indirect impacts of the POLITUR project are those derived by applying the knowledge developed and acquired within the project to other related projects undertaken during the same period. This type of impact is complex, as it means that it was not the POLITUR project itself that generated benefits in society, but that POLITUR served as the basis for action in other applied projects that had social impact. This would mean an indirect impact from POLITUR and a direct impact from the new research (Van der Weijden et al., 2012).

Time dimension

Following the same rationale as in the previous dimension, due to the short-term assessment of the social impact of the project, it is not possible to detect issues such as economic improvements brought on by the results of the research. Communication and promotion is a dimension that stands out in short-term impacts, as it mainly consists of publications, mentions and participation in conferences and congresses that seek to disseminate the results, as well as the resulting dissemination through media, digital social media and the Internet. The other thematic areas generally have long-term impacts, as a consequence of applying the research results once they have been transferred.

Geographical dimension

The project analyzed specific regional and local spaces mostly in Catalan tourism destinations. This is what we call the local dimension in our analysis and, when analyzing the social impact of the project, it refers to the effects produced in the areas of study where it the research was conducted. However, the acquired knowledge could have been applied to other geographical areas not analysed within the POLITUR project, through the transfer of this knowledge to other applied projects. This is the extended dimension of the social impacts of the project. POLITUR has mainly impacted the local dimension, although it is hoped that in the long term the results will be applied to other tourist areas. Such extended impact would have to be monitored adequately.

Sustainability dimension

POLITUR made contributions to eight of the SDGs set out by the United Nations in 2015, as shown in Figure 3. SDG 4, on quality education, was promoted through the project’s support of doctoral students and through the high impact of the resulting scientific publications. This has led to useable and applicable results for society. SDG 8, on decent work and economic growth, was attained through proposals seeking to increase tourist spending and to improve the economy of the analysed areas. SDG 9, on industry, innovation and infrastructure, was covered through the development of new tools for the assessment and analysis of tourism flows. SDG 11, on sustainable cities and communities; SDG 12, on responsible production and consumption; and SDG 13, on climate action, were pursued through ongoing proposals to mitigate the environmental impact of tourism, the identification of more sustainable actions and the involvement of stakeholders. The analysis of the effects of tourism on vulnerable populations, the study of social exclusion of certain populations derived from tourism and the search for solutions are closely related to the achievement of SDG 10, on reducing inequalities. All contributions supported SDG 17, on partnerships to achieve the goals.

Discussion

Beyond the analysis of the ex-post social impact of the results of POLITUR, the specific application of the evaluation framework in this project can be useful to advance the validation of the proposed tool. In this vein, it can be highlighted first that classifying the information about social impacts into six thematic areas provides a mechanism to clearly identify the types of impacts and their roles as components of the societal consequences of research.

The area of communication and promotion is fundamental for any research project since generating social impact requires transferring scientific knowledge to institutions and to society in general (Flecha, 2018). Successful communication between research and societal stakeholders is also fundamental (Bornmann, 2014). There may be many policy-relevant research topics (Bornmann et al., 2016), but whether the communication and promotion processes actually had an impact must be assessed separately. Researchers in an expert role of providing policy recommendations (Muhonen et al., 2020) are significant, but it is the follow-up of the process that will determine whether there has been an impact. Projects that have captured social impact through the formulation of policies aimed at solving social problems (Tellado et al., 2020) should be adequately referred in relevant policy documents (Bornmann et al., 2016). The application of tourism research can also generate economic benefits such as increased productivity (Chams et al., 2020), job creation (Samuel and Derrick, 2015) and the sale of new products (Cunha et al., 2012). This has been the most studied area within social impact (Bornmann, 2013), even though this framework only considers it as one of the six pillars of the evaluation. The social impact of tourism research has also included the development of technological resources (Bornmann et al., 2016; Eschenbach, 2017). This is an area that affects both private and public organisations (Cunha et al., 2012). The environmental field has widespread social concern (Wolf, 1982). In a world increasingly concerned about climate change and gas emissions, it seems appropriate for projects to focus on reducing this problem (Bornmann et al., 2016) as it has been identified through the application of the designed evaluation framework. The last area included in the evaluation framework is the social improvements generated by research. Research can help in the fight to reduce social inequalities (Tellado et al., 2020). In this area, aspects such as social equity, social inclusion, social justice and gender equality should be analysed so that research can be the key to a fairer society.

On the other hand, the analysis of dimensions allows for reflection on the importance of planning the project and the ex-post evaluation impact; otherwise, most of the impacts will be identified as direct and short-term. This study also highlights that indirect impacts must include the impacts derived from the development of other initiatives, projects or alliances. Furthermore, the analysis reinforces the need for research monitoring in the long term and in geographical areas other than those analysed in the project. International projection is one of the goals that should be considered in tourism research projects, with the aim of extending research to other destinations and increasing its impact in the geographical dimension.

Conclusion

This article seeks to advance the methodology of social impact assessment in tourism research. To this end, an analytical framework was created with various tools, evaluation moments and dimensions. The application of the ex-post evaluation to the POLITUR project as a pilot test opens discussion on how to design an internationally accepted assessment system that aligns with the SDGs of the United Nations.

The proposed analytical framework helps to plan and organise the social impact assessment of a research project. It identifies impacts at all stages of research and avoids the bias of a single method of data collection. In this way, this research reduces the problems encountered in other social impact assessment tools (Viana-Lora and Nel-lo Andreu, 2021). The problem of attribution, which is the main problem with social impact evaluations, arises when social impact is linked to a project under study (Bornmann, 2012; Bornmann et al., 2019; Lauronen, 2020; Sivertsen and Meijer, 2020; Spaapen and Van Drooge, 2011; Tahamtan and Bornmann, 2020). The evaluation of social impact should be improved so that it does not depend on each project’s methodology (Tellado et al., 2020); instead, clear processes of attribution should be increasingly defined from a grounded process of analysis and evaluation of particular research projects, such as the one presented in this paper.

Tourism research has additional particularities that make it even more difficult to assess social impact. For one, tourism is a complex and dynamic social phenomenon that encompasses different branches of research (Picornell, 2015), so evaluating this impact must also involve a multidisciplinary team. Additionally, tourism research has not yet highlighted a broad concept of its social impact, instead highlighting some of the impacts of its activity. This article aims to raise awareness among tourism researchers about the importance of the social impact of their research activity.

A limitation of this study is that it only applies one part of the proposed analytical framework, the ex-post evaluation. Nevertheless, the authors affirm the need to apply the framework from the beginning of the research and to follow up at all stages. Planning and targeting research from strategic priorities, such as the SDGs, will help in achieving social benefits, thus making it easier to generate social impact (Hill, 2016; Lauronen, 2020).

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Science and Innovation (MICINN), the European Social Fund (ESF) (PRE2018-085470), the POLITUR project (CSO2017-82156-R), the ADAPTOUR project (PID2020-112525RB-100) and the Department of Research and Universities of the Catalan Government (2017SGR22).

ORCID iDs

Alba Viana-Lora

Marta G Nel-lo-Andreu

Salvador Anton-Clavé

References

Ahmadvand

Karami

(2017) Social impacts evaluation and insider-outsider paradigm: floodwater spreading project on the Gareh-Bygone plain as an illustrative case. Evaluation and Program Planning 65: 69–76.

Alla

Hall

Whiteford

, et al. (2017) Research impact for (mental) health policy. Health Res Policy Sys 15: 1.

Alvare-Bornstein

Montesi

(2019) Who is interacting with researchers on Twitter?: a survey in the field of information science. JLIS 10(2): 87–106.

Andereck

(1995) Environmental consequences of tourism: a review of recent research. In: Linking tourism, the environment, and sustainability. Annual meeting of the national recreation and park association, 77–81.

Andereck

Nyaupane

(2011) Exploring the nature of tourism and quality of life perceptions among residents. Journal of Travel Research 50(3): 248–260.

Andereck

Valentine

Knopf

, et al. (2005) Residents’ perceptions of community tourism impacts. Annals of Tourism Research 32(4): 1056–1076.

Archer

Cooper

Ruhanen

(2005) The positive and negative impacts of tourism. Global Tourism 3: 79–102.

Aucamp

Lombard

(2018) Can social impact assessment contribute to social development outcomes in an emerging economy? Impact Assessment and Project Appraisal 36(2): 173–185.

Becker

(2001) Social impact assessment. European Journal of Operational Research 128(2): 311–321.

10.

Bornmann

(2012) Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO Reports 13(8): 673–676.

11.

Bornmann

(2013) What is sicietal impact of research and how can it be assessed? A literature Survey. Journal of American Society for Information Science and Technology 64(20): 217–233.

12.

Bornmann

(2014) Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics 8(4): 895–903.

13.

Bornmann

Marx

(2014) How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics 98(1): 211–219.

14.

Bornmann

(2015) Usefulness of altmetrics for measuring the broader impact of research: a case study using data from PLOS and F1000Prime. Aslib J. Inf. Manag 67(3): 305–319.

15.

Bornmann

Haunschild

Marx

(2016) Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents? Scientometrics 109(3): 1477–1495.

16.

Bornmann

Haunschild

(2018) Normalization of zero-inflated data: an empirical analysis of a new indicator family and its use with altmetricsdata. Journal of Informetrics 12(3): 998–1011.

17.

Bornmann

Haunschild

Adams

(2019) Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics 13(1): 325–340.

18.

Boshoff

de Jong

(2020) Conceptualizing the societal impact of research in terms of elements of logic models: a survey of researchers in sub-Saharan Africa. Research Evaluation 29(1): 48–65.

19.

Brougham

Butler

(1981) A segmentation analysis of resident attitudes to the social impact of tourism. Annals of Tourism Research 8(4): 569–590.

20.

Burdge

Williams

Llewellyn

, et al. (1995) Guidelines and principles for social impact assessment. Environmental Impact Assessment Review 15(1): 11–43.

21.

Chams

Guesmi

Gil

(2020) Beyond scientific contribution: assessment of the societal impact of research and innovation to build a sustainable agri-food sector. Journal of Environmental Management 264: 110455.

22.

Cho

(2017) A comparative study of the impact of Korean research articles in four academic fields using altmetrics. Performance Measurement and Metrics 18: 1467–8047.

23.

Corsi

Pagani

Kovaleski

, et al. (2019) Technology transfer for sustainable development: social impacts depicted and some other answers to a few questions. Journal of Cleaner Production 245: 118522.

24.

Cunha

Ferreira

Araújo

, et al. (2012, April) Social return of R&D investments in manufacturing sector: some insights from an exploratory case study. AIP Conference Proceedings 1431(1): 43–53. American Institute of Physics.

25.

Dardas

Woodward

Scott

, et al. (2019) Measuring the social impact of nursing research: an insight into altmetrics. Journal of Advanced Nursing 75(7): 1394–1405.

26.

De Jong

Barker

Cox

, et al. (2014) Understanding societal impact through productive interactions: ICT research as a case. Research Evaluation 23(2): 89–102.

27.

De Silva

Vance

(2017) Assessing the societal impact of scientific research. In: Scientific Scholarly Communication. Cham: Springer, pp. 117–132.

28.

Deery

Jago

Fredline

(2012) Rethinking social impacts of tourism research: a new research agenda. Tourism Management 33(1): 64–73.

29.

Duque

Gairal

Molina

, et al. (2020) How the psychology of education contributes to research with a social impact on the education of students with special needs: the case of successful educational actions. Frontiers in Psychology 11: 439.

30.

Eschenbach

(2017) Bridging the gap between observational oceanography and users. Ocean Science 13(1): 161–173.

31.

Esko

Miettinen

(2019) Scholarly understanding, mediating artefacts and the social impact of research in the educational sciences. Research Evaluation 28(4): 295–303.

32.

Esko

Tuunainen

(2019) Achieving the social impact of science: an analysis of public press debate on urban development. Science and Public Policy 46(3): 404–414.

33.

Flecha

(2018) Evaluación del impacto social de la investigación. Revista de Fomento Social: 485–502.

34.

Garcovich

Adobes Martin

(2020) Measuring the social impact of research in paediatric dentistry: an altmetric study. International Journal of Paediatric Dentistry 30(1): 66–74.

35.

Gómez

Elboj

Capllonch

(2013) Beyond action research: the communicative methodology of research. International Review of Qualitative Research 6(2): 183–197.

36.

Gómez

Padrós

Ríos

, et al. (2019). Reaching social impact through communicative methodology. Researching with rather than on vulnerable populations: the Roma case. Frontiers Education 4: 9.

37.

Hanna

Gatting

Boyd

, et al. (2020) Evidencing the impact of cancer trials: insights from the 2014 UK research excellence framework. Trials 21: 1–13.

38.

Helming

Diehl

Bach

, et al. (2011) Ex ante impact assessment of policies affecting land use, Part A: analytical framework. Ecology and Society 16(1): 29.

39.

Hill

(2016) Assessing (for) impact: future assessment of the societal impact of research. Palgrave Communications 2(1): 1–7.

40.

Holbrook

Frodeman

(2011) Peer review and the ex ante assessment of societal impacts. Research Evaluation 20(3): 239–246.

41.

Jamali

Alimohammadi

(2015) Blog citations as indicators of the societal impact of research: content analysis of social sciences blogs. International Journal of Knowledge Content Development & Technology 5(1): 15–32.

42.

Kale

Siravuri

Alhoori

, et al. Predicting research that will be cited in policy documents. In Proceedings of the 2017 ACM on Web Science Conference, Troy, NY, USA, 25–28 June 2017, pp. 389–390.

43.

Kolahi

Khazaei

(2018) Altmetric analysis of contemporary dental literature. British Dental Journal 225(1): 68–72.

44.

Kousis

(1989) Tourism and the family in a rural Cretan community. Annals of Tourism Research 16(3): 318–332.

45.

Kvam

(2018) Evaluación Del Impacto Social: Integrando Los Aspectos Sociales En Los Proyectos De Desarrollo. Inter-American Development Bank.

46.

Lauronen

(2020) The dilemmas and uncertainties in assessing the societal impact of research. Science and Public Policy 47(2): 207–218.

47.

Chen

Hui

ECM

, et al. (2014) A methodology for ex-post assessment of social impacts of an affordable housing project. Habitat International 43: 32–40.

48.

Lima

GDMR

Wood

(2014) The social impact of research in business and public administration. Revista de Administração de Empresas 54(4): 458–463.

49.

Macombe

Loeillet

(2017) Instruments to assess the social impacts of value chains. In: Sustainable Development and Tropical Agri-chains. Dordrecht: Springer, pp. 257–265.

50.

McCombes

Vanclay

Evers

(2015) Putting social impact assessment to the test as a method for implementing responsible tourism practice. Environmental Impact Assessment Review 55: 156–168.

51.

Milano

Novelli

Cheer

(2019) Overtourism and degrowth: a social movements perspective. Journal of Sustainable Tourism 27(12): 1857–1875.

52.

Molas-Gallart

Tang

(2011) Tracing ‘productive interactions’ to identify social impacts: an example from the social sciences. Research Evaluation 20(3): 219–226.

53.

Muhonen

Benneworth

Olmos-Peñuela

(2020) From productive interactions to impact pathways: understanding the key dimensions in developing SSH research societal impact. Research Evaluation 29(1): 34–47.

54.

Olmos-Peñuela

Castro-Martínez

d’Este

(2014) Knowledge transfer activities in social sciences and humanities: explaining the interactions of research groups with non-academic agents. Research Policy 43(4): 696–706.

55.

Ozanne

Davis

Murray

, et al. (2017) Assessing the societal impact of research: the relational engagement approach. Journal of Public Policy & Marketing 36(1): 1–14.

56.

Pejić Bach

Pulido

Suša Vugec

, et al. (2020) Fostering social project impact with twitter: current usage and perspectives. Sustainability 12(15): 6290.

57.

Picornell

(2015) Los impactos del turismo. Papers de turisme (11): 65–91.

58.

Pulido

Mara

Ionescu

, et al. (2020) Social impact of psychological research on well-being shared in social media. Frontiers in Psychology 11: 135.

59.

Reale

Avramov

Canhial

, et al. (2018) A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation 27(4): 298–308

60.

Redondo-Sama

Díez-Palomar

Campdepadrós

, et al. (2020) Communicative methodology: contributions to social impact assessment in psychological research. Frontiers in Psychology 11: 286.

61.

Samuel

Derrick

(2015) Societal impact evaluation: exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 24(3): 229–241.

62.

Sedighi

(2020) Evaluating the impact of research using the altmetrics approach (case study: the field of scientometrics). Global Knowledge, Memory and Communication. in press.

63.

Seetanah

(2011) Assessing the dynamic economic impact of tourism for island economies. Annals of Tourism Research 38(1): 291–308.

64.

SIOR (2017) Social Impact Open Repository. Available at https://sior.ub.edu/(accessed on 1 June 2021).

65.

Sivertsen

Meijer

(2020) Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? Research Evaluation 29(1): 66–70.

66.

Smith

(2001) Measuring the Social Impact of Research: Difficult but Necessary. BMJ.

67.

Sordé Martí

Flecha

Rodríguez

, et al. (2020) Qualitative inquiry: a key element for assessing the social impact of research. Qualitative Inquiry 26(8–9): 948–954.

68.

Spaapen

Van Drooge

(2011) Introducing ‘productive interactions’ in social impact assessment. Research Evaluation 20(3): 211–218.

69.

Tahamtan

Bornmann

(2020) Altmetrics and societal impact measurements: match or mismatch? A literature review. El profesional de la información (EPI) 29(1): e290102.

70.

Tellado

Lepori

Morla-Folch

(2020) WIEGO: communicative daily life stories to assess social impact in the lives of informal workers. Qualitative Inquiry 26(8–9): 962–969.

71.

Tonetti

(2019) Leadership in publishing. Journal of Dentistry 87: 28–31.

72.

Van den Akker

Spaapen

Maes

(2017) Productive Interactions: Societal Impact of Academic Research in the Knowledge Society. League of European Research Universities (LERU). https://www.leru.org/publicafions/producfive-interacfions-societal-impact-of-academic-research-in-the-knowledge-society

73.

Van den Besselaar

PAA

Flecha

Radauer

(2018) Monitoring the Impact of EU Framework Programmes. Brussel: European Commission.

74.

Van der Weijden

Verbree

Van Den Besselaar

(2012) From bench to bedside: the societal orientation of research leaders: the case of biomedical and health research in the Netherlands. Science and Public Policy 39(3): 285–303.

75.

Vanclay

(2002) Conceptualising social impacts. Environmental Impact Assessment Review 22(3): 183–211.

76.

Vanclay

Esteves

Aucamp

, et al. (2015) Evaluación de Impacto Social: Lineamientos para la evaluación y gestión de impactos sociales en proyectos (BID, trad.). Fargo, Dakota del Norte: Asociación Internacional para la Evaluación de Impactos.

77.

Viana-Lora

Nel·lo-Andreu

(2020) Alternative Metrics for Assessing the Social Impact of Tourism Research. Sustainability 12(10): 4299.

78.

Viana-Lora

Nel·lo-Andreu

(2021) Approaching the social impact of research through a literature review. International Journal of Qualitative Methods 20.

79.

Wilsdon

Allen

Belfiore

, et al. (2015) The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363

80.

Wolf

(1982) Social impact assessment. Impact Assessment 1(1): 9–19.

81.

Yen

Kerstetter

(2008) Residents’ view of expected tourism impact, attitude, and behavioral intention. Tourism Analysis 13(5–6): 545–564.