Developing and Applying a Framework for Assessing the Research Quality of Qualitative Project Methods in the EU Project SECUR-ED

Abstract

Qualitative research plays a vital role in political development and in the design of statutes and directives. Consequently, ensuring the quality of this research is important. However, the current literature on evaluation of quantitative and qualitative research does not reflect this importance, and we have identified a need to establish guidelines for evaluating qualitative research projects for quality. Therefore, and based on existing research, we have developed a framework for assessing the research quality of large complex projects using qualitative methods. In our study, as presented in this article, we operationalize and apply the framework to evaluate six specific methods in the large European research project, Secured Urban Transportation—A European Demonstration (SECUR-ED); each method is assessed according to the quality criteria of “transferability,” “systematic design/reliability,” and “transactional validity.” Overall, we find that half of the SECUR-ED project methods demonstrate thorough documentation and transferability, and that half of the methods lack consistent usage and therefore score low on both reliability and validity. We also find that one method, the capacity mapping matrix, scores high on all quality parameters. Accordingly, we suggest that future European Union (EU) projects replicate the documentation efforts demonstrated in relation to several of the SECUR-ED methods, and consider the capacity mapping matrix as “best practice” standard. We conclude that the framework represents a novel approach to quality assessments of qualitative project methods across research topics and contexts.

Keywords

research quality project methods quality assessment framework qualitative assessment criteria social sciences

Introduction

Qualitative research plays a vital role in political development and the design of statutes and directives, including research projects funded by the European Commission. Consequently, ensuring the quality of this research is important. Moreover, a large and still growing body of scientific literature addresses the issue of quality and quality criteria in research. Within this body, greater agreement exists regarding quality criteria in quantitative research compared with qualitative, particularly in terms of validity and reliability (Bryman, Becker, & Sempik, 2008). An explanation for this state of affairs lies in the diagnostic exploratory nature of qualitative research (de Ruyter & Scholl, 1998; Kapoulas & Mitic, 2012), which rests on the belief that the social world is comprised of subjective experiences and understandings that can change over time and social contexts (Dew, 2007). This multifaceted, contextual nature of qualitative/exploratory research makes it difficult to establish common quality criteria, as reflected in the lack of consensus among researchers regarding how to approach quality criteria in qualitative research (Cohen & Crabtree, 2008; Mays & Pope, 2000; Sandelowski, 2015; Stige, Malterud, & Midtgarden, 2009). However, clear guidelines for how to judge the quality of the particular research account or project remain desirable (Hammersley, 2007; Leung, 2015).

The lack of clear quality guidelines in qualitative research inspired us to design a framework for assessing the research quality of qualitative project methods employed in the European Commission financed project Secured Urban Transportation—A European Demonstration (SECUR-ED). The SECUR-ED project, which ran from April 2011 to September 2014, was a complex security demonstration project involving 40 European partners across industry, research, and transport operations. The project aimed to provide public transport operators with a set of tools to improve urban transport security, identified through development and testing of a number of security capacities (which can be security processes, technological tools, or training approaches) in demonstration scenarios in several European cities. The project came up with security solutions and recommendations that can bring value to public transport security stakeholders, as outlined in the “White Paper for Public Transport Stakeholders” (SECUR-ED, 2014b).

The SECUR-ED project applied a wide range of project methods, including common research methods such as interviews, questionnaires, and literature studies; project management methods such as mailing lists, cooperation portals, physical meetings, and teleconferences; methods that support project coordination, cooperation, and collaboration; and methods developed for the transport operator partners (capacities/solutions). However, the assessment framework presented in this article had a different focus on methods; it was designed by the authors to study the qualitative methods that the SECUR-ED project developed to handle its size and complexity. The six selected methods are qualitative by nature: common glossary, interoperability notation, scenario description with business process model and notation (BPMN), capacity mapping matrix, demo city dashboards, and capacity evaluation approach. (The following sections will discuss these evaluations and methods.)

The agenda for this article is to describe our study and approach to the development of the assessment framework, the framework itself, and application of the framework to a selection of SECUR-ED project methods. In addition, we discuss the outcome of our assessment process, including the novelty and possible broader application of the framework, to assess similar research projects. It is important to note that our analysis did not address whether the methods would have been sufficient or necessary to achieve the SECUR-ED project’s overall goals regardless of how well they were implemented. In other words, our assessments do not say anything about the degree of success of the SECUR-ED project as a whole.

Method

For an overview of current research into quality and quality criteria, we searched electronic online databases, such as Academic Search Elite, ScienceDirect, Web of Science, and SwetsWise, for combinations of the key words: research quality, qualitative research, validity, and reliability. Based on the resulting publications and reference lists (snowballing) (Greenhalgh & Peacock, 2005; Webster & Watson, 2002), we identified the quality criteria of transferability, systematic design/reliability, and transactional validity, which we integrated into a framework to assess the quality of the SECUR-ED project methods. The criteria identification and framework integration are described and illustrated in the next section.

As partners of the SECUR-ED consortium, we had access to working papers in progress and peer-reviewed research reports from the project that described and discussed SECUR-ED methods, which were evaluated in our study. These reports constituted important empirical data for our evaluation of the SECUR-ED methods and for developing and testing the framework. In addition to these documents, we had access to project partners, covering researchers, industry partners, and transport operators. Specifically, individuals involved in the development and application of the methods provided oral and written input based on their experiences. Also, our assessments of each method as well as different versions of our assessment framework were presented to project members (including dedicated reviewers) on several occasions, such as in meetings, teleconferences, and emails. Thus, written and oral contributions were successively collected and triangulated over time. In sum, we obtained specific rich information to enable comparison and review of a selection of SECUR-ED methods.

Developing the Assessment Framework

As described, qualitative research provides important insights and input for development of policies and statutes, which implies the need for guidelines to judge the quality of a particular research account or project. Accordingly, in this section, we argue for the identified assessment criteria and the integration of the criteria into a framework to assess project methods.

Assessment Criterion: Transferability

Polit and Beck (2010) suggested that “generalization is an act of reasoning that involves drawing broad conclusions from particular instances—that is, making an inference about the unobserved based on the observed” (p. 1451). Some qualitative researchers have questioned generalizability of any type of findings. In their view, findings are always embedded within a context, making it problematic to extrapolate “the particular” (Erlandson, Harris, Skipper, & Allen, 1993; Lincoln & Guba, 1985). Therefore, an alternate approach of transferability should be considered. Transferability, or the case-to-case translation model in Firestone (1993), assumes that the researcher’s job is to describe the time and context in which the particular findings were true, whereas the reader’s job is to determine the extent to which the findings may apply in another context (Polit & Beck, 2010). Therefore, the degree of transferability between two given contexts depends on the thoroughness of the researcher’s contextual descriptions that allow other researchers to determine the applicability of the findings (Lincoln & Guba, 1985).

Applying the same understanding to the SECUR-ED project methods, the logic becomes the following: The better the context or background described in documents, the better the reader will be able to assess the applicability and transferability of the method to another setting both within the transportation sector and in other sectors. In this study, we are the readers who will assess transferability, based on the available descriptions of context and background. On other occasions, readers may be decision makers or advisers within the private sector or public authorities.

Assessment Criterion: Systematic Design/Reliability

For quantitative research, the concept of reliability implies “dependability, stability, consistency, predictability, accuracy” (Kerlinger, 1973, p. 422), and is considered a precondition for validity. More specifically, Kirk and Miller (1986) identified three types of reliability: The degree to which a measurement, given repeatedly, remains the same; the stability of a measurement over time; and the similarity of measurements within a given time period. Careful considerations of measurements are imperative in quantitative research. However, the quantitative researcher’s focus on measurements and stability does not align well with the qualitative researcher’s view of the social world as complex, multifaceted, and contextual (Dew, 2007; Moran-Ellis et al., 2006). Therefore, an alternative perspective suggests that reliability in qualitative research can be reached through systematic operation at the design level (methods and techniques, interview protocols, and so forth; de Ruyter & Scholl, 1998, p. 13), which implies keeping “detailed account of the research steps undertaken” (Kapoulas & Mitic, 2012, p. 361).

To assess the degree of systematic research design employed in the SECUR-ED project methods, we reviewed project documents and gathered oral and written inputs from individuals involved in the project.

Assessment Criterion: Transactional Validity

Within qualitative research, validity has traditionally been defined as the researcher’s efforts to determine the degree of correspondence between claims about knowledge and the reality being investigated (Eisner & Peshkin, 1990). This definition is comparable with how quantitative research applies internal validity and is understood as a means to determine whether observations and measurements truly capture what they are intended to capture (LeCompte & Goetz, 1982). More recently, the concept of validity in qualitative research has evolved to an approach Cho and Trent (2006) labeled transactional validity, which “assumes that qualitative research can be more credible as long as certain techniques, methods, and/or strategies are employed during the conduct of the inquiry” (p. 322). Based on this development of transactional validity, we considered the following techniques or strategies to assess the SECUR-ED project methods (Onwuegbuzie & Leech, 2007):

Prolonged engagement: Entails investing enough time to understand the culture, establish trust with study participants, and check for distortions such as a priori values and constructions (Lincoln & Guba, 1985). In the context of the SECUR-ED project, prolonged engagement translates to the participants’ investment to understand and use project methods. As suggested by Billups (2015) and Matthews and Kostelis (2011), this investment cannot be measured in numbers, but must be long enough to establish a familiarity with project methods.

Persistent observations: Aims to identify the characteristics and elements in a situation that are most relevant to the phenomena under investigation and to focus on them extensively to achieve depth (Lincoln & Guba, 1985). In the context of the SECUR-ED project, persistent observations refer to the participants’ investment to identify and observe the most relevant characteristics of project methods over time, which implies that prolonged engagement to understand and use the methods is required. This investment cannot be measured but must be long enough to enable identification and observation of the most relevant characteristics of project methods.

Triangulation: Involves combining multiple and different sources of information to deepen and widen understanding of the multifaceted, complex nature of the social world and phenomena (Moran-Ellis et al., 2006; Olsen, 2004). Triangulation reduces the possibility of chance associations and prevalent, systematic biases, and thereby increases the confidence in the research data and interpretations (Denzin, 1970; Fielding & Fielding, 1986; Maxwell, 1992; Thurmond, 2001).

Member-checking/informant feedback: Member-checking, also known as informant feedback, is a continuous process in which the researcher seeks feedback on collected data, analytic categories, interpretations, and conclusions from the study group (Cho & Trent, 2006; Lincoln & Guba, 1985). In the context of the SECUR-ED project, member-checking entails participants’ active and continuous discussions of project methods with other project members.

Peer debriefing/review: Constitutes an external and logical evaluation of the research process, such as procedures, methods, interpretations, and conclusions (Glesne & Peshkin, 1992; Lincoln & Guba, 1985; Maxwell, 1996).

Rich and thick description: The researcher provides rich and thick description of the setting, participants, methods, data collection (such as observations of specific events and behaviors), data analysis, interpretations, the researcher’s role, and so forth (Becker, 1970). This detailed information strengthens the credibility of findings and enables the reader to assert whether the findings can be transferred to another setting because of shared similarities (Erlandson et al., 1993).

To summarize, we can determine the transactional validity of the selected project method and the overall validity of the project methods by assessing whether and to what degree these techniques/strategies apply to each SECUR-ED project method.

The Quality Assessment Framework

The three identified quality criteria—transferability, systematic design/reliability, and transactional validity—combined with our operationalization approaches and sources of data form a framework for assessing the quality of SECUR-ED project methods (Figure 1). The figure shows how quality was assessed through a five-step process: First, the assessment criteria were identified, and then operationalized. Next, the assessment process was operationalized, and by extension, the method’s intended use was evaluated against its actual application, and quality was assessed using the three identified criteria. Thereafter, the identified and operationalized criteria were applied to selected SECUR-ED methods. These methods were first identified and their use within the project was described. Against that baseline, the quality assessment of the project methods was conducted. Data for the assessment included written project documents and oral and written input from project partners, which was documented and systematically reviewed. Finally, the quality aspect of the project methods was assessed and synthesized.

Figure 1.

The quality assessment framework.

Applying the Quality Assessment Framework on SECUR-ED Developed Methods

To assess each project method, we reviewed and compared the data (documents and input from project members) and resolved discrepancies and disagreements in our assessments and descriptions to reach a consensus (Cho & Trent, 2006; Denzin, 1978; Eisner & Peshkin, 1990; Patton, 1990). The process started with the simple formula of identifying the method intent (the intention behind the particular method) and method use (the method’s actual use during the project), which subsequently informed our assessment of transferability, reliability, and validity in the quality assessment step illustrated in Figure 1.

We subsequently assessed each SECUR-ED method qualitatively, according to the three criteria of transferability, reliability, and validity. The results of this evaluation were summarized and merged into a few variables and presented together in a matrix (Table 1): research quality measured by transferability, reliability, and validity, using a 3-point scale of satisfactory, less satisfactory, and unsatisfactory. This matrix provides a holistic picture of the quality dimensions of the applied methods.

Table 1.

An Overview of Our Quality Evaluations of the Applied SECUR-ED Project Methods.

SECUR-ED method	Quality criteria
SECUR-ED method	Transferability	Reliability	Validity
Common glossary	Satisfactory	Unsatisfactory	Unsatisfactory
Interoperability notation	Satisfactory	Less satisfactory	Less satisfactory
Scenario description with BPMN	Less satisfactory	Unsatisfactory	Unsatisfactory
Capacity mapping matrix	Satisfactory	Satisfactory	Satisfactory
Demo city dashboards	Less satisfactory	Unsatisfactory	Unsatisfactory
Capacity evaluation approach	Less satisfactory	Less satisfactory	Less satisfactory

Note. BPMN = business process model and notation; SECUR-ED = Secured Urban Transportation—A European Demonstration.

Evaluation Results

This section presents the results of our assessment for each of the SECUR-ED project methods. It follows the schema of method intent, method use, and quality assessment.

First Method Assessed: Common Glossary

Method Intent

The common glossary document provided a set of security terminology, definitions, and acronyms used throughout the SECUR-ED project, with the main intention to create a common communication platform. The document also was intended to be a living document, and therefore to be continuously updated according to the needs and circumstances throughout the project’s life.

Our evaluations of the quality of the different methods applied in the SECUR-ED project are summarized in Table 1, using a simple 3-point scale: If the method did not comply with good practice as described by a criterion, we enter a score of “unsatisfactory” along this criterion; if there was limited compliance with a criterion, the method was scored as performing “less satisfactory” along this criterion; and if the criterion was roughly met, we enter a score of “satisfactory.”

As shown in Table 1, we have assessed six SECUR-ED project methods. Only one of these (capacity mapping matrix) demonstrated satisfactorily in the quality criteria of transferability, reliability, and validity. One method (common glossary) demonstrated satisfactorily in transferability but unsatisfactorily in both reliability and validity. One method (interoperability notation) also demonstrated satisfactorily in terms of transferability, but less satisfactorily in both reliability and validity. Two methods (scenario description with BPMN and demo city dashboards) scored similarly: They both demonstrated less satisfactorily in transferability and unsatisfactorily in both reliability and validity. Finally, one method (the capacity evaluation approach) scored uniformly less satisfactorily in all quality criteria of transferability, reliability, and validity.

Overall Assessment of Project Methods

Three of the six SECUR-ED project methods assessed in our study were accompanied by detailed descriptions and/or represent established methods or notations, which demonstrate potential transferability to other contexts both within and outside the transportation sector (see Table 1). On the flipside, the scoring shown in Table 1 demonstrates that three of the six methods had unsatisfactory reliability; therefore, clearer requirements and better internal communication and information-sharing could be obtained if such project-wide methods were used consistently and in a way that ensured comparable results. Stability and consistency are prerequisites for rich and thick descriptions, triangulation, persistent observation, and prolonged engagement, and therefore, three of the six SECUR-ED project methods had unsatisfactory contributions to the validity to the project because of the identified variations and inconsistent usage. It should be noted that four of the six methods scored somewhere in between (less satisfactorily) on one or more of the quality parameters, implying that the overall score picture has many nuances. Nevertheless, a ranking can be made as follows. The capacity mapping matrix comes out best, interoperability notation second, capacity evaluation approach third, and common glossary fourth, and scenario description with BPMN and demo city dashboards both came fifth. By scoring high on all quality parameters, the capacity mapping matrix should be of particular importance as “best practice” standard for future European Union (EU) projects. Moreover, we suggest improved coordination and routines for project internal information-sharing be applied to the design and implementation of methods in future projects, and that these projects replicate the documentation efforts demonstrated in relation to several of the SECUR-ED methods. It must be noted the scoring could turn out totally differently if applied in another project.

Limitations and Strengths of the Study

Viswanathan (2005) pointed out the general difference between conceptual and operational in measurements and scientific research. Time and length can be defined conceptually as the shortest distance between two points, and measurement follows directly. Weight and temperature involve more abstract conceptual definitions as well as larger distances between the conceptual and the operational. In the field of social science, for example, when measuring of attitudes toward objects, the distance between conceptual and operational can be large. As the distance increases, so do the different ways to measure something and measurement error, and accurate measurement is central to scientific research. The difference between conceptual and operational—which can be huge—is very relevant for this evaluation and could be considered a weakness of our study.

However, one can also argue that qualitative data and findings are embedded in a given context and subjective experiences, and therefore can change from time to time and place to place (Dew, 2007). This fact makes any extrapolation of “the particular” in qualitative research highly challenging (Erlandson et al., 1993; Lincoln & Guba, 1985), including attempts at repeated and stable measurements, and therefore necessarily less accurate (see our discussion in determining the reliability criterion). Consequently, while we acknowledge that the way we assessed and measured quality criteria in our study—using a 3-point scale of satisfactory, less satisfactory, and unsatisfactory—was largely experimental and not an accurate science, our approach nevertheless complies with the nature of qualitative research.

We compensated for lack of measurement precision with triangulation of data across several researchers and the use of several methods (meetings, teleconferences, and emails) and sources of data (documents and inputs from project members), which increases our study’s credibility. We also documented and applied our assessment framework in a systematic and consistent manner, which strengthened both the transferability and reliability of our study.

Conclusion

SECUR-ED was one of the largest demonstration projects in the EU, with 40 partners operating in different cultural and legal environments. Managing such a comprehensive consortium with partners from different countries, with different mother tongues, cultures, and understandings of how to do the job, is a daunting endeavor. A huge amount of tasks had to be performed within a short timeline, and one failure to deliver would delay somebody’s work somewhere in the consortium. This fact is important to take into consideration as it is much easier to get a favorable score if you have full control of the activity and people involved.

As a contribution to existing research, we have demonstrated how to conceptualize, operationalize, and apply quality criteria in the assessment of project methods that are qualitative in nature, thereby answering the identified need for clear quality guidelines within qualitative research. Therefore, our assessment approach and framework is a novel approach that we believe can hold value in future project evaluations across research areas and sectors. More specifically, we believe that the three quality criteria at the core of the framework and assessment process constitute higher level concepts; that is, the concepts are not embedded in the particular findings and are therefore contextual independent and applicable to other settings and participants (Erlandson et al., 1993; Glaser, 2002; Lincoln & Guba, 1985; Misco, 2007; Morse, 2004). Findings resulting from explorations based on the framework can potentially be compared across research areas and sectors. Consequently, we recommend that research projects conducted across contexts and topics should employ this framework to facilitate further development and validation of the existing assessment framework and process. To take this initiative further along into standards development would provide additional value to policy and statute development.

Footnotes

Acknowledgements

The authors wish to thank the cities and demonstrators as well as other individuals involved for their contributions to the SECUR-ED project and this work. A special appreciation goes to the following persons who contributed with comments and input to our work: Anne-Kari Valdal at Proactima AS, Christian Carling and Karolina Gasinska at the Swedish Defence Research Agency (FOI), and Jean-François Sulzer at Thales Communications and Security SA. They also wish to thank the reviewers for their suggestions and contributions to the article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The European Commission funded this work through the SECUR-ED project and under Grant Agreement number 261605.

Author Biographies

Sindre A. Høyland received his PhD degree in Risk Management and Societal Safety from the University of Stavanger in 2013. His research interests include organisational safety, system complexity, information systems, safe work practices, interdisciplinary teamwork and skills, research design and concepts, and qualitative methodologies. Høyland is currently a post doctor at the University of Stavanger and a senior research scientist at the International Research Institute of Stavanger.

Janne M. Hagen received her MSc degree in industrial economy from the University of Linköping in 1989 and her PhD in information security in 2009. She has been working at different education and research institutions, including the Norwegian Defence Research Establishment and the University of Stavanger, particpating in the reported study and other research on emergency preparedness and critical infrastructure protection. Hagen is currently working as Head engineer at the Norwegian Water Resources and Energy Directorate.

Wolf Engelbach is the head of department for Digitalisation, Climate Protection and Europe in the State Ministry for Mobility of Baden-Württemberg (Germany). The reported study was done as part of his prior position as Head of the Competence Team for Information Management at Fraunhofer IAO, with a research focus on information quality and data governance as well as crisis management and civil security collaboration.

References

Becker

H. S.

(1970). Sociological work: Method and substance. New Brunswick, NJ: Transaction Books.

Billups

(2015). The quest for rigor in qualitative studies—Strategies for institutional researchers. Association for Institutional Research. Retrieved from https://www.airweb.org/eAIR/specialfeatures/Documents/ArticleFBillups.pdf

Bryman

Becker

Sempik

(2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11, 261-276.

Cho

Trent

(2006). Validity in qualitative research revisited. Qualitative Research, 6, 319-340.

Cohen

D. J.

Crabtree

B. F.

(2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. Annals of Family Medicine, 6, 331-339.

de Ruyter

Scholl

. (1998). Positioning qualitative market research: Reflections from theory and practice. Qualitative Market Research, 1, 7-14.

Denzin

N. K.

(1970). The research act: A theoretical introduction to sociological methods. Chicago, IL: Aldine.

Denzin

N. K.

(1978). Sociological methods. New York, NY: McGraw-Hill.

Dew

(2007). A health researcher’s guide to qualitative methodologies. Australian and New Zealand Journal of Public Health, 31, 433-437.

10.

Eisner

Peshkin

(1990). Qualitative inquiry in education: The continuing debate. New York, NY: Teachers College Press.

11.

Erlandson

D. A.

Harris

E. L.

Skipper

B. L.

Allen

S. D.

(1993). Doing naturalistic inquiry: A guide to methods. Newbury Park, CA: Sage.

12.

Fielding

N. G.

Fielding

J. L.

(1986). Linking data. Beverly Hills, CA: Sage.

13.

Firestone

W. A.

(1993). Alternative arguments for generalizing from data as applied to qualitative research. Educational Researcher, 22(4), 16-23.

14.

Glaser

B. G.

(2002). Conceptualization: On theory and theorizing using grounded theory. International Journal of Qualitative Methods, 1(2), 1-31.

15.

Glesne

Peshkin

(1992). Becoming qualitative researchers: An introduction. White Plains, NY: Longman.

16.

Greenhalgh

Peacock

(2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. British Medical Journal, 331, 1064-1065.

17.

Hammersley

(2007). The issue of quality in qualitative research. International Journal of Research & Method in Education, 30, 287-305.

18.

Kapoulas

Mitic

(2012). Understanding challenges of qualitative research: Rhetorical issues and reality traps. Qualitative Market Research: An International Journal, 15, 354-368.

19.

Kerlinger

F. N.

(1973). Foundations of behavioral research (2nd ed.). New York, NY: Holt, Rinehart & Winston.

20.

Kirk

Miller

M. L.

(1986). Reliability and validity in qualitative research. Beverly Hills, CA: Sage.

21.

LeCompte

M. D.

Goetz

J. P.

(1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52, 31-60.

22.

Leung

(2015). Validity, reliability, and generalizability in qualitative research. Journal of Family Medicine and Primary Care, 4, 324-327.

23.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Newbury Park, CA: Sage.

24.

Matthews

T. D.

Kostelis

K. T.

(2011). Designing and conducting research in health and human performance (1st ed.). San Francisco, CA: Jossey-Bass.

25.

Maxwell

J. A.

(1992). Understanding and validity in qualitative research. Harvard Educational Review, 62, 279-300.

26.

Maxwell

J. A.

(1996). Qualitative research design. Newbury Park, CA: Sage.

27.

Mays

Pope

(2000). Assessing quality in qualitative research. British Medical Journal, 320, 50-52.

28.

Misco

(2007). The frustrations of reader generalizability and grounded theory: Alternative considerations for transferability. Journal of Research Practice, 3, 1-11.

29.

Moran-Ellis

Alexander

V. D.

Cronin

Dickinson

Fielding

Sleney

Thomas

(2006). Triangulation and integration: Processes, claims and implications. Qualitative Research, 6, 45-49.

30.

Morse

J. M.

(2004). Constructing qualitatively derived theory: Concept construction and concept typologies. Qualitative Health Research, 14, 1387-1395.

31.

Olsen

W. K.

(2004). Triangulation in social research: Qualitative and quantitative methods can really be mixed. In Haralambos

Holborn

(Eds.), Developments in sociology (pp. 1-30). Ormskirk, UK: Causeway Press.

32.

Onwuegbuzie

A. J.

Leech

N. L.

(2007). Validity and qualitative research: An oxymoron? Quality & Quantity, 41, 233-249.

33.

Patton

M. Q.

(1990). Qualitative evaluation & research methods. Newbury Park, CA: Sage.

34.

Polit

D. F.

Beck

C. T.

(2010). Generalization in quantitative and qualitative research: Myths and strategies. International Journal of Nursing Studies, 47, 1451-1458.

35.

Sandelowski

(2015). A matter of taste: Evaluating the quality of qualitative research. Nursing Inquiry, 22, 86-94.

36.

Secured Urban Transportation—A European Demonstration. (2014a). D22.5 interoperablity framework and guidelines. Paris, France: Thales Group. Retrieved from http://www.secur-ed.eu/?page_id=33

37.

Secured Urban Transportation—A European Demonstration. (2014b). White Paper for public transport stakeholders. Paris, France: Thales Group. Retrieved from http://www.secur-ed.eu/?page_id=33

38.

Stige

Malterud

Midtgarden

(2009). Toward an agenda for evaluation of qualitative research. Qualitative Health Research, 19, 1504-1516.

39.

Thurmond

V. A.

(2001). The point of triangulation. Journal of Nursing Scholarship, 33, 253-258.

40.

Viswanathan

(2005). Measurement error and research design. Thousand Oaks, CA: Sage.

41.

Webster

Watson

R. T.

(2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26, xiii-xxiii.