An interpretive structural modelling of the features influencing researchers’ selection of reference management software

Abstract

Reference management software packages are established as research software packages to help scholars organize their work, improve workflows and ultimately save time. The number of citation management software packages has increased in recent years and therefore choosing an appropriate one has become a challenge for researchers. Scholars always explore the features of an appropriate RMS prior to making an investment to invest in one. Hence, the purpose of this study was to identify and analyze the prominent features in the selection of appropriate reference management software based on an extensive literature review and further to validate this through experts’ opinions. We have utilized the valuable opinions of experts to develop a hierarchical model based on the interpretive structural modelling approach to demonstrate the contextual interrelationship among these factors. Furthermore, the Matrice d’Impacts Croisées-Multiplication Appliquée à un Classement analysis approach has been utilized to classify the identified features based on their dependences and driving power, and to validate the developed interpretive structural modelling-based conceptual model. The developed model in this study can help reference management software developers to understand the correlations among the identified features and their interdependences to further enhance the quality of their products. The academic and practical contributions of the study are discussed.

Keywords

Citation management features interpretive structural modelling MICMAC reference management software

Introduction

Reference management software (RMS), first introduced in 1980s, is used by researchers and academics to manage the bibliographic citations they encounter in their research (Francese, 2011; Reiswig, 2010){Francese, 2011 #1023}. With these software packages, scholars keep track of the scientific literature they read to facilitate the editing of the scientific papers they write. These software packages are known by different names: ‘personal bibliographic software’ (East, 2003), ‘bibliographic citation management software’ (Cibbarelli, 1995), ‘bibliographic management software’ (Fitzgibbons and Meert, 2010), but also ‘reference management software’ (Basak, 2014).

RMS is considered essential for all levels of researchers. A study by Fitzgibbons and Meert (2010) indicated that RMS has become established as time-saving software for researchers writing academic papers. RMS decreases researchers’ workload in terms of editing, proofreading and avoiding formatting errors (Aronsky et al., 2005). RMS not only appraises and codes search results but also organizes and stores them (King et al., 2011). The primary reasons for using them for writing research literature reviews are: promoting the accuracy in reference citations, decreasing the time in reformatting information to meet the style requirements of different journal publishers and managing a large quantity of reference data (Emanuel, 2013; Steele, 2008).

Several different RMS exist, with sometimes different features and purposes, namely Mendeley (Medaille, 2010), Zotero (Arellano, 2010), EndNote (Reichardt, 2010a), CiteULike (McMullen, 2010), RefWorks (Reichardt, 2010b). All these are well known in the scientific community (Duong, 2010; Hull et al., 2008; Norman, 2010). Today’s packages offer sophisticated functionalities, and the basic functions of older versions have been extended through advanced features which vary from package to package. Most RMS packages take the best of the web environment providing application programming interfaces (APIs) which allow integration with other software or other virtual environments, sharing and enriching the data, collecting them from different sources, alongside many other features.

The features of RMS are important in researchers’ selection of citation management software. Many researchers who are not familiar with RMS may need to consult a reference librarian or other expert in their decision to select citation management software and before purchasing a specific proprietary package. Users who have experience in working with different types of RMS can provide valuable information in that regard. In addition, for a college student, the Internet may also be a valuable resource to search for information about RMS. Several publications have been published comparing RMS packages using a set of criteria and features. Specific comparative studies of features have been performed by Dell’Orso (2010); Kessler and Van Ullen (2005); Steeleworthy and Dewan (2013).

Since RMS programs are practical software used in a real-case context, it is worth investigating the importance of their features and assessing them based on those features. In addition, while the RMS programs have been previously assessed, based on a number of factors like ease of use and quality of support documentation, it seems an opportune time to evaluate the features of RMS and determine their levels of importance from the researchers’ perspective when they choose an RMS program.

Hence, the purpose of this paper is to identify the prominent features that influence scholars and researchers to select a particular RMS package by answering the following research questions:

What are the main features to be considered for the selection of an appropriate RMS?

What is the contextual interrelationship among these features?

Four reference management programs, Endnote, Mendeley, Zotero and RefWorks that have been frequently used in prior studies were selected for this study. To answer the research questions, we derived the main features of RMS through an extensive literature review and further analysed these based on experts’ opinions, utilizing the interpretive structural modelling (ISM) approach and Matrice d’Impacts Croisées-Multiplication Appliquée à un Matrice d’Impacts Croises Multipication Applique’ an Classment (MICMAC) analysis technique.

The remainder of this paper is organized as follows. In the next section, the literature review of related work on RMS programs and their features is presented. This is followed by a description of the ISM methodology and a discussion of the results of the study. Finally, conclusions, implications, limitations and future work are presented.

Literature review

Related work

Literature about RMS can be dived into two main themes: firstly, we find description, comparison and technical analysis of the features offered by the software packages; secondly, we find papers about library initiatives in training and promotion of the packages. These two main threads are confirmed by Martin (2009) and McMinn (2011). Several articles have been written about bibliographic management products. Some have compared their features (Gilmour and Cobus-Kuo, 2011; Hensley, 2011; Ovadia, 2011), some researchers have investigated the bibliographic accuracy of the cited papers (Eichorn and Yankauer, 1987; Gupta et al., 2005), others have investigated their use among students (Emanuel, 2013; Salem and Fehrmann, 2013), or how faculty perceive them (Martin, 2009). In addition, a range of short reviews of different packages of RMS have been provided for example for Mendeley (Medaille, 2010), Zotero (Arellano, 2010), EndNote (Reichardt, 2010a), CiteULike (McMullen, 2010), RefWorks (Reichardt, 2010b).

Ram et al. (2014) conducted a study through an online survey with the aim of assessing the perception, awareness and use of bibliography management software (BMS) by Library and Information Science (LIS) professionals in India. Kali (2016) conducted a review to illustrate the essential features, advantages and limitations of popular referencing tools that would be beneficial for medical students and scholars in their research endeavour and academic development. Using the unified theory of acceptance and use of technology (UTAUT) model, Rempel and Mellinger (2015) explored how researchers select a bibliographic management tool and what makes them continue using this tool. The findings of their research indicated that participants (researchers) adopt bibliographic management tools because of an expectation of enhanced research productivity. The research findings also demonstrated that participants persist in using these tools because of ease-of-use experiences. Cibbarelli (1995) undertook a survey on the usage of RMS. The researcher asked her respondents to rate different aspects of the software (such as available documentation, ease of use, reliability, etc.) on a scale from 0 to 10. The results seemed positive, setting the average rating at around 8, and the comments provided by the respondents seemed encouraging for further attention to the subject. Her survey was addressed to the customers of the software companies: she questioned people who were already using such a software package; she did not calculate the level of popularity. Kessler and Van Ullen (2005) also point at the role of libraries, and reference librarians in particular, in providing information and support on managing bibliographies and citations. They examined the accuracy of citations generated by NoodleBib, EasyBib and EndNote. The three products differed in the types of errors they generated, as well as in their handling of print and electronic sources, with NoodleBib having the lowest error rate. Bornmann and Haunschild (2015) used the Mendeley API to download the Mendeley counts (broken down by different user types of publications in Mendeley) for a comprehensive F1000Prime data set. Kraker et al. (2015) analysed the adequacy and applicability of readership statistics recorded in social reference management systems for creating knowledge domain visualizations. An analysis of the distribution of subject areas in user libraries of educational technology researchers on Mendeley showed that 69.2% of the journal articles in an average user library can be attributed to a single subject area.

Homol (2014) examined APA and MLA bibliographies created using the citations generated by RefWorks, EndNote Basic, Zotero, and EDS to determine the frequency and type of errors each program’s citations contained. A survey by Lorenzetti and Ghali (2013) showed that of the 78 researchers, 79.5% had used a reference management software package to prepare their review. EndNote, Reference Manager and RefWorks were the programs of choice for more than 98% of authors who used this software. The article by Fitzgibbons and Meert (2010) is a critical analysis of RMS reliability. Their study compared the results between searches conducted in academic databases’ search interfaces versus the EndNote search interface. The results showed that mixed search reliability depends on the database and type of search performed. Steele (2008) focused on the accuracy of the use of citations in research papers, providing useful hints for those who have to make decision about adoption of this type of software.

The use of citation management software does not guarantee the absence of reference errors within a manuscript. Bibliographic errors in capitalization, punctuation, dates, and volume or issue number are frequent occurrences and have been identified with the use of both free and proprietary computer programs. Sahu (2003) indicated that incorrect referencing always frustrates researchers in terms of searching for specific articles. Studies conducted by Evans et al. (1990), Eichorn and Yankauer (1987) and Gupta et al. (2005) found errors in major surgical journals in terms of citation and quotation. Steeleworthy and Dewan (2013) reviewed popular citation management systems, RefWorks, Zotero, WizFolio and Mendeley. To compare these software packages, they examined the import capabilities as well as the organizing, searching, annotating and sharing functions of these RMS.

Zhang (2012) examined the popular reference management software, EndNote, EndNote Web and RefWorks, and also the free software packages of Mendeley and Zotero. In addition to a demonstration of the main features, they provided a comparison of the products, including the advantages and disadvantages of each one. Courraud (2014) focused on Zotero, a free and open-source program originally developed by the Center for History and New Media at George Mason University and the Corporation for Digital Scholarship in the USA. They introduced Zotero software and listed all strengths and limitations of this software as a bibliographic management product. Basak (2014) conducted a comparative analysis of Zotero and Mendeley RMS to see which software can import data more accurately. This aim was achieved through the specific objectives: to identify which software is more accurate in terms of importing citations, reference and ease of use; to identify the similarities for the specific fields of the RMS. Brahmi and Gall (2006) compared EndNote and Reference Manager’s citations to the instructions provided for references by top medical journals. It was discovered that both programs had difficulties formatting the author, article title, journal title and punctuation according to the journals’ standards.

Gomis et al. (2008) examined EndNote and Refworks in the health sciences literature. They found that both software packages can achieve similar results and can be useful for acquiring new literature using established personal search techniques. Fitzgibbons and Meert (2010) also did similar work and examined EndNote for library database searching through their web interfaces. Kessler and Van Ullen (2005) compared NoodleBib and EasyBib with EndNote in relation to accuracy. In a recent study by Zhang (2012), the researcher compared the main features of EndNote, Zotero, Connotea and Medeley from the viewpoint of medical researchers.

RMS features

Various types of software packages are available that scholars and researchers can use to manage the bibliographic details of the information and documents that they find during their research activities. Hence, consulting with individuals who are expert in using RMS, like reference librarians or other scholars, regarding the necessary features is highly recommended before using a specific tool and, most importantly, proprietary packages (Steele, 2008).

In order to extract the features which are common among the well-known RMS, we reviewed different types of literature including public reports comparing RMS (e.g. Technische Universität München Library 2015), scholarly articles investigating the features of RMS (e.g. Hensley, 2011) and the source websites of RMS. In the first round we extracted 17 features from these resources. To make the list of features short and more comprehensible, we used general terms for the features with the same meaning. For example, in different resources the compatibility of RMS to different operating systems (e.g. Windows, Linux, etc.) was mentioned separately. Instead, we used the general term of OS Compatibility for the purpose of this paper.

After revising the initial list, we identified 13 important features which were common among the reviewed RMS. Table 1 presents the features.

Table 1.

Common features among the selected software packages.

	OS Compatibility	Data import	Collaboration	Word processor integration	Intuitive user interface	Cite-as-you-write	Metadata extraction	Mobile compatibility	Database search	Completion of metadata	Export from databases	Linking of references	Duplicate checking
RefWorks¹	●	●	●	●	●	●	○	●	●	●	●	○	●
EndNote²	●	●	●	●	○	●	●	○	●	●	●	●	●
Zotero³	●	●	●	●	●	●	●	●	○	●	●	●	●
Mendeley⁴	●	●	●	●	●	●	●	●	○	●	●	○	●

Notes:

● Available

○ Not available

www.refworks.com, ² www.endnote.com, ³ www.zotero.org, ⁴ www.mendeley.com

ISM methodology

In this study we were interested in gathering feedback from expert academic researchers regarding the features that individuals consider when selecting appropriate RMS for their citation and bibliographic purposes. First introduced by Warfield (1974), ISM methodology was devised to deal with complex issues. It enables individuals or a group of experts to develop a map of complex relationships between many factors in a complex situation (Ansari et al., 2013). The method of ISM is used to interpret complex situations together with the development of courses of actions to solve the target problem (Sohani and Sohani, 2012). This method has been used by many prestigious organisations, such as NASA, to solve complicated problems (Meena and Thakkar, 2014).

Statistical analyses and hypotheses testing of emerging research areas are enabled first by assessing the field qualitatively (Edmondson and McManus, 2007). However, investigating a research field using quantitative research methods extensively while it has not achieved maturity ‘… is not likely to produce compelling field research’ (Edmondson and McManus, 2007: 1168). In the same way, while the existing literature is not rich enough for the identification of features and the relationship between them to select an appropriate RMS, a qualitative ISM method is considered suitable for the purpose of this study (Bolaños et al., 2005).

Since the focus of this research was to investigate the personal preferences of researchers, we identified focus groups as an appropriate data collection method. Focus groups are considered an exploratory methodology and are specifically good for ‘… understanding both what people think about a topic and why they think that way …’ (Beck and Manuel, 2008: 78). Furthermore, in order to ensure that the same results would be obtained by a repetition of operations, we collected the data utilizing various techniques like brainstorming during focus groups sessions.

The results of the current literature survey showed that there is no study that uses ISM for assessing the interrelationship between the RMS features from the researchers’ perspective. While previous studies compare RMS programs using several criteria and features, in this study we evaluate those features from the researchers’ perspectives when they are making a decision about selecting an appropriate reference management program.

Three modelling languages construct ISM including words, diagraphs and discrete mathematics, which together offer a methodology to structure the complex issue. ISM is interpretive as the judgement of working participants in a group decide whether and how the factors of a complex situation are related (Ansari et al., 2013). To develop an ISM several steps should be taken as follows (Eswarlal et al., 2011):

Step 1: identification of variables to be studied. In this study, features that influence scholars to use specific RMS were identified.

Step 2: examining the contextual relationship among the variables identified in Step 1.

Step 3: indicating pair wise relationship between variables, and developing a structural self-interaction matrix (SSIM).

Step 4: developing the reachability matrix from the SSIM. Checking the transitivity of the matrix. Transitivity is the basic assumption in ISM which states if variable X is related to variable Y, and variable Y is related to variable Z, then necessarily variable X is related to variable Z.

Step 5: through the reachability matrix developed in Step 4, partitioning of levels is done.

Step 6: based on the contextual relationships resulting from the matrix, a directed graph is drawn and the transitive links are removed.

Step 7: converting the diagraph to an interpretive structural model by replacing variable nodes with statements.

Step 8: reviewing the model developed in Step 7 for any possible contextual inconsistencies and modifications.

Figure 1 illustrates the necessary steps to be taken for preparing ISM.

Figure 1.

Flow diagram for preparing ISM model adapted from Kannan et al. (2009).

Focus group arrangement and participants

We recruited 10 academics who were expert in using RMS for their research activities from Universiti Teknologi Malaysia. Prior to conducting the focus group sessions, we presented participants with an outline of the study’s purposes and processes, and secured their permission to have their voices tape-recorded for the purpose of transcription. The participants were selected based on their experience in conducting research and their expertise in using RMS for their research activities. Table 2 exhibits the profile of focus groups participants in this study.

Table 2.

Profile of focus group participant.

Title	No.	Gender	No.	Age	No.	Research experience (years)	No.	RMS	Proficiency level
PhD Candidate	6	Male	6	< 25	0	3–5	1	Endnote	High
Doctor	3	Female	4	25–35	7	5–8	4	Mendeley	High
Assoc. Prof.	1			35–55	3	> 8	5	Zotero	Medium
				> 55	0			RefWorks	Medium

We conducted two focus group sessions in March 2016. Each session was held in a discussion room available in the Faculty of Computing featuring a video projector, whiteboard, and comfortable table and chairs for the participants to gather around.

Application of ISM

Following the suggested ISM procedure depicted in Figure 1, firstly, we performed an extensive literature review on RMS literature and then conducted brainstorming sessions during established focus groups with the selected academic experts. During the initial focus group session, all the invited experts participated in the session and the list of all extracted features from the literature (see Table 1) was circulated among them. The list of RMS features was given to the experts for developing consensus on the features included on the list. The experts decided to group the features related to their functionalities, in order to decrease the number of items in the initial list during the first session. Accordingly, they defined a new feature termed Technical specification features which was used as an umbrella term to include any technical and technology-related features such as ‘OS compatibility’, ‘word processing integration’ and ‘mobile compatibility’. The general term of Citing features replaced the two features of ‘cite-as-you-write’ and ‘linking of references’, as this new term contains the functionalities of the other two features. The term Import features was introduced during the session by the experts as a general feature to include the features of ‘data import’, ‘metadata extraction’ and ‘import of references’, since all these features are linked to importing data and information to the RMS. The general term of Collaboration features was introduced since some of the RMS packages facilitate collaboration through social network platforms. The term Ease of use features was recommended by experts instead of the term ‘intuitive user interface’. This term is one of the mostly used terms in the context of technology adoption introduced in the technology acceptance model (TAM) (Davis, 1986). It has been an important feature for researchers when selecting software packages (Emanuel, 2013). In this study, ease of use refers to the integration of a software package with other programs and computer interfaces and easily finding the software functions. The general term View/search features was introduced to replace the features of ‘database search’ and ‘export from databases’. During the session, experts introduced two new general terms and recommended their inclusion in the list – Editing features and Data format features. Editing features provide researchers with the capabilities of creating references, editing references, selecting publication type, creating folders/groups and ‘completion of metadata’. Document types (e.g. book, book chapter, journal article, etc.) that are supported by the RMS and the related fields assigned to each document type are related to data format features introduced by the experts.

After 10 days, another focus group session was established to finalize the features important for the selection of RMS. Figure 2 depicts the finalized selected RMS features confirmed by the experts at the end of the second focus group session.

Figure 2.

Features influencing RMS selection.

Development of structural self-interaction matrix

As discussed above, during the group discussion and brainstorming session conducted with academic experts the contextual relationships among the identified features were identified. We have used four symbols to denote the relationships between the variables in the development process of the structural self-interaction matrix (SSIM).

These four symbols are as follows:

V – Feature ‘i’ leads to feature ‘j’;

A – Feature ‘j’ leads to feature ‘i’;

X – Feature ‘i’ and feature ‘j’ lead to each other;

O – Feature ‘i’ and ‘j’ are unrelated.

Based on the contextual relationships among the identified features, the SSIM was developed (see Table 3).

Table 3.

Structured self-interaction matrix (SSIM) for the features influencing RMS selection.

S. no	Features influencing RMS selection	8	7	6	5	4	3	2
1	Technical specifications features	A	A	A	A	A	X	V
2	Import features	A	A	A	A	A	X
3	Data format features	A	X	X	X	X
4	Editing features	A	X	X	X
5	View/search features	A	X	X
6	Collaboration features	A	X
7	Citing features	A
8	Ease of use features

Feature 1 leads to Feature 2 so symbol ‘V’ has been given to the cell (1, 2); Feature 8 leads to Feature 1 so symbol ‘A’ has been given to cell (1, 8); Features 3 and 7 lead to each other, hence symbol ‘X’ has been given to the cell (3, 7); and no features lead to each other. The number of pair wise comparison to construct the SSIM is (N × (N – 1) / 2), where N is the number of identified factors.

Initial reachability matrix

The initial reachability matrix (RM) is constructed by converting the SSIM developed in the previous section to a binary matrix. The target binary matrix is developed by substituting the symbols ‘V’, ‘A’, ‘X’ and ‘O’ by ‘1’ or ‘0’ based on the following rules:

If the value of the cell (i,j) in the SSIM is the symbol ‘V’, then, in initial RM the values of (i,j) and (j,i) are ‘1’ and ‘0’, respectively; If the value of the cell (i,j) in the SSIM is the symbol ‘A’, then, in initial RM the values of (i,j) and (j,i) are ‘0’ and ‘1’, respectively;

If the value of the cell (i,j) in the SSIM is the symbol ‘X’, then, in initial RM the values of (i,j) and (j,i) are both ‘1’;

If the value of the cell (i,j) in the SSIM is the symbol ‘O’, then, in initial RM the values of (i,j) and (j,i) are both ‘0’;

For example, for V(1,2) in the SSIM, ‘1’ has been given in cell (1,2) and ‘0’ in cell (2,1) in initial RM; for A(1,8) in the SSIM, ‘0’ has been given in cell (1,8) and ‘1’ in cell (8,1); for X(3,7) in the SSIM, ‘1’ has been given in cells (3,7) and (7,3) in initial RM.

An initial RM has been developed based on the abovementioned rules for the features to be considered for the selection of an appropriate RMS (see Table 4).

Table 4.

Initial reachability matrix.

F.no	Features influencing RMS selection	1	2	3	4	5	6	7	8
1	Technical specifications features	1	1	1	0	0	0	0	0
2	Import features	0	1	1	0	0	0	0	0
3	Data format features	1	1	1	1	1	1	1	0
4	Editing features	1	1	1	1	1	1	1	0
5	View/search features	1	1	1	1	1	1	1	0
6	Collaboration features	1	1	1	1	1	1	1	0
7	Citing features	1	1	1	1	1	1	1	0
8	Ease of use features	1	1	1	1	1	1	1	1

Final reachability matrix

The final RM is developed based on Step 4 of the rules explained in the Methodology section, which states if variable X is related to variable Y, and variable Y is related to variable Z, then, necessarily variable X is related to variable Z.

The final reachability matrix was obtained from the previously developed initial RM (see Table 4). Table 5 exhibits the final RM of the features influencing the selection of RMS. From this matrix, the driving power and dependence power of each feature is calculated by adding all 1s in the rows and all 1s in the columns, respectively.

Table 5.

Final reachability matrix.

S. no	Features influencing RMS selection	1	2	3	4	5	6	7	8	Driving Power ↓
1	Technical specifications features	1	1	1	1^*	1^*	1^*	1^*	0	7
2	Import features	1^*	1	1	1^*	1^*	1^*	1^*	0	7
3	Data format features	1	1	1	1	1	1	1	0	7
4	Editing features	1	1	1	1	1	1	1	0	7
5	View/search features	1	1	1	1	1	1	1	0	7
6	Collaboration features	1	1	1	1	1	1	1	0	7
7	Citing features	1	1	1	1	1	1	1	0	7
8	Ease of use features	1	1	1	1	1	1	1	1	8
Dependence power →		8	8	8	8	8	8	8	1	57/57

Incorporating the ‘rule of transitivity’.

Partitioning of levels

In order to reveal the importance level of each feature, the partitioning level technique is performed. From the final RM (see Table 5), the reachability set and antecedent set (Warfield, 1974) for each feature was obtained. The reachability set of a particular feature is a set of features influenced by that and the feature itself, whereas the antecedent set of a particular feature is a set of features influencing that feature and itself. Specifically, the reachability set of feature i is the set of features with values of ‘1’ and ‘1^*’ in the row i of final RM and antecedent set of feature i is the set of features with values of ‘1’ and ‘1^*’ in the column i of the final RM.

Reachability, antecedent and intersection sets of all features have been found. A feature having the same reachability set and intersection set has been assigned as the top level feature in the ISM hierarchy (Warfield, 1974) (see Table 6).

Table 6.

First iteration for partitioning the levels of features.

Feature no.	Reachability set	Antecedent set	Intersection set	Level
1	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
2	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
3	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
4	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
5	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
6	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
7	1,2,3,4,5,6,7	1,2,3,4,5,6,7,8	1,2,3,4,5,6,7	1st
8	1,2,3,4,5,6,7,8	8	8	2nd

Feature in Level 1 is discarded to find further levels. In this study, after performing the first iteration only two layers of features were identified and there was no necessity for further iterations. Table 7 exhibits the final levels of features.

Table 7.

Various levels of features.

S. no.	Level no.	Features influencing RMS selection
1	1st	• Technical specifications features• Import features• Data format features• Editing features• View/search features• Collaboration features• Citing features
2
3
4
5
6
7
8	2nd	• Ease of use

In our study, we identified two levels of features. Ease of use was identified as the only bottom level feature, whereas the other seven features were found to be most important top level features.

ISM-based model formation for the features to be considered for RMS selection

Upon understanding the levels of features (see Table 7) and utilizing the RM (see Table 4), the structural model can be generated graphically by the aid of vertices and edges (Ansari et al., 2013). Out of 8 features identified for the selection of appropriate RMS by academic experts (see Methodology section), citing features (CIF), collaboration features (COF), view/search features (VSF), editing features (EDF), data format features (DFF), import features (IMF), and technical specifications features (TSF) are lying at the top level of the model. Ease of use features (EUF) is lying at the bottom layer of the structural model. In order to classify these eight features, further MICMAC analysis has been performed.

Based on the ISM methodology described above, after removing the transitivity, the ISM model called a digraph was created and is depicted in Figure 3.

Figure 3.

ISM-based model for the features influencing RMS selection.

Classification of features: MICMAC analysis

MICMAC (cross-impact matrix multiplication applied to classiﬁcation) (Duperrin and Godet, 1973) is an approach to graphically classify factors of a complicated situation based on their driving power and dependence power. Based on driving and dependence powers, factors are classified into four clusters of Autonomous, Independent, Linkage and Dependent. Independent factors are the most important ones with high driving power and low dependency. Variables with intermediate importance are Linkage factors with not only high driving power but also high dependence power. Dependent factors are those driven by independent variables in which they have low driving power and high dependence power. The stand-alone factors are categorized under Autonomous variables. Both driving power and dependence power of these variables are low but they are still essential parts of the system.

The purpose of this section is to investigate the driving and dependence powers of features influencing RMS selection and classify them utilizing MICMAC analysis. High value of dependence power for a feature means that large number of features should be improved to enhance that feature, and high value of driving power means a large number of features would be enhanced upon the improvement of that feature. Figure 4 illustrates the result of MICMAC analysis of features based on their driving and dependence powers.

Figure 4.

MICMAC analysis for the features influencing RMS selection.

From the results (see Figure 4), we can see that ease of use is the only Independent feature in the criteria for selecting RMS. Technical specification features, import features, data format features, editing features, view/search features, collaboration features, and citing features are all Linkage features. There are no autonomous and dependent features in the model.

Discussion

The results of this study show that the feature with the highest driving power to influence users to select RMS compared with other features is Ease of use. Perceived ease of use is an important construct investigated in technology adoption studies and is defined as ‘the degree to which a person believes that using a particular system should be free of effort’ (Davis et al., 1989: 320). In the study conducted by Emanuel (2013), the researcher asked the users what features they look for when selecting RMS and they found that ease of use was the number one response with more than 69% of users specifying it. For an RMS package with a complicated interface which is difficult to learn, there would be a human resource cost in developing instructional materials and teaching workshops (Hensley, 2011). An intuitive user interface design would impact on the design of other features of the RMS such as editing, search, import/export etc. Accordingly, the findings of this study are in line with previous studies highlighting the importance of ease of use in the technology adoption context.

Strong elements residing at the lowest level of the hierarchy are considered as the root cause factors of other variables in the model. This approach would assist RMS developers to fit their RMSs more towards the demands of users. Following the MICMAC analysis, the driving power and dependence power (see Table 5) of individual factors were taken into consideration to further classify features into four groups of Dependent, Independent, Linkage and Autonomous drivers. While there were no factors in two groups of Independent and Autonomous, seven features were categorized under Linkage factors, and one feature was categorized under Dependent factors (see Figure 4).

Autonomous features

The stand-alone factors are categorized under Autonomous variables. Both driving power and dependence power of these variables are low but they are still essential parts of the system.

Autonomous features which are somehow disconnected from the system have less driving and dependence power compare with other factors. In our study no factors fall in this category. This indicates that all the features were identified carefully for the purpose of this study. An empty cluster of the autonomous group suggests that all the identified factors are influencing the overall decision-making process of the system and none has autonomic property.

Linkage features

Linkage features gained relatively high driving power and driving dependence. These drivers are unstable and any change to them affects other drivers and themselves.

Seven features of citing features (CIF), collaboration features (COF), view/search features (VSF), editing features (EDF), data format features (DFF), import features (IMF) and technical specifications features (TSF) were categorized under this group. It means that any change to any of these features would affect other features of RMS. Since all these seven features gained the same values for their driving power (7) and dependence power (8), they are all distributed around the same point in the diagram depicted in Figure 4 related to the Linkage cluster.

Dependent features

Dependent factors are the ones that are driven by independent variables in which they have low driving power and high dependence power. In our model, none of the features were categorized under this group.

Independent features

Independent features are the most important ones with high driving power and low dependency. These factors are considered as ‘key factors’ which are also considered as the root cause of other factors in the complex problem. In our study, only one feature of ease of use (EUF) was categorized as an Independent factor with high driving power and low dependence power.

Conclusions and further study

In this research, an effort was made to develop a hierarchical model using the ISM approach to assess the contextual interrelationships among the features of RMS programs from the researchers’ perspective and to analyse their driving and dependency power. The features of the RMS, namely: ease of use features (EUF), citing features (CIF), collaboration features (COF), view/search features (VSF), editing features (EDF), data format features (DFF), import features (IMF) and technical specifications features (TSF) were selected from the literature. The data for this research was collected via a focus group including 10 academics who were expert in utilizing RMS in their research.

Limitations of the study

In this study we developed a model of features influencing the selection of an appropriate RMS package based on experts’ point of views and the literature. There is a necessity to test the model in the real world to check the features and the relationship among them. In a real case application, the identified features may be incomplete or their relationships different. Although the ISM-based model provides a good understanding of relationships between these features, it does not indicate how and to what extent each feature influences others in the selection of an RMS program.

Future studies

The current study elicited features to be considered for the selection of an RMS package from the literature, and further analysed these using ISM methodology and MICMAC analysis approach. The scope of a future study could be:

empirically test the model utilizing structural equation modelling (SEM) to investigate the significance and effect of each feature on other features based on their hypothesized relationships;

the interrelationship among these features could be quantified utilizing multi-criteria decision making models (MCDMs) such as analytical hierarchy process (AHP), analytical network process (ANP), interpretive ranking process (IRP), etc.;

increasing the sensitivity of MICMAC analysis by considering additional possible interaction among the features rather merely binary interaction, which is called fuzzy MICMAC analysis.

Implications of the study

The results of this study contribute both to research and practice.

Research contributions

The appropriate selection and using of RSM is an important issue for scholars from different disciplines. This study represents one of the first focusing on the features influencing the selection of an appropriate RMS package utilizing ISM methodology. This research made an original contribution in defining a model for the features of RMS selection.

By enriching our understanding of the influence of features on the selection of an RMS package, the model sheds light on how scholars and researchers select an appropriate software package for their reference management purposes.

Practical contributions

The results of this study also provide practical implications for RMS developers wishing to develop or maintain high levels of software usage. RMS developers may face challenges in identifying these features and then working on them to increase the quality of their products. In this paper an attempt has been made to identify the most important features which developers can consider to enhance the quality of their RMS. Furthermore, the proposed SSIM would help developers better understand the features that mostly influence RMS selection and the relationships between them. Features with highest driving power are more critical and developers can use the results of this study in their tactical and strategic decisions for the improvement of their output product. Following the results of this study, the ease of use feature was selected as the most important one which developers should consider in developing RMS. Not only should the interface be easy to learn but also all other features and functionalities should be intuitive to learn without additional costs of learning for the users.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the Research Management Center (RMC) of Universiti Teknologi Malaysia (UTM) through its postdoctoral fellowship scheme with Reference No.: PY/2016/06979.

Author biographies

Mehrbakhsh Nilashi received his PhD in Computer Science in the Faculty of Computing, Universiti Teknologi Malaysia in 2014. His research is mainly in the field of soft computing, machine learning, big data, multi-criteria decision making, information retrieval, recommender systems with a special focus on multi-criteria recommender systems, health informatics, health information technology and decision support systems. His contributions have been published in prestigious peer-reviewed journals and international conferences.

Mohammad Dalvi Esfahani received his PhD in Information Systems in the Faculty of Computing, Universiti Teknologi Malaysia in 2016. He holds an MSc in Information Technology from UTM and BSc in Software Engineering from University of Isfahan, Iran. His research interests are in IS and environmental sustainability, Green IT/IS, eco-innovation, business process management, multi-criteria decision making, tourism management, IT diffusion and adoption. He has published articles on Green IT/IS, business process improvement, wavelet analysis and quality of experiences. He also serves as an expert corporate trainer for the international certificates of ITIL, COBIT5 and Prince2.

Morteza Zamani Roudbaraki received his Master’s degree in Information Technology in the School of Computer Sciences from Universiti Sains Malaysia (USM). His research is mainly in the field of cloud computing, mobile cloud computing, mobile healthcare systems, decision support systems, health informatics, information retrieval and information systems.

T. Ramayah is currently a professor of Technology Management at the School of Management, Universiti Sains Malaysia, visiting professor at King Saud University, Kingdom of Saudi Arabia and adjunct professor at Multimedia University and Universiti Tenaga Nasional, Malaysia. He has taught courses in Statistics, Operations Management, Research Methods and at undergraduate level. He has supervised numerous PhD/MA/MBA students in the fields of information systems, operations management, marketing management and organizational behaviour. He has also presented numerous papers at local and international conferences having won 5 Best Papers awards. His publications have appeared in many international peer-reviewed journals. He also serves on the editorial boards and program committees of several international journals and conferences of repute. His full profile can be accessed at .

Othman Ibrahim is an associate professor in the Faculty of Computing, Universiti Teknologi Malaysia (UTM). His interests are in information systems, information systems theory, knowledge management, e-government and e-business. He received his PhD from University of Manchester Institute of Science and Technology (UMIST).

References

Ansari

Kharb

Luthra

. (2013) Analysis of barriers to implement solar power installations in India using interpretive structural modeling technique. Renewable and Sustainable Energy Reviews 27: 163–174.

Arellano

(2010) A case for Zotero. Public Services Quarterly 6(4): 364–366. DOI: 10.1080/15228951003772546

Aronsky

Ransom

Robinson

(2005) Accuracy of references in five biomedical informatics journals. Journal of the American Medical Informatics Association 12(2): 225–228.

Basak

(2014) A Comparative Analysis of Zotero and Mendeley Reference Management Software. World Academy of Science, Engineering and Technology. Available at: http://waset.org/publications/9999631/a-comparative-analysis-of-zotero-and-mendeley-reference-management-software (accessed 29 August 2016).

Beck

Manuel

(2008) Practical Research Methods for Librarians and Information Professionals. New York: Neal-Schuman.

Bolaños

Fontela

Nenclares

. (2005) Using interpretive structural modelling in strategic decision-making groups. Management Decision 43(6): 877–895. DOI: 10.1108/00251740510603619

Bornmann

Haunschild

(2015) Which people use which scientific papers? An evaluation of data from F1000 and Mendeley. Journal of Informetrics 9(3): 477–487.

Brahmi

Gall

(2006) EndNote® and Reference Manager® Citation Formats Compared to ‘Instructions to Authors’ in top medical journals. Medical Reference Services Quarterly 25(2): 49–57.

Cibbarelli

(1995) Cibbarelli’s surveys: User ratings of bibliographic citation management software. Computers in Libraries 15(4): 25–28, 30–32, 34–40.

10.

Courraud

(2014) Zotero: A free and open-source reference manager. Medical Writing 23(1): 46–48.

11.

Davis

(1986) A technology acceptance model for empirically testing new end-user information systems: Theory and results. PhD Dissertation, Massachusetts Institute of Technology, USA.

12.

Davis

Bagozzi

Warshaw

(1989) User acceptance of computer technology: A comparison of two theoretical models. Management Science 35(8): 982–1003.

13.

Dell’Orso

(2010) Bibliography Management Software with a Detailed Analysis of Some Packages: Bibus, EndNote Web, RefWorks, Procite, EndNote, Reference Manager. 3rd edn. Available at: http://www.riccardoridi.it/esb/dellorso/bms-dasp/text/index.html (accessed 29 August 2016).

14.

Duong

(2010) Rolling out Zotero across campus as a part of a science librarian’s outreach efforts. Science & Technology Libraries 29(4): 315–324.

15.

Duperrin

Godet

(1973) Méthode de Hiérarchisation des Eléments d’un Système. Rapport Economique du CEA 4541. Paris: CEA. Available at: http://lara.inist.fr/bitstream/handle/2332/1601/XX_CNE-LIPSOR_001414.pdf%3Fsequence%3D1 (accessed 29 August 2016).

16.

East

(2003) Z39.50 and personal bibliographic software. Library Hi Tech 21(1): 34–43.

17.

Edmondson

McManus

(2007) Methodological fit in management field research. Academy of Management Review 32(4): 1246–1264.

18.

Eichorn

Yankauer

(1987) Do authors check their references? A survey of accuracy of references in three public health journals. American Journal of Public Health 77(8): 1011–1012.

19.

Emanuel

(2013) Users and citation management tools: Use and support. Reference Services Review 41(4): 639–659.

20.

Eswarlal

Dey

Shankar

(2011) Enhanced renewable energy adoption for sustainable development in India: Interpretive structural modeling approach. In: World renewable energy congress 2011, Linköping, Sweden, 8–13 May 2011.

21.

Evans

Nadjari

Burchell

(1990) Quotational and reference accuracy in surgical journals: A continuing peer review problem. JAMA 263(10): 1353–1354.

22.

Fitzgibbons

Meert

(2010) Are bibliographic management software search interfaces reliable?: A comparison between search results obtained using database interfaces and the EndNote online search function. Journal of Academic Librarianship 36(2): 144–150.

23.

Francese

(2011) Mendeley, dal social bookmarking al reference management: Uno strumento innovativo per la gestione delle citazioni bibliografiche. Biblioteche Oggi 29(2): 41–46.

24.

Gilmour

Cobus-Kuo

(2011) Reference management software: A comparative analysis of four products. Issues in Science and Technology Librarianship 66(66): 63–75.

25.

Gomis

Gall

Brahmi

(2008) Web-based citation management compared to EndNote: options for medical sciences. Medical Reference Services Quarterly 27(3): 260–271. DOI: 10.1080/02763860802198804

26.

Gupta

Yadav

Mohta

. (2005) References in Indian pediatrics: Authors need to be accurate. Indian Pediatrics 42(2): 140–145.

27.

Hensley

(2011) Citation management software: Features and futures. Reference & User Services Quarterly 50(3): 204–208.

28.

Homol

(2014) Web-based citation management tools: Comparing the accuracy of their electronic journal citations. Journal of Academic Librarianship 40(6): 552–557.

29.

Hull

Pettifer

Kell

(2008) Defrosting the digital library: Bibliographic tools for the next generation web. PLoS Computational Biology 4(10): e1000204.

30.

Kali

(2016) Reference management: A critical element of scientific writing. Journal of Advanced Pharmaceutical Technology & Research 7(1): 27–29.

31.

Kannan

Pokharel

Sasi Kumar

(2009) A hybrid approach using ISM and fuzzy TOPSIS for the selection of reverse logistics provider. Resources, Conservation and Recycling 54(1): 28–36.

32.

Kessler

Van Ullen

(2005) Citation generators: Generating bibliographies for the next generation. Journal of Academic Librarianship 31(4): 310–316.

33.

King

Hooper

Wood

(2011) Using bibliographic software to appraise and code data in educational systematic review research. Medical Teacher 33(9): 719–723.

34.

Kraker

Schlögl

Jack

. (2015) Visualization of co-readership patterns from an online reference management system. Journal of Informetrics 9(1): 169–182.

35.

Lorenzetti

Ghali

(2013) Reference management software for systematic reviews and meta-analyses: An exploration of usage and usability. BMC Medical Research Methodology 13(1): 141.

36.

McMinn

(2011) Library support of bibliographic management tools: A review. Reference Services Review 39(2): 278–302.

37.

McMullen

(2010) CiteULike. Public Services Quarterly 6(4): 368–369. DOI: 10.1080/15228951003772595

38.

Martin

(2009) Course instructor perceptions of computer-generated bibliographic citations. Reference Services Review 37(3): 304–312.

39.

Medaille

(2010) Mendeley: www.mendeley.com. Public Services Quarterly 6(4): 360–362.

40.

Meena

Thakkar

(2014) Development of balanced scorecard for healthcare using interpretive structural modeling and analytic network process. Journal of Advances in Management Research 11(3): 232–256. DOI: 10.1108/JAMR-12–2012–0051

41.

Nashelsky

Earley

(1991) Reference management software: Selection and uses. Library Software Review 10(3): 174–178.

42.

Norman

(2010) From Sci-Mate to Mendeley: A Brief History of Reference Managers. Available: http://occamstypewriter.org/trading-knowledge/2010/06/08/this_is_an_edited_version/ (accessed May 2016).

43.

Ovadia

(2011) Managing citations with cost-free tools. Behavioral & Social Sciences Librarian 30(2): 107–111.

44.

Ram

Paul Anbu

(2014) The use of bibliographic management software by Indian library and information science professionals. Reference Services Review 42(3): 499–513.

45.

Reichardt

(2010a) EndNote X3. Public Services Quarterly 6(4): 367–368. DOI: 10.1080/15228951003772579

46.

Reichardt

(2010b) Refworks. Public Services Quarterly 6(4): 366–367. DOI: 10.1080/15228951003772561

47.

Reiswig

(2010) Mendeley. Journal of the Medical Library Association (JMLA) 98(2): 193–194.

48.

Rempel

Mellinger

(2015) Bibliographic management tool adoption and use: A qualitative research study using the UTAUT model. Reference & User Services Quarterly 54(4): 43–53.

49.

Sahu

(2003) Accuracy of references in Indian Journal of Surgery: Comments. Indian Journal of Surgery 65: 290–291.

50.

Salem

Fehrmann

(2013) Bibliographic management software: A focus group study of the preferences and practices of undergraduate students. Public Services Quarterly 9(2): 110–120.

51.

Sohani

(2012) Developing interpretive structural model for quality framework in higher education: Indian context. Journal of Engineering Science & Management Education 5(2): 495–501.

52.

Steele

(2008) Bibliographic citation management software as a tool for building knowledge. Journal of Wound Ostomy & Continence Nursing 35(5): 463–468.

53.

Steeleworthy

Dewan

(2013) Web-based citation management systems: Which one is best? Partnership: The Canadian Journal of Library and Information Practice and Research 8(1).

54.

Technische Universität München Library (2015) Reference Management Software Comparison. 5th update. Available at: https://mediatum.ub.tum.de/doc/1274008/1274008.pdf (accessed 29 August 2016).

55.

Warfield

(1974) Developing interconnection matrices in structural modeling. Systems, Man and Cybernetics, IEEE Transactions on SMC 4(1): 81–87. DOI:10.1109/TSMC.1974.5408524

56.

Zhang

(2012) Comparison of select reference management tools. Medical Reference Services Quarterly 31(1): 45–60.