Citation Patterns of Conference Proceedings in Master’s and Doctoral Studies: A Case Study of Information Technology and Systems

Abstract

This study reports on an analysis of the use of conference proceedings by postgraduate students at master’s and doctoral levels in the field of information technology and systems at the Cape Peninsula University of Technology (CPUT) over a period of 10 years. The analysis was based on master’s theses and doctoral dissertations submitted by the Department of Information Technology and deposited into the web-based institutional repository and involved analyzing the various references cited. The results of the study show that conference papers are the fourth most cited materials in theses and dissertations after journal articles, books, and online sources. The types of conferences cited relate to the subject under investigation in the theses or dissertation. The conferences proceedings cited are from various international and local conferences at which the students and lecturers from CPUT have or frequently presented. This study contributes to the knowledge on the trends and patterns of conference proceedings citations in theses and dissertations. This study also contributes to the knowledge on the use of conference proceedings by postgraduate students at master’s and doctoral levels in the information systems and computer sciences. It provides an analytical framework that can be used by library managers in the collection development processes in support of postgraduate students’ research information needs.

Keywords

computer science conference proceedings bibliometrics citation analysis dissertations information science theses social sciences

Introduction

The purpose of this study is to analyze how master’s and doctoral students in Information Systems at the Cape Peninsula University of Technology (CPUT) cite conference proceedings in their research. The final results of the assessment offer valuable insights into the patterns and trends of postgraduate use of conference proceedings and how this information can be used by library managers to improve their library collections in support of research activities at the university.

The presentation of research results at conferences has always been one important way of sharing and disseminating information and knowledge in some of the scientific and social science disciplines. Published online proceedings are additional sources for research information (Glänzel, Schlemmer, Schubert, & Thijs, 2006), especially in rapidly developing areas such as computer and information science. Shamir (2010, p. 1) emphasizes that “conference papers have traditionally been a quick form of research communication and an important source of information for scientists in addition to the standard journal papers.” Furthermore, Shamir (2010) points out that in some disciplines the common method of reporting scientific results is through peer-reviewed papers presented at conferences and published in conference proceedings.

Conference proceedings are however seen to be less relevant in most disciplines outside computer science and engineering (Lisee, Lariviere, & Archambault, 2008). The citation of conference proceedings is one of the bibliometric indicators that many researchers include in studies on citation patterns and trends in various scientific disciplines. The impact factor of journal articles is significantly higher than that of conference papers in computer science. The importance of conference proceedings can still be measured by the number of citations they receive (Michels & Fu, 2013).

Lisee et al. (2008) argue that although the importance of conference proceedings as a source of scientific information and knowledge has been expressed by many authors, not enough investigation has been done to verify their claims. The absence of “conference proceedings” as a separate category in citation indexes also presents a problem in the evaluation of their impact. The situation is further complicated by the conversion of some conference papers into journal articles that creates another complex dimension in their measurement. Lin, Tsai, and Ke (2014) observed that journal papers maybe of better quality in some subject areas because of the reviewing process that is much longer than for conference papers. In software engineering, for example, Montesi and Owen (2008) found that extending conference papers for publication in journals is a common way of carrying out research in the discipline. But still Lisee et al. (2008) found that in some fields, conference proceedings are considered the final product of scientific research; and in some fields they represent the source of the newest ideas and/or findings because of their high turnover rate in the reviewing process (Lin et al., 2014).

Master’s theses and doctoral dissertations are an important part of academic qualifications (Schopfel, Zendulkova, & Fatemi, 2014) making significant contributions to the body of knowledge in many fields. Today many theses and dissertations reside in open institutional repositories and can be easily and openly accessed through the Internet. And along with journal articles they are said to be the “most important content of open archives” (Schopfel et al., 2014). In South Africa, all the 23 higher education institutions maintain open electronic theses and dissertation (ETD) archives platforms, and at the national level, these are harvested through the National ETD Portal jointly maintained by the National Research Foundation (NRF) and Committee of Higher Education Libraries in South Africa (CHELSA). In the near future, they will form part of an envisioned national digital library open to all in the public and private sectors.

As part of the research process, master’s theses and doctoral dissertations in many subject areas make use of published scientific papers. Citation studies on these theses and dissertations show the citation trends and patterns of many other types of literature that are cited (Lisee et al., 2008). The importance of conferences is emphasized in many postgraduate programs where students are encouraged to present and share their preliminary and final research findings at various conferences, workshops, and seminars. This presents them with the opportunity to enter, grow, and network in the scientific community. In some institutions and programs, the presentation of preliminary research results for a master’s or a doctoral degree at a conference is a prerequisite for graduation.

In this study, we analyzed the citation patterns of conference proceedings in master’s theses and doctoral dissertations in the field of Information Systems at CPUT. The discipline of Information Systems is offered within the Department of Information Technology that specializes in the areas of applications development (software development), communication networks, and multimedia technology. The department also supports a large graduate and postgraduate program where research is directed at both community and industry interests. The Department of Information Technology is within the Faculty of Informatics and Design, one of the six faculties at CPUT.

Literature Review

Conferences are one of the means by which scientists share their research findings and get feedback from peers and thus are part of the process of scientific knowledge creation. They also allow scientists “to stay abreast of current research trends in their field and learn about cutting edge developments in the speciality” (Gonzalez-Albo & Bordons, 2011, p. 369). Conference literature is published in two common ways—first, as conference proceedings or second, conversion of presented papers into journal articles (Glänzel et al., 2006). In journals, they can appear as a normal article alongside other submissions or editors might decide to dedicate special issues to a particular conference. Kademani, Sagar, and Kumar (2009) stress the point that “conference papers are an invaluable source of knowledge for sciences and engineers as they provide a battle ground to argue, sharpen, modify and refine ones ideas” (p. 166). Papers published in leading conferences are said to likely generate a more significant citation impact in related fields (Ke, Lin, Tsai, & Hu, 2014). The challenge of some conference proceedings is that in many cases they are neglected as sources of information as they are not adequately covered in leading abstracting and indexing tools. Some of the leading indexing and abstracting tools like Scopus and Web of Science do not include conference proceedings as a specific category of cited materials, making the evaluation of their use difficult. However, if conference papers are cited in articles in leading journals, they also get valuable attention as cited literature sources.

The importance and usefulness of conference proceedings is however determined by the discipline, having more relevance in some and less in others. Several studies have looked at specific types of conferences to determine their impact, for example, in engineering. McMinn and Fleming (2011) tracked the use of engineering papers presented at Stapp Car Crash Conferences; Shamir (2010) reviewed the effect of conference proceedings on scholarly communication in computer science and engineering; Cocosila, Serenko, and Turel (2011) carried out a scientometric study on the identity and development of management information systems through three global conferences (International Conference on Information Systems [ICIS], Pacific Asia Conference on Information Systems [PACIS], and Administrative Sciences Association of Canada [ASAC] Annual Conference); and Montesi and Owen (2008) reported on how conference papers in software engineering are extended to journal articles, whereas Eckmann, Rocha, and Wainer (2012) studied the relationship between high-quality journals and conferences in computer vision. In most of these studies, results show the relevance and importance of the research presented at conferences and how the literature is cited in journals that are indexed in leading abstracting and indexing tools. Gonzalez-Albo and Bordons (2011) concluded that articles had more citations compared with proceedings papers due to a number of factors, especially as articles are more focused and represent a more complete and influential research than proceedings papers that have more authors and are dominated by the need of authors to attend conferences as coauthors. These conclusions are however not universal as they are based on an evaluation of one discipline.

Glänzel et al. (2006) point to proceedings literature as an additional important data source for bibliometric analysis. And more importantly Eckmann et al. (2012, p. 617) also emphasize that “in computer science as opposed to many disciplines, papers published in conference and workshop proceedings count as formal publications when evaluating the scholarship of an academic.” There are several studies covering the citation patterns in master’s theses and doctoral dissertations in many different disciplines that have been published (Abeyrathne, 2015; Condic, 2015; Keat & Kaur, 2008; Kushkowski & Parsons, 2003; Swanepoel, 2008). In most of the studies, the bibliometric indicators include various types of materials such as journal articles, monographs, Internet sources, gray literature, and conference proceedings and in some cases, materials categorized as “other” (such as standards, reports, working papers, and guides). The contribution and importance of conference proceedings as a source and means of communicating scientific information has also been covered in some research articles (Glänzel et al., 2006; Kademani et al., 2009; Shamir, 2010). Montolio, Dominguez-Sal, and Larriba-Pey (2013) have explored the endogamy of a large set of computer science conferences and journals and have observed a correlation between the quality of those conferences and the endogamy of their authors, that is, conferences where researchers collaborate with new peers having more quality than conferences in which researchers work in groups over a long time.

In information and computer sciences, according to Lisee et al. (2008), “proceedings play an important role, where they account for close to 20% of the references” (p. 1776). Eckmann et al. (2012) emphasizes that in computer science, researchers have several options of disseminating research and these include submitting papers to workshops, conferences, and journals. Workshops and conference presentations in the form of talks or posters are published and provide the ability of the researcher to present findings in person. Shamir (2010, p. 1) points out that “in the disciplines of computer science and engineering, a vast majority of peer-reviewed publications are in the form of conference proceedings, which have become a primary channel of research communication in these disciplines.” Furthermore, although Montesi and Owen (2008) emphasize the importance of papers presented at conferences in software engineering and computing science disciplines as being considered formal and counted when evaluating the research productivity of academic staff, they also point out that most presentations are extended for publication in academic journals.

Aims and Objectives

The focus of this study was on master’s theses and doctoral dissertations at the CPUT in the Department of Information Technology of the Faculty of Informatics and Design and was restricted to the citation of conference proceedings in these submissions. The aim was to highlight the importance of conference proceedings as important sources of literature in the field of Information Systems and to encourage future studies to investigate the importance of conference outputs in knowledge production at universities.

There are a number of important and potentially useful national and international conferences for South African master’s and doctoral students. High costs of travel obviously limit overseas travel, but local conferences provide an avenue for presenting their research outputs. In terms of citation of conference proceedings in their research projects, there are many access opportunities within universities as most universities subscribe to leading international full text databases and abstracting and indexing tools. At CPUT, the Faculty of Informatics and Design postgraduate students in information technology and systems are engaged in various research projects with a focus on industry and societal needs. The objectives of this study were therefore as follows

To establish the highly cited conferences in master’s theses and doctoral dissertations submitted to the faculty

To establish the conferences at which the highly cited proceeding were presented

The years in which the most cited conferences were held as well as published

Covers the 10-year period from 2005 to 2014.

Method

Data were collected from master’s theses and doctoral dissertations submitted to the Department of Information Technology in the Faculty of Informatics and Design and deposited with the CPUT’s institutional repository—Digital Knowledge. Although 55 masters and eight doctoral degrees were conferred from 2005 to 2014, only 28 master’s theses and seven doctoral dissertations have been uploaded into Digital Knowledge and were included in the study.

The references in each thesis and dissertation were checked and those identified as relating to conferences were extracted. The extracted data were uploaded and analyzed from spread sheets. In total, there were 374 conference citations. The extracted data included the conference name, year and country, as well as the type of proceeding (book, online, journal). For online proceedings, the type of access, for example, subscription-based, free, or restricted, was listed. The number of papers delivered by CPUT staff and students was recorded, and it was also noted whether the conference was hosted at CPUT.

Five methodological procedures were followed in this process:

All citations were checked against the Internet for accuracy of data.

Where data were incomplete or incorrect, the correct data were noted. This mainly pertained to the correct name of the conference as well as the year and place of the conference. The correct data were regarded as essential for the analysis.

A search was done for the proceedings of each cited conference to determine the format of the proceedings, for example, book, journal, CD, or online. It was also noted if access was free, restricted, based on subscription, or for purchase.

All conferences were checked against the available CPUT research reports to determine the conferences as which articles were delivered and those for which articles were included in proceedings.

The research report was also checked to determine the number of master’s and doctorate degrees conferred in the field information technology for each year.

Results and Discussion

Types of Materials Cited

In their submissions, and as shown in Figure 1, the 35 students cited a total of 5,644 references. A breakdown by the types of materials showed that with 2,113 references, journals had the highest occurrence of citations. Books (1,447 citations) and online sources (1,079 citations) were a distant second and third, while theses (103), government documents (152), and “Other” formats (299) had the least references. “Other” formats included a variety of materials types such as newspapers, meeting notes, and discussion papers. There were only 374 conference-related references (proceedings, papers, keynote addresses, presentations, and reports) representing 7% of the total citations. However, although with only 7% of the total citations, conference-related references are the fourth most cited material type.

Figure 1.

Breakdown of the types of items cited by M&D students from 2005 to 2014.

CPUT Libraries currently only has 61 conference proceedings listed in the catalog; of which, only 31 were published after 2000. Access to conference proceedings is, therefore, largely via database subscription.

Conference Citation Trends and Types

An analysis of the referencing of conference materials by master’s students showed a moderate average increase from 7.3 citations per student before 2010 to 10.5 citations per student after 2010. Only one master’s student (in 2005) recorded no references for conference materials. Citation of conference materials by doctorate students showed a slight decline with an average of 21 citations per student before 2010 compared with 16.8 citations per student after 2010.

According to Greenstein (2009), one of the key purposes of published conference papers is to share ideas and discuss recent research with others in the field in a timely manner. The study revealed that of the 374 citations, 38 were for papers presented by CPUT staff and students. Although 86% (Figure 2) of the citations were for conference proceedings, addresses, discussion papers, papers, presentations, and conference reports were also cited. In a few instances (eight or 2%), it was not possible to determine the format of the item cited.

Figure 2.

Types of conference citations.

It was also discovered that many of the conference proceedings cited related directly to the topic of the thesis or dissertation rather than general information technology. This was particularly notable in areas such as design, health, and education. In most cases, these conferences were only cited once. There were 122 such citations.

Forty-two conferences were cited multiple times, with the most cited conferences being the Conference of the Society for Imaging Science and Technology (IS&T) and Hawaii International Conference on System Sciences (HICSS), both with 23 citations, closely followed by the local series of the Conference on World Wide Web Applications (ZA-WWW) with 20 citations and the European Conference on Information Systems (ECIS) with 19 citations.

Figure 3 illustrates the most cited conferences.

Figure 3.

Most cited conferences.

The high figures for some of the conferences were due to the number of citations for conferences cited for specific years. The 2004 IS&T Conference was cited 11 times, making it the single most cited conference. Second was the ECIS held in Poland in 2002 with six citations. The European Conference on Information Technology Evaluation (ECITE) held in France in 2002 had five citations, and the 2002 HICSS with four citations completed the list for the most cited conferences.

The most cited South African conferences were the local series of the ZA-WWW with 20 citations followed by the IST–Africa Conference (IST-Africa; seven citations) and the Annual Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT; six citations). The ZA-WWW Conference is well supported by CPUT with many papers by CPUT staff and postgraduate students. Graduate students are encouraged to deliver poster presentations at the conference to demonstrate their research.

Other conferences that were often cited include the Americas Conference on Information Systems (AMCIS; 12 citations) and the ICIS (10 citations). Of all the conferences cited, four were hosted at CPUT and six were student conferences held locally and abroad.

Master’s and doctorate students and staff lecturing in the fields of information science presented a total of 47 papers at 35 conferences and had 94 papers published in 26 conference proceedings for the period 2007 to 2013. These conferences are both local (e.g., ZA-WWW, IST-Africa, and SAICSIT) as well as international (e.g., ECIS, IEEE Conference on Commerce and Enterprise Computing, and the International Federation for Information Processing [IFIP] Conference).

Michels and Fu (2013) indicate that, in some fields, conference proceedings can be seen either as a substitute for or as an alternative to journal articles. Initially conference proceedings were published as books (UNESCO & International Federation for Documentation, 1962), but more recently, they have also been published as journal titles. However, both of these forms of publication are quite slow. Some conferences use other forms of publication such as CDs and more recently online access to proceedings. Online access is currently the quickest method for making the presented papers available, either as individual papers or combined into proceedings.

Format of Conference Proceedings

Table 1 provides a breakdown of the format of proceedings cited.

Table 1.

Breakdown of Format of Proceedings.

Abstracts	1
Book	112
CD	5
Journal	47
Online	175
Summary	3
Unpublished	2
Unknown	29

The majority of citations are for proceedings or papers (175) found online, followed by those published as books (112). Of the items accessed online, 25 had restricted access and 48 were available via a CPUT subscription. The remaining 106 were available online free of charge. Selected papers forming part of conference proceedings of some conferences, such as the Information Resources Management Association (IRMA) International Conference, and published as books or journals are also freely available online. However, these numbers are fairly small.

Very few (40) conference proceedings or papers published prior to the year 2000 were cited. The oldest conference citation was for the ICIS that was held in the United States in 1981. Figure 4 indicates that the highest number of citations occurred during 2004 (49 citations) followed closely by the year 2002 with 43 citations. The reasons for these particularly high numbers are the 11 citations for the 2004 IS&T conference and the combined 15 citations for the 2002 ECITE, HCISS, and ECIS conferences.

Figure 4.

Number of citations per year of conference.

The country in which the conference was held was listed in each citation. A total of 52 countries were listed, and there were 13 citations for which a country could not be identified. Of the listed countries, 24 were listed only once. The United States of America had the most listings with 95 conferences cited followed by South Africa with 45 citations. The cited conferences were mainly from North America (140 citations) or Europe (115 citations). Of the European countries, conferences in Slovenia (12), France (11), the United Kingdom (11), and Italy (10) were cited more than 10 times. The North American conferences included 23 in Hawaii and 19 in Canada.

Outside South Africa with 45 citations, only Mozambique (4) and Turkey (3) within the Africa and Middle Eastern region had conferences that were cited more than once. Australia had 16 citations, followed by Japan (7), China (6), and New Zealand (4). A South American conference was cited only once and was held in Peru.

Conclusion

The citing of conference proceedings and papers relies on easy access to these items which CPUT Libraries provides mainly via subscriptions to conference database or via interlibrary loan services. Lecturing staff also make students aware of conferences by exposing them to conferences by including posters presentations during their undergraduate studies and by producing joint presentations with students during their postgraduate studies. Of the 38 cited papers by CPUT staff or students, only six were by a single author. While most conferences cited relate directly to the topic of the thesis or dissertation, there are selected conferences covering a variety of information technology and systems topics that are regularly cited. Conference proceedings, however, remain an important primary source of peer-reviewed literature in computer and information science literature, and the acceptance of papers at conference is in many cases considered prestigious and also highly important for promotion and tenure (Shamir, 2010). Based on these conclusions, it is important then that collection development processes at tertiary institutions continue to consider the value of online-based conference proceedings to enable students’ easy access, especially in the fields of information and computer sciences in which conference proceedings remain an important source of research literature.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Elisha R. T. Chiware

Author Biographies

Elisha R. T. Chiware is the director of libraries at Cape Peninsula University of Technology. He has previously worked as a lecturer and assistant librarian in Namibia, Zimbabwe, and Botswana.

Deborah Becker is the quality assurance and research support librarian at CPUT Libraries of Cape Peninsula University of Technology, Cape Town, South Africa. She has also held positions of branch librarian and systems librarian.

References

Abeyrathne

D. K.

(2015). Citation analysis of dissertations for collection development. Collection Building, 34(2), 30-40.

Cocosila

Serenko

Turel

(2011). Exploring the management information systems discipline: A scientometric study of ICIS, PACIS and ASAC. Scientometrics, 87, 1-16.

Condic

K. S

. (2015). Citation analysis of student dissertations and faculty publications in reading and educational leadership at Oakland University. Journal of Academic Librarianship, 41(5), 548-557. doi:10.1016/j.acalib.2015.07.007

Eckmann

Rocha

Wainer

(2012). Relationship between high-quality journals and conferences in computer vision. Scientometrics, 90, 617-630.

Glänzel

Schlemmer

Schubert

Thijs

(2006). Proceedings literature as additional source for bibliometric analysis. Scientometrics, 68, 457-473.

Gonzalez-Albo

Bordons

(2011). Articles vs. proceedings papers: Do they differ in research relevance and impact? A case study in Library and Information Science field. Journal of Informetrics, 5, 369-381.

Greenstein

(2009). IEEE transactions on wireless communications homepage: Conference vs. journal papers (Conference papers offered as journals). Retrieved from http://www.comsoc.org/twc/conference-vs-journal-papers

Kademani

B. S.

Sagar

Kumar

(2009, May 1-2). Conference papers of BARC scientists and engineers: A citation based study. Proceedings of the National Conference on Putting Knowledge to Work: Best Practices in Librarianship, Scientific Information Resource Division, Knowledge Management Group, Bhabha Atomic Research, Mumbai, India.

S. W.

Lin

W. C.

Tsai

C. F.

Y. H.

(2014). Citation impact analysis of research papers that appear in oral and poster sessions. Online Information Review, 38, 738-745.

10.

Keat

Y. C.

Kaur

(2008). Citation study of library and information science dissertations for collection development. Malaysian Journal of Library and Information Science, 13(2), 29-47.

11.

Kushkowski

J. D.

Parsons

K. A.

(2003). Master’s and doctoral thesis citations: Analysis and trends of a longitudinal study. Retrieved from http://works.bepress.com/jeffrey_kushkowski/4

12.

Lin

W. C.

Tsai

C. H.

S. W.

(2014). Correlation analysis for comparison of the citation impact of journals, magazines and conferences in computer science. Online Information Review, 39, 310-325.

13.

Lisee

Lariviere

Archambault

(2008). Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society of Information Science and Technology, 59, 1776-1784.

14.

McMinn

H. S.

Fleming

(2011). Tracking the use of engineering conference papers: Citation influence of the Stapp Car Crash Conference. Collection Building, 30(2), 76-85. doi:10.1108/01604951111127443

15.

Michels

(2013). Systematic analysis of coverage and usage of conference proceedings in web of science. Innovation Systems and Policy Analysis, 33, 1-56. doi:10.1007/211192014-1309-4

16.

Montesi

Owen

J. M.

(2008). From conference to journal publication: How conference papers in software engineering are extended for publication in journals. Journal of the American Society for Information Science and Technology, 59, 816-829.

17.

Montolio

S. L.

Dominguez-Sal

Larriba-Pey

J. L.

(2013). Research endogamy as an indicator of conference quality. SIGMOD Record, 42(2), 11-16.

18.

Schopfel

Zendulkova

Fatemi

(2014). Electronic theses and dissertations in CRIS. Procedia Computer Science, 33, 110-117.

19.

Shamir

(2010). The effect of conference proceedings on the scholarly communication in computer science and engineering. Scholarly and Research Communication, 1(2), 1-7.

20.

Swanepoel

A. J.

(2008). Citation analysis of theses and dissertations submitted at the Tshwane University of Technology. South African Journal of Higher Education, 22, 1097-1113.

21.

UNESCO & International Federation for Documentation. (1962). Scientific conference papers and proceedings: Contents, influence, value, availability. Retrieved from http://unesdoc.unesco.org/images/0002/000202/020249eo.pdf