Use of Google Scholar public profiles in orthopedics

Introduction

Almost all orthopedic surgeons would be aware of the classic academic imperative, “to publish or perish.” Research articles and other publications have always been an essential aspect of academic medicine and considered one of the keys to career advancement with respect to university appointments, tenure, and successful grant applications. Citation counts have long been regarded as an indirect measure of the quality of scholarly output, ^{1 –3} although critical review and careful evaluation clearly remain the best measures of a paper’s substance and value. ^{4 –10}

With the advent of computerized databases, the Internet, and web-based link analysis algorithms, basic information regarding academic contributions, including both the number of publications and citation counts, has now become very easy to acquire. ^{11 –19} Surprisingly, many orthopedic surgeons are unaware of the additional performance metrics that have been developed, are now routinely applied to their academic productivity, ^{4,7,13,20 –23} and that all this information is publically available. Most importantly, specific metrics such as the h-index are now commonly used when making decisions with regard to academic appointments and when awarding grants. ^{5,10,24 –31}

Although a minority of orthopedic surgeons are aware of the public availability of this information, fewer still recognize many of the potential implications. The most prudent course of action would be to monitor and closely manage this information, and thereby have some control over its presentation. However, the vast majority of orthopedic surgeons are largely unaware that this information is widely available and that potential employers, promotion committees, and grant adjudication panels may use this data as a critical discriminator.

Launched in 2004, Google Scholar ^{9,13,16,24,32,33} is a free online search platform that now rivals the more established commercial databases of PubMed, ³⁴ Scopus, ¹⁷ and the ISI Web of Knowledge. ¹⁸ Multiple authors have compared the search efficiency of Google Scholar to the established commercial databases ^11,15,35,36 and have shown that it is a user friendly, fast, and accurate research tool. ^{6,14,24,32,37} Google Scholar has now added two new features, Google Metrics and “My Citations.” ¹³ Primarily a tool to assess individual journals, Google Metrics tracks citations and impacts factors following user defined criteria. “My Citations” is a platform where individual researchers can create a unique profile, track their publications and citation counts, link themselves to co-authors, and receive notifications of papers relevant to their field of interest. Citation analysis is also available in the form of several indices, including the h-index. ¹³ Researchers can declare multiple fields of interest and choose for their profile to be private or public.

The use of Internet search engines continues to increase globally; investigating the pattern of use by orthopedic surgeons can provide valuable insight into how this is being currently used, and trends may indicate what aspects of this use will become more prominent in the near future. The primary purpose of this study is to investigate the growth of Google Scholar profiles in orthopedics over a period of 12 months, as represented by the number of individuals creating a public profile. The secondary purpose was to provide better understanding of how this specific website is being used by various groups globally, when stratified with respect to gender, nationality, and research interests.

Materials and methods

The investigation was conducted as an observational study, and data was prospectively acquired over the course of one year from June 2013 to June 2014. As an extramural study, no institutional review board or ethics approval has been sought; all data obtained is publically available through the Internet. Google Scholar (www.scholar.google.com) was accessed on five separate occasions (June 24, 2013, September 22, 2013, December 21, 2013, March 20, 2014, and June 24, 2014). This constitutes a search conducted every 90 days; the fifth and final audit was performed after 95 days so as to complete one full year. After connecting with Google Scholar, navigation to the individual profiles was achieved through the “My Profile” tab on the menu bar. Entering a query in the search box then provides access to a full listing of all (public access) individual profiles that correspond with the specific query. To identify as many individuals associated with orthopedics as possible, the search was performed serially on each date with six related queries: “orthopedics,” “orthopedic surgery,” and “orthopedic surgeon,” as well as the accepted alternative spellings including “orthopaedics,” “orthopaedic surgery,” and “orthopaedic surgeon.” This would not be an exhaustive search of every possible alternative, but these acted as very consistent search parameters that almost certainly capture the vast majority of those public profiles associated with the specialty of orthopedics and orthopedic surgery, including both clinicians and basic scientists.

These results were recorded on each of the five intervals, and duplicates obtained in the multiple queries were identified and removed; the remaining unique individual public profiles were then ranked by total citation count (in descending order). The individual profiles were further allocated into six different geographic regions: United States, United Kingdom, Europe, Asia-Pacific, Australia, and Other. Additional variables were recorded where available including gender, nationality (current work location), and whether an individual was either a clinician (orthopedic surgeon) or a basic scientist. This final distinction was based on both the areas of interest identified by the users, as well as review of their lists of publications. Gender was not always clear based on the profile data and where ambiguous remained unassigned.

Descriptive statistics were determined for all continuous variables; categorical variables are presented where applicable as percentages and frequencies. The overall rate of increase in the number of participating individuals was determined, and projections of growth were calculated and displayed in a graph using exponential smoothing. To determine differences between the growth of Google Scholar public profile registrations and citation counts, repeated measures of analysis of variance (RMANOVA) were used. Bonferroni corrections were performed for multiple comparisons, and a Kolmogorov–Smirnov test was used to determine the normality of the data. Similarly, the differences in regional growth were analyzed using RMANOVA. Multiple two-way contingency tables were created, and a two-tailed Pearson χ ² test with Yates corrections was then used to compare gender to clinical and basic science, clinical and basic science to the regions United States/Canada and rest of world, citation count to the regions United States/Canada and rest of world, and initial and final total number of Google Scholar public profiles to the regions United States/Canada and rest of world. All statistical analysis was performed with Systat (version 13; Systat, Chicago, IL, USA). A p-level of <0.05 was considered significant.

Results

Statistical analysis confirmed rapid growth (p = 0.001) over the observed 12-month interval. RMANOVA revealed statistically significant differences (F = 119, p = 0.0001) for regional growth within each region (Table 1). The largest growth was observed in the United Kingdom (F = 22.8, p = 0.009, 289%), followed by the Asia-Pacific region (F = 36.3, p = 0.004, 177%), “other” (F = 27.8, p = 0.006, 172%), Australia (F = 40, p = 0.003, 129%), Europe (F = 83.2, p = 0.001, 74%), and the United States (F = 108, p = 0.0001, 68%). Significant between-country growth differences were observed for United States–United Kingdom (p = 0.002), United States–Europe (p = 0.005), United States–Asia-Pacific (p = 0.004), United States–Australia (p = 0.006), United States–Other (p = 0.001), United Kingdom–Europe (p = 0.003), Europe–Asia-Pacific (p = 0.022), Europe–Australia (p = 0.011), and Europe–Other (p = 0.005).

OPEN IN VIEWER

Table 1.

Gender distribution and regional growth of Google Scholar individual public profiles.

Date	Gender			Region
	M	F	Uncertain	United States	United Kingdom	Europe	Asia-Pacific	Australia	Other
June 24, 2013	161	16	0	78	9	43	22	7	18
September 22, 2013	193	18	1	86	15	50	33	8	20
December 21, 2013	230	22	1	96	22	54	45	9	27
March 20, 2014	277	29	1	116	26	74	56	10	35
June 24, 2014	331	36	0	131	35	75	61	16	49

The mean growth per 90-day interval is 19.9% (within-group RMANOVA F = 73.2, p = 0.003) and the mean 12-month growth is 107% (within-group RMANOVA p = 0.05; Table 2). The growth curve with exponential smoothing forecasts a pattern of exponential growth (Figure 1). At this rate, the number of researchers and authors associated with orthopedics who create a public profile on Google Scholar will double approximately every 10.6 months. If this rate of increase was to continue unabated, this naïve model predicts that by November 2018 over 10,000 orthopedic surgeons will have created a public profile on Google Scholar.

OPEN IN VIEWER

Table 2.

Total growth of Google Scholar orthopedic public profiles, median citation count, and number of citations divided into >10.000, 10,000–1,000, 999–100, 99–10, and <10 citations per author.

Date		Citation count					Median
	Total	>10,000	10,000–1000	999–100	99–10	<10
June 24, 2013	177	3	43	74	37	20	259
September 22, 2013	212	4	48	97	38	25	258
December 21, 2013	253	4	57	114	50	28	215
March 20, 2014	307	4	71	131	65	36	222
June 24, 2014	367	7	82	153	84	41	214

OPEN IN VIEWER

Figure 1.

The growth curve with exponential smoothing forecasts a pattern of exponential growth. At this rate, the number of researchers and authors associated with orthopedics who create a public profile on Google Scholar will double approximately every 10.6 months.

RMANOVA revealed statistically significant growth (F = 72.9, p = 0.0001) for citation count within each and between intervals: >10.000: p = 0.001; >1000: p = 0.003; >100: p = 0.003; >10: p = 0.003, <10: p = 0.004. While the number of researchers with a citation count of more than 10,000 was low (n = 3), it increased by 133%, followed by researchers with citation counts of >10 (128%), <10 (106%), >100 (105%), and >1000 (91%) (Table 2). The median citation value has also consistently decreased over the year observed (Table 2). Both these findings suggest that researchers with fewer citations are more frequently registering a public profile on Google Scholar.

There was significant within and between growth for male and female gender growth (F = 65.3, p = 0.001; Table 1). Significant within-region growth was observed for all regions (F = 71.2, p = 0.0001). Significant growth between regions was observed for all regions (F = 119.1, p = 0.0001) with the exception of the following comparisons: Australia–United Kingdom (p = 0.31), United Kingdom–Other (p = 0.1), Asia-Pacific–Australia (p = 0.074), Asia-Pacific–Other (p = 0.14), and Australia–Other (p = 0.13) (Table 1).

Contingency table analysis revealed statistical differences between gender (male vs. female) and basic and clinical sciences (χ ² = 25.9, p = 0.0001) (Table 3); between regions (United States/Canada vs. all other countries) and basic and clinical sciences (χ ² = 24.9, p = 0.0001) (Table 4); and regional growth between regions (United States/Canada vs. all other countries) (χ ² = 7.2, p = 0.03) (Table 5). There were significant differences for citation count (>1000 vs. <1000) and regions (United States/Canada vs. all other countries) (χ ² = 22.1, p = 0.0001) (Table 6).

OPEN IN VIEWER

Table 3.

Final gender distribution between clinical and basic science, June 2014.

	Male	Female	Uncertain	Total
Clinical	236 (71.3%)	10 (27.8%)	0	246 (67%)
Basic science	95 (28.7%)	26 (72.2%)	0	121 (33%)
Total	331	36	0	367

OPEN IN VIEWER

Table 4.

Regional distribution between basic science and clinical, June 2014.

	Basic science	Clinical	Total
United States/Canada	68 (50%)	68 (50%)	136
All other countries	55 (23.8%)	176 (76.2%)	231
Total	123	244	367

OPEN IN VIEWER

Table 5.

Regional growth of Google Scholar participation from June 2013 through June 2014.

	United States/Canada	All other countries	Total
Initial June 2013	80	97	177
Final June 2014	136	231	367
Growth over 12 months	56 (70% increase)	134 (138% increase)	190 (107% increase)

OPEN IN VIEWER

Table 6.

Citation counts comparing United States/Canada and all other countries, June 2014.

	United States/Canada	All other countries	Total
>1000	52 (38.2%)	37 (16.0%)	89 (24.3%)
<1000	84 (61.8%)	194 (84.0%)	278 (75.7%)
Total	136	231	367

Discussion

The most significant finding of this study is that, over the course of the year examined, Google Scholar public profiles grew in number at an exponential rate. At this current rate, participation within orthopedics is doubling every 10.6 months. This growth is significantly related to increasing popularity with users in the United Kingdom, India, and the Middle East. Participation is increasing twice as fast in the rest of the world as it is in the United States and Canada. Growth is also occurring through greater participation by presumably younger authors and researchers, with fewer publications and citations. There is a significant difference in use when stratified by gender, and women are more likely to be involved in basic science research while men are more likely clinically active as orthopedic surgeons. This almost certainly reflects established differences in the distribution of women within our profession. ^{38 –40}

The published academic contributions of an individual surgeon have always been in the public domain; ^{1 –3} however, over the course of the past decade, this information has become widely available, to virtually anyone, anywhere. ^16,24,37,32 Internet search engines and web-based link analysis algorithms now provide immediate public access in unprecedented detail. ^13,14 This allows potential employers, promotion committees, and grant adjudication panels a method to directly compare candidates for a position, tenure, or funding. Google Scholar public profiles provide a very convenient means for an individual orthopedic surgeon to monitor, and manage, at least some aspects of this information. ^13,14

It is widely recognized that certain articles are of greater interest than others, especially those closely related to our own areas of subspecialty focus. But is there a method to determine the intrinsic value of a particular article? For scholarly publications, citation counts generally serve as a well-defined and objective measure of a paper’s relevance and visibility and indirectly indicate the quality of a given paper. ^7,22 Determining the number of publications, as well as their citation counts, therefore provides a convenient measure of not only the quantity of an individual’s scholarly output, but also the quality.

Bibliometric measures and various indices based on citation counts have been developed specifically to assess the quantity and quality of the academic contributions of particular journals, groups, or individuals. ^{15,16,19,41 –44} When a body of papers is considered together, we can then use the combined citation counts for the individual papers as a more general measure of their overall impact. Commonly done for a particular journal over a period of one or more years, ^{1 –3,8,20,21,45} the concept can also be applied to the collected works of one author over a career.

Citation analysis then potentially allows the academic standing of one orthopedic surgeon to be compared to another, recognizing certain caveats. ^{8,9,14,26,28,30,43,44,46 –53} The more important questions then become, should these indices or citation counts be considered when selecting candidates for academic appointments? How should committees deciding on promotions, tenure, or grants use this information? This has been a topic of considerable debate among academics for several years and this interest is now spreading to the medical community, ^{6,10,12,24,29,31,42,44 –47,49,54} if belatedly to orthopedics. ^{4,7,20 –23}

Developed by Hirsch in 2005 ⁵⁰ , the h-index ^{5,10,24 –28,31} is a measure of an individual’s research standing considering both quantity (the number of publications) and quality (the number of citations); it is now widely accepted as a robust measure of scientific impact. ^6,14 It can be applied to an individual, a group, department, or discipline. A value of “h” means that the individual or group has published at least “h” papers, each of which have been cited at least “h” times; an h-index of 20 means that there have been 20 papers published which have each been cited at least 20 times. The value of the h-index thus relies on accurately identifying the number of publications and the number of citations for an individual author. ^6,50

Historically, ^{1 –3} there were two main sources of bibliometric data: Thomson Reuters Web of Science ¹⁸ and Scopus from Elsevier. ¹⁷ Both databases, while extensive, have several limitations that have been widely discussed in the literature. Publications in languages other than English are underrepresented in these databases, and they often fail to capture book chapters, and the abstracts of conference proceedings. ⁶ Furthermore, access to both Web of Science and Scopus is usually limited to large institutions due to significant subscription fees. Google Scholar is a search engine rather than a database and will therefore detect non-journal references and citations, thus affecting the calculated h-index. ^{15,16,24,32,33,37} Several studies have compared the h-index value across the Web of Science, Scopus, and Google Scholar; given its increased rate of capture of publications and citations, the h-index from Google Scholar has been shown to be larger than that determined by either Web of Science or Scopus. ^{6,10,11,15,26,32,33,35 –37,47,54}

The principal limitation of this study is that it has not been independently verified. This data has been accumulated using the search queries specified and very accurately reflects the content of the Google Scholar website over the period examined. It is possible that the citation counts of an individual have been artificially inflated or otherwise manipulated or that an individual may even be entirely fictitious; however, we regard this second possibility as highly unlikely. Gender was determined based on the first name displayed in the profile, or their general appearance in an optional photograph; where ambiguous, gender remained unassigned. Areas of interest (either clinical or basic science) were determined based on review of the associated publications; there are no formal criteria available to make this distinction objectively. Some of the public profiles identified in this study used a primary identifier of “orthopedics” or one of the other specified search criteria but further include an additional label of veterinarian, orthodontics, dental engineering, physiotherapist, robotics, or other qualifiers. The use of multiple labels to identify oneself within the same subspecialty can potentially create difficulties when attempting within group comparisons. ²⁴

The h-index and its calculation are becoming more relevant to the everyday clinician and researcher alike. Increasingly, indices such as the h-index are being used as an adjunct when evaluating applications for employment, research grants, or tenure. These indices clearly have the potential to affect critical decisions that may influence the direction our careers may take. It is therefore important for orthopedic surgeons to understand what data is currently readily available publically and to better appreciate how this information is now being quantified and assessed. ⁶

Conclusion

These results demonstrate authors in the field of orthopedic surgery are very rapidly creating a Google Scholar public profile, and this is currently increasing exponentially. If this current rate of growth continues, participation is expected to double every 10.6 months. Although potentially reaching a total of 10.000 orthopedic authors with a Google Scholar public profile by November 2018, it is unlikely that growth will continue at this rate indefinitely. Women are more likely to report they are involved in basic science research, while men are more likely to report they are clinically active as orthopedic surgeons. Much of this growth reflects greater participation by authors based in the United Kingdom, India, and the Middle East. The median number of citations for each author is gradually decreasing, and this suggests the growth observed most likely reflects greater participation from younger, less experienced authors in orthopedics.