An Evaluation of Mobile Applications for Reproductive Endocrinology and Infertility Providers

Abstract

Objective: To identify and rate reproductive endocrinology and infertility (REI) mobile applications (apps) targeted toward REI providers. Design: A list of REI apps was found in both the Apple iTunes and Google Play stores using the following seven MeSH terms: reproductive endocrinology, REI, infertility, fertility, In Vitro Fertilization, IVF, and embryology. Patient-centered apps were excluded. The remaining apps were then evaluated for accuracy using reliable references. Setting: Mobile technology. Patients/Interventions: None. Main Outcome Measures: Accurate apps were evaluated for comprehensiveness (the extent of the ability to aid in clinical decision-making) and rated with objective and subjective components using the APPLICATIONS scoring system. Results: Using the seven REI-related MeSH terms, 985 apps and 1,194 apps were identified in the Apple iTunes and Google Play stores, respectively. Of these unique apps, only 20 remained after excluding patient-centered apps. Upon further review for applicability to REI specifically and content accuracy, only seven apps remained. These seven apps were then rated using the APPLICATIONS scoring system. Conclusion: Only 0.32% of 2,179 apps reviewed for this study were useful to REI providers. There is potential for further mobile resource development in the area of REI, given the limited number and varying comprehensiveness and quality of available apps.

Introduction

The advent of smartphone technology has created reliance on medical applications or “apps” as a resource for medical information. According to Cisco's most recent global mobile data traffic forecast in 2014, 88% of global mobile data traffic was considered “smart” traffic defined by advanced computing or multimedia capabilities and a minimum of 3G connectivity. That figure is expected to rise to 97% by 2019.¹ However, as with medical Web sites, the content and accuracy of these mobile apps must be analyzed since medical providers are now using apps to guide them in clinical decision-making.² One study of medical students, residents, and faculty showed that 77% used at least one medical app (medical reference, calculators, or program-specific apps) for information management regularly.³ Identifying high-quality and accurate mobile apps poses a challenge to healthcare providers given the plethora of apps available.⁴

A previous study by Farag et al. demonstrated that <15% of 1,800 Apple Operative System apps were considered potentially useful to obstetrician-gynecologists (OB-GYNs).⁵ To enhance filtering of irrelevant and inaccurate apps, the APPLICATIONS scoring system was created by Chyjek et al.,⁶ and while not formally validated, the scoring system has been used in multiple studies analyzing apps for OB-GYN specialties.^5

–9 To date, there have been no studies determining the availability, accuracy, and usefulness of apps for reproductive endocrinology and infertility (REI) providers. The aim of this study is to identify and evaluate apps targeted toward REI providers from both the Apple iTunes and Google Play stores using the previously developed APPLICATIONS scoring system.

Materials and Methods

This study was exempt from the Institutional Review Board as the research did not involve or pose any risk to human subjects. A complete list of REI apps was compiled and included apps from the previous study by Farag et al.⁵ Using similar methodology for finding apps, REI MeSH terms were identified and updated based on the 2014 list of MeSH terms, and these terms were searched in the Apple iTunes store twice on December 4, 2014 and June 18, 2015 and once in the Google Play store on July 20, 2015. The iTunes store was searched twice due to the 6-month time period between the first search and the second search. A second search was necessary given the rapidly changing environment of mobile apps. The following seven MeSH terms were searched: “reproductive endocrinology,” “REI,” “infertility,” “fertility,” “In Vitro Fertilization,” “IVF,” and “embryology.” Different iterations for each MeSH term were used to capture as many apps related to REI as possible. Any new apps were added to the existing list. The apps were then categorized based on the original classification system outlined by Farag et al.⁵ Apps containing patient-centered language and nonmedical jargon were classified as patient-centered apps and hence excluded. In addition, student-centered apps, apps advertising commercial products, non-English apps, private office or hospital advertisement apps, provider locator apps, gaming apps, conference apps, student-centered simulator apps, and other extraneous non-OB-GYN apps were excluded. Finally, any apps considered not specific to REI were excluded.

The remaining apps were then classified into the following subcategories: topic specific, journals, dictionaries, books, calculators, and sponsored educational apps. Journal apps were not rated as they all required a subscription and were electronic versions of the journal publication. Dictionary apps were also not rated as they included non-REI-specific terms. Books were excluded as they were the electronic version of the paper text. The rest of the apps, subsequently referred to as REI apps, were downloaded. Authors, K.L.S. and J.K., assessed the accuracy of the REI apps by checking the information and references cited within the apps to ensure that the information provided was correct. Inaccurate apps were excluded from further analysis.

Characteristic information about the accurate REI apps was collected. This included the download date, developer or seller, version, interplatform compatibility (iPhone, iPad, or both; Android tablet), and price. The popularity index was calculated from the product of the average app rating and number of reviews.^6,10 The APPLICATIONS scoring system was applied to these accurate REI apps, as shown in Table 1.⁶

Table 1.

The APPLICATIONS Scoring System

COMPONENT	SCORE	DESCRIPTION
App comprehensiveness	3	See Table 2
Price	1	0 = Priced, 1 = free
Paid subscription	1	0 = Required, 1 = not required
Literature used	1	0 = No references, 1 = references used
In-app purchase	1	0 = Present, 1 = absent
Connectivity	1	0 = Internet required, 1 = Internet not required
Advertisements	1	0 = Present, 1 = absent
Text search field	1	0 = No search field, 1 = search field present
Interplatform compatibility	1	0 = iOS device or Android Tablet, 1 = iOS device and Android Tablet
Other components
Images/figures	1	0 = Absent, 1 = present
Videos	1	0 = Absent, 1 = present
Special features	1	0 = Absent, 1 = present
Navigation ease	1	0 = Ease of navigation score <3, 1 = ease of navigation
	1	score ≥3
Subjective presentation		0 = Subjective presentation score < 3, 1 = subjective presentation ≥3
Total	16

App, application.

The app comprehensiveness score was determined based on the app's ability to aid in the clinical decision-making criteria outlined by Ventola and shown in Table 2.²

Table 2.

Application Comprehensiveness Criteria for Clinical Decision-Making

CLINICAL DECISION-MAKING CRITERIA	DESCRIPTION
Clinical decision support systems	Active knowledge system that uses two or more items of patient data to generate case-specific advice
Clinical treatment guidelines	Recommends how to treat a specific disease process
Disease diagnosis aids	Constellation of symptoms focused to one diagnosis
Differential diagnosis aids	Constellation of symptoms providing you with range of diagnoses
Medical calculators	Provides a numerical output with one or more inputs
Laboratory test ordering	Recommends laboratory tests for initial workup of a symptom(s)
Laboratory test interpretation	Clarifies laboratory test result
Medical examinations	Physical examination aid

A score of 0 was assigned if the app had no clinical decision-making capacity. A score of 1 was assigned if the app included 1–2 criteria. A score of 2 was assigned if the app included 3–5 criteria. A score of 3 was assigned if the app included 6–8 criteria.

Adapted from Ventola (2014).²

Navigation ease and subjective presentation were evaluated on a Likert scale with 1 = poor, 2 = below average, 3 = average, 4 = above average, and 5 = excellent. Authors' (K.L.S., S.F., K.C.M., and K.T.C.) scores were averaged. An average rating of <3 received no points and an average rating of 3 or greater received one point. These four authors also independently scored each app to determine if there was any interobserver variability in objective components and reconciled any discrepancies by discussion for an eventual 100% agreement. Objective and subjective components of each REI app were obtained between October 2, 2015 and December 2, 2015 from both the Apple iTunes and Google Play stores.

Results

Ten REI-related apps were identified from the initial search published by Farag et al.⁵ Of these apps, seven were excluded for the following reasons: one was no longer accessible on the Apple iTunes store, four were not specific to REI, one was a private office advertisement, and one was functional only with purchase of the company's software. Hence, three apps remained for analysis from the initial search. No unique apps in the Google Play store were found. The new search results of MeSH terms in the Apple iTunes and Google Play stores are shown in Table 3. Of the 2,179 total apps found, 2,169 apps were excluded due to the aforementioned exclusion criteria. Of the 10 remaining REI apps from the new search, 5 were not REI specific and hence excluded, and 1 app was excluded for inaccuracy. Hence, a final list of seven accurate apps for REI providers was available for rating between the original search by Farag et al. and the updated search in this study. Characteristic information and the APPLICATIONS scores are shown in Table 4.

Table 3.

Reproductive Endocrinology and Infertility MeSH Terms Searched in Apple iTunes and Google Play Stores

	APPLE iTUNES STORE
MeSH TERM	iPHONE	iPAD	GOOGLE PLAY STORE
Embryology	21	18	192
Fertility	387	179	250
Infertility	74	37	220
In vitro fertilization	5	1	134
IVF	60	30	127
REI	102	56	250
Reproductive endocrinology	8	7	21
Total apps found	657	328	1,194

REI, reproductive endocrinology and infertility.

Table 4.

Characteristics and APPLICATIONS Scores of Apps for Reproductive Endocrinology and Infertility Providers

								APPLICATIONS SCORE
APP NAME (DEVELOPER)	VERSION	iPHONE	iPAD	ANDROID TABLET	PRICE	POPULARITY INDEX	SUBCATEGORY	NAVIGATION	COMPREHENSIVENESS	PRICE	PAID SUBSCRIPTION	LITERATURE USED	IN-APP PURCHASES	CONNECTIVITY	ADVERTISEMENTS	TEXT SEARCH FIELD	INTERPLATFORM COMPATIBILITY	IMAGES/FIGURES	VIDEOS	SPECIAL FEATURES	PRESENTATION	TOTAL
Embryo (National Library of Medicine)	2.0	X	X		Free	0	Topic-specific information	1	0	1	1	1	1	1	0	1	0	1	0	1	1	10
IVF (Novel Approach)	3.3.2	X	X		$0.99	0	Topic-specific information	1	1	1	1	1	1	1	0	1	0	1	0	0	0	8
IVF4U—Dr. Marjorie Dixon (Purple Forge)	4.53	X	X		Free	1	Sponsored educational apps	1	0	1	0	1	1	1	0	1	0	1	1	1	1	12
Endo App (European Society of Human Reproduction and Embryology)	1.2	X	X	X	Free	0	Topic-specific information	2	0	1	1	1	1	1	0	1	1	1	1	0	1	14
Primo Vision Viewer (Cryo Mgmt Ltd.)	1.1.2		X		Free	0	Topic-specific information	0	0	1	1	1	0	1	1	1	0	1	0	0	1	7
iSaveFertility (Northwestern University)	1.0	X	X		Free	25	Sponsored educational apps	1	1	1	1	1	1	1	0	1	0	1	1	1	1	10
ReproductiveTech (Aleksey Kostin)	1.6	X	X		$9.99	0	Topic-specific information	2	1	1	0	1	1	1	0	1	1	1	1	1	0	8

The mean score on the APPLICATIONS scoring system was 9.8 with the lowest app scoring 7 points and the highest app receiving 14 points from a possible total of 16 points. The objective component reporting error rate was 8 out of 84 (9.5%), meaning that the authors independently assigned the same score to a given objective component of an app 90.5% of the time.

The seven apps for REI providers fell into the following two major categories: “topic-specific information” or “sponsored educational apps.” An app was classified as providing “topic-specific information” if it discussed an REI-related topic in detail. This category included the following apps: Embryo, IVF, Endo App, Primo Vision Viewer, and Reproductive Tech. The highest scoring apps in this category were Endo App and Embryo. Endo App, published by the European Society of Human Reproduction and Embryology (ESHRE), received the highest score (14 points) and incorporated treatment guidelines specific to endometriosis. The app was free for 30 days after which the app required ESHRE membership. Embryo also scored highly with 10 points and provided detailed videos of fertilization and each stage of ovulation, as well as three-dimensional imaged embryo data. The app that received the lowest score, Primo Vision Viewer, required a PIN code specific to the company's software and was unable to be analyzed.

“Sponsored educational apps” included IVF4U and iSaveFertility. The IVF4U app was sponsored by a private REI, Dr. Marjorie Dixon, and received the second highest score of 12 points. The app comprises links to information and videos from ASRM, as well as a news feed with the most up-to-date REI publications. iSaveFertility was a sponsored educational app by both The Hormone Foundation and The Oncofertility Consortium^®, providing REI providers with resources for discussions regarding fertility preservation options with cancer patients and their families. This app received 10 points for its useful “pocket guides” separated for both provider and patient and further subdivided into information specific to men, women, and children.

Discussion

In this study, only 0.32% of the 2,179 apps reviewed were considered useful to REI providers. Although this statistics is impressive, it is not surprising. In fact, the findings in this study are consistent with previous research in this field noting that a high proportion of apps were not applicable to OB-GYN providers.^5,7 Most of the apps in this study were excluded due to being patient centered rather than inaccuracy of content, which had been demonstrated in a previous study.⁵

The results of our study must be interpreted in the context of the study's limitations. First, we only searched the Apple iTunes and Google Play stores given their dominance in the market. Although searching all available platforms with our search terms may have decreased ascertainment bias, it was beyond the scope of the study. Second, our findings are limited to the apps that were available at the time that our search was performed and could be downloaded. Due to the fast pace of app development, there may be currently available apps that are not included in this study.

Healthcare providers may have difficulty choosing which apps to download based on the overwhelming number of apps available.¹¹ Our study shows that the popularity index does not correlate with the APPLICATIONS score as most of the apps had not received enough reviews to generate a rating within the app stores. Furthermore, app evaluation is limited by the lack of an accepted standard across medical specialties. The APPLICATIONS scoring system was developed in response to a review of the literature and is the only published systematic scoring system we found within the obstetrics and gynecology literature. Components of the system are adjusted to fit the type of apps being evaluated. For example, we chose the clinical decision-making criteria to include in the comprehensiveness component.

A committee of practitioners who lack conflict of interest to review apps pertinent to each discipline would increase the degree to which individual providers utilize apps in their practices. Increasing participation of physicians and other healthcare professionals in app development could improve the quality of apps available for providers. For example, two of the authors in this study assisted the American College of Obstetrics and Gynecology to reinvent the pregnancy wheel.¹²

Conclusion

In conclusion, we identified a very small number of apps that provide comprehensive, accurate user-friendly information, which can serve as a resource to REI providers. Periodic identification and evaluation of available apps are important to help update providers and advance mobile health technology in REI. Given the varying comprehensiveness and quality of available apps, a systematic evaluation of provider-centered apps is necessary before their use in clinical settings.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Cisco & Cisco. Cisco Visual Networking Index (VNI) mobile forecast projects nearly 10-fold global mobile data traffic growth over next five years. Available at http://newsroom.cisco.com/press-release-content?articleId=1578507 (last accessed December 22, 2015 ).

Ventola

. Mobile devices and apps for health care professionals: Uses and benefits. P T, 2014; 39:356–364.

Wallace

, Clark

, White

. “It's on my iPhone”: Attitudes to the use of mobile computing devices in medical education, a mixed methods study. BMJ Open, 2012; 2:e001099.

Cummings

, Borycki

, Roehrer

. Issues and considerations for healthcare consumers using mobile applications. Stud Health Technol Inform, 2013; 183:227–231.

Farag

, Chyjek

, Chen

. Identification of iPhone and iPad applications for obstetrics and gynecology providers. Obstet Gynecol, 2014; 124:941–945.

Chyjek

, Farag

, Chen

. Rating pregnancy wheel applications using the APPLICATIONS scoring system. Obstet Gynecol, 2015; 125:1478–1483.

Perry

, Lunde

, Chen

. An evaluation of contraception mobile applications for providers of family planning services. Contraception, 2016; 93:539–544.

Farag

, Fields

, Chyjek

, Pereira

, Chen

. Identification and rating of gynecologic oncology applications using the APPLICATIONS scoring system. Telemed J E Health, 2016 [Epub ahead of print]. doi: 10.1089/tmj.2016.0040.

Moglia

, Nguyen

, Chyjek

, Chen

, Castano

. Evaluation of smartphone menstrual cycle tracking applications using an adapted APPLICATIONS scoring system. Obstet Gynecol, 2016; 127:1153–1160.

10.

Tripp

, Hainey

, Liu

, Poulton

, Peek

, Kim

, et al. An emerging model of maternity care: Smartphone, midwife, doctor?. Women Birth, 2014; 27:64–67.

11.

Powell

, Landman

, Bates

. In search of a few good apps. JAMA, 2014; 311:1851–1852.

12.

ACOG Office of Communications. ACOG reinvents the pregnancy wheel: Launces new due date app. Available at www.acog.org/About-ACOG/News-Room/News-Releases/2016/ACOG-Reinvents-the-Pregnancy-Wheel (last accessed January 31, 2016 ).