Identification and Rating of Gynecologic Oncology Applications Using the APPLICATIONS Scoring System

Introduction

Mobile technology has become widespread in healthcare, aiding in both communication and gathering information. Owning a handheld device allows healthcare providers to quickly and effectively achieve a given task using an application (app). ¹ However, there are thousands of medical apps in the Apple iTunes and Google Play Stores. ² Navigating the overload of apps can be frustrating and time-consuming, often yielding nonuseful or inaccurate apps. In fact, we have previously shown that less than 15% of 1,800 iOS apps were considered potentially useful to obstetrician-gynecologists (ob-gyns). ³

To date, there have been no studies determining the availability, usefulness, and accuracy of gynecologic oncology (gyn-onc) apps. In 2011, Padey et al. identified a total of 77 oncology apps within the Apple iTunes Store, of which 55.8% provided scientifically validated data. ⁴ This study did not specifically name any of the apps found nor did it divide the apps found to different medical subspecialties. Hence, there is a need to help healthcare providers in finding useful and accurate gyn-onc apps. The aim of this study is to identify and rate gyn-onc apps from both the Apple iTunes and Google Play Stores, using our previously developed APPLICATIONS scoring system. ⁵

Materials and Methods

This study was exempt from the institutional review board as the research did not involve or pose any risk to human subjects. A complete list of gyn-onc apps was compiled to include apps that were found using gyn-onc MeSH terms from our previous study, ³ which was a search of the Apple iTunes Store only. In addition, a new search of gyn-onc MeSH terms based on the 2014 list was performed in the Apple iTunes Store on February 15, 2015 and Google Play Store on March 14, 2015. Different iterations for each MeSH term were also added to our search; for example, “uterine cancer,” “uterus cancer,” “endometrial cancer,” and “endometrium cancer.” Any new apps were added to our list. The apps were then categorized based on our original classification system. ³ Apps that used patient-centered language and nonmedical jargon were considered consumer apps by the authors and were excluded. The exclusion criteria from our original publication were used to remove apps that would not be useful to providers: student-centered apps, apps advertising commercial products, non-English apps, private office or hospital advertisement apps, provider locator apps, gaming apps, conference apps, student-centered simulator apps, and other extraneous apps. ³ In addition, any apps in our list that were no longer available on the Apple iTunes Store, non-gyn-onc specific, or breast cancer related were excluded.

The remaining apps were classified into the following subcategories: journals, dictionaries, books, calculators, risk assessments, cancer staging, guideline specific, and topic specific. Journal apps were not rated as they all required a subscription and were electronic versions of the journal publication. Dictionary apps were also not rated as they included non-gyn-onc specific terms. The rest of the apps, subsequently referred to as gyn-onc apps, were downloaded. Authors S.F., J.F., and E.P. assessed the accuracy of the gyn-onc apps; each of the three authors reviewed each app entirely and ensured that all of the information were correct based on several references. ^{6 –14} Any inaccuracies deemed the app inaccurate and excluded the app from further analysis.

Characteristic information about the accurate gyn-onc apps was collected. This included the download date, developer or seller, version, interdevice compatibility (iPhone, iPad, or both; Android phone, Android tablet, or both), and price. The popularity index was calculated from the product of the average app rating and number of reviews. ¹⁵ The APPLICATIONS scoring system was applied to these gyn-onc apps, as shown in Table 1. ⁵

The app comprehensiveness score was dependent on the purpose of the gyn-onc app. The apps were divided into the following types: calculator apps, screening apps, staging apps, or informational apps. Points were given based on the criteria developed as shown in Table 2.

Navigation ease and subjective presentation were evaluated on a Likert scale with 1 = poor, 2 = below average, 3 = average, 4 = above average, and 5 = excellent. Authors' (S.F., J.F., K.C., and K.T.C.) scores were averaged. An average rating of less than three received no points and an average rating of three or greater received one point. These four authors also independently scored each app using the APPLICATIONS score to determine if there was any interobserver variability. Objective and subjective measures of each gyn-onc app were obtained between May 11, 2015 and May 20, 2015 from both the Apple iTunes and Google Play Stores.

Results

Eighteen oncology-related apps were identified from our initial search, as published by Farag et al. ³ Of these apps, three were no longer available on the Apple iTunes Store, seven were not specific to gyn-onc, and two described non-American guidelines; these 12 apps were excluded. The new search results of MeSH terms in the Apple iTunes and Google Play Stores are shown in Table 3. Of the 748 total apps found, 736 apps were excluded due to the aforementioned exclusion criteria. Of the list of gyn-onc apps, five were excluded as being inaccurate. Two of the apps were in both the Apple iTunes and Google Play Stores. Hence, a final list of 11 nonconsumer accurate gyn-onc apps was available for rating (Fig. 1).

OPEN IN VIEWER

Fig. 1.

Flowchart of apps included from search in Apple iTunes and Google Play Stores.

Characteristic information and the APPLICATIONS scores are shown in Table 4. To account for interobserver differences, the APPLICATIONS scoring system was validated by having four of the authors (S.F., J.F., K.C., K.T.C.) individually rate each app. Eight apps were rated using Apple devices and five apps were rated using Android devices for a total of 13 individual scores, each including 10 objective criteria (App comprehensiveness, Price, Paid subscription, Literature used, In-app purchase, Connectivity, Advertisements, Text search field, Interdevice compatibility, and Other components). The four authors' scores were compared and six errors were found; hence, the objective reporting error rate was 6/130 (4.6%). In contrast, subjective measures (Navigation ease and Subjective presentation) naturally lend themselves to user bias and were averaged between reviewers.

Discussion

In this study, only 1.5% of the apps reviewed were determined to be both useful and accurate to gyn-onc providers. Furthermore, 5 of 16 nonconsumer gyn-onc apps were noted to be inaccurate, which is 31% of these apps. This high proportion of inaccurate apps is indicative of the nonreliability of currently published apps available for purchase. Without a systematic and stringent review process, providers can mistakenly download such inaccurate apps for usage, potentially hindering patient care and knowledge base.

Although the popularity index did not correlate with each app's APPLICATIONS score, the apps with the highest popularity indices were screening apps, specifically pap smear screening apps. Of these, the ASCCP app received the highest score and popularity index. This is likely due to the inclusion of both pap smear screening guidelines, as well as follow-up colposcopy management guidelines.

The app with the highest overall score was the Gynecologic cancers @point of care app. This app can be used as a reference tool regarding ovarian cancer and cervical cancer. However, it does not include any information about endometrial cancer, the most prevalent gyn cancer in the United States, nor does it include information regarding less prevalent cancers such as vaginal and vulvar cancer. The FIGO staging app contains a wider scope of information and includes information about all of the gyn cancers, providing useful information to both general ob-gyns and gyn-onc specialists. The FIGO staging app does not include a text search field and is priced, causing the app's final score to be less than Gynecologic cancers @point of care.

Apps have driven the increased usage of mobile devices in the medical field. ¹⁹ As we continue to identify and rate apps within different subspecialties of obstetrics and gynecology, our goal is to encourage the appropriate usage of mobile technology in the workplace. With medical information at each provider's fingertips, better clinical decision-making and improved patient outcomes can ensue. ²⁰ Healthcare providers use apps for information management, time management, health record maintenance and access, communications and consulting, reference and information gathering, clinical decision-making, patient monitoring, and medical education and training. ¹ Using our methodology for identifying and rating apps, we have found apps to aid specifically in reference and information gathering, clinical decision-making, and medical education and training; the other uses are beyond the scope of this study. In addition, our scoring system is of high quality as it fulfills the eight-question checklist described by BinDhim et al. ²¹

In this study, we limited our search to the Apple iTunes and Google Play Stores. About 89% of healthcare providers use these platforms, with a dominant proportion using iPhones versus Android phones (69% vs. 19.9%). ²² Of note, 8 of our 11 featured apps were found in the Apple iTunes Store and 5 of the 11 apps were found in the Google Play Store; only two of the apps were found in both stores. Searching all other available platforms could have decreased ascertainment bias, but was beyond the scope of this study.

Given the dynamic nature of the app stores, several factors limit our analysis. First, new apps are created and others are removed each day. This was evident at the start of our search when three apps were no longer available in the Apple iTunes Store from our prior publication. Second, we cannot be sure that all gyn-onc apps were captured in our extensive search in both the Apple iTunes and Google Play Stores. Even using third-party websites like 148 apps, apps hopper, and touch arcade to search app stores would present a new set of limitations such as noninclusion of potentially useful apps or delay of inclusion of apps on the host websites. ²³ Third, with the inherent lag-time that exists between app search, app review, and app scoring, disseminated information loses exigency with passing time. Finally, this study focused on gyn-onc specific apps; general oncology apps were excluded, as they were deemed too broad. Potentially useful information within these apps could be missed due to this exclusion.

Apps are useful for providing patient care and in helping with decision-making in this new era of mobile technology. ¹ We hope to make using such apps easy for ob-gyns by providing a list of useful accurate apps in each subspecialty of Obstetrics and Gynecology. In addition, we believe the APPLICATIONS scoring system is versatile enough to be used in other specialties to ease provider uncertainty about using apps.