Abstract
Abstract
Background:
Fast Facts Mobile (FFM) was created to be a convenient way for clinicians to access the Fast Facts and Concepts database of palliative care articles on a smartphone or tablet device. We analyzed usage patterns of FFM through an integrated analytics platform on the mobile versions of the FFM application.
Objective:
The primary objective of this study was to evaluate the usage data from FFM as a way to better understand user behavior for FFM as a palliative care educational tool.
Design:
This is an exploratory, retrospective analysis of de-identified analytics data collected through the iOS and Android versions of FFM captured from November 2015 to November 2016.
Measurements:
FFM App download statistics from November 1, 2015, to November 1, 2016, were accessed from the Apple and Google development websites. Further FFM session data were obtained from the analytics platform built into FFM.
Results:
FFM was downloaded 9409 times over the year with 201,383 articles accessed. The most searched-for terms in FFM include the following: nausea, methadone, and delirium. We compared frequent users of FFM to infrequent users of FFM and found that 13% of all users comprise 66% of all activity in the application.
Conclusions:
Demand for useful and scalable tools for both primary palliative care and specialty palliative care will likely continue to grow. Understanding the usage patterns for FFM has the potential to inform the development of future versions of Fast Facts. Further studies of mobile palliative care educational tools will be needed to further define the impact of these educational tools.
Background
W
Attitudinal barriers such as the “confidence-competency gap”—when clinicians have an overinflated sense of confidence about their base knowledge and clinical skill-set—have been recognized as impediments to improving generalists' knowledge in palliative care.7,8 To better overcome these attitudinal barriers, various types of concise palliative care educational materials have been designed, so that generalist clinicians can access these materials in the moment, to address a targeted clinical problem that the clinician is encountering. 9
One example of such an educational tool is fast facts and concepts (FFC), a collection of concise, peer-reviewed, evidence-based summaries on key palliative care topics regarding symptom management, pharmacology, communication, prognosis, and ethical issues in HPM, designed with the principles of “just-in-time” education in mind. 8
For over 15 years, FFC has been available as a website-based platform and been a widely utilized resource by clinicians, trainees, and educators throughout the world. New FFC articles are created by the palliative care community, peer reviewed by the FFC editorial board, and published both online and, since 2008, in the Journal of Palliative Medicine.
On May 1, 2014, a mobile application version of FFC was released on the Apple App Store called fast facts mobile (FFM). An Android version of the application was subsequently released on Jan 31, 2016. Since their release, the iOS and Android versions of the FFC reference have been downloaded over forty thousand times worldwide with over 2000 active users per month.
Little is known about how FFM is used. To better understand how and if the FFM app was meeting the “just-in-time” educational needs of clinicians, we analyzed usage patterns through an integrated analytics platform on the mobile versions of the FFM app.
Objectives
The primary objective of this study was to evaluate the usage data from FFM as a way to better understand user behavior and gain insights on the user population for FFM as a palliative care educational tool. A secondary objective was to examine search terms, articles accessed, articles shared, and articles bookmarked of the most frequent users of FFM and compared this to infrequent users in the hopes of informing and improving future versions of FFM.
Design
This is an exploratory, retrospective analysis of de-identified analytics data collected through the iOS and Android versions of FFM captured from November 2015 to November 2016. This study was reviewed and exempt by the institutional review board at Partners Healthcare.
Methods
Analytics platform
An analytics platform (Amplitude Analytics, San Francisco, CA) was integrated with version 1.0.3 of the FFM iOS application and version 1.0 of the FFM Android application. On implementation of the analytics platform, a notice was posted on the description on the application pages in both the Apple App Store and the Google Play Store. This disclaimer/terms of use was also placed in the “About Fast Facts” section of FFM.
The analytics platform logs every time a user accesses FFM and the action performed, such as when an article is opened, what terms are searched for, and duration of time of a session. All logged items are defined as events in the platform. All analytics events were implemented as per the Amplitude analytic platform instructions.
Event-based and user metadata, including device type and general location data (city, region, and country), were sent encrypted to the Amplitude web server. The raw data from November 1, 2015, to November 1, 2016, were exported from Amplitude as JSON files, de-identified, and converted to a comma separated value (.csv) file. All statistical analysis and reporting were performed using Python version 2.7.10 (Python Foundation, Wilmington, DE) and Microsoft Excel, version 1702 (Microsoft Corporation, Seattle, WA). All user analysis data were limited to users in the United States.
Data collection
FFM App download statistics, including number of downloads, platform, and territory, from November 1, 2015, to November 1, 2016, were accessed from the Apple and Google development websites. Lifetime downloads denote downloads from initial release of the application to 11/1/2016. A total of 18,700 downloads were omitted from the lifetime download statistics results due to an Apple iOS software update that caused a duplicate FFM download to occur for all existing users on iOS.
We also investigated the temporal relationship of FFM usage by evaluating average total number of events per day of the week. Further characterization of FFM user behavior was performed by describing the event frequency of articles being opened, shared, bookmarked, and searched.
Characteristics of user behavior
A session was defined as one full event starting from when the user opened the FFM application to when the user closed out of FFM. Events were defined as a single interaction within FFM. Examples of events include opening an article, sharing an article, and searching for a topic. Bar charts showing the distribution of users based on total number of sessions along with total number of events (Fig. 1) were created using Microsoft Excel, version 1702 (Microsoft Corporation, Seattle, WA).
Comparison of Overall Users of Fast Facts to Total App Events Segmented into Infrequent Users and Frequent Users.
Frequent and infrequent user analysis
Frequent Users of FFM were defined as users who logged ten or more sessions on the FFM application during this study period. All users who had less than ten sessions were defined as Infrequent Users. Less than ten sessions was used as the cut point between Frequent Users and Infrequent Users based on a sharp decline in volume of users with ten or greater sessions (Fig. 1).
Article access analysis between Frequent and Infrequent Users was performed. The top ten most accessed articles were sorted by Frequent and Infrequent users. The estimated number of times an article was accessed per user was calculated for both Frequent and Infrequent users. This was done by taking the total number of article open events for both populations and dividing by the respective total number of users in each group. This was defined as Mean Article Access Per User for both the Frequent User and Infrequent User cohorts.
Search term analysis was performed on the top ten most searched-for terms for both Frequent and Infrequent Users. Each search term included all instances where a user searched for the word along with all letter prefixes of the search word greater than three letters, determined by constructing a prefix tree of all search terms.
The estimated times a search term was searched per user was calculated for both Frequent and Infrequent users by taking the total number of search events for both populations and dividing by the total number of users in each group. This was defined as Mean Searches Per User for both the Frequent User and Infrequent User groups.
User information retrieval analysis
We performed an information retrieval analysis to evaluate how well the search function of FFM was aligned with user article viewing behavior. Information retrieval analysis was performed on the top five search terms by linking the search term from all users to the most accessed article based on user logs. In addition, access of articles mapped to its search term was then ranked based on popularity to the top five most accessed articles per search term.
The most accessed article for each query was cross-referenced and coded as a Search Rank defined as the order of article appearance after performing a search of the the full search term. The current top article for each search term was also logged and cross-referenced as the Accessed Article Rank representing the popularity of the article based on user access. If the current article did not appear as one of the most accessed articles based on search term, it was denoted as Not Applicable (NA).
Results
Characteristics of user behavior
A total of 9409 downloads of FFM occurred during the study period. The majority of downloads, 6780 (72%), occurred on iOS devices. Of the iOS downloads, 6120 (90%) utilized the application on an iPhone. Overall, including both iOS and Android platforms, 8583 (91%) downloads of FFM were on smartphone devices with the remainder on tablets. There were 9186 distinct users during the study period. Most users of FFM (98%) utilize FFM on one device only.
The average number of total events throughout the week showed the highest use on Wednesdays with an average of 35,255 events. There is an average of 32,914 events on the weekdays. On weekends, the average events decreased to 18,405, representing a 44% decrease in events on weekends compared to weekdays.
The most utilized function in FFM was opening articles with 201,383 article open events over the study period. Outside of opening articles, the second most utilized function in FFM was the Search function with 68,865 events. The sharing function of FFM was the least utilized function by users with 4008 events. Article bookmarking was slightly more utilized (6728) compared to sharing.
The top 20 most opened articles in FFM accounted for 78,013 article open events, representing thirty nine percent of all article open events during the study period. During that same timeframe, a total of 201,383 article open events occurred throughout the FFM user base. The most searched-for terms in FFM included Nausea (1553 total events), Methadone (1502 events), Delirium (926), Opioids (872), Prognosis (865), Constipation (855), Dyspnea (849), Dementia (643), Bone (632), and Hiccups (592).
Frequent and infrequent user analysis
Of the 9186 users, 1560 were Frequent Users defined as users who opened the FFM application ten times or more. User behavior of Frequent Users compared to Infrequent Users (users who opened the FFM app less than ten times) showed increases in rates of article opening, search, bookmarking, and sharing. Frequent Users, who comprise 13% of all users of FFM, accounted for 66% of all events during the study period (Fig. 1).
Overall, Frequent Users accessed and searched for articles on FFM more often than Infrequent Users. Frequent Users accessed articles 4.2 times more often than Infrequent Users and utilized the search function 5.6 times more frequently. Frequent Users shared and bookmarked articles 3.4 and 3.7 times more than Infrequent Users.
Frequent Users showed greater usage of search in all of the most searched-for terms. Both Frequent Users and Infrequent Users searched for the same terms in the same rank order.
On average, Frequent Users searched for the top ten Fast Facts articles 4.7 times more often than Infrequent Users. Search terms with the highest ratio of Frequent User searches to Infrequent User searches (ratio >5) included “Nausea,” “Delirium,” “Constipation,” “Dementia,” and “Hiccups.” (Table 1).
The top ten most accessed articles on FFM showed consistently higher access by Frequent Users for all articles. The most accessed articles for both Frequent Users and Infrequent Users were largely the same with some differences in rank order. The “Diagnosis and Treatment of Terminal Delirium” article was the most accessed article for both Frequent Users and Infrequent Users. “Dementia prognosis” and “Delirium: Drug Therapy” articles uniquely appear in the top ten most accessed articles list for Frequent Users, while “Morphine and Hastened Death” and “Tube Feed or not Tube Feed” appear only on the Infrequent Users list (Table 2a).
Mean Article Access Per User of the ten most frequently accessed articles show that Frequent Users access these articles more often than Infrequent Users. Articles with the highest ratio of Average per User Article Access Frequent Users to Infrequent Users (>5) include “Bowel Obstruction: Medical Management,” “Palliative Performance Scale,” “Delirium: Drug Therapy,” “Methadone Basics,” “Steroids for Bone Pain,” and “Methadone Starting Doses” (Table 2b).
Information retrieval of top five articles
User information retrieval analysis was performed to better understand how well the search function of FFM aligned with what articles users most access for viewing. Of the top five searched-for terms, only one article (13: Prognostication in Cancer) accessed by a search term (“Prognosis”) displayed as the number one article by search rank and accessed article rank (Table 3). One article (36: Opioid Conversion Calculations, search term: “Opioids”) appeared at a search rank of 22. Two of the top articles (161: Screening for ICU Delirium and 295: Opioid Induced Constipation) did not appear in the top five most accessed articles based on their respective search terms. The remaining two articles appeared as accessed article ranked 2 (25: Opioids and Nausea) and 3 (171: Methadone Neuropathic Pain).
NA, not applicable.
Discussion
Frequent Users, who make up a minority of FFM users (13%), accounted for the majority of FFM usage (66%). The search behavior of Frequent Users and Infrequent Users was similar: both groups searched for the same terms in similar frequencies relative to each group. Depending on the search term, Frequent Users utilized the search feature anywhere from three to five times more often than Infrequent Users.
One possible explanation for this usage pattern is that Frequent Users are using FFM as a “just in time” clinical reference tool, while Infrequent Users are using FFM for more passive educational purposes. In this scenario, Frequent Users of FFM utilize the FFM app to find answers to clinical questions that surface throughout the course of the workday. As a clinical reference tool, FFM would likely be accessed multiple times throughout the workday, and search terms and articles addressing common clinical questions would be accessed most frequently and largely based on the clinical issues that the end user experiences throughout the day.
Infrequent Users access and search for similar topics as Frequent Users, but are not reaccessing the content as frequently. This behavior could be consistent with a user who is using FFM as an educational tool to review concepts instead of utilizing FFM for clinical work.
Another possible explanation for the divide in Frequent Users and Infrequent Users of FFM could be based on end user comfort with utilizing clinical mobile applications, Frequent Users being more adept and willing to utilize mobile applications than Infrequent Users.
We also found variation in FFM about how search term results are displayed to the user and what the user actually selects. The current search engine in FFM ranks each article by the total number of times a term is listed somewhere in the body of the text. When searching, users often selected an article other than the first article returned, suggesting an opportunity to improve the search ranking algorithm. There is an opportunity to improve search functionality, such as providing additional context to how search results are ranked in FFM.
Knowledge of what topics users are most searching for, what articles users are accessing most frequently, and the characteristics of the FFM base could be used to create targeted calls to the readership of FFM for new articles. The data provide a better understanding of the current relationship that users have with the Fast Facts articles and could inform future direction of the apps and the overall database of articles.
Mobile applications have the potential to provide FFM users new ways of interacting with Fast Facts. Features such as dynamic notifications, interactive quizzes, and personal annotations to FFM articles are all possible directions for future versions of FFM. Future development of the application will also benefit from the insight of the usage data. Near-term updates to the applications are also possible based on the insights from this study—a new section of the application listing the articles by most viewed or most searched-for keywords is one possible near-term update to FFM.
Further work is needed to characterize the Fast Facts user base and their needs. By leveraging some of the insights from this study, other modalities, including the use of qualitative studies, could be utilized to measure Frequent User and Infrequent User sentiment for FFM. In addition, a better understanding of key demographic information such as user profession and training level is needed.
Limitations
Limitations of this study include the lack of direct demographics of the existing user base of FFM. Because of this, no knowledge of profession or experience can be analyzed within the usage data. As such, we can only infer user characteristics based on articles access and in-app behavior. In addition, only events that were coded to be recorded by the analytics platform were recorded, limiting the detail of recorded behaviors. More nuanced user behaviors such as whether users were accessing articles from the “recently viewed” section of the application over the “bookmarked” section of the application were not recorded. This study also focused on usage of FFM in the United States only and does not evaluate the usage needs of international users of FFM. Importantly, this study is not able to evaluate the impact of FFM on quality or efficiency of care.
Conclusion
This study describes usage patterns over a year in a popular palliative care mobile reference application. We found that a relatively small proportion of users accounted for the majority of all use of the application. It is unclear exactly what separates Frequent Users of FFM from Infrequent Users, although we suspect that comfort with mobile tools and reason for use (educational vs. just in time clinical reference) are likely factors in this division. Further characterization of these two segments of users will likely be invaluable for improving future versions of FFM.
Footnotes
Acknowledgments
The authors would like to acknowledge the all-volunteer software development efforts of the Fast Facts Mobile team: David Liu (iOS lead developer), Jess Smith (Android lead developer), and Mike Caterino (original iOS developer). Without their ongoing efforts, Fast Facts Mobile would not exist. In addition, we also want to acknowledge Dr. David Weissman for his many contributions to Fast Facts over the years.
Author Disclosure Statement
No competing financial interests exist.