Abstract
Introduction
Two-way text-messaging systems have been applied to a wide range of clinical domains, such as smoking cessation, 1 –4 diabetes, 5 –9 and medication adherence. 10 –15 Medication adherence is particularly important for the successful treatment of human immunodeficiency virus (HIV), as nonadherence can enable HIV replication, mutation, and ultimately, medication resistance. 16 Suboptimal medication adherence among HIV-positive patients continues to be a frequent cause of poor treatment outcomes. 17
Preliminary results indicate that text-messaging technology is acceptable to HIV-positive patients and shows potential for improving adherence outcomes. 12 A common barrier to adherence is forgetfulness, and mobile messaging systems can serve as successful reminders. 18 –20 Nonetheless, mobile messaging systems will only be successful if patients are actively engaged in the system. Little research has addressed the specific characteristics of messaging systems that appeal to users or their viability as medication-monitoring systems. To begin to explore these issues, we present quantitative and qualitative findings from a randomized controlled clinical trial that used a pager-based text-messaging system as part of an intervention to promote medication adherence among HIV-positive patients initiating highly active antiretroviral therapy (HAART). The impact of the intervention on adherence and other health outcomes has been published previously. 21 The aims of this analysis were to examine (1) response rates to the pager system, (2) user evaluations of the system, (3) demographic and psychosocial correlates of use, and (4) the performance of the system as a tool to assess adherence.
Materials and Methods
Data were collected from 2003 to 2007 at the adult HIV primary care outpatient clinic at Harborview Medical Center, a public institution affiliated with the University of Washington in Seattle, Washington. 21 Advertisements for the study were posted in the clinic waiting room, and all providers were informed of the study. Referrals came from clinic providers, the patients themselves, and a nurse at the clinic who approached all potentially eligible patients at each clinic session.
Equipment
Participants received two-way alphanumeric pagers with programmable silent or audible alerts. The messages could be 20 lines long, but only 4 lines of text could be viewed concurrently. Participants could respond with a preprogrammed response or by composing their own replies using a “soft key” interface. The investigators used commercially available proprietary software (Talaria) to enter, edit, and store messages. The software enabled the investigator to create a personalized algorithm of the number and types of messages sent to a participant each day (e.g., two reminder messages for a twice-daily regimen, one educational message, and one joke). The investigator could review the outgoing messages and replies using a custom Web interface that organized messages by patient, time, or content.
Participant Training
Eligible participants were (a) at least 18 years of age, (b) proficient in English, (c) living within the service area of the pager, and (e) initiating or changing at least two medications of a HAART regimen. Individuals who were cognitively impaired, actively psychotic, or had a known history of harming others were excluded. After enrollment, each participant described daily work, food, sleep, and medication schedules to a member of the research staff so that a daily medication schedule could be written. The functioning of the pager and the nature and purpose of the messages were demonstrated. The participants were asked to respond to all pager messages during the 3-month duration of the intervention. Users could respond to messages with a single button press. If a response was not received within 10 min, a second duplicate message was sent out. We asked users to respond to all messages for two reasons. First, it was the only way that we could tell whether or not they had received the message. Second, we believed that participants would be less likely to ignore pages if they knew that there was an expectation of a response.
Pager Messages
The participants were sent between one and eight personalized messages daily with content that varied from day to day. Messages included medication reminders (e.g., “Good morning, JR. Time for your Indinavir”) and a question about adherence (“Hi, JR, how many doses of your antiviral medications did you miss in the past 3 days?”) once a week for assessment purposes. Some messages provided educational information, whereas others included nonclinical content, including jokes and inspirational quotes.
Data Collection
Three months after receiving the pager, participants responded to a 29-item survey about their attitudes toward the pager system, frequency of use, prediction of future use, the preferred quantity of messages, and reasons for nonuse. Adherence to HAART was assessed with the pager and the Medication Event Monitoring System (MEMS), a pill bottle capable of recording the exact time and date of each opening. After completion of the study, three focus groups were conducted, each consisting of one female and three male participants (N = 12). A focus group guide was utilized to explore experiences with the pager and barriers to and facilitators of participation.
Data Analyses
Response rates
Individual response rates were calculated as the number of user responses divided by the total number of messages received by the device. Messages that were not received because of insufficient signal strength or inactive device (i.e., the device being turned off) were excluded from the response rate. Analysis of variance, bivariate correlations, and multiple linear regression models were used to assess relationships between response rates and user characteristics.
User evaluations
User attitudes were assessed with surveys (Table 1) and focus groups. Focus group data were transcribed from audiotapes, coded, and analyzed with ATLAS.ti qualitative software. Two research team members independently analyzed the transcripts for categories and concepts and used a process of open coding to assign codes.
Pager Evaluation Descriptives
Percentages were calculated out of total responders specific to each item.
Assessment of adherence
We examined the pager as a tool for assessing medication adherence, using MEMS data for comparison. We calculated the frequency that the pager-reported proportion of doses was taken as (1) equal to, (2) less than, and (3) greater than the proportion calculated by MEMS.
Results
A total of 224 HIV-positive individuals were enrolled in the study, 110 of whom were originally randomized to receive a pager. Three participants were excluded from analysis after randomization—two had <10 days of pager data available, and one died—resulting in a final sample of 107. Eighty-five participants (79%) were male and the mean age was 40.7 years. Fifty four (50%) were Caucasian, 31 (29%) were African American, 7 (7%) were American Indian or Alaskan Native, and 15 (14%) were from other races, more than one race, or did not report their race. Eighteen (17%) had less than a high-school education, 32 (30%) had a high-school degree or equivalent, 48 (45%) had some college or an associate's degree, and 9 (8%) had a bachelor's degree or higher. Fifty percent of the sample had an annual income of less than $6,636; only one individual had an annual income over $42,000. Fifty nine (55%) identified as gay or lesbian, 40 (37%) as heterosexual, and 9 (8%) as bisexual; 21 (20%) were currently married or had a steady domestic partner.
Message Characteristics and Response Rate
Medication reminders accounted for 44% of all messages, jokes and quotes for 27%, educational messages for 20%, adherence questions for 6%, and side-effect information for 4%. The system sent an average of 2.4 messages per day per participant. Of those, an average of 1.8 messages were received by the pager. The overall response rate, calculated with a denominator of all messages received by the pager, was 42.8% (standard deviation [SD] = 30.7%). The response rate dropped significantly over time, from 49.9% (SD = 35.6%) during the first 3 weeks of the trial to 27.8% (SD = 33.6%) for the last 3 weeks (F (12, 1,380) = 5.84, p < 0.001).
Response rates were significantly correlated with the proportion of messages received, such that participants who received a higher proportion of the messages sent out by the system, possibly due to signal reception or to keeping their pagers turned on more often, were more likely to respond to the ones they got (R 2 (105) = 0.53, p < 0.001). Specifically, among participants who received over 90% of the messages (n = 43), the mean response rate was 61.0% (SD = 28.5%) compared with 23.2% for those who received fewer than 50% of the messages (n = 64; SD = 22.8%). The response rate did not vary by message type.
Gay/lesbian participants were more likely to respond to the pager messages compared with heterosexual or bisexual participants (M = 51.4%, SD = 29.6% vs. M = 31.6%, SD = 29.2%, respectively; F (2, 104) = 6.07, p < 0.001). A higher response rate was also associated with Caucasian race (M = 49.1%, SD = 30.4%) versus all other races (M = 35.6%, SD = 30.0%) (F (1, 105) = 5.28, p = 0.02). The response rate was not, however, related to gender, age, ethnicity, education, income, or marital/partnered status. Moreover, the response rate did not vary significantly by any of the psychosocial variables assessed (with the assessment instrument used for each referenced): mental health, 22 physical functioning, 22 quality of life, 23 depression, 24 social support, 25 alcohol dependence, 26 and drug dependence. 27
Of the total number of responses received over the course of the trial, 58% were received within 10 min (“quick response”). In bivariate analyses, a higher frequency of quick response was associated with male gender (R 2 (104) = − 0.20, p = 0.04), Caucasian race (R 2 (105) =0.22, p = 0.02), and gay or lesbian sexual orientation (F (2, 104) =6.95, p < 0.01). The frequency of quick response was not associated with any other demographic or psychosocial characteristics. In multivariate models that included gender, race, and sexual orientation as predictors, only sexual orientation was significant (β = 0.18, standard error = 0.07, p = 0.009 [response rate]; and β = 0.15, standard error =0.05, p = 0.006 [quick response rate]).
Survey Evaluation
Of the 104 participants (97% of the sample) who responded to the pager evaluation survey, 94 reported ever using the pager. Sixty-nine respondents (73%) reported generally liking the pager, believing that it helped them to take their medications correctly, that they learned about HAART medications and their side effects, and that it was convenient and easy to use (Table 1). Forty-eight respondents (51%) found getting messages sent again after no response irritating, and 56 respondents (60%) stopped reading the informational and entertainment pages over time. Forty-five respondents (48%) believed that they would have done just as well taking their medications without the pager. When asked whether they would continue to use the pager for no charge, 44 (47%) responded they would. When asked about the highest monthly fee that they would pay, 18 respondents (31%) said they would not pay for the service, 23 (39%) would pay between $1 and $10, 10 (17%) would pay between $11 and $20, and 8 (14%) would pay more than $20.
Focus Group Evaluation
Three themes emerged from focus group participants' responses regarding perceptions of usefulness of the pager for medication adherence. These were (1) pager as useful in establishing a routine, (2) pager as a useful reminder to take medications, and (3) problems with pager use/function. Themes were consistent across the three focus groups.
Pager usefulness in establishing a routine
One participant in each of the three focus groups experienced the pager as being a useful tool in establishing a medication routine. These participants noted that the pager was useful in creating a medication schedule. Two participants noted that the pager acted as a catalyst for acquiring a phone with an alarm. These participants now utilize the alarm on their cell phone as an effective reminder to take their mediations.
Pager usefulness as a reminder to take medications
In each focus group, one participant stated that the pager was a useful memory aid in their efforts to adhere to their medication schedule. For one participant, the pager was used by her family as “a signal for medication time.”
Problems with pager use/function
The majority of focus group participants experienced one or more problems with the functioning of their pager. Four participants noted that the pager did not vibrate or beep loud enough, resulting in missed pages. Other participants experienced the pager as “difficult to figure out” and “did not know how to work it.” These participants stated that they did not use their pager or turned it off because of these difficulties. One participant inquired as to the reasons for providing participants with new pagers instead of “using the cell phone we already have to send pages or text messages.” He noted, “this would be easier because we are more familiar with our own phones.”
Adherence Assessment
The average response rate to adherence messages was 46.9% (SD = 33.0%). For an average of 93.9% (1306/1391) of the days that participants were sent adherence questions on their pagers, adherence data were also available through MEMS tracking (SD = 15.8%). Adherence data from MEMS were excluded in cases where the participants' antiretroviral prescription data were unavailable (6.1%). According to pager data, the average reported 3-day adherence was 90.8% (SD = 15.8%). Overall adherence reported by MEMS was 53.6% (SD = 37%). When data were available from both pager and MEMS for the same 3-day window, adherence was concordant for 53.0% of the time. The pager-reported adherence was higher than that reported by MEMS (40.8% of the time), and MEMS-reported adherence was higher than that reported by the pager (6.2% of the time).
Discussion
Response Rate and User Evaluations
This study examined factors associated with user engagement and adherence assessment in a health-related text-messaging system among HIV-positive men and women. Although most participants evaluated the pager system positively, the overall response rate to system messages was relatively low (42.8%) and dropped significantly over the course of the 3-month intervention. The low response rate could be explained by several factors. First, a sizable number of participants in the sample were economically disadvantaged, and it is plausible that patients faced barriers to responding to the pager in the presence of other life stressors. Second, participants in the focus groups reported technical usability issues with the pager. Response rates also tended to drop linearly over time, at an average of 2.5% per week, which parallels self-report findings that a majority of participants stopped responding to informational and entertainment messages. Collectively, these findings suggest that more effort is needed to maintain interest over time, perhaps by varying content, tapering the nonessential messages, allowing users to opt out of certain features (like repeat messaging), and addressing device problems that may erode usage. Nonetheless, participants reported that the pager helped them set up a routine and served as a useful reminder. Response rate did not appear to be associated with sociodemographic and psychosocial variables, with the exception of Caucasian race and gay/lesbian sexual orientation. The post hoc findings of a higher response rate and a higher frequency of quick response within particular sociodemographic subgroups may reflect different attitudes, preferences, or experience with this type of technology. Several previous studies have found that technology use varies by race, ethnicity, and sexual orientation. 28 –30 Given that the response rate did not vary significantly by most major sociodemographic and psychosocial variables, the system appears to be applicable to a diverse population. Although the overall response rate was low, results suggest that a substantial proportion of participants responded to the system to some degree despite formidable financial and health-related stressors.
Adherence Assessment
In cases where adherence data from both the pager and MEMS were available, the average adherence reported by the pager was significantly higher. Given that the pager is a self-report tool, the higher reported adherence could be partially explained by social desirability bias. Indeed, other studies have shown that self-report overestimates adherence compared with MEMS. 31 Conversely, MEMS may have underestimated the true adherence in cases where participants were not using the device to dispense their medications or removed multiple doses with a single bottle opening. 32
Limitations
These analyses are limited by the nature of response rate as a measure of user engagement. Individuals may have relied upon the messaging system for its reminder function without actually responding to the messages, which would explain the discrepancy between the high proportion of survey responders who believed the pager helped them with medication adherence and the lower response rate. In the future, messaging technology that automatically detects when a message has been read should be utilized. In addition, although all of the messages in this intervention were addressed to participants by name and the medication reminders were customized to each participant's unique medication schedule, the educational content was not tailored. This is an important area for future research, as tailoring has been shown to enhance relevance and stimulate greater cognitive activity in a wide variety of health interventions. 33 –35 Finally, our study was limited by the requirement to carry a pager. Only 64 respondents (68%) found the pager easy to carry. When this study was begun in 2003, U.S. mobile phone penetration was only around 50%. 36 Further, 50% of our sample had an annual income of less than $6,636, suggesting that mobile phone ownership within our sample was likely to be even lower than the U.S. average. Given that mobile phone penetration is now over 90% in the United States 36 and many other countries worldwide, future studies in populations, where mobile phone penetration is expected to be high, should use participant's own mobile phones rather than provide study-specific devices that may be unfamiliar and inconvenient.
Conclusions
This study is an important step, in an area with limited research, toward understanding the factors that characterize patient engagement with health-related messaging systems. Although a majority of our study participants reported liking the pager system, usage dropped significantly over time and it did not appear to be reliable as an adherence assessment tool when compared with MEMS. Further work is needed to assess how to maintain user engagement over time and how to better adapt health-related messaging systems to meet individual patient needs and preferences.
Footnotes
Acknowledgments
Thomas Hooton was an investigator on the grant and director of the clinic where the data were collected, and Pamela Frick Schwartz helped with the grant and wrote the pager messages. This work was supported by a grant from the National Institutes of Health (R01 MH58986).
Disclosure Statement
P.J. Dunbar owns stock in Talaria, Inc., the commercial entity that was paid by the study grant for use of their product (the pager-based messaging system). None of the other authors has commercial associations that might have created a conflict of interest in the conduct of this research.