Quality,functionality,and features of a newly developed smartphone app for diabetes: A comparative analysis using Mobile app rating scale

Phase-I: Searching for the apps

In the initial phase of the study, a systematic method for identifying appropriate applications was implemented, adhering to the standards outlined in the PRISMA 2020 guidelines, as detailed below:

Search Strategy: On April 24, 2023, two separate primary reviewers (TPS and SW) conducted searches on the app platform [specifically, Google Play Store (GPS)] to identify health-related applications (Figure 1). Keywords for app searching included diabetes, blood glucose, diabetes prevention, diabetes control/management, and diabetes treatment.^19,23 Subsequently, a rating scale (Appendix I) as previously used by Ranjani et al. ²² was utilized for rating included apps from the GPS. The selection process was guided by four key parameters, which assessed their presence on the GPS: (a) the app's overall rating, (b) the number of downloads, (c) the app's size, and (d) the number of reviews. Each primary reviewer assigned ratings on a scale of 1–4, where 1 indicating the lowest and 4 indicating the highest rating. The total rating for each app was calculated as the sum of these four ratings, with a maximum possible score of 16 (4*4) and a total rating score of 9 or higher from both reviewers as the criterion for shortlisting the apps.^20,22

OPEN IN VIEWER

Figure 1.

Flowchart showing the systematic search of the apps on google play store according to the PRISMA 2020 guidelines.

Eligibility Criteria: Applications were considered for inclusion if they were employed in the context of diabetes self-care and encompassed a minimum of one of the subsequent elements: management of blood glucose levels, control of dietary and physical activity, medication adherence, or the prevention of diabetes-related comorbid conditions. The criteria for exclusion encompassed medical applications involved in diagnostic or clinician-driven healthcare, apps offering consultation services, pharmacy-related apps, non-Asian dietary guides (particularly those catering to American or Western diets that did not align with Indian dietary preferences and habits), apps featuring lifestyle quotes, unconventional weight loss diets, apps endorsing specific organizations or products, and promotional apps (e.g., those associated with fitness bands or fitness centers). Additionally, apps requiring prescription from a clinician for log-in the app and apps with premium features that required payment from users were omitted, while those offering a basic version without any extra charges were included.^20–22

Phase-II: Grading of apps based on AADE-7 criteria for self-care behaviors (mHealth support in diabetes self-care)

In the second phase of this study, we have screened all the apps selected in Phase-I for the features offered by these apps in diabetes self-care and management. For this screening, we have utilized the list of minimum features for diabetes apps and developed and validated by Salari et al. ²⁴ This list consists of 23 minimum features (Appendix II) which are referenced to the American Association of Diabetes Educators (AADE-7) ⁶ guidelines for providing a collection of prerequisites for diabetes mobile apps. AADE-7 guidelines unequivocally narrate about improving any of the seven therapeutic elements mentioned below for diabetic patients: 1. healthy eating, 2. being active, 3. monitoring, 4. taking medications, 5. problem-solving, 6. reducing risks, and 7. healthy coping.

All the apps selected in Phase-I were screened for these minimum features by the primary reviewers and the apps which have fulfilled AADE-7 criteria and offered minimum 20 out of 23 features were further selected for qualitative app rating study utilizing the Mobile App Rating Scale (MARS).^6,22,24,25

Phase-III: Qualitative screening of selected apps by MARS

Mobile App Rating Scale serves as a tool for evaluating the quality of mobile medical apps. MARS evaluates mHealth apps using 19 criteria across four dimensions: engagement (5), functionality (4), aesthetics (3), and information (7). The assessment is executed using a 5-point Likert scale, ranging from 1 (indicating inadequacy) to 5 (indicating excellence), with the provision for selecting “Not Applicable” (N/A) in instances where specified criteria are deemed irrelevant. Furthermore, a subjective quality assessment (4) segment is incorporated to evaluate: the intention to recommend the app, the anticipated frequency of app usage, the willingness to pay for the app, and an overall star rating for the app.^25,26

All the apps selected in Phase-II along with “Madhumeh Setu” were subjected to quality assessment by 6 secondary reviewers (including 2 Physicians, 2 Nurses, and 2 Pharmacists) as invited experts, utilizing MARS during May 2023. These reviewers took primary training on the MARS instrument through an online tutorial developed by primary developers of MARS instrument at University of Queensland, Australia. ²⁸ Rating was done for all 19 criteria along with assessing the app subjective quality segments (4). Mean scores for each dimension: engagement, functionality, aesthetics, information, and app subjective quality were calculated, and overall mean scores were generated for each app. The inter-reviewer (rater) reliability between reviewers was assessed using the K-Alpha statistics [G. Marzi, M. Balzano, and D. Marchiori]. ³⁰ The reliability estimate, determined through Krippendorff's Alpha Calculator, ³¹ yielded a value of 0.74, indicating moderate agreement. Considering the inherently subjective nature of the domains within this scale, the obtained reliability estimate was regarded as highly favorable.^27,32

Phase-IV: Comparative analysis of Madhumeh Setu and marketed apps by type 2 diabetes patients utilizing MARS

In Phase IV of the study, the newly developed Madhumeh Setu app was subjected to a qualitative comparative analysis against two established diabetes management applications: MySugr and BeatO.^22,26 The selection of these comparators was purposive and strategic, aiming to elicit patient perspectives regarding the features, usability, and overall quality of “Madhumeh Setu” in relation to leading market alternatives. MySugr, developed in Austria and widely adopted internationally, is among the most extensively cited and clinically validated diabetes self-management applications. It benefits from endorsement by healthcare professionals and adherence to the stringent data protection standards stipulated by the European Union's General Data Protection Regulation (GDPR). ²⁸ In contrast, BeatO, an Indian-origin application, is specifically tailored to the Indian demographic, incorporating considerations of regional healthcare infrastructure, dietary practices, and linguistic diversity. Furthermore, BeatO's integration with local diagnostic laboratories and insurance services renders it a contextually relevant comparator for Madhumeh Setu. Both MySugr and BeatO demonstrate high user ratings, significant download volumes, and favorable clinical appraisals, frequently featuring in systematic reviews and meta-analyses assessing the efficacy of mHealth interventions for diabetes management.^23,29–31 Consequently, their inclusion provided a robust framework for the qualitative assessment of the clinical relevance, user-centered design, and technical soundness of the Madhumeh Setu application.

Participants (type 2 diabetes patients) were chosen according to specific inclusion criteria; individuals of both genders aged 18 years and above, possession or regular access to an Android phone, confirmed diagnosis of type 2 diabetes by a Physician or Endocrinologist, proficiency in reading and understanding English, and willingness and capability to complete data collection through a mobile app. Conversely, exclusion criteria encompassed individuals with any diabetes type other than type 2, those experiencing end-stage hepatic or renal failure, individuals lacking education or training in the use of an Android mobile phone, and those unwilling to participate in data collection through a mobile app. Table 1 provides the demographics of the selected patients for this comparative study. The study was approved by the ethics committee of Delhi Pharmaceutical Sciences and Research University (DPSRU-BREC/2023/A/018). Participants were recruited by convenience sampling from a local community in Delhi, and to be able to detect 0.5 score change in MARS with the power of 80% and desired a significance level of 0.05, a sample size of 45 was estimated carrying 10% dropouts. All the participants were trained in MARS instrument utilization through an online tutorial, ²⁸ fostering rating uniformity among assessors. The reliability of the ratings, as measured by Krippendorff's Alpha (0.71), was deemed satisfactory, particularly given the subjective nature of the MARS scale.^33,34 Mean MARS scores were employed to estimate the top-quality applications.^{23,26–32,35}

OPEN IN VIEWER

Table 1.

Demographics of the type 2 diabetes patients (N = 45) selected for the comparative study through MARS.

Characteristics	Participants
Female, n (%)	21 (46.7)
Age in years, mean (SD)	50.7 (6.5)
Diabetes duration in years, mean (SD)	10.6 (3.4)
Baseline HbA1c (%), mean (SD)	8.3 (1.33)
Self-monitoring of blood glucose per week, mean (SD)	5.7 (3.8)
BMI, kg/m2, mean (SD)	33.4 (3.44)
Education level, n (%)
Undergraduate/Graduate	30 (66.7)
Post-graduate or above	15 (33.3)
Annual income, n (%)
Below Rs. 1,00,000/-	4 (8.9)
Rs. 1,00,000/- to Rs. 5,00,000/-	11 (24.4)
Above Rs. 5,00,000/-	30 (66.7)
Physical activity level, n (%)
Sedentary (bare minimum activity, e.g., desk  job, simple cooking)	5 (11.1)
Lightly active (Light exercise/sports for 1–3  days per week)	22 (48.9)
Moderately active (Exercise/sports for 3–5  days per week)	18 (40.0)

Statistical analyses

Means (M) and standard deviations (SDs) for the MARS sum score and all subscales were calculated. Further, one-way ANOVA with post hoc Tukey HSD (Honestly Significant Difference) was employed at a significance level of p < 0.05 to assess the differences in MARS ratings given by diabetes patients.

Screening and selection of apps

Among 2116 apps searched in the initial screening, only 11 apps met the final screening criteria which were employed in phases I and II of this study. An overview of the selection process and exclusion categories is depicted in Figure 1 through a comprehensive PRISMA flow diagram.

The descriptive data for the 11 apps picked through the screening process are presented in Table 2. Based on the criteria developed by Rajani et al. (Appendix I), which primarily considers the number of downloads, reviews, and user ratings on the play store, the BeatO app got the highest score (15 out of 16). In this category, 2 other apps with similar results included My Sugr (14 out of 16) and Diabetes:M (13 out of 16). Furthermore, in the second screening where minimum app features were evaluated, My Sugr (23) received the highest score, followed by BeatO (22) and DiabTrend (21). During this screening, internally developed Madhumeh Setu app was also rated using MARS. Given that Madhumeh Setu is a recent addition to the Google Play Store, the reviewers, bypassed the initial criterion (Ranjani et al., 2021) for app rating. However, adhering to the second criterion (Salari et al., 2019), Madhumeh Setu achieved a score of 21 out of 23, meeting the minimum requirements for diabetes self-management features.

Qualitative assessment of selected apps through MARS

Table 3 presents the mean ratings of mobile apps across various categories in MARS, encompassing all chosen apps including Madhumeh Setu. Initially, individual dimensions such as engagement, functionality, aesthetics, information, and subjective quality were rated by six secondary reviewers, resulting in mean scores. Subsequently, overall mean scores were computed for each app. The highest overall mean score was achieved by My Sugr app (4.40 ± 0.14), followed by BeatO (4.12 ± 0.04), Diabetes: M (4.07 ± 0.08), and Madhumeh Setu (4.05 ± 0.05), whereas the lowest score was attributed to Social diabetes (3.62 ± 0.19) app. Similarly, subjective quality ratings assigned by these reviewers were averaged, with the highest rating given to My Sugr (4.2 ± 0.34), followed by BeatO (4.1 ± 0.41), and Diabetes:M (4.00 ± 0.39), while the lowest rating was assigned to Diabetes Connect app (3.0 ± 0.59). Madhumeh Setu app obtained a mean quality score of 4.05 ± 0.05 and a subjective quality score of 3.9 ± 0.48, comparable to scores of the marketed apps examined in this study.

Comparative analysis of best apps with Madhuemeh Setu

Within the last phase of this study, two best scorer apps namely: My Sugr and BeatO were qualitatively compared with Madhumeh setu by type 2 diabetes patients in the community. As many as 45 patients from the local community in New Delhi, India, filled the Mobile App Rating Scale for these apps. The cumulative mean scores for these apps were 4.46 ± 0.16, 4.14 ± 0.14, and 4.09 ± 0.10, respectively. Figure 3 illustrates the criteria-wise rating of these three apps as per the MARS scale. For the comparative assessment of these apps, one-way ANOVA was applied employing the level of significance as p < 0.05 (Figure 2). The value of F ratio was obtained as 97.79 with a p < 0.00001, further the post hoc Tukey's test for pairwise comparison explained that Madhumeh Setu app was significantly different when compared with My Sugr app [Q = 15.88 (p = 0.00000)]; however, no significant difference was observed between the BeatO and Madhuemeh setu [Q = 2.27 (p = 0.0246)] explaining that the Madhumeh Setu was qualitatively comparable to BeatO, while My Sugr was of a higher quality within the purview of MARS rating given by type 2 diabetes patients.

OPEN IN VIEWER

Figure 3.

MARS criteria-wise average scores provided to selected apps by diabetes patients.

User testimonials gathered through this study, reported that MySugr provides better engagement features such as; gamification and motivational prompts, the app consistently received the highest patient scores, owing to its playful and visually rich interface, provided more comprehensive, internationally standardized educational material but was considered less region specific. BeatO also perceived as moderately engaging but was critiqued for limited personalization, offered moderate-quality educational content, largely focusing on device integration and monitoring without sufficient depth in lifestyle education. Madhumeh Setu demonstrated moderate to high-level engagement due to its personalized goal setting, regional food options, and vernacular language support. Patients appreciated the intuitive design, minimal technical glitches, and efficient loading times though a minority of users suggested it could benefit from more modern, polished graphics, and educational support.