Greek validation of the user version of the Mobile Application Rating Scale (uMARS)

The English-language version of the uMARS

The uMARS provides an evaluation of the quality of an app with a 20-question scale divided into five subscales. Every item is rated on a 5-point Likert scale ranging from 1 (poor) to 5 (excellent). Four subscales—engagement (N = 5), functionality (N = 4), aesthetics (N = 3), and information (N = 4)—are objective and are associated with a quality rating; one subscale (N = 4) is subjective. An additional subscale of six items measures the perceived impact of the app being evaluated. ¹²

Cross-cultural adaptation and translation process

This was a qualitative prospective study. The Greek translation and cross-cultural validation of the English-language uMARS followed the World Health Organization guidelines ¹⁴ and consisted of forward and backward translation, pretesting, and cognitive evaluation and production of the final version.

The translation and adaptation processes involved the following steps: (i) The English-language version of uMARS was independently translated into Greek by two native Greek bilingual speakers (one physician and one non-medical professional) who were fluent in English. (ii) The two Greek translations were discussed within the research team to ensure meanings were as close as possible to the original English version. (iii) The Greek version was then back-translated into English by one bilingual researcher (a native Greek with a degree in English literature) and (iv) the result was reviewed and compared with the original English version by the developers of the original questionnaire, who provided feedback on the accuracy of the translation and suggestions for edits. The comments and suggestions from the original developers were discussed within the research group and integrated to develop the final version of the Greek uMARS.

Cultural adaptions were made in every step of the translation by changing the wording of some items to better fit the Greek context. The goals were to adapt uMARS to the Greek culture and language, consider cultural and linguistic differences between English and Greek, identify and resolve inadequate expressions or concepts in the translation, and ensure equivalence.

Selection of the mobile app for validation

When the Greek-language translation was ready, a mobile app was selected according to the following inclusion criteria: available in the Google Play and Apple App stores, free of charge, targeting adults (≥18 years of age) of either sex, containing features that promote interaction, relevant to health management (e.g., questions and answers, the exchange of images, reports, or numerical results).

After discussion and consensus, the research team chose the UTHme app, an accessible and free-of-charge app widely used by students of the University of Thessaly. The app requires the student’s academic credentials and is therefore downloadable exclusively for students at the University of Thessaly.

Although emerging, health apps in the Greek language are still scarce. The choice of the UTHme app was appropriate because the app already incorporates many features offered by English-language health apps, some of which are:

Quick access to students’ grades, which are entered by academic personnel—a feature commonly used by various health-tracking apps.

Further benefits of the UTHme app included: (i) accessibility to all students and used to provide easy, on-the-go access to some University of Thessaly services, (ii) does not store user data, which offered compliance with current GDPR and further convenience to our research, and (iii) availability on Android and iOS devices, with a rating of 4.9/5 stars (1,000+ ratings) and more than 10,000 downloads.

Sample and procedure

All participants were 5th- and 6th-grade medical students at the University of Thessaly. All students were native Greek speakers. A paper version of the Greek translation of the uMARS was distributed on demand to individuals who agreed to participate in the project. The final translated version of the uMARS was hand-distributed to a sample group of 91 medical students from the University of Thessaly.

Each participant completed the uMARS twice—once after using the app for a month and then again 4 to 6 weeks later. If necessary, students were allowed to use the app between the two evaluations. Participants were instructed to complete the questionnaire individually and without assistance. Participants handed in completed questionnaires immediately after completion.

Ethics approval was not required because the study involved the translation and validation of the uMARS questionnaire with the participation of medical students who were informed about the study objective. No patients participated and no form of treatment was used in the study.

Data analysis

The internal consistencies of the uMARS subscales and the total score were calculated using Cronbach's alpha. Test–retest reliabilities were calculated for the subscales and total scores of the uMARS after 1 month of app use post baseline (i.e., a test–retest period of 1 month). Interclass correlation coefficients (ICCs)^15–17 were used because they provide weighted values of agreement and assess proximity rather than equality of ratings. To calculate ICCs, a random-effects average measures model with absolute agreement was used. The r_WG index was chosen given the skewness of responses. ¹⁵ The r_WG index is preferred over ICCs for Likert scale measurements and especially when a single group of participants is used. ¹⁸

Data were analyzed with SPSS version 23 (SPSS Inc., Armonk, NY, USA). The reporting of this study conforms to the Standards for Reporting Qualitative Research guidelines. ¹⁹

Internal consistency of the uMARS

The total uMARS score showed excellent internal consistency (Cronbach’s alpha =0.86). The internal consistency of the subscales was also very high, with Cronbach’s alphas of 0.71 for engagement, 0.71 for functionality, 0.67 for aesthetics, 0.63 for information, and 0.61 for subjective satisfaction.

The uMARS total score demonstrated almost perfect agreement levels in most of the subscales according to the r_WG index from baseline to 1 month. The r_WGs for the engagement, functionality, aesthetics, information, and subjective subscales were 0.97, 0.98, 0.98, 0.97, and 0.97, respectively. The question with the lowest interrater agreement queried the willingness of participants to pay for the app (0.73). Test–retest reliability results are presented in detail in Table 1.