Evaluation of Usability,Perception of Usefulness,and Efficiency of an Application in Interpreting Imaging Examinations and Supporting Decision-Making in Orthopedics

Abstract

Background:

Medical smartphone applications have rendered positive results in daily practice and also serve as potentially valuable educational tools. In orthopedics, radiograph analysis and angle measurements comprise an essential mechanism in the diagnosis, treatment, planning, and evaluation of the results in orthopedic surgery.

Purpose:

Thus, the present study aimed to evaluate an application for mobile devices for orthopedic doctors and orthopedic residents. The application proposes the analysis and interpretation of radiographic imaging examinations through assessments and angle measurements. In this survey, we analyzed the parameter usability and usefulness of a mobile application for the iOS system, in addition to conducting its validation.

Patients and Methods:

The application was tested by 26 volunteers: 13 orthopedists and 13 orthopedic residents.

Results:

After applying the System Utility Score (SUS) questionnaire, which assesses the usability of an application, a score of 84.5 was obtained, a value that represents an A+ rating on the Sauro and Lewis scale. There was no statistically significant difference when comparing the SUS between the orthopedic residents and orthopedists. According to the usefulness perception analysis, 90% of the residents responded positively to the questions, while among orthopedists, the percentage was 75%, denoting a statistically significant difference (p = 0.002). Subsequently, the application underwent validation to assess whether the calculated angle measurements were reliable when compared with those obtained using the goniometer. No significant differences were observed when comparing the angles measured by the two evaluated methods.

Conclusion/Clinical Relevance:

In conclusion, the developed application can reliably contribute to the measurement of angles in radiographs, assisting in therapeutic decision-making, and may be used as an instrument to promote the teaching and learning processes in orthopedics.

Introduction

In orthopedics, the evaluation of radiographs and measurement of angles comprise a fundamental component in the assessment of a patient's clinical condition. Measuring joint angles and range of motion is essential to establish the severity of joint deformity and the level of postinjury disability.¹ Determining whether angular deformities are outside acceptable parameters is often used to guide practitioners in considering clinical or surgical interventions.² Thus, measuring angles in radiographs constitutes a valuable tool in the diagnosis, treatment, planning, and prognosis in orthopedic surgery.^2,3

Angle measuring can be performed manually using a goniometer or with systems for direct measurement on a computer. However, they are not always easy to use and accessible.⁴ Moreover, digital radiograph measurement tools are not portable or useful when consulting patients outside the hospital environment or medical facilities lacking a computer network.⁵

The most utilized method for measuring angles on nondigital radiographs is conducted using a goniometer. Although there is a gradual replacement of printed radiographs with digitally processed images, many surgeons still use this previous modality in clinical practice, especially in developing countries.³ Therefore, an alternative method for the goniometer can be the use of smartphone applications for angle measurement, which can be practical and time-saving.^3,4

Advances in image reproduction and sharing technology enable the remote radiographic assessment of pathologies. The use of mobile imaging is becoming increasingly prevalent in the clinical setting, allowing doctors to transmit images quickly through multimedia messaging technology.⁴ In particular, emergency physicians, as well as surgery and orthopedics residents, generally rely on their cell phones to transmit X-ray and computed tomography images to consultants at home or to senior residents.⁶

In this context, the present study aimed to develop and evaluate the usability and perceived usefulness of an application for a portable device (smartphone), in addition to performing its validation. The application proposes the interpretation of radiographic imaging examinations through measurements of size, length, distance, and angle.

Materials and Methods

The present survey comprised an analytical, transversal, descriptive, and quantitative evaluation. The first step consisted of selecting the target group and choosing the topics to be explored in the application, according to the daily needs of orthopedists and orthopedic residents. Next, the development (execution and complementation) of the application was initiated with the participation of a multidisciplinary staff composed of professionals in the areas of computing and medicine (orthopedics).

The application, known as OrtopeX, proposes the evaluation and interpretation of radiographic imaging examinations, with distance assessments and angle measuring (Fig. 1). To do so, the application uses photographs of printed examination images and photographs received by e-mail, WhatsApp, or that are stored on the physician's smartphone.

Fig. 1.

Illustration of the application. Measuring of the desired angle. Guides the user to trace the lines of the angles.

In the next step, in which the user has already obtained the value of the desired measurement, a flowchart is provided, with reference values for normality and abnormality and related treatment options, assisting in decision-making in orthopedic pathologies. In addition to assessing measurements, the topic “Relevant Information” was implemented in the software, with instructions and illustrations for performing these measurements as an option, if needed by the user, due to lack of habit or knowledge regarding the pathology proposed for measurement (Fig. 2).

Fig. 2.

Illustration of the application. Selection of the body segment and the angle to be measured and the screen showing the topic “Relevant Information.”

USABILITY TEST AND PERCEPTION OF USEFULNESS

The usability of an application can be assessed by harmonizing its main attributes: the potential of the interface to be understood by the user (“effectiveness”), through easy navigation (“efficiency”), which should be user-friendly (“satisfaction”).^7
–9

The sample required for the usability and usefulness tests was composed of residents and orthopedists with degrees in orthopedics and traumatology from the Brazilian Society of Orthopedics and Traumatology. According to Sauro,¹⁰ the minimum sample size to produce statistically significant data is 12 participants. The professionals took part in the test by signing a free and informed consent form and by attesting their interest in taking the test. The sample size was 13 residents and 13 orthopedists.

In the next stage of the study, the usability and perception of usefulness tests were conducted using a survey based on existing questionnaires, which consisted of four parts: Part 1, developed to obtain information concerning experiences with the profession and the experience of each participant using mobile applications, in general. Part 2, based on the System Utility Score (SUS) questionnaire,¹¹ validated in Portuguese¹² in 2011, which aims to collect information regarding the easiness of use (usability) of the developed application and the simplicity in learning how to use it (ease of learning). Part 3, based on Davis's Technology Acceptance Model,¹³ aimed at identifying the level of usefulness of the system, perceived by users (perception of usefulness) when measuring angles and interpreting images with the use of the application. Part 4, comprised two subjective questions, which record the participants' opinions regarding the positive and negative points, and suggestions of improvements for the application.

The usability test is characterized as a method of easy application to check a given system's usability, where each question contains five answer options that follow the Likert scale¹⁴ of five points (disagree entirely, disagree, indifferent, agree, and agree entirely). The SUS scale is characterized as an easy-to-apply model for assessing the usability of systems.¹⁵

In the SUS calculation, one point was subtracted from the score in the positive responses (odds), while five points were subtracted from the score in the negative responses (evens). Afterward, the resulting scores were summed, and the value was multiplied by 2.5 to obtain the final score, which can range from 0 to 100. As stated by Sauro and Lewis, SUS focuses on analyzing two main factors of the system: usability and learning capacity.^10,16,17

Considering the questions that assessed the usefulness of the system, the sum of the 4 responses was multiplied by 5, thus obtaining the final score, which can range from 0 to 100.

All tests were applied by the same evaluator to minimize any bias that may interfere with the final evaluation of the product.¹⁸ The time required for the angles to be measured by the application was not evaluated.

The final scores of the SUS and the questions that assessed the usefulness of the system were expressed as means and standard deviations, submitted to the Kolmogorov–Smirnov normality test, and compared between groups using the Mann–Whitney test (nonparametric data). The SUS and the questions that assessed system usefulness were compared with each other using the Wilcoxon test. The obtained data were tabulated in Microsoft Excel and exported to the Statistical Package for the Social Sciences (SPSS) software, version 20.0 for Windows, adopting a 95% confidence interval (CI), with significance established at p < 0.05.

APPLICATION VALIDATION

After the application was developed, we proceeded with its validation. Angles were measured by five randomly selected orthopedic doctors with degrees in orthopedics and traumatology by the Brazilian Society of Orthopedics and Traumatology.

Initially, 10 angles were selected from among the list of angles available in the application. Two angles of each segment were chosen (shoulder and elbow, hand and wrist, pelvis, knee, foot and ankle), and then different radiographs of the body segments were selected, totaling 10 distinct angles, each one with a corresponding radiograph. Each angle/radiograph was numbered from 1 to 10.

Angle measurement was performed in two moments. At first, each orthopedist measured angles 1 to 5 using the goniometer, and angles 6 to 10 with the application. In the second moment, the same professionals carried out the assessment through method inversion, that is, angles 6 to 10 were measured using the goniometer, and 1 to 5 with the application. This second moment was conducted more than 60 days after the first measurement to avoid the possible influence of memorizing the angles measured in the first moment.

In the next step, the statistical analysis of the angles obtained by the two measurement methods took place. The angles were expressed as mean and standard deviation and were submitted to the paired t-test.

The data were tabulated in Microsoft Excel and exported to the Statistical Package for the Social Sciences (SPSS) software, version 20.0 for Windows, adopting a 95% CI and significance of p < 0.05.

Results

The results revealed that the usability assessment of the application was suitable, with an average score of 84.5, based on the responses from the SUS questionnaire (Table 1). No statistically significant difference was observed when comparing the SUS between orthopedic residents and orthopedists.

Table 1.

Outline of Part 1 of the Assessment

VARIABLE	TOTAL SAMPLE	RESIDENTS	ORTHOPEDISTS
Sample size	26	13	13
Mean SUS	84.5	85.9	83.0
Confidence interval	70.7–98.3	76.3–95.5	69.7–96.3
Margin of error	13.8	9.6	13.3
Level of significance, %	95	95	95

Source: provided by the author.

SUS, System Utility Score.

In accordance with the scale described by Bangor, Kortum, and Miller (2009),¹⁹ the OrtopeX application fell into category B, which corresponds to an “excellent” result, according to the better level of usability in all classes (residents, orthopedists, and overall). Meanwhile, Sauro and Lewis¹⁶ developed a scale on which OrtopeX obtained an A+ score for residents and overall, considered the best level of usability according to this form of categorization. When the orthopedists were analyzed separately, the application received grade A (Table 2).

Table 2.

Classification of the Application's Usability According to the Scales Developed by Bangor et al.¹⁹ and Sauro and Lewis¹⁶

OrtopeX SYSTEM	SUS	CLASSIFICATION BY BANGOR, KORTUM, AND MILLE RANGE SCORE	CLASSIFICATION BY SAURO AND LEWIS RANGE SCORE
Overall	84.5	80–90 B (Excellent)	84.1–100 A+
Residents	85.6	80–90 B (Excellent)	84.1–100 A+
Orthopedists	83.0	80–90 B (Excellent)	80.8–84 A

Source: provided by the author.

Bangor, Kortum, and Mille's classification—Score B—equivalent to grade “excellent.”

Sauro and Lewis' classification—A+ represents the highest possible score, and D the lowest.

Sample stratification and comparison were carried out according to the following criteria: use or not of a medical application on a smartphone, used operating system, year of residency in orthopedics (for residents), years since the end of medical residency (for orthopedists), and whether the participant uses a goniometer in daily professional life. A statistically significant difference was observed regarding the SUS for participants who use the goniometer daily in relation to those who do not use it with such periodicity (p = 0.020) and for users who have already used a smartphone medical application in comparison with those who do not use such technology (p = 0.032).

The perception of usefulness assessment of the application is shown in Table 3. The results demonstrate that, in the usefulness assessment, the OrtopeX application received a score of 82.88% positivity, when adding the responses of the two evaluated groups (residents and orthopedists). When separating the sample, residents had 90.38% positive responses, while orthopedists had 75.38%.

Table 3.

Results from the Assessment of the Perception of Usefulness of the Application

ASSESSMENT CRITERIA	TAM (0–100%)
ASSESSMENT CRITERIA	RESIDENTS	ORTHOPEDISTS	p
1. It seems to be a useful technology for measuring angles in radiographic images.	92.3	78.4	0.045
2. I believe that standardization through the step-by-step method proposed by the application may assist in the learning process of residents in orthopedics and traumatology regarding angle measurements in orthopedics.	87.6	84.6	0.644
3. It helped me to better understand the concepts related to angle measurements in radiographic images.	92.3	66.15	<0.001
4. Would you use the application on a daily basis in emergency situations or at clinical practices?	89.2	72.3	0.015
General	90.3 ± 8.2	75.3 ± 13.6	0.002

Source: provided by the author.

TAM, Technology Acceptance Model.

When comparing the responses of the classes of professionals to the questions in which the perception of usefulness is evaluated, based on Davis's model,¹³ there was a statistically significant difference between the topics “It seems to be a useful technology for measuring angles in radiographic images” (p = 0.045); “It helped me to better understand the concepts related to angle measurements in radiographic images” (p < 0.001); “Would you use the application on a daily basis in emergency situations or clinical practices” (p = 0.002); and the general assessment (p = 0.002). There was no statistically significant difference concerning the topic “I believe that standardization through the step-by-step method proposed by the application may assist in the learning process of residents in orthopedics and traumatology regarding angle measurements in orthopedics” (p = 0.644).

In the perception of usefulness analysis, sample stratification and comparison were carried out according to some criteria, aside from the orthopedists or residents: use or not of a medical application on a smartphone, used operating system, year of residency in orthopedics (for residents), years since the end of medical residency (for orthopedists), and whether the participant uses the goniometer in daily professional life. A statistically significant difference (p = 0.006) was observed in the perception of usefulness only for participants who use the goniometer daily when compared with those who do not use the device with such frequency.

No significant differences were found when comparing the 10 angles measured by the professionals using the two evaluated methods (Table 4).

Table 4.

Outline of the Angles Determined by the Goniometer and the Application

	GONIOMETER	APPLICATION	p ^*
Angle 1	32.00 ± 0.71	32.40 ± 1.14	0.587
Angle 2	0.60 ± 0.55	0.60 ± 0.55	1.000
Angle 3	27.20 ± 1.48	26.60 ± 0.89	0.426
Angle 4	34.60 ± 0.55	35.20 ± 0.45	0.208
Angle 5	3.80 ± 0.45	4.80 ± 0.84	0.089
Angle 6	11.40 ± 0.89	10.80 ± 1.30	0.305
Angle 7	21.80 ± 1.10	21.80 ± 1.30	1.000
Angle 8	26.40 ± 0.89	28.40 ± 0.89	1.000
Angle 9	6.80 ± 1.10	6.00 ± 2.00	0.242
Angle 10	127.00 ± 1.00	127.60 ± 2.07	0.374

Source: provided by the author.

p < 0.05, paired t-test (mean ± standard deviation).

Discussion

The most traditional way of evaluating and measuring angles in orthopedics has been, and still is, performed manually. To do so, the professional needs the printed image, a negatoscope, and instruments such as pencils, rulers, protractors, goniometers, and an eraser. With a planning strategy in mind, the orthopedist scores points, draws lines, and, through these parameters, estimates the conduct to be followed. The main advantage of this type of planning is the low cost.^4,20 However, since several materials are required, doctors may decide not to use this method, induced by the lack of necessary equipment, which could compromise the results in decision-making.^5,20

Currently, the development of technologies and applications focused on health (m-health) has been increasing, thus contributing to the construction of a new modality of health care.^21,22

The present study outlines the development of the OrtopeX mobile application and evaluates its usability and usefulness according to orthopedists and residents. It also assesses the efficiency of measurements of orthopedic radiographic angles.

The good results of the application in terms of the usability evaluation deserve emphasis. Usability is a central idea in human/computer interaction. It is defined as the effectiveness, efficiency, and satisfaction with which specific users obtain their goals in particular environments.²³ The mean SUS of 84.5 categorizes the usability of the application as “excellent.” This result is similar to studies that evaluated the usability of medical applications through the SUS questionnaire and obtained results in the same category (excellent), according to the classification utilized by Bangor et al.¹⁹ Among these studies are two that stand out, one that evaluated the usability of an application for planning total knee arthroplasty,²⁰ and another that assessed the usability of an application used as a support in decision-making in preoperative evaluations using mobile devices.²⁴ These data highlight the effectiveness and efficiency of several applications, in addition to user satisfaction.

To analyze the perception of usefulness of the application, Davis's model¹³ was used, aiming at identifying the level of usefulness of the system, perceived by users (perception of usefulness), when measuring angles and interpreting images using the application. This is a useful system that helps users to improve their performance when conducting a given task. According to the author, usability constitutes an essential factor, but if the user does not realize the usefulness of the system, he/she will not use it.^25,26

Considering the factors that implied a good evaluation of usefulness (the possibility of learning and supporting assistance information for the measurement of angles, the better understanding of the results, aiding in decision-making, and the possibility of use in emergency environments and practices), the OrtopeX application has proven to be a useful system, without neglecting the margin for improvement obtained from the participants' assertions and suggestions. The discrepancy in the results is noteworthy when separating the sample into residents and orthopedists: residents had 90.38% positivity in the answers, and orthopedists had 75.38%. As a result, a higher general acceptance by residents was observed. However, the answer in question 12, which asks the user if the step-by-step method proposed by the application could assist in the learning process of residents, was the only one in the perception of usefulness assessment that did not present a statistically significant difference. Due to the higher percentage of positivity among the questions answered by orthopedists, it can be noted that even this class believes that the application's primary purpose is of learning for residents, rather than useful for themselves. Thus, the relevance of the application in the teaching and learning process can be verified, since, in addition to the practical part with angle measuring, there is also theoretical content regarding each angle available in the software.

Balance is still sought between the scientific teaching offered by universities and training that develops the competencies and skills essential to medical practice.²⁷ The responsibility of preceptors and teachers involved in medical residency increases, and it becomes even more relevant to discuss how the teaching-learning process takes place in this postgraduate modality. Active methodologies, such as that proposed by the use of the OrtopeX application to complement this process, should be applied, considering professional training during medical residency as an educational process and viewing something more than training, basing the teaching and learning process on the coordinated development of different forms of knowledge and skills, and on the acquisition of technical and relational attributes.

Therefore, greater perception of usefulness when evaluating residents and professionals in training shows that the influence of technology on medical learning is a trend and immerses itself in the discussion of mobile learning (or m-learning) that arose from the use of mobile and wireless technologies as part of an integrated learning model.²⁸ According to Peng et al.,²⁹ m-learning is an evolution of e-learning through the addition of mobile technologies. A range of methods/procedures have been developed in the past decades to build new technological tools that improve the teaching-learning process and the performance of users in the most diverse contexts. Advantages such as convenience, efficiency, and the potential for accelerating learning are easily noticed by users. However, disadvantages such as distraction, dependence, and poor regulation of knowledge can render learning and using mobile applications questionable.³⁰

Following the analysis of perception of usefulness and usability, sample stratification and comparison were carried out according to some criteria, aside from the orthopedists or residents. A statistically significant difference was observed only for the participants who use goniometers daily in relation to those who do not use it with such frequency. These results show that the participants who use the device more frequently both find the application easier to handle and more useful in the orthopedic routine. The better response regarding usability may have occurred due to the user's greater familiarity with the measured angles, rendering the process more agile by taking steps related to research/instructions concerning the angles.

For use in orthopedic environments, it is necessary to evaluate the reliability of smartphone applications that simulate the goniometer. In a study by Ege et al.³¹ and Qiao et al.,³² the authors compared the measurements made using a smartphone with computerized and manual measurement radiographs. The agreement between the iPhone and the other angle measurements was considered excellent.

In this study, to evaluate the reproducibility and replicability of the assessed measurements with the application in comparison with those performed with the goniometer, the application was also validated. The obtained result was similar to existing literature, with no significant difference between the goniometer measurements when compared with those measured by the OrtopeX application, which proved to be reliable when compared with the manual measurement method.

Conclusion

It can be concluded that the developed application contributes to situations where there is a need to reliably measure angles on radiographs, assisting the orthopedist or resident physician in making therapeutic decisions, in addition to enabling its use as an instrument to favor the processes of teaching and learning in orthopedics.

Footnotes

Disclosure Statement

No competing financial interests exist.

Funding Information

The authors report no conflict of interest. F.S.M. reports no salary from the Christus University Center during the conduct of this study. Dr. Edgar Marçal reports salary from Federal University of Ceará during the conduct of the study. Dr. Paulo G. de B. Silva reports salary from Christus University Center during the conduct of the study. J.P.M.L.R. reports salary from Christus University Center during the conduct of the study.

References

Lam

, Preketes

, Gates

. Mobile phone photo messaging assisted communication in the assessment of hand trauma. ANZ J Surg, 2004; 74:598–602.

Tay

, Zhang

, Gayagay

. Use of iPhone technology in improving acetabular component position in total hip arthroplasty. Arthroplasty Today, 2017; 3:167–170.

Dinato

MCM

, de Faria Freitas

, Milano

, Valloto

Jr ., Ninomiya

, Pagnano

RG.

Reliability of two smartphone applications for radiographic measurements of Hallux Valgus angles. J Foot Ankle Surg, 2017; 56:230–233.

Paryavi

, et al. Reliability and effectiveness of smartphone technology for the diagnosis and treatment planning of pediatric elbow trauma. J Pediatr Orthop, 2016; 36:483–487.

Popat

, Mohan

, Branford

. Current uses of smartphones and apps in orthopaedic surgery. Br J Hosp Med, 2013; 74:672–676.

Sorelle

Take two apps and call my iPhone in the morning. Emerg Med News, 2010; 32:1–7.

Nielsen

Ten Usability Heuristics. 1995. Available at https://www.nngroup.com/articles/ten-usability-heuristics (last accessed October 27, 2018 ).

Nielsen

Usability 101: Introduction to usability. 2003. Available at www.useit.com/alertbox/20030825.html (last accessed October 27, 2018 ).

Nielsen

, Budiu

. Mobile usability, [In Portuguese]., 1st ed. Rio de Janeiro: Elsevier, 2014. Translation: Sergio Facchim. Rev. Técnica: Yvi Alan da Silva Lacerda.

10.

Sauro

Measuring Usability with The System Usability Scale (SUS). 2011. Available at www.measuringusability.com/sus.php (last accessed October 27, 2018 ).

11.

Brooke

SUS-A quick and dirty usability scale. Usability Eval Industry, 1996; 189:4–7.

12.

Tenório

, et al. Development and evaluation of an electronic protocol for care and monitoring of patients with celiac disease. J Theoretical A Info, 2010; 17:210–220.

13.

Davis

FD.

Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q, 1989; 319–340.

14.

Likert

A technique for the measurement of attitudes. Arch Psychol, 1932; 22:55.

15.

Zbick

, et al. A web-based framework to design and deploy mobile learning activities: Evaluating its usability, learnability and acceptance. In: Proceedings—IEEE 15th International Conference on Advanced Learning Technologies: Advanced Technologies for Supporting Open Access to Formal and Informal Learning. Hualien, Taiwan: ICALT 2015, 2015:88–92.

16.

Sauro

, Lewis

. Quantifying the user experience: Practical statistics for user research. Waltham, MA: Morgan Kaufmann, 2012.

17.

Sauro

A practical guide to the system usability scale: Background, benchmarks & best practices. Denver, CO: Measuring Usability LLC, 2011.

18.

Hertum

, Jacobsen

. The evaluator effect: A chilling fact about usability evaluation methods. Int J Hum Comput Interact, 2001; 13:421–443.

19.

Bangor

, Kortum

, Miller

. Determining what individual SUS scores mean: Adding an adjective rating scale. J Usability Stud, 2009; 4:114–123.

20.

Nogueira

JBS

, et al. Evaluation of benefits and accuracy of a mobile application in planning total knee arthroplasties. Rev Bras Ortop, 2018; 53:142–150.

21.

Banos

, et al. Design, implementation and validation of a novel open framework for agile development of mobile health applications. Biomed Eng Online, 2015; 14:6–12.

22.

Buijink

AWG

, Visser

, Marshall

. Medical apps for smartphones: Lack of evidence undermines quality and safety. Evid Based Med, 2012; 18:90–92.

23.

Hornbaek

Current practice in measuring usability: Challenges to usability studies and research. Int J Hum Comput Stud, 2006; 64:79–102.

24.

Pereira

RVS

, Kubrusly

, Nogueira

, Gondim

VJT

, Marcal

. Development of an application to support in-service training of anesthesiologists on preoperative evaluation in a public hospital in Brazil. J Eval Clin Pract, 2019; 25:850–855.

25.

Akman

, Turhan

. User acceptance of social learning systems in higher education: An application of the extended Technology Acceptance Model. Innov Educ Teach Int, 2015; 54:229–237.

26.

Chang

, Huang

. Technology acceptance model, consumser personality and smartphone users' satisfaction. In: Robinson

Jr. L.

ed. Marketing Dynamism & Sustainability: Things Change, Things Stay the Same…. Developments in Marketing Science: Proceedings of the Academy of Marketing Science. Cham: Springer International Publishing, 2015;710–712.

27.

Heaven

, Clegg

, Maguire

. Transfer of communication skills training from workshop to workplace: The impact of clinical supervision. Patient Educ Couns, 2006; 60:313–325.

28.

Marçal

, Andrade

, Rios

. Learning using Mobile Devices with Virtual Reality Systems. Renote, 2005; 3:1–7.

29.

Peng

, et al. Ubiquitous knowledge construction: Mobile learning re-defined and a conceptual framework. Innov Educ Teach Int, 2009; 46:171–183.

30.

Bullock

, Webb

. Technology in postgraduate medical education: A dynamic influence on learning?. Postgrad Med J, 2015; 91:646–650.

31.

Ege

, Kose

, Koca

, Demiralp

, Basbozkurt

. Use of the iPhone for radiographic evaluation of hallux valgus. Skeletal Radiol, 2012; 42:269–273.

32.

Qiao

, Liu

, Xu

, Wu

, Zheng

, Zhu

, Qian

, Qiu

. Reliability analysis of a smartphone-aided measurement method for the Cobb angle of scoliosis. J Spinal Disord Tech, 2012; 25:67–71.