Using a Two-Stage System Evaluation Approach to Explore the New Adoption of an Information System in the Mobile Era

Abstract

Currently, e-learning systems are being widely used in all stages of education. However, it is difficult for school administrators to accurately assess the actual usage performance of a new system, especially when an organization wishes to update the system for users from different backgrounds using new devices such as smartphones. To allow school administrators to conduct upgrades of e-learning systems that take into consideration students' current usage conditions, this study proposed a two-stage system evaluation approach to explore the adoption of new systems. We collected 352 samples in Stage I. The goal of this Stage I was to propose a research model to understand the usage intentions of college students toward campus e-learning systems and also the factors which showed significant differences between PC and smartphone usage. A total of 30 trained students participated in Stage II. The goal of Stage II was to propose a system performance evaluation method to evaluate the performance of the new and existing systems according to the concerned factors of smartphone users after actual system use. Finally, based on our research model and system performance evaluation method, we put forward conclusions and suggestions that schools could use as references for future system procurements and updates.

Keywords

system performance evaluation information system evaluation prepurchase assessment intention to use system e-learning with smartphones learning management systems mobile learning

Get full access to this article

View all access options for this article.

References

Adamson

K. A.

Prion

(2013) Reliability: Measuring internal consistency using Cronbach's α. Clinical Simulation in Nursing 9(5): 179–180. doi:10.1016/j.ecns.2012.12.001.

Agarwal

Ferratt

T. W.

(2001) Crafting and HR strategy to meet the need for IT workers. Communications of the ACM 44(7): 58–64.

Arbaugh

J. B.

Duray

(2002) Technological and structural characteristics, student learning and satisfaction with web-based courses an exploratory study of two on-line MBA programs. Management Learning 33(3): 331–347.

Ashbrook, D., Baudisch, P., & White, S. (2011). Nenya: subtle and eyes-free mobile input with a magnetically-tracked finger ring. In CHI '11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, Canada, 7–12 May, pp. 2043–2046.

Balakrishnan

Paul

H. P.

(2008) A study of the effect of thumb sizes on mobile phone texting satisfaction. Journal of Usability Studies 3(3): 118–128.

Bauldry

(2015) Structural equation modeling. In: Wright

J. D.

(ed.) International encyclopedia of the social & behavioral sciences, 2nd. Oxford: Elsevier, pp. 615–620. doi:10.1016/B978-0-08-097086-8.44055-9.

Bruun

Stage

(2015) New approaches to usability evaluation in software development: Barefoot and crowdsourcing. Journal of Systems and Software 105: 40–53. doi:10.1016/j.jss.2015.03.043.

Chang

C. C.

(2013) Exploring the determinants of e-learning systems continuance intention in academic libraries. Library Management 34(1/2): 40–55. doi:10.1108/01435121311298261.

Chen

L. H.

(2008) Job satisfaction among information system (IS) personnel. Computers in Human Behavior 24(1): 105–118.

10.

Chen

R. S.

Liu

I. F.

(2013) Research on the effectiveness of information system in reducing the rural-urban knowledge divide. Computer & Education 63: 437–445. doi:10.1016/j.compedu.2013.01.002.

11.

Chen

Rau

P. L.

Chen

(2014) How to design finger input of Chinese characters: A literature review. International Journal of Industrial Ergonomics 44(3): 428–435. doi:10.1016/j.ergon.2013.11.009.

12.

Cho

Cheng

T. E.

Lai

W. J.

(2009) The role of perceived user-interface design in continued usage intention of self-paced e-learning tools. Computers & Education 53(2): 216–227.

13.

Clark

R. C.

Mayer

R. E.

(2011) E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning, John Wiley & Sonsdoi:10.1002/pfi.4930420510.

14.

Clarke

(2004) Measuring API usability. Doctor Dobbs Journal 29(5): 1–5.

15.

Curran

E. A.

Stokes

M. J.

(2003) Learning to control brain activity: A review of the production and control of EEG components for driving brain–computer interface (BCI) systems. Brain and Cognition 51(3): 326–336.

16.

Davis

F. D.

(1989) Perceived usefulness, perceived ease of use, and user acceptance of information system. MIS Quarterly 13(3): 319–340.

17.

Echeverría

García-Campo

Nussbaum

Gil

Villalta

Améstica

Echeverría

(2011) A framework for the design and integration of collaborative classroom games original research article. Computers & Education 57(1): 1127–1136. doi:10.1016/j.compedu.2010.12.010.

18.

Egea

J. M. O.

González

M. V. R.

(2011) Explaining physicians' acceptance of EHCR systems: An extension of TAM with trust and risk factors. Computers in Human Behavior 27(1): 319–332. doi:10.1016/j.chb.2010.08.010.

19.

Fornell

Larcker

(1981) Evaluating structural equation models with unobservable variable and measurement error. Journal of Marketing Research 18: 39–50.

20.

Gandhi

(2014) Brain-computer interfacing for assistive robotics: Electroencephalograms, recurrent quantum neural networks, and user-centric graphical interfaces, Academic Press, pp. 95–115. doi:10.1016/B978-0-12-801543-8.00004-1.

21.

Godwin-Jones

(2008) Emerging technologies mobile-computing trends: Lighter, faster, smarter. Language Learning & Technology 12(3): 3–9.

22.

Gong, J. & Tarasewich, P. (2004). Guidelines for handheld mobile device interface design. In Proceedings of DSI 2004 Annual Meeting, Boston, U.S.A., 20–23 November, pp. 3751–3756. Retrieved from https://personal.cis.strath.ac.uk/sotirios.terzis/classes/52.504/2010/GuidelinesGongTarase.pdf.

23.

Green, K. C., Armstrong, J. S., & Graefe, A. (2007). Methods to elicit forecasts from groups: Delphi and prediction markets compared. Retrieved from http://repository.upenn.edu/marketing_papers/157.

24.

Horsky

McColgan

Pang

J. E.

Melnikas

A. J.

Linder

J. A.

Schnipper

J. L.

Middleton

(2010) Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles. Journal of Biomedical Informatics 43(5): 782–790. doi:10.1016/j.jbi.2010.05.010.

25.

Hoyle

R. H.

Panter

A. T.

(1995) Writing about structural equation models, Thousand Oaks, CA: Sage.

26.

Hsiao

M. H.

Chen

L. C.

(2015) Smart phone demand: An empirical study on the relationships between phone handset, Internet access and mobile services. Telematics and Informatics 32: 158–168. doi:10.1016/j.tele.2014.06.001.

27.

Ingrid

G. L.

Hicks

(2015) The participatory design of a performance oriented monitoring and evaluation system in an international development environment. Evaluation and Program Planning 48: 21–30. doi:10.1016/j.evalprogplan.2014.09.003.

28.

Kelloway

E. K.

(1998) Using LISREL for structural equation modeling: A researcher's guide, Thousand Oaks, CA: Sage.

29.

Kelly

Novak

(2012) What cost development? Building performance in transitional societies. Performance 4(2): 70–77.

30.

Kim

Jung

E. S.

(2014) Optimal control location for the customer-oriented design of smart phones. Information Sciences 257: 264–275. doi:10.1016/j.ins.2013.05.034.

31.

Konu

(2015) Developing nature-based tourism products with customers by utilising the Delphi method. Tourism Management Perspectives 14: 42–54. doi:10.1016/j.tmp.2015.03.003.

32.

Kunz

(2015) Objectivity and subjectivity in performance evaluation and autonomous motivation: An exploratory study. Management Accounting Research 27: 27–46. doi:10.1016/j.mar.2015.01.003.

33.

Lai

T. L.

(2004) Service quality and perceived value's impact on satisfaction, intention and usage of short message service (SMS). Information Systems Frontiers 6(4): 353–368.

34.

Lan

Y. J.

Sung

Y. T.

Chang

K. E.

(2007) A mobile-device-supported peer-assisted learning system for collaborative early EFL reading. Language Learning & Technology 11(3): 130–151.

35.

Lau

C. M.

Oger

(2012) Behavioral effects of fairness in performance measurement and evaluation systems: Empirical evidence from France Original. Advances in Accounting 28(2): 323–332. doi:10.1016/j.adiac.2012.09.009.

36.

S. C. S.

(2014) Adoption of three new types of computers in Taiwan: Tablet PCs, netbooks, and smart phones. Computers in Human Behavior 35: 243–251. doi:10.1016/j.chb.2014.03.001.

37.

Linstone

H. A.

Turoff

(1975) The Delphi method: Techniques and applications, Boston, MA: Addison-Wesley.

38.

Liu

I. F.

Chen

M. C.

Sun

Y. S.

Wible

Kuo

C. H.

(2010) Extending the TAM model to explore the factors that affect Intention to use an online learning community. Computers & Education 54(2): 600–610. doi:10.1016/j.compedu.2009.09.009.

39.

Metros

S. E.

Hedberg

J. G.

(2002) More than just a pretty (inter) face: The role of the graphical user interface in engaging e-learners. Quarterly Review of Distance Education 3(2): 191–205.

40.

Mijailović

Milićev

(2014) Empirical analysis of GUI programming concerns. International Journal of Human-Computer Studies 72(10–11): 757–771. doi:10.1016/j.ijhcs.2014.04.002.

41.

Mohammadi

(2015) Factors affecting the e-learning outcomes: An integration of TAM and IS success model. Telematics and Informatics 32(4): 701–719. doi:10.1016/j.tele.2015.06.004.

42.

Muji

(2015) Logical operators for the data-oriented design of the user interfaces. Procedia Technology 19: 810–815. doi:10.1016/j.protcy.2015.02.116.

43.

Neiderman

Sumner

(2004) Effects of tasks, salaries, and shocks on job satisfaction among IS professionals. Information Resources Management Journal 17(4): 49–72.

44.

Ngai

E. W. T.

Poon

J. K. L.

Chan

Y. H. C.

(2007) Empirical examination of the adoption of WebCT using TAM. Computers & Education 48: 250–267.

45.

Nielsen, J. (2009). Mobile usability, alertbox. Retrieved from http://www.useit.com/alertbox/mobile-usability-study-1.html.

46.

Nielsen, J. (2011). Mobile usability update, alertbox. Retrieved from http://www.useit.com/alertbox/mobile-usability.html.

47.

Nunnally

J. C.

Bernstein

I. H.

(1994) Psychometric theory, 3rd ed. New York, NY: McGraw-Hill.

48.

Nuzzo, R. L. (2014). P values, the ‘gold standard’ of statistical validity, are not as reliable as many scientists assume. International Weekly Journal of Science. Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700.

49.

Oha

Yoonb

S. J.

(2014) Validation of haptic enabling technology acceptance model (HE-TAM): Integration of IDT and TAM. Telematics and Informatics 31(4): 585–596. doi:10.1016/j.tele.2014.01.002.

50.

Petter

DeLone

McLean

(2008) Measuring information systems success: Models, dimensions, measures, and interrelationships. European Journal of Information Systems 17(3): 236–263.

51.

Reed

W. M.

Giessler

S. F.

(1996) Prior computer-related experiences and hypermedia metacognition. Computers in Human Behavior 11(3): 581–600.

52.

Rivera, J., & van der Meulen, R. (2014). Gartner says worldwide traditional PC, tablet, ultramobile and mobile phone shipments to grow 4.2 percent in 2014, Gartner. Retrieved from http://www.gartner.com/newsroom/id/2791017.

53.

Roca

J. C.

Chiu

C. M.

Martínez

F. J.

(2006) Understanding e-learning continuance intention: An extension of the technology acceptance model. International Journal of Human-Computer Studies 64(8): 683–696.

54.

Samsung. (2015). Samsung galaxy s4 laser keyboard. Retrieved from http://www.samsung.com/us/.

55.

Schumacker

R. E.

Lomax

R. G.

(1996) A beginner's guide to structural equation modeling, Mahwah, NJ: Lawrence Erlbaum Associates.

56.

Shi

W. Z.

F. G.

(2015) An investigation of the performance of novel chorded keyboards in combination with pointing input devices. Applied Ergonomics 46(A): 1–7. doi:10.1016/j.apergo.2014.05.004.

57.

Shih

P. C.

Munoz

Sánchez

(2006) The effect of previous experience with information and communication technologies on performance in a web-based learning program. Computers in Human Behavior 22(6): 962–970.

58.

Shneiderman

(2010) Designing the user interface: Strategies for effective human-computer interaction, Delhi, India: Pearson Education.

59.

Song

Singleton

E. S.

Hill

J. R.

Koh

M. H.

(2004) Improving online learning: Student perceptions of useful and challenging characteristics. The Internet and Higher Education 7(1): 59–70.

60.

Suh

Han

(2003) The impact of customer trust and perception of security control on the acceptance of electronic commerce. International Journal of Electronic Commerce 7(3): 135–161.

61.

Udo

G. J.

Bagchi

K. K.

Kirs

P. J.

(2011) Using SERVQUAL to assess the quality of e-learning experience. Computers in Human Behavior 27(3): 1272–1283. doi:10.1016/j.chb.2011.01.009.

62.

Wang

C. H.

(2015) Using quality function deployment to conduct vendor assessment and supplier recommendation for business-intelligence systems. Computers & Industrial Engineering 84: 24–31. doi:10.1016/j.cie.2014.10.005.

63.

Wang

S. K.

Yang

(2005) The interface design and the usability testing of a fossilization web-based learning environment. Journal of Science Education and Technology 14(3): 305–313.

64.

Wang

W. T.

Wang

C. C.

(2009) An empirical study of instructor adoption of web-based learning systems. Computers & Education 53(3): 761–774.

65.

Winkler

Kuklinski

C. P. J.

Moser

(2015) Decision making in emerging markets: The Delphi approach's contribution to coping with uncertainty and equivocality. Journal of Business Research 68(5): 1118–1126. doi:10.1016/j.jbusres.2014.11.001.

66.

Yang

J. Y.

Che

P. C.

(2015) Improving college students english learning with Dr. Eye Android Mid. The Turkish Online Journal of Educational Technology 14(2): 101–109.

67.

Yodsaneha

Sopeerak

(2013) Factors Influencing the success of Rajamangala University of Technology Thanyaburi's ubiquitous learning. Procedia—Social and Behavioral Sciences 103(26): 400–403. doi:10.1016/j.sbspro.2013.10.352.