REID (Research and Evaluation in Education)
Keywords
e-learning, satisfaction, usage-based metrics, SUS
Document Type
Article
Abstract
Nowadays, in the pandemic of COVID-19, e-learning systems have been widely used to facilitate teaching and learning processes between lecturers and students. Assessing lecturers' and students' satisfaction with e-learning systems has become essential in improving the quality of education for higher learning institutions. Most existing approaches have attempted to assess users' satisfaction based on System Usability Scale (SUS). On the other hand, different studies proposed usage-based metrics (completion rate, task duration, and mouse or cursor distance) which assess users' satisfaction based on how they use and interact with the system. However, the cursor or mouse distance metric does not consider the effectiveness of navigation in e-learning systems, and such approaches measure either lecturers' or students' satisfaction independently. Towards this end, we propose a lostness metric to replace the click or cursor distance metric for assessing lecturers' and students' satisfaction with using e-learning systems. Furthermore, to obtain a deep analysis of users' satisfaction, we tandem the usage-based metric (i.e., completion rate, task duration, and lostness) and the SUS metric. The evaluation results indicate that the proposed approach can precisely predict users' satisfaction with e-learning systems.
Page Range
118-131
Issue
2
Volume
7
Digital Object Identifier (DOI)
10.21831/reid.v7i2.39642
Source
https://journal.uny.ac.id/index.php/reid/article/view/39642
Recommended Citation
Sandiwarno, S. (2021). Empirical lecturers' and students' satisfaction assessment in e-learning systems based on the usage metrics. REID (Research and Evaluation in Education), 7(2), 118-131. https://doi.org/10.21831/reid.v7i2.39642
References
Ahn, J., Kim, K., & Proctor, R. W. (2018). Comparison of mobile web browsers for smartphones. Journal of Computer Information Systems, 58(1), 10-18. https://doi.org/10.1080/08874417.2016.1180652
AlGhannam, B. A., Albustan, S. A., Al-Hassan, A. A., & Albustan, L. A. (2017). Towards a standard Arabic system usability scale: Psychometric evaluation using communication disorder app. International Journal of Human-Computer Interaction, 34(9), 1-6. https://doi.org/10.1080/10447318.2017.1388099
Almarashdeh, I. (2016). Sharing instructors experience of learning management system: A technology perspective of user satisfaction in distance learning course. Computers in Human Behavior, 63, 249-255. https://doi.org/10.1016/j.chb.2016.05.013
Asoodar, M., Vaezi, S., & Izanloo, B. (2016a). Framework to improve e-learner satisfaction and further strengthen e-learning implementation. Computers in Human Behavior, 63, 704-716. https://doi.org/10.1016/j.chb.2016.05.060
Asoodar, M., Vaezi, S., & Izanloo, B. (2016b). Framework to improve e-learner satisfaction and further strengthen e-learning implementation. Computers in Human Behavior, 63, 704-716. https://doi.org/10.1016/j.chb.2016.05.060
Berkman, M. Ä°., Karahoca, D., & Karahoca, A. (2018). A measurement and structural model for usability evaluation of shared workspace groupware. International Journal of Human-Computer Interaction, 34(1), 35-56. https://doi.org/10.1080/10447318.2017.1326578
Borkowska, A., & Jach, K. (2017). Pre-testing of Polish translation of System Usability Scale (SUS). Advances in Intelligent Systems and Computing, 521, 143-153. https://doi.org/10.1007/978-3-319-46583-8_12
Bringula, R. P. (2013). Influence of faculty- and web portal design-related factors on web portal usability: A hierarchical regression analysis. Computers and Education, 68, 187-198. https://doi.org/10.1016/j.compedu.2013.05.008
Caputi, V., & Garrido, A. (2015). Student-oriented planning of e-learning contents for Moodle. Journal of Network and Computer Applications, 53, 115-127. https://doi.org/10.1016/j.jnca.2015.04.001
Casamayor, A., Amandi, A., & Campo, M. (2009). Intelligent assistance for teachers in collaborative e-learning environments. Computers and Education, 53(4), 1147-1154. https://doi.org/10.1016/j.compedu.2009.05.025
Chen, P.-H., & Adesope, O. (2016). The effects of need satisfaction on EFL online learner satisfaction. Distance Education, 37(1), 89-106. https://doi.org/10.1080/01587919.2016.1155962
Cohen, A., & Baruth, O. (2017). Personality, learning, and satisfaction in fully online academic courses. Computers in Human Behavior, 72, 1-12. https://doi.org/10.1016/j.chb.2017.02.030
Curcio, K., Santana, R., Reinehr, S., & Malucelli, A. (2019). Usability in agile software development: A tertiary study. Computer Standards and Interfaces, 64, 61-77. https://doi.org/10.1016/j.csi.2018.12.003
Gameel, B. G. (2017). Learner satisfaction with massive open online courses. American Journal of Distance Education, 31(2), 98-111. https://doi.org/10.1080/08923647.2017.1300462
Haron, H., Aziz, N. H. N., & Harun, A. (2017). A conceptual model participatory engagement within e-learning community. Procedia Computer Science, 116, 242-250. https://doi.org/10.1016/j.procs.2017.10.046
Harrati, N., Bouchrika, I., Tari, A., & Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Computers in Human Behavior, 61, 463-471. https://doi.org/10.1016/j.chb.2016.03.051
Hong, J. C., Tai, K. H., Hwang, M. Y., Kuo, Y. C., & Chen, J. S. (2017). Internet cognitive failure relevant to users' satisfaction with content and interface design to reflect continuance intention to use a government e-learning system. Computers in Human Behavior, 66, 353-362. https://doi.org/10.1016/j.chb.2016.08.044
Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student perception of Moodle learning management system: A satisfaction and significance analysis. Interactive Learning Environments, 23(4), 515-527. https://doi.org/10.1080/10494820.2013.788033
Ifinedo, P., Pyke, J., & Anwar, A. (2018). Business undergraduates' perceived use outcomes of Moodle in a blended learning environment: The roles of usability factors and external support. Telematics and Informatics, 35(1), 93-102). https://doi.org/10.1016/j.tele.2017.10.001
Kerimbayev, N., Kultan, J., Abdykarimova, S., & Akramova, A. (2017). LMS Moodle: Distance international education in cooperation of higher education institutions of different countries. Education and Information Technologies, 22(5), 2125-2139. https://doi.org/10.1007/s10639-016-9534-5
Kim, J. (2013). Influence of group size on students' participation in online discussion forums. Computers & Education, 62, 123-129. https://doi.org/10.1016/j.compedu.2012.10.025
Koohang, A., Paliszkiewicz, J., Gołuchowski, J., & Nord, J. H. (2016). Active learning for knowledge construction in e-learning: A replication study. Journal of Computer Information Systems, 56(3), 238-243. https://doi.org/10.1080/08874417.2016.1153914
Ku, H. Y., Tseng, H. W., & Akarasriworn, C. (2013). Collaboration factors, teamwork satisfaction, and student attitudes toward online collaborative learning. Computers in Human Behavior, 29(3), 922-929. https://doi.org/10.1016/j.chb.2012.12.019
Liberona, D., & Fuenzalida, D. (2014). Use of Moodle platforms in higher education: A Chilean case. Communications in Computer and Information Science, 446, 124-134. https://doi.org/10.1007/978-3-319-10671-7_12
Lin, J. W. (2018). Effects of an online team project-based learning environment with group awareness and peer evaluation on socially shared regulation of learning and self-regulated learning. Behaviour and Information Technology, 37(5), 445-461. https://doi.org/10.1080/0144929X.2018.1451558
Mentes, A., & Turan, A. H. (2012). Assessing the usability of university websites: An empirical study on Namik Kemal University. Turkish Online Journal of Educational Technology, 11(3), 61-69. http://www.tojet.net/articles/v11i3/1136.pdf
Muñoz, A., Delgado, R., Rubio, E., Grilo, C., & Basto-Fernandes, V. (2017). Forum participation plugin for Moodle: Development and discussion. Procedia Computer Science, 121, 982-989. https://doi.org/10.1016/j.procs.2017.11.127
Navimipour, N. J., & Zareie, B. (2015). A model for assessing the impact of e-learning systems on employees' satisfaction. Computers in Human Behavior, 53, 475-485. https://doi.org/10.1016/j.chb.2015.07.026
Sadikin, M. (2017). Mining relation extraction based on pattern learning approach. Indonesian Journal of Electrical Engineering and Computer Science, 6(1), 50-57. https://doi.org/10.11591/ijeecs.v6.i1.pp50-57
Sadikin, M., Fanany, M. I., & Basaruddin, T. (2016). A new data representation based on training data characteristics to extract drug name entity in medical text. Computational Intelligence and Neuroscience, 3483528. https://doi.org/10.1155/2016/3483528
Sandiwarno, S. (2016). Perancangan model e-learning berbasis collaborative video conference learning guna mendapatkan hasil pembelajaran yang efektif dan efisien. Jurnal Ilmiah FIFO, 8(2), 191-200. https://doi.org/10.22441/fifo.v8i2.1314
Smith, P. A. (1996). Towards a practical measure of hypertext usability. Interacting with Computers, 8(4), 365-381. https://doi.org/10.1016/S0953-5438(97)83779-4
Sun, J. (2016). Multi-dimensional alignment between online instruction and course technology: A learner-centered perspective. Computers and Education, 101, 102-114. https://doi.org/10.1016/j.compedu.2016.06.003
Tullis, T., & Albert, B. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.). Elsevier. https://doi.org/10.1016/C2011-0-00016-9
Wagner, N., Hassanein, K., & Head, M. (2014). The impact of age on website usability. Computers in Human Behavior, 37, 270-282. https://doi.org/10.1016/j.chb.2014.05.003
Zhang, S., Liu, Q., Chen, W., Wang, Q., & Huang, Z. (2017). Interactive networks and social knowledge construction behavioral patterns in primary school teachers' online collaborative learning activities. Computers and Education, 104, 1-17. https://doi.org/10.1016/j.compedu.2016.10.011