Comparing Heuristic Evaluation and Cognitive Walkthrough in Educational Management Systems: An Expert-Based Assessment
DOI:
https://doi.org/10.22034/TJT.3.1.74Keywords:
User-Centered Design, Heuristic evaluation, Cognitive walkthrough, Computer-Assisted Instruction, Distance EducationAbstract
Designing user interfaces for educational platforms is a complex yet essential process for achieving effective learning outcomes. The primary objective of this study was to compare two expert-based usability evaluation methods, heuristic evaluation (HE) and cognitive walkthrough (CW), to determine their effectiveness in identifying usability problems in the Hamava and Sipad. Four evaluators independently assessed the systems using both HE and CW methods. The two approaches were compared based on the number and severity of usability problems, as well as the overall coverage of usability aspects. A total of 173 usability issues were identified across the two systems. The most prominent problems in both systems concerned memorability. There was no significant difference in performance between the two evaluation methods in either system (P > 0.05). CW is better suited to evaluating systems used by novice users, whereas HE is better suited to evaluating systems used by experienced users. These findings indicate that system developers should choose usability evaluation methods based on the target group and system objectives. The findings of this study can inform the adoption of more effective evaluation strategies and the selection of learning management systems with higher usability.
References
1. Abdurahman D, Ishlah MSN, Maesya A. Usability Evaluation of Learning Management System (LMS) Interface Universitas Pakuan Mobile Version with Heuristic Walkthrough Method. Komputasi: Jurnal Ilmiah Ilmu Komputer dan Matematika. 2025;22(1):108-19. DOI: https://doi.org/10.33751/komputasi.v22i1.25
2. Alihosseini S, Samad-Soltani T, Katebi K, Pourabbas A. Evaluation of the Usability of the NAVID Learning Management System by Cognitive Walkthrough Method. Brieflands; 2025. Report No.: 1735-1391. DOI: https://doi.org/10.5812/semj-161772
3. Then SJ, Surja S, editors. Analyzing the Usability of an E-Learning Application Using the Heuristics Evaluation Method. 2024 11th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE); 2024: IEEE. DOI: https://doi.org/10.1109/ICITACEE62763.2024.10762770
4. Liu M, Yu D. Towards intelligent E-learning systems. Education and Information Technologies. 2023;28(7):7845-76. DOI: https://doi.org/10.1007/s10639-022-11479-6
5. Loo CKE, Lim CY, Abdullah NA. A review of usability evaluation framework for tertiary education Learning Management Systems. IEEE Access. 2024;12:189941-51. DOI: https://doi.org/10.1109/ACCESS.2024.3516000
6. Khajouei R, Peute LW, Hasman A, Jaspers MW. Classification and prioritization of usability problems using an augmented classification scheme. Journal of Biomedical Informatics. 2011;44(6):948-57. DOI: https://doi.org/10.1016/j.jbi.2011.07.002
7. Jokela T, Iivari N, Matero J, Karukka M, editors. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241-11. Proceedings of the Latin American conference on Human-computer interaction; 2003. DOI: https://doi.org/10.1145/944519.944525
8. Montazeri M, Khajouei R, Montazeri M. Evaluating hospital information system according to ISO 9241 part 12. Digital Health. 2020;6:2055207620979466. DOI: https://doi.org/10.1177/205520762097946
9. Khajouei R, Farahani F. A combination of two methods for evaluating the usability of a hospital information system. BMC Medical Informatics and Decision Making. 2020;20:1-10. DOI: https://doi.org/10.1186/s12911-020-1083-6
10. Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. International journal of medical informatics. 2013;82(5):e73-e86. DOI: https://doi.org/10.1016/j.ijmedinf.2012.10.007
11. Khajouei R, de JONGH D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Medical Informatics in a United and Healthy Europe: IOS Press; 2009. p. 532-6. Available from: https://pure.uva.nl/ws/files/1396800/88770_thesis.pdf#page=53
12. Blecken A, Bruggemann D, Marx W, editors. Usability evaluation of a learning management system. 2010 43rd Hawaii international conference on system sciences; 2010: IEEE. DOI: https://doi.org/10.1109/HICSS.2010.422
13. Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. Journal of the American Medical Informatics Association. 2017;24(e1):e55-e60. DOI: https://doi.org/10.1093/jamia/ocw100
14. Wechsung I. An evaluation framework for multimodal interaction. T-Labs Series in Telecommunication Services Cham: Springer International doi. 2014;10:978-3. DOI: https://doi.org/10.1007/978-3-319-03810-0
15. Jangi M, Khajouei R, Tara M, Mazaheri Habibi MR, Ghalibaf AK, Zangouei S, et al. User testing of an admission, discharge, transfer system: Usability evaluation. Frontiers in Health Informatics. 2021;10(1). DOI: https://doi.org/10.30699/fhi.v10i1.282
16. Yen P-Y, Bakken S, editors. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol–an example from a web-based communication tool for nurse scheduling. AMIA annual symposium proceedings; 2009: American Medical Informatics Association. Available from: https://pmc.ncbi.nlm.nih.gov/articles/PMC2815403
17. Rabiei R, Moghaddasi H, Hosseini AS, Paydar S. A study on radiology management information system (RMIS) in teaching hospitals affiliated to medical universities in Tehran. Health Information Management. 2015;12(1):28-37. Available from: https://him.mui.ac.ir/article_11387_en.html
18. Polson PG, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies. 1992;36(5):741-73. DOI: https://doi.org/10.1016/0020-7373(92)90039-N
19. Polson PG, Lewis CH. Theory-based design for easily learned interfaces. Human–Computer Interaction. 1990;5(2-3):191-220. DOI: https://doi.org/10.1080/07370024.1990.9667154
20. 20. Nielsen J. Usability 101: Introduction to usability. 2012. Available from: https://dl.acm.org/doi/pdf/10.1145/214748.214758
21. Chen SY, Macredie RD. The assessment of usability of electronic shopping: A heuristic evaluation. International journal of information management. 2005;25(6):516-32. DOI: https://doi.org/10.1016/j.ijinfomgt.2005.08.008
22. Farzandipour M, Nabovati E, Zaeimi G-H, Khajouei R. Usability evaluation of three admission and medical records subsystems integrated into nationwide hospital information systems: Heuristic evaluation. Acta Informatica Medica. 2018;26(2):133. DOI: https://doi.org/10.5455/aim.2018.26.133-138
23. Tang Z, Johnson TR, Tindall RD, Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemedicine Journal & E-Health. 2006;12(1):24-34. DOI: https://doi.org/10.1089/tmj.2006.12.24
24. Khajouei R, Ameri A, Jahani Y. Evaluating the agreement of users with usability problems identified by heuristic evaluation. International journal of medical informatics. 2018;117:13-8. DOI: https://doi.org/10.1016/j.ijmedinf.2018.05.012
25. Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. International journal of medical informatics. 2009;78(5):340-53. DOI: https://doi.org/10.1016/j.ijmedinf.2008.10.002
26. Tariq A, Westbrook J, Byrne M, Robinson M, Baysari MT. Applying a human factors approach to improve usability of a decision support system in tele-nursing. Collegian. 2017;24(3):227-36. DOI: https://doi.org/10.1016/j.colegn.2016.02.001
27. Huenerfauth MP. Design approaches for developing user-interfaces accessible to illiterate users. University College Dublin, Ireland. 2002:23-32. Available from: https://cdn.aaai.org/Workshops/2002/WS-02-08/WS02-08-005.pdf
28. Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, et al. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. Journal of the American Medical Informatics Association. 2013;20(e1):e2-e8. DOI: https://doi.org/10.1136/amiajnl-2012-001458
29. Khajouei R, Gohari SH, Mirzaee M. Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. Journal of Biomedical Informatics. 2018;80:37-42. DOI: https://doi.org/10.1016/j.jbi.2018.02.016
