Comparing the Relevance of the USE Questionnaire and Heatmap Analysis in Evaluating Usability and User Experience: A Case Study of the Unjani Information Systems Study Program Community Website
DOI:
https://doi.org/10.59141/jiss.v6i8.1828Keywords:
Academic Community Usability, Heatmap, User Experience, USE QuestionnaireAbstract
Usability and user experience are crucial aspects in designing and evaluating digital systems, as both directly affect user satisfaction, efficiency, and engagement. However, there often exists a gap between users' perceptions of the system’s usability and their actual behavior when using it. This research explores the potential of such gaps by analyzing user perceptions (measured through the USE questionnaire) and actual behavior data (collected from Microsoft Clarity heatmaps). A quantitative approach and visual observation involving 115 questionnaire respondents and 333 recorded user interaction sessions through Microsoft Clarity are employed. The study results indicate a general alignment between users’ positive views and behavioral engagement, such as high usability scores correlating with active duration on the Berita page. However, discrepancies were also identified on the Diskusi pages, where high satisfaction scores did not align with user behaviors like Quick Backs and Dead Clicks, indicating barriers in usability. These findings underscore the importance of integrating subjective and objective data: the USE questionnaire captures views on ease of use, satisfaction, and benefits, while heatmaps reveal behavioral barriers that may be hidden. In conclusion, both approaches complement each other, and when used together, they produce a more comprehensive and implementable user experience evaluation.
References
Anggoro, E. D. W., Prasetyo, N. A., & Shintia Dwi Alika. (2022). Analisis Usability Testing Website Desa Wisata Adiluhur Kebumen Menggunakan Metode Use Questionnaire. Prosiding Seminar Nasional Teknologi Dan Sistem Informasi, 2(1), 221–230. https://doi.org/10.33005/sitasi.v2i1.301
Bujang, M. A., Omar, E. D., Foo, D. H. P., & Hon, Y. K. (2024). Sample size determination for conducting a pilot study to assess reliability of a questionnaire. Restorative Dentistry and Endodontics, 49(1), 1–8. https://doi.org/10.5395/rde.2024.49.e3
Corrêa, G., Pereira, R., Silveira, M. S., & Gasparini, I. (2024). Exploring Usability and User Experience Evaluation Methods: A Tertiary Study. International Conference on Enterprise Information Systems, ICEIS - Proceedings, 2(Iceis), 357–368. https://doi.org/10.5220/0012606100003690
Darma, B. (n.d.). STATISTIKA PENELITIAN MENGGUNAKAN SPSS (Uji Validitas, Uji Reliabilitas, Regresi Linier Sederhana, Regresi Linier Berganda, Uji t, Uji F, R2). GUEPEDIA. https://books.google.co.id/books?id=acpLEAAAQBAJ
Davila, F., Paz, F., & Moquillaza, A. (2023). Usage and Application of Heatmap Visualizations on Usability User Testing: A Systematic Literature Review. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 14032 LNCS(July), 3–17. https://doi.org/10.1007/978-3-031-35702-2_1
Fachrizal, M. R., Paramitha Fadillah, A., & Budiarto, A. (2023). UI/UX Prototype Usability Analysis of E-Commerce Websites. 2023 International Conference on Informatics Engineering, Science & Technology (INCITEST), 1–6. https://doi.org/10.1109/INCITEST59455.2023.10396947
Firdaus, F., Ariandi, Firman, & Rasydianah. (2022). Pengukuran Usability eLearning UNSULBAR selama masa Pandemi COVID-19 (Studi Kasus: Prodi Pendidikan Biologi). BIOMA: Jurnal Biologi Dan Pembelajarannya, 4(1), 23–30. https://doi.org/10.31605/bioma.v4i1.1607
Gada, T. (2024). Enhancing User Engagement and Retention in Fintech: A Study on Effective UX Strategies and Design Principles. International Journal of Science and Research (IJSR), 13(5), 1260–1263. https://doi.org/10.21275/sr24520104932
Goodhue, D. L., & Thompson, R. L. (1995). Task-Technology Fit and Individual Performance. MIS Quarterly, 19(2), 213. https://doi.org/10.2307/249689
Hajesmaeel-Gohari, S., & Bahaadinbeigy, K. (2021). The most used questionnaires for evaluating telemedicine services. BMC Medical Informatics and Decision Making, 21(1), 1–11. https://doi.org/10.1186/s12911-021-01407-y
Hidayat, R., Asnawi, M. F., & Rohman, S. (2021). Analisis Usability Dengan Metode Use Questionnaire Pada Aplikasi Myindihome. Journal of Economic, Business and Engineering (JEBE), 3(1), 168–173. https://doi.org/10.32500/jebe.v3i1.2155
Husseniy, N., Abdellatif, T., & Nakhil, R. (2021). Improving the Websites User Experience (UX) Through the Human-Centered Design Approach (An Analytical Study Targeting Universities Websites in Egypt). Journal of Design Sciences and Applied Arts, 2(2), 24–31. https://doi.org/10.21608/jdsaa.2021.29802.1029
Kapusy, K., & Lógó, E. (2022). User Experience Evaluation Methodology in the Onboarding Process: Snapchat Case Study. Ergonomics in Design: The Quarterly of Human Factors Applications, 30(3), 4–10. https://doi.org/10.1177/1064804620962270
Kholik, A., Soegiarto, A., Sari, W. P., & Negeri, U. (2024). Strategi Komunikasi Visual dalam User Interface ( UI ) dan User Experience ( UX ) Untuk Membangun Kepuasan Pengguna. 4.
Köhler, T., & Usability, T. (2020). Usability evaluation of personalized adaptive e-learning system using USE questionnaire Didik Hariyanto Recommended citation : Usability evaluation of personalized adaptive e-learning system using USE questionnaire Didik Hariyanto. Knoledge Management & E-Learning: An International Journal (KM&EL), 12(1), 85–105.
Lemon, C., Huckvale, K., Carswell, K., & Torous, J. (2020). A Narrative Review of Methods for Applying User Experience in the Design and Assessment of Mental Health Smartphone Interventions. International Journal of Technology Assessment in Health Care, 36(1), 64–70. https://doi.org/10.1017/S0266462319003507
Liu, W., Gori, J., Rioul, O., Beaudouin-Lafon, M., & Guiard, Y. (2020). How Relevant is Hick’s Law for HCI? Conference on Human Factors in Computing Systems - Proceedings, 1–11. https://doi.org/10.1145/3313831.3376878
Lund, A. M. (2001). Measuring usability with the USE questionnaire. Usability and User Experience, 8(2).
Marques, L., Matsubara, P. G., Nakamura, W. T., Ferreira, B. M., Wiese, I. S., Gadelha, B. F., Zaina, L. M., Redmiles, D., & Conte, T. U. (2021). Understanding ux better: A new technique to go beyond emotion assessment. Sensors, 21(21), 1–26. https://doi.org/10.3390/s21217183
Masri Singarimbun. (2005). Metode Penelitian Survei:Masri Singarimbun & Sofian Effendi. LP3ES.
Melin, J., Bonn, S. E., Pendrill, L., & Lagerros, Y. T. (2020). A questionnaire for assessing user satisfaction with mobile health apps: Development using rasch measurement theory. JMIR MHealth and UHealth, 8(5). https://doi.org/10.2196/15909
Mortazavi, E., Doyon-Poulin, P., Imbeau, D., Taraghi, M., & Robert, J.-M. (2024). Exploring the Landscape of UX Subjective Evaluation Tools and UX Dimensions: A Systematic Literature Review (2010–2021). Interacting with Computers, 36(4), 255–278. https://doi.org/10.1093/iwc/iwae017
Nadifa, R. M., Pradana, F., & Bachtiar, F. A. (2024). Evaluation of Usability in LetsCode Digital Learning Media Using the Usability Testing Method and Use Questionnaire. Journal of Information Technology and Computer Science, 9(2), 152–163. https://doi.org/10.25126/jitecs.92599
Nakamura, W. T., Ahmed, I., Redmiles, D., Oliveira, E., Fernandes, D., de Oliveira, E. H. T., & Conte, T. (2021). Are ux evaluation methods providing the same big picture? Sensors, 21(10). https://doi.org/10.3390/s21103480
Nurazizah, E. O., Ermatita, & Astriratma, R. (2021). Analisis pengukuran usability menggunakan metode Use Questionnaire pada aplikasi Shopee Indonesia. Seminar Nasional Mahasiswa Ilmu Komputer Dan Aplikasinya (SENAMIKA), 2(2), 688–697. https://conference.upnvj.ac.id/index.php/senamika/article/view/1629%0Ahttps://conference.upnvj.ac.id/index.php/senamika/article/download/1629/1391
Paul M. Fitts. (1954). The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement. Journal of Experimental Psychology, 47(6), 381–391. http://www2.psychology.uiowa.edu/faculty/mordkoff/InfoProc/pdfs/Fitts 1954.pdf
Priyadi, A., Sediyono, E., & Purnomo, H. D. (2021). Evaluasi Kebergunaan (Usability) dan Rekomendasi Penggunaan Google Classroom untuk Blended Learning di Perguruan Tinggi. J. Sistem Info. Bisnis, 11(2), 105–116. https://doi.org/10.21456/vol11iss2pp105-116
Purwinarko, A., Subagja, M., & Yanuarto, A. (2020). The Evaluation of Final Assignment System Using the USE Questionnaire Approach. Scientific Journal of Informatics, 7(2), 2407–7658. http://journal.unnes.ac.id/nju/index.php/sji
Putra, Y. S. M., & Tanamal, R. (2020). Analisis Usability Menggunakan Metode USE Questionnaire Pada Website Ciputra Enterprise System. Teknika, 9(1), 58–65. https://doi.org/10.34148/teknika.v9i1.267
Raka, S. J., & Setyohadi, D. B. (2021). Measuring User Satisfaction in Website Usability by Considering Stress Level. JOIV : International Journal on Informatics Visualization, 5(3), 333. https://doi.org/10.30630/joiv.5.3.512
Rezaiguia, H., & Djeffal, A. (2022). Usability evaluation from client-side traces using heatmap and system usability scale. 2022 International Symposium on INnovative Informatics of Biskra (ISNIB), 1–6. https://doi.org/10.1109/ISNIB57382.2022.10075994
Sari, D. F., Efendi, A., & Sumaryati, S. (2024). Examining Student Satisfaction with Learning Management System-Based E-Learning: A Self-Determination Theory Perspective. Voices of English Language Education Society, 8(1), 52–63. https://doi.org/10.29408/veles.v8i1.25297
Sasongko, A., Jayanti, W. E., & Risdiansyah, D. (2020). USE Questionnaire Untuk Mengukur Daya Guna Sistem Informasi e-Tadkzirah. Jurnal Khatulistiwa Informatika, 8(2). https://doi.org/10.31294/jki.v8i2.9135
Zaki, T., & Islam, M. N. (2021). Neurological and physiological measures to evaluate the usability and user-experience (UX) of information systems: A systematic literature review. Computer Science Review, 40, 100375. https://doi.org/10.1016/j.cosrev.2021.100375
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Meisya Putri Utami, Hasan Basri, Faiza Renaldi, Dea Destiani
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International. that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
