This alert has been successfully added and will be sent to:
You will be notified whenever a record that you have chosen has been cited.
To manage your alert preferences, click on the button below.
Recommender systems have become one of the main tools for personalized content filtering in the educational domain. Those who support teaching and learning activities, particularly, have gained increasing attention in the past years. This growing interest has motivated the emergence of new approaches and models in the field, in spite of it, there is a gap in literature about the current trends on how recommendations have been produced, how recommenders have been evaluated as well as what are the research limitations and opportunities for advancement in the field. In this regard, this paper reports the main findings of a systematic literature review covering these four dimensions. The study is based on the analysis of a set of primary studies (N = 16 out of 756, published from 2015 to 2020) included according to defined criteria. Results indicate that the hybrid approach has been the leading strategy for recommendation production. Concerning the purpose of the evaluation, the recommenders were evaluated mainly regarding the quality of accuracy and a reduced number of studies were found that investigated their pedagogical effectiveness. This evidence points to a potential research opportunity for the development of multidimensional evaluation frameworks that effectively support the verification of the impact of recommendations on the teaching and learning process. Also, we identify and discuss main limitations to clarify current difficulties that demand attention for future research.
Anelli, V. W., Bellogín, A., Di Noia, T., & Pomo, C. (2021). Revisioning the comparison between neural collaborative filtering and matrix factorization. Proceedings of the Fifteenth ACM Conference on Recommender Systems, 521–529.
Ashraf E, Manickam S, and Karuppayah S A comprehensive review of curse recommender systems in e-learning Journal of Educators Online 2021 18 23-35 https://www.thejeo.com/archive/2021_18_1/ashraf_manickam__karuppayah
Barraza-Urbina, A., & Glowacka, D. (2020). Introduction to Bandits in Recommender Systems. Proceedings of the Fourteenth ACM Conference on Recommender Systems, 748–750.
Becker F Teacher epistemology: The daily life of the school 1993 1 Editora VozesBeel J, Langer S, and Genzmehr M Aalberg T, Papatheodorou C, Dobreva M, Tsakonas G, and Farrugia CJ Sponsored vs. Organic (Research Paper) Recommendations and the Impact of Labeling Research and Advanced Technology for Digital Libraries 2013 Springer Berlin Heidelberg 391-395
Betoret F The influence of students’ and teachers’ thinking styles on student course satisfaction and on their learning process Educational Psychology 2007 27 2 219-234
Bobadilla J, Serradilla F, and Hernando A Collaborative filtering adapted to recommender systems of e-learning Knowledge-Based Systems 2009 22 4 261-265
Bobadilla J, Ortega F, Hernando A, and Gutiérrez A Recommender systems survey Knowledge-Based Systems 2013 46 109-132
Buder J and Schwind C Learning with personalized recommender systems: A psychological view Computers in Human Behavior 2012 28 1 207-216
Çano, E., & Morisio, M. (2015). Characterization of public datasets for Recommender Systems. (2015 IEEE 1 st ) International Forum on Research and Technologies for Society and Industry Leveraging a better tomorrow (RTSI), 249–257.
Cazella SC, Behar PA, Schneider D, Silva KKd, and Freitas R Developing a learning objects recommender system based on competences to education: Experience report New Perspectives in Information Systems and Technologies 2014 2 217-226
Cechinel C, Sánchez-Alonso S, and García-Barriocanal E Statistical profiles of highly-rated learning objects Computers & Education 2011 57 1 1255-1269
Cechinel C, Sicilia M-Á, Sánchez-Alonso S, and García-Barriocanal E Evaluating collaborative filtering recommendations inside large learning object repositories Information Processing & Management 2013 49 1 34-50
Chen SY and Wang J-H Individual differences and personalized learning: A review and appraisal Universal Access in the Information Society 2021 20 4 833-849
Cremonesi P, Garzotto F, and Turrin R Kotzé P, Marsden G, Lindgaard G, Wesson J, and Winckler M User-centric vs. system-centric evaluation of recommender systems Human-Computer Interaction – INTERACT 2013, 334–351 2013 Springer Berlin Heidelberg
Dacrema MF, Boglio S, Cremonesi P, and Jannach D A troubling analysis of reproducibility and progress in recommender systems research ACM Transactions on Information Systems 2021 39 2 1-49
Dermeval, D., Coelho, J.A.P.d.M., & Bittencourt, I.I. (2020). Mapeamento Sistemático e Revisão Sistemática da Literatura em Informática na Educação. Metodologia de Pesquisa Científica em Informática na Educação: Abordagem Quantitativa. Porto Alegre. https://jodi-ojs-tdl.tdl.org/jodi/article/view/442
Drachsler H, Hummel HGK, and Koper R Identifying the goal, user model and conditions of recommender systems for formal and informal learning Journal of Digital Information 2009 10 2 1-17 https://jodi-ojs-tdl.tdl.org/jodi/article/view/442
Drachsler H, Verbert K, Santos OC, and Manouselis N Ricci F, Rokach L, and Shapira B Panorama of Recommender Systems to Support Learning Recommender Systems Handbook 2015 Springer 421-451
Erdt M, Fernández A, and Rensing C Evaluating recommender systems for technology enhanced learning: A quantitative survey IEEE Transactions on Learning Technologies 2015 8 4 326-344
Felder R Learning and teaching styles in engineering education Journal of Engineering Education 1988 78 674-681 Washington
Fernandez-Garcia AJ, Rodriguez-Echeverria R, Preciado JC, Manzano JMC, and Sanchez-Figueroa F Creating a recommender system to support higher education students in the subject enrollment decision IEEE Access 2020 8 189069-189088
Ferreira, V., Vasconcelos, G., & França, R. (2017). Mapeamento Sistemático sobre Sistemas de Recomendações Educacionais. Proceedings of the XXVIII Brazilian Symposium on Computers in Education, 253-262.
Garcia-Martinez S and Hamou-Lhadj A Educational recommender systems: A pedagogical-focused perspective Multimedia Services in Intelligent Environments. Smart Innovation, Systems and Technologies 2013 25 113-124
George G and Lal AM Review of ontology-based recommender systems in e-learning Computers & Education 2019 142 103642-103659
Harrathi M and Braham R Recommenders in improving students’ engagement in large scale open learning Procedia Computer Science 2021 192 1121-1131
Herpich F, Nunes F, Petri G, and Tarouco L How Mobile augmented reality is applied in education? A systematic literature review Creative Education 2019 10 1589-1627
Huang L, Wang C-D, Chao H-Y, Lai J-H, and Yu PS A score prediction approach for optional course recommendation via cross-user-domain collaborative filtering IEEE Access 2019 7 19550-19563
Iaquinta, L., Gemmis, M. de,Lops, P., Semeraro, G., Filannino, M.& Molino, P. (2008). Introducing serendipity in a content-based recommender system. Proceedings of the Eighth International Conference on Hybrid Intelligent Systems, 168-173,
Isinkaye FO, Folajimi YO, and Ojokoh BA Recommendation systems: Principles, methods and evaluation Egyptian Informatics Journal 2015 16 3 261-273
Ismail HM, Belkhouche B, and Harous S Framework for personalized content recommendations to support informal learning in massively diverse information Wikis IEEE Access 2019 7 172752-172773
Khan KS, Kunz R, Kleijnen J, and Antes G Five steps to conducting a systematic review Journal of the Royal Society of Medicine 2003 96 3 118-121
Khanal SS, Prasad PWC, Alsadoon A, and Maag A A systematic review: Machine learning based recommendation systems for e-learning Education and Information Technologies 2019 25 4 2635-2664
Khusro S, Ali Z, and Ullah I Kim K and Joukov N Recommender Systems: Issues, Challenges, and Research Opportunities Lecture Notes in Electrical Engineering 2016 Springer 1179-1189
Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007–001. Keele University and Durham University Joint Report. https://www.elsevier.com/data/promis_misc/525444systematicreviewsguide.pdf.
Kitchenham B, Pearl Brereton O, Budgen D, Turner M, Bailey J, and Linkman S Systematic literature reviews in software engineering – A systematic literature review Information and Software Technology 2009 51 1 7-15
Klašnja-Milićević A, Ivanović M, and Nanopoulos A Recommender systems in e-learning environments: A survey of the state-of-the-art and possible extensions Artificial Intelligence Review 2015 44 4 571-604
Klašnja-Milićević A, Vesin B, and Ivanović M Social tagging strategy for enhancing e-learning experience Computers & Education 2018 118 166-181
Kolb, D., Boyatzis, R., Mainemelis, C., (2001). Experiential Learning Theory: Previous Research and New Directions Perspectives on Thinking, Learning and Cognitive Styles, 227–247.
Krahenbuhl KS Student-centered Education and Constructivism: Challenges, Concerns, and Clarity for Teachers The Clearing House: A Journal of Educational Strategies, Issues and Ideas 2016 89 3 97-105
Kunaver M and Požrl T Diversity in recommender systems – A survey Knowledge-Based Systems 2017 123 154-162
Manouselis N, Drachsler H, Vuorikari R, Hummel H, and Koper R Ricci F, Rokach L, Shapira B, and Kantor P Recommender systems in technology enhanced learning Recommender Systems Handbook 2010 Springer 387-415
Manouselis N, Drachsler H, Verbert K, and Santos OC Recommender systems for technology enhanced learning 2014 Springer
Manouselis, N., Drachsler, H., Verbert, K., & Duval, E. (2013). Challenges and Outlook. Recommender Systems for Learning, 63–76.
Maravanyika M and Dlodlo N An adaptive framework for recommender-based learning management systems Open Innovations Conference (OI) 2018 2018 203-212
Maria, S. A. A., Cazella, S. C., & Behar, P. A. (2019). Sistemas de Recomendação: conceitos e técnicas de aplicação. Recomendação Pedagógica em Educação a Distância, 19–47, Penso.
McCombs, B. L. (2013). The Learner-Centered Model: Implications for Research Approaches. In Cornelius-White, J., Motschnig-Pitrik, R. & Lux, M. (eds), Interdisciplinary Handbook of the Person-Centered Approach, 335–352. 10.1007/ 978-1-4614-7141-7_23
Medeiros RP, Ramalho GL, and Falcao TP A systematic literature review on teaching and learning introductory programming in higher education IEEE Transactions on Education 2019 62 2 77-90
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, and PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement Systematic Reviews 2015 4 1 1
Moubayed A, Injadat M, Nassif AB, Lutfiyya H, and Shami A E-Learning: Challenges and research opportunities using machine learning & data analytics IEEE Access 2018 6 39117-39138
Nabizadeh AH, Gonçalves D, Gama S, Jorge J, and Rafsanjani HN Adaptive learning path recommender approach using auxiliary learning objects Computers & Education 2020 147 103777-103793
Nafea SM, Siewe F, and He Y On Recommendation of learning objects using Felder-Silverman learning style model IEEE Access 2019 7 163034-163048
Nascimento PD, Barreto R, Primo T, Gusmão T, and Oliveira E Recomendação de Objetos de Aprendizagem baseada em Modelos de Estilos de Aprendizagem: Uma Revisão Sistemática da Literatura Proceedings of XXVIII Brazilian Symposium on Computers in Education- SBIE 2017 2017 213-222
Nguyen QH, Ly H-B, Ho LS, Al-Ansari N, Le HV, Tran VQ, Prakash I, and Pham BT Influence of data splitting on performance of machine learning models in prediction of shear strength of soil Mathematical Problems in Engineering 2021 2021 1-15
Nichols, D. M. (1998). Implicit rating and filtering. Proceedings of the Fifth Delos Workshop: Filtering and Collaborative Filtering, 31–36.
Okoye, I., Maull, K., Foster, J., & Sumner, T. (2012). Educational recommendation in an informal intentional learning system. Educational Recommender Systems and Technologies, 1–23.
Pai M, McCulloch M, Gorman JD, Pai N, Enanoria W, Kennedy G, Tharyan P, and Colford JM Jr Systematic reviews and meta-analyses: An illustrated, step-by-step guide The National Medical Journal of India 2004 17 2 86-95
Petri G and Gresse von Wangenheim C How games for computing education are evaluated? A systematic literature review Computers & Education 2017 107 68-90
Petticrew M and Roberts H Systematic reviews in the social sciences a practical guide Blackwell Publishing 2006
Pinho PCR, Barwaldt R, Espindola D, Torres M, Pias M, Topin L, Borba A, and Oliveira M Developments in educational recommendation systems: a systematic review Proceedings of 2019 IEEE Frontiers in Education Conference (FIE) 2019
Pöntinen S, Dillon P, and Väisänen P Student teachers’ discourse about digital technologies and transitions between formal and informal learning contexts Education and Information Technologies 2017 22 1 317-335
Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation framework for recommender systems. Proceedings of the fifth ACM conference on Recommender systems, 157–164.
Rahman MM and Abdullah NA A personalized group-based recommendation approach for web search in E-Learning IEEE Access 2018 6 34166-34178
Ricci, F., Rokach, L., & Shapira, B. (2015). Recommender Systems: Introduction and Challenges. I Ricci, F., Rokach, L., Shapira, B. (eds), Recommender Systems Handbook, 1–34.
Rivera, A. C., Tapia-Leon, M., & Lujan-Mora, S. (2018). Recommendation Systems in Education: A Systematic Mapping Study. Proceedings of the International Conference on Information Technology & Systems (ICITS 2018), 937–947.
Salazar C, Aguilar J, Monsalve-Pulido J, and Montoya E Affective recommender systems in the educational field. A systematic literature review Computer Science Review 2021 40 100377
Santos IM and Ali N Exploring the uses of mobile phones to support informal learning Education and Information Technologies 2012 17 2 187-203
Sergis S and Sampson DG Learning object recommendations for teachers based on elicited ICT competence profiles IEEE Transactions on Learning Technologies 2016 9 1 67-80
Shani G and Gunawardana A Ricci F, Rokach L, Shapira B, and Kantor P Evaluating recommendation systems Recommender Systems Handbook 2010 Springer 257-297
Tahereh, M., Maryam, T. M., Mahdiyeh, M., & Mahmood, K. (2013). Multi dimensional framework for qualitative evaluation in e-learning. 4th International Conference on e-Learning and e-Teaching (ICELET 2013), 69–75.
Tarus JK, Niu Z, and Yousif A A hybrid knowledge-based recommender system for e-learning based on ontology and sequential pattern mining Future Generation Computer Systems 2017 72 37-48
Tarus JK, Niu Z, and Mustafa G Knowledge-based recommendation: A review of ontology-based recommender systems for e-learning Artificial Intelligence Review 2018 50 1 21-48
Verbert K, Manouselis N, Ochoa X, Wolpers M, Drachsler H, Bosnic I, and Duval E Context-aware recommender systems for learning: A survey and future challenges IEEE Transactions on Learning Technologies 2012 5 4 318-335
Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., & Duval, E. (2011). Dataset-Driven Research for Improving Recommender Systems for Learning. Proceedings of the 1st International Conference on Learning Analytics and Knowledge, 44–53.
Wan S and Niu Z A learner oriented learning recommendation approach based on mixed concept mapping and immune algorithm Knowledge-Based Systems 2016 103 28-40
Wan S and Niu Z An e-learning recommendation approach based on the self-organization of learning resource Knowledge-Based Systems 2018 160 71-87
Wan S and Niu Z A hybrid E-Learning recommendation approach based on learners’ influence propagation IEEE Transactions on Knowledge and Data Engineering 2020 32 5 827-840
Watkins KE and Marsick VJ Informal and incidental learning in the time of COVID-19 Advances in Developing Human Resources 2020 23 1 88-96
Wu D, Lu J, and Zhang G A Fuzzy Tree Matching-based personalized E-Learning recommender system IEEE Transactions on Fuzzy Systems 2015 23 6 2412-2426
Wu Z, Li M, Tang Y, and Liang Q Exercise recommendation based on knowledge concept prediction Knowledge-Based Systems 2020 210 106481-106492
Yanes N, Mostafa AM, Ezz M, and Almuayqil SN A machine learning-based recommender system for improving students learning experiences IEEE Access 2020 8 201218-201235
Zapata A, Menéndez VH, Prieto ME, and Romero C Evaluation and selection of group recommendation strategies for collaborative searching of learning objects International Journal of Human-Computer Studies 2015 76 22-39
Zhang S, Yao L, Sun A, and Tay Y Deep learning based recommender system ACM Computing Surveys 2020 52 1 1-38
Zhong J, Xie H, and Wang FL The research trends in recommender systems for e-learning: A systematic review of SSCI journal articles from 2014 to 2018 Asian Association of Open Universities Journal 2019 14 1 12-27