Evidence Based Education
Do Learning Management Systems Work?
According to Saygili Hatice, LMS is defined as a “software that supports access to a wide variety of web-enabled tools for conducting, compiling, managing and documenting educational activities. LMS is software that combines online learning tools in learning and teaching using internet technologies. It combines online learning tools in learning and teaching using internet technologies. That can be classified as open source products, commercial products and customized software. The most common of open source learning management systems include Moodle (used in 220 countries around the world), Claroline, Interact, Ilias, Sakai, Canvas, which were utilized in many countries.”
Personally, I interpret this as meaning an LMS is a sophisticated software tool, specifically meant to enable more effective asynchronous online learning. This might seem like an increasingly enticing concept, as technology improves. LMS systems will overtime have increasingly sophisticated AI, curriculums, and engaging systematic learning games. However, so much of teaching is not just the pedagogy and the curriculum, but rather than the relationships and intrinsic motivation, which classroom teachers build. A 2021 meta-analysis of 43 experimental design studies by Saygili Hatice and Cetin Hatice, examined the efficacy of LMS and overall the results were not very promising.
Discussion
While the mean result of this meta-analysis showed an effect size of .50, most of that result is being powered by the Norvell 2011 study, which posted the absurdly high effect size of 7.81. This study was an obvious outlier and I would argue should have been removed from the calculations. Indeed the Zervni 2020 study also appeared to have an inflated effect size of 2.52. While I have not specifically read these papers, usually when we see effect sizes this large, the sample size is too low and it has caused a distorted SD. If we remove these clear outliers, we actually get a mean effect size of .20. Which is exactly on the cusp of statistical significance according to Cohen’s d interpretation guidelines. Meaning that there is little to no evidence that LMS improves education results.
If we break down the results according to individual outcomes, we do see a ton of diversity in the impacts. However, the only uses which showed above average effect sizes and had more than one study supporting said effect sizes were for primary students and for algebra. That being said, if we look at the use of LMS across different time horizons we see diminishing returns if used longer than 10 hours, with 0 statistical signs of efficacy beyond 15 hours. This suggests that the positive results we do see for LMS are from the novelty factor and not from the software itself.
While on average, LMS does show a positive impact, almost 25% of all studies showed negative effects on math outcomes, with some studies showing extremely negative outcomes. This means that not only is the potential benefit of LMS small and short lived, but it also comes with significant risk for impeding student learning. Of course, one might suggest, maybe LMS would be best used as an addendum to regular class instruction, instead of as a replacement. However, the 2021 Hatice meta-analysis actually showed substantially lower outcomes for hybrid usage, opposed to online only usage. Overall it appears there is little to no benefit for LMS, except possibly for Algebra and primary students.
Written by Nathaniel Hansford
Last Edited 2022-02-24
References:
SAYGILI, H., & ÇETÄ°N, H. (2021). The Effects of Learning Management Systems (LMS) on Mathematics Achievement: A Meta-Analysis Study. Necatibey Faculty of Education Electronic Journal of Science & Mathematics Education, 15(2), 341–362. https://doi-org.ezproxy.lakeheadu.ca/10.17522/balikesirnef.1026534