Evidence Based Education
What Can We Learn About the Science of Reading for ELL/ESL Instruction?
Much of the conversation on the science of reading has been focused on how to help students in grades K-3 learn how to read. However, one question that I see consistently come up is how do we help students who are learning English as another language. Unfortunately, most of the major studies on the science of reading, did not specifically look at this issue. I decided to dig deeper into the research on this topic, by looking at what the meta-analyses show. I searched on the Education Source data-base for every meta-analysis on the topic. In total, I was able to find 15 meta-analyses dating back to 2004.
​
It is important to look at education science via meta-analysis, because they combine all of the studies on a given topic, to find the mean result. This is crucial in education, because we see a large amount of variability in education studies and meta-analysis helps to correct for this problem. That said, it is important to remember that a meta-analysis is only as valid as the studies in it, and the inclusion criteria was quite weak for many of these studies. Indeed, only 3 of these meta-analyses excluded studies without control groups. Meaning, that many low quality studies were included. This lack of high quality studies, might be less reflective of the meta-analysis authors having low standards, but rather a lack of available research for them to pull studies from. As ELL instruction is a niche topic, there are less total studies. This all said, I think the best practice is to base our understanding of the science, based on the available research, knowing that we may have to update that understanding as more high quality research is conducted.
All but one of these meta-analyses was looking at how best to support a specific literacy skill, for ELL students. So to make things easier, I have organized the findings of these 15 meta-analyses according to each researched literacy skill.
Oral:
In 2021, Jia et al. conducted a meta-analysis of 13 studies that examined the impact of decoding instruction on listening outcomes. The inclusion criterion for this study was weak, which could lead to inflated effect sizes. Nonetheless, the authors found a mean Hedge’s g effect size of .55, CI= [.34, .75], with the strongest results observed for elementary-aged students (g= .87 CI= [.46, 1.28].).
Decoding instruction refers to the process of teaching students how to break down written words into their individual sounds, which is crucial for developing reading skills. This meta-analysis suggests that decoding instruction can also improve listening outcomes, which could have important implications for language learners. However, the weak inclusion criteria used in this study mean that the results should be interpreted with caution.
In the same year, TESOL conducted a meta-analysis of 17 studies that investigated the factors that influence pronunciation instruction. Like the previous study, the inclusion criteria for this meta-analysis were weak, which raises questions about the reliability of the findings. Despite this limitation, the authors found a mean Cohen’s d effect size of .61CI= [.47, .74]. p=.001 for the explicit instruction of pronunciation. This suggests that explicit instruction can be effective for improving pronunciation accuracy in language learners.
The authors also noted that teachers should focus on improving segmental accuracy at the phoneme level, which is the primary correlate of accentedness. However, they also cautioned that becoming native-like in pronunciation is rare for adult learners. This highlights the importance of realistic goals and expectations when it comes to pronunciation instruction.
​
Finally, in 2014, Roessingh et al. conducted a literature review of 12 studies on effective ESL programs in high school. The authors identified several effective pedagogies, including ESL buddies, collaboration, thematic organization, and high-level questions. However, the reliability of these findings is limited by the fact that the inclusion criteria and descriptive statistics were not reported.
​
In conclusion, these three studies provide some insights into the effectiveness of different instructional approaches for language learners. However, the limitations of weak inclusion criteria and incomplete reporting of statistics highlight the importance of rigorously designed and reported studies in this area. Future research should strive to address these limitations to provide more reliable evidence on the effectiveness of different instructional approaches.
Grammar:
Language learners face many challenges, including mastering complex grammatical structures. In 2005, Goldschneider et al. conducted a study that examined error rates for different morphemes and found that students need instruction on several types, including ing, plural, possessive, regular, past tense, third person singular, and articles. This highlights the importance of targeted instruction to address specific grammatical challenges faced by language learners.
In 2020, Schenck et al. conducted a meta-analysis of 15 studies on grammar feedback. While the inclusion criteria for this study were weak, the results suggest that several strategies can be effective for providing grammar feedback to language learners. The authors found that meta-linguistic feedback (written) had the largest effect size g= 1.99, followed by direct feedback g=1.54.
Direct feedback involves explicitly pointing out errors and providing the correct form, while indirect feedback provides more subtle cues that guide the learner towards the correct form. Meta-linguistic feedback involves explaining the rules behind the correct form, and recast involves restating the correct answer. Elicitation involves prompting the learner to produce the correct form.
​
These findings suggest that different strategies may be more or less effective depending on the context and the learner's needs. For example, direct feedback may be more effective for learners who are struggling with a particular grammar structure, while recast may be more effective for learners who need more practice with natural conversation. The effectiveness of these strategies may also depend on factors such as the learner's proficiency level. I have graphed, these results as to how they impact different levels of student ability below.
In conclusion, these two studies highlight the importance of targeted instruction and feedback for language learners. By identifying specific grammatical challenges faced by learners and using effective feedback strategies, instructors can help learners to overcome these challenges and develop their grammar skills. However, the limitations of weak inclusion criteria and the need for further research underscore the importance of rigorously designed studies in this area.
Fluency:
In 2018, Fitton, et al conducted a meta-analysis of 54 studies on the impact of shared reading interventions on elementary-aged ESL students. The goal of the shared reading intervention is to improve the English language skills of students by engaging them in reading activities with an adult. The adult may use interactive techniques such as dialogic reading to reinforce specific words or ideas from the text.
The results of the meta-analysis showed that shared reading interventions were effective in improving the English language skills of ESL students. The mean effect size was d= .28 CI=[.18, .38]., which means that the intervention had a low effect on improving reading fluency. However, the inclusion criteria for the studies in the meta-analysis were weak, and the effect sizes may be inflated. When the researchers looked at the five highest quality studies, the effect size remained unchanged; however, the confidence intervals were different [-.53, 1.14], indicating that study quality did not make a meaningful difference in research results, for shared reading.
​
In 2016, Eun-Young Jeon, et al conducted a meta-analysis of 77 studies on the impact of extended personal reading time. The goal of this intervention is to increase the amount of time that students spend reading on their own, which has been theorized to improve vocabulary, comprehension, and overall language proficiency. 57 of these studies were experimental and showed a mean effect size of d=.57, CI = [.46, .68]. This indicates that the increasing student reading time had on average a moderate effect on improving ELL students reading fluency.
It appears reading fluency instruction has been understudied for ELL students. There were no meta-analyses found, which had examined the impact of varied reading or repeated reading for the fluency development of ELL students. Shared reading interventions and extended personal reading time have been shown to improve the reading fluency of ESL students. While the evidence base for shared reading interventions is not as strong due to weaker inclusion criteria, both interventions can help students improve their vocabulary, comprehension, and overall language proficiency.
Comprehension:
In 2011, Adesope, et al, conducted a meta-analysis of 14 studies on the impacts of phonics on ESL students' comprehension. The study found that a mean Hedge's g effect size was .32 [.22, .42] for comprehension. The results were strongest when phonics was systematic, in grades 1-3, and lasted longer than 100 hours. These findings suggest that phonics instruction can improve reading comprehension for ESL students, particularly when it is implemented consistently over a longer period of time.
​
In 2020, C Thompson conducted a meta-analysis of 20 studies on the impact of video game-based learning for vocabulary outcomes. The study found a mean effect size of .69; however, the inclusion criteria was weak, and the results may be inflated. The study also showed that results were positive for learners of all levels of ability. These findings suggest that video game-based learning can be an effective way to improve vocabulary outcomes for ESL students.
In 2014, Unkyoung Maeng, et al, conducted a meta-analysis of 37 studies on comprehension strategy instruction. The study found that teaching cognitive strategies is helpful, but teaching only metacognitive strategies is not. For fixed effect studies, the mean effect size was d= .46, k=45, p=0, [.40, .51]. The study also compared the use of cognitive strategies, meta-cognition strategies, and the combining of both types of strategies. The authors found that using only cognitive strategies showed the highest results.
In 2010, Yong-Hyo Park, et al conducted a meta-analysis of 20 studies on the correlation of comprehension outcomes and comprehension strategy used. The study found a moderate impact of strategy use for university students and a statistically insignificant correlation for secondary students. These findings suggest that comprehension strategies may be more effective for university-level ESL students.
In 2022, Rui Li conducted a meta-analysis of 17 studies on the use of mobile-assisted learning for K-12 ESL reading comprehension instruction. The study found a mean result of g = 0.813, 95% CI = [0.566, 1.060] p= <.001. The moderator variables showed that the type of intervention matters, with skill and drill interventions having a higher effect size of 1.07 and collaborative interventions having a lower effect size of .60. These findings suggest that mobile-assisted learning can be an effective way to improve outcomes for K-12 ESL students, particularly when using skill and drill interventions.
Rui Li 2022 Moderator Variables:
The findings summarized here suggest that phonics, game-based learning, comprehension strategies, and mobile-assisted learning can be effective instructional tools for ESL students, but the type of intervention and the student's level may affect the intervention's effectiveness.
Writing
In 2011, Adesope, et al, conducted a meta-analysis of 14 studies on the impacts of phonics on ESL students. The researchers found a mean effect size of g=.53 for writing, indicating that the use of phonics instruction had a moderate impact on students' writing skills. The study also found that the effectiveness of phonics instruction was highest when it was systematic, in grades 1-3, and lasted longer than 100 hours. These findings suggest that a comprehensive and sustained phonics instruction program can significantly benefit ESL students' writing skills.
​
In 2021, Ren, et al conducted a meta-analysis of 17 studies on online vs. in-person feedback. The study aimed to compare the effectiveness of online and in-person feedback, as well as automated feedback. The inclusion criteria for the study was weak, which may have inflated the results. The results showed that all types of feedback had a positive impact on learning outcomes. In-person feedback had the highest mean effect size of 2.24, but the confidence intervals suggest that the effect size was either underpowered or not statistically significant. Online feedback and automated feedback also showed positive results, with mean effect sizes of .77 and .69, respectively.
These findings suggest that the type of feedback is likely unimportant, as all types of feedback showed positive results. However, given the weak inclusion criteria, it is important to interpret the findings with caution. Further research is needed to better understand the impact of different types of feedback on ESL learning outcomes.
Decoding
Phonics instruction has long been a staple in literacy education, and many studies have investigated its effectiveness for English as a second language (ESL) students. In 2011, Adesope and colleagues conducted a meta-analysis of 14 studies to examine the impact of phonics instruction on decoding skills in ESL students.
​
The researchers found a mean effect size of g = .18, with a confidence interval ranging from -.07 to .43. This suggests that phonics instruction has a negligible to moderate positive effect on decoding skills in ESL students. However, the confidence interval also indicates some degree of variability across the studies.
When Adesope and colleagues examined the moderator variables, they found that the effect size was highest when phonics instruction was systematic and provided to students in grades 1-3. Furthermore, the positive effects were most apparent when phonics instruction lasted longer than 100 hours.
​
These findings highlight the importance of implementing phonics instruction in a structured and systematic way, particularly for younger ESL students. Additionally, the results suggest that longer-term instruction may be necessary to see the greatest gains in decoding skills. Overall, the study adds to the growing body of research on phonics instruction for ESL students, and highlights the potential benefits of this approach for improving literacy outcomes in this population. However, as with any meta-analysis, the results should be interpreted with caution and further research is needed to fully understand the impact of phonics instruction on decoding skills in ESL students.
General:
Computer-based learning has been a popular topic of research in recent years, particularly in relation to its impact on K-12 ESL students. In 2022, R. Li conducted a meta-analysis of 40 studies that examined the impact of computer-based learning on these students. The inclusion criteria for the study was weak, which means that the results may be inflated.
Despite this limitation, the authors found a statistically significant mean effect size of 0.68, with a 95% confidence interval ranging from 0.51 to 0.85 (p= 0.0001). This suggests that computer-based learning has a positive impact on K-12 ESL students' learning outcomes, although the strength of this impact may vary across different studies.
It is worth noting that the impact of computer-based learning may depend on various factors, such as the specific type of program used, the duration and frequency of use, and the level of individual student engagement. As such, future research in this area should consider these moderating factors in order to gain a more nuanced understanding of the effectiveness of computer-based learning for K-12 ESL students.
Secondary Meta-Analysis:
In order to better synthesize all of this information, I conducted a secondary meta-analysis of the studies reviewed in this article. A secondary meta-analysis, also known as a meta-meta-analysis, is a type of meta-analysis that combines the results of multiple meta-analyses on a particular topic. In other words, it is an analysis of analyses. Instead of aggregating primary studies, a secondary meta-analysis aggregates the findings of multiple meta-analyses.
​
Secondary meta-analyses are useful for education research because they can provide a more comprehensive understanding of the evidence base for a particular educational intervention or practice. By synthesizing the results of multiple meta-analyses, a secondary meta-analysis can help identify patterns and inconsistencies across studies, and can provide a more robust estimate of the overall effect size of an intervention or practice.
​
One potential limitation of secondary meta-analyses is that they are only as good as the quality of the meta-analyses they include. If the included meta-analyses have methodological limitations, bias, or inconsistencies in their inclusion criteria, the results of the secondary meta-analysis may be similarly flawed or limited. Additionally, the secondary meta-analysis may be limited by the availability of meta-analyses on a particular topic, and by the potential for publication bias, where studies with null or negative results may be less likely to be published or included in meta-analyses.Despite these limitations, secondary meta-analyses can be a powerful tool for synthesizing and evaluating the evidence base for educational interventions and practices.
To help control for study quality, I have color coded the effect sizes below, based on the strength of the inclusion criteria shown. Effect sizes based on only standardized test results for experimental studies are green. Effect sizes that are based on mixed measurements, but experimental studies only are blue. Effect sizes that are based on a mixture of case studies and experimental studies are orange.
Written by Nathaniel Hansford
Last Edited 2023-03-26
​
References:
Adesope OO, Lavin T, Thompson T, Ungerleider C. Pedagogical strategies for teaching literacy to ESL immigrant students: a meta-analysis. Br J Educ Psychol. 2011 Dec;81(Pt 4):629-53. doi: 10.1111/j.2044-8279.2010.02015.x. Epub 2011 Jan 6. PMID: 22050311.
Block, N. C. (2020). Evaluating the efficacy of using sentence frames for learning new vocabulary in science. Journal of Research in Science Teaching, 57(3), 454–478. https://doi-org.ezproxy.lakeheadu.ca/10.1002/tea.21602
Boram Kim. (2009). Proficiency Level and the Relative Effects of Different Corrective Feedback Options on EFL Student Writing. English Teaching, 64(4), 203–222.
"Clougherty, Leah, ""Emergent bilinguals and academic language acquisition through the use of sentence frames"" (2019). Cal Poly Humboldt theses and projects. 247.
https://digitalcommons.humboldt.edu/etd/247"
Eun-Young Jeon, & Day, R. R. (2016). The effectiveness of ER on reading proficiency: A meta-analysis. Reading in a Foreign Language, 28(2), 246–265.
Fitton, L., McIlraith, A. L., & Wood, C. L. (2018). Shared Book Reading Interventions With English Learners: A Meta-Analysis. Review of Educational Research, 88(5), 712–751. https://doi.org/10.3102/0034654318790909
Goldschneider, J. M., & DeKeyser, R. M. (2005). Explaining the “Natural Order of L2 Morpheme Acquisition” in English: A Meta-analysis of Multiple Determinants. Language Learning, 55, 27–77. https://doi-org.ezproxy.lakeheadu.ca/10.1111/j.0023-8333.2005.00295.x
Jia. (2021). Toward a set of design principles for decoding training: A systematic review of studies of English as a foreign/second language listening education. Educational Research Review., 33, N.PAG.
Li, R. (2022). Effects of blended language learning on EFL learners’ language performance: An activity theory approach. Journal of Computer Assisted Learning, 38(5), 1273–1285. https://doi.org/10.1111/jcal.12697
Lv, X., Ren, W., & Xie, Y. (2021). The Effects of Online Feedback on ESL/EFL Writing: A Meta-Analysis. Asia-Pacific Education Researcher (Springer Science & Business Media B.V.), 30(6), 643–653. https://doi-org.ezproxy.lakeheadu.ca/10.1007/s40299-021-00594-6
ÖLMEZER-ÖZTÜRK, E., & ÖZTÜRK, G. (2016). Types and Timing of Oral Corrective Feedback in Efl Classrooms: Voices from Students. Novitas-ROYAL, 10(2), 113–133.
Roessingh, H. (2004). Effective High School ESL Programs: A Synthesis and Meta-analysis. Canadian Modern Language Review, 60(5), 611–636. https://doi-org.ezproxy.lakeheadu.ca/10.3138/cmlr.60.5.611
Rui Li. (2022). Effects of Mobile-Assisted Language Learning on EFL/ESL Reading Comprehension. Journal of Educational Technology & Society, 25(3), 15–29.
Schenck, Andrew. (2020). Using meta-analysis of technique and timing to optimize corrective feedback for specific grammatical features. Asian-Pacific Journal of Second and Foreign Language Education. 5. 1-20. 10.1186/s40862-020-00097-9.
Thompson, C. (2020). Video-game based instruction for vocabulary acquisition with English language learners: A Bayesian meta-analysis. Educational Research Review., 30, N.PAG.