As schools used their federal Covid-relief aid to address student learning loss in the 2021-22 school year, a team of researchers drawn from institutions across the country began tracking how these recovery efforts were working. The Road to Recovery project worked with 12 mid-sized to large school districts collectively enrolling about 600,000 students across 10 states. The research team concentrated on students receiving five types of interventions—tutoring, small group push-in and pull-out programs, afterschool and Saturday school offerings, virtual learning, and extended school calendars—and compared their performance to similar students who were not receiving the extra support. The programs are supported with federal Elementary and Secondary School Emergency Relief (ESSER) funds. FutureEd associate director Phyllis Jordan spoke with Dan Goldhaber, one of the research team’s leaders, about the project’s first-year findings. Goldhaber is director of the Center for Analysis of Longitudinal Data in Education Research at the American Institutes for Research and the director of the Center for Education Data & Research at the University of Washington.
What was your key takeaway from the research on these various responses to learning loss?
Outside of elementary math, the pace of achievement looks flat, meaning it’s broadly consistent with what we saw pre-pandemic. What that suggests is that we are not on a trajectory for a rapid recovery.
Why do you think that is?
When we looked at the plans for these targeted interventions, there were a lot of implementation challenges, especially in the scope of the program. We saw that in a lot of cases, the programs did not pan out as ambitiously as they were planned.
But even if the scale and intensity of the programs were as ambitious as they were originally planned to be, we don’t think that that would have been enough to significantly move the needle. That’s why I’ve got a growing concern about what we call an “urgency gap.” There may not be enough urgency in society around the extent to which we are in a hole, the depth of the hole that we’re in, and how we fill it.
You tracked five major categories of interventions: tutoring, small group instruction, virtual learning and various extended time approaches. Do you have any sense that any of them were more effective than others?
We didn’t see a pattern that jumped out at us. They all suffered from some implementation challenges. We also saw that in some districts all of the interventions were relatively successful and in other districts they weren’t.
What sort of shape did these implementation challenges take?
Kids were out with COVID. Teachers were out with COVID. The labor market is tight. There were scheduling challenges so that raises coordination issues. At least one of the districts put out an RFP for tutoring and got several hundred tutoring applications from external vendors, and you have to go through a procurement process. Because of a lot of these challenges, there were districts where the programs didn’t fully ramp up until mid-to-late spring 2022.
Do you have any sense of whether student mental health concerns played a role in disrupting learning?
The mental health issues that students have that are real. As a caveat, we’re focused on math and reading test scores, and I think those are important, but they’re not everything. We don’t know the degree to which a lot of the investments in social-emotional learning might be helping to re-engage in school.
What’s at stake if schools don’t address this quickly?
We know that the pandemic had a disparate impact on students. That impact was generally more severe along existing lines of inequity in educational outcomes. If we don’t address that inequity that we can now see in the snake, it’s going to sort of move throughout the snake’s body. And when these kids are out in the labor market, we’re going to see greater inequity in labor market outcomes, which makes for a more inequitable society. One other thing we have a hard time measuring is the missed potential. We won’t know what we missed exactly, which kids who didn’t get an adequate education could have gone on to be the next Einstein.
And when I’m talking about academic recovery, I’m just talking about getting back to pre-pandemic levels of achievement. And those levels were too low for too many kids.
Are there political implications given the tremendous infusion of federal aid for schools to address learning loss?
What I’m worried about is what happens if we get to the end of the ESSER road and people look back and say, “Well, there was $200 billion, and there’s not much evidence that kids are much further back to where they should have been, and we didn’t track it all that well, and we don’t have much evidence of what worked or what didn’t.” Beyond what that means for the students themselves, it’s kind of a public-relations disaster for public schools.
So given what you found, what’s the way forward? How do we increase the urgency?
I’m worried that all the policy wonks know from the NAEP scores that things are a disaster, but the parents don’t. I’m worried that grades are what parents focus on. And there’s some evidence that grading standards changed over the pandemic, and so it appears the kids are doing better if grades closely aligned with objective standards. So if you’re a parent and your kid is taking home all As, but they are several grade levels behind, schools are not sending you the right message about the urgency of getting supplemental help.
Given the politics, some of these programs are ultimately going to have to be voluntary. Saturday school, summer school, on-demand tutoring—it’s hard to mandate that kids get those things. So you need to make sure that parents are aware of where their kids stand and that the language that’s used to convey that is transparent.
You found a lot of variation in what people consider tutoring or what they consider pull-out, pull-in programs. Is there a role for the state or for federal authorities to provide some direction or guidance?
There is a real role for the feds or states to provide templates to help districts add up the impact of recovery programs—to show that what’s planned adds up to recovery and to track the pace of achievement. I have more mixed feelings about directing what the nature of the programs should be, because they are going to vary depending on the needs of the students. Sometimes it’s only folks who are close to the students on the ground who can best identify those needs.
You’re continuing the research this year. Are you seeing any reasons for hope or optimism?
It feels like there are fewer disruptions this year. Last year, when we interviewed some districts in the fall, they had programs up and running. And then there were other districts that didn’t even have superintendents sign off or board approval. And the programs did not get up and running until the spring. And we had the Omicron surge.
This school year, I feel like the labor market has settled down. Procurement problems have probably been smoothed out, and hopefully, students are more likely to be actually attending. We’ve learned something from some of the glitches that came up last year around scheduling and coordination. So for all of those reasons, I think this year things are likely to be smoother.
