In November 2023 we, at Harvard’s Program on Education Policy and Governance, released a new state-by-state ranking of the performance of charter school students on the National Assessment of Educational Progress, often called the Nation’s Report Card. The ranking is based on charter students’ scores on 24 NAEP tests of math and reading administered between 2009 and 2019. Ours is the first ranking of charter student performance on the same set of tests administered to samples of all students throughout the United States.
For the most part, the ranking has been well received. The head of the KIPP Foundation, the nation’s largest charter school network, says in one news report that the results “confirm our experience.” The National Alliance for Public Charter Schools comments that “the new data are ‘sobering in many respects,’ showing that charter schools in many places have ‘room to grow.’” And, of course, the ranking has been received enthusiastically by policymakers in states like Alaska, Massachusetts, New Hampshire, and Oklahoma—all of which came in at the top end of the standings. Even middle- to bottom-ranking states have not chosen to criticize the ranking procedures—though one charter-school advocate who did not like the below-average placement of his home state, objected on the rather bizarre grounds that “[o]nly a randomly selected sample of … students take the NAEP test,” a denial of the reliability of an approach regularly employed by the U.S. Census Bureau.
But in a recent blog post, Matthew Ladner, executive editor of NextSteps, a publication of the school-choice advocacy group Step Up For Students, has expressed his own doubts about our findings. He says we failed to adjust for the share of charter students who are in special education programs or are English Language Learners, we relied on information that fluctuates from one test to the next, and that charter students should have been ranked on state proficiency tests instead of the NAEP.
These criticisms either are wrong, mislead, or fail to take into account what was said in the technical version of the paper published in the Journal of School Choice.
We take particular exception to the erroneous claims that we “were unable to control for the rates of special education and English Language Learner status” on the NAEP. Those charges, if true, would be serious. But as reported in the abridged version that appeared in Education Next, scores are adjusted “to take into account the age of the test-taker, parents’ education levels, gender, ethnicity, English proficiency, disability status, eligibility for free and reduced school lunch, student-reported access to books and computers at home, and location [emphasis added].” In the unabridged version, we inform readers that eligibility for special education and English Language Learner status is ascertained by NAEP from school administrative records.
Ladner misleads when he notes that math scores of Texas 8th grade charter students tested by NAEP fluctuated substantially between 2017 and 2019. Although that is certainly correct, it is the very reason we use information from multiple tests over an extended period. As we say in the technical paper, “By combining results from 24 tests over an 11-year period, the chances of obtaining reliable results are greatly enhanced.”
Ladner argues it would be preferable to use data from the Stanford Education Data Archives, or SEDA, a source that provides student performance on every state’s proficiency tests. We in fact report a ranking obtained from the SEDA data, which is calculated in a manner comparable to the one used to construct the PEPG ranking, in Table A.11 in the appendix to the paper available in the Journal of School Choice. That ranking correlates with the PEPG ranking at the 0.7 level, which suggests the two data sources yield broadly similar results. As we discuss in our article, however, the NAEP tests are preferable because they allow for a ranking of students’ scores on the same set of tests. Ranking states based on SEDA data requires the strong assumption that state tests may all be placed on the same scale. Also, state proficiency tests are high-stakes tests used to evaluate both charter schools and their teachers, providing incentives to manipulate test results. NAEP is a low-stakes test that is not used for student, teacher, or school evaluations. Lastly, SEDA excludes over 32 percent of all charter schools from its sample. By contrast, PEPG’s NAEP sample includes over 99 percent of all charter student observations in NAEP.
But Ladner would have us use the problematic SEDA test data because SEDA reports changes in student performance in each school district and charter school from one year to the next. That requires yet another assumption: that there is no change in the composition of a school cohort from one year to the next, a particularly strong assumption for a school of choice.
As we concluded in both versions of the paper, “the PEPG rankings are not the last word on charter-school quality.” We are hopeful that assessments of charter school quality will continue to improve in the coming years. But we can only make progress if criticism is accurate and straightforward.
Paul E. Peterson is a professor of government at Harvard University, director of its Program on Education Policy and Governance, and senior editor at Education Next. M. Danish Shakeel is professor and the director of the E. G. West Centre for Education Policy at the University of Buckingham, U.K.
