Charter School City:
What the End of Traditional Public Schools in New Orleans Means for American Education
By Douglas N. Harris
University of Chicago Press, 2020, $60 cloth, $20 paperback; 352 pages.
As reviewed by Jay P. Greene
New Orleans is the new Finland in Douglas N. Harris’s new book, Charter School City. An obsession with Finland swept through the education policy world in the first years of this century, when that country’s students posted particularly strong results on some international tests. Researchers, advocates, and policymakers flocked to Finland to identify the secret of their success. Most of them returned home trumpeting whatever policy they had already been championing as the “key” to superior performance in Finland. If they opposed testing, they noted Finland’s low reliance on standardized testing. If they favored higher teacher pay and stricter credentialing, they touted Finland’s teacher preparation and compensation policies. If they favored progressive education pedagogy, they focused on the priority Finland gives to the arts and child-centered learning.
In truth, these analysts had no way of isolating which features of the Finnish education system or society might be responsible for that country’s high test scores. All of these policies were in place at the same time, and any of them could have helped, hurt, or had no effect on Finnish results. For all we know, strong test performance there was caused by the type of fish the Finns eat—or it was merely a fluke. Undeterred, those determined to learn from Finland chose to focus on a particular feature and tell a plausible story about how that feature produced Finnish success, as if that were persuasive evidence of a causal effect.
In Charter School City, Harris convincingly documents that something good appears to have happened in New Orleans in the years following the deadly Hurricane Katrina of August 2005. The challenge lies in disentangling the broad set of policies and practices that have been adopted in New Orleans, so as to isolate which, if any, of these factors are responsible for positive results in that one city. The New Orleans school system changed in many ways at the same time, some of which we may be able to name and discuss and some we may not even have noticed. Almost every school in the city was converted into a charter school, greatly expanding choice and competition. Those newly opened schools differed greatly from their predecessors in terms of their pedagogy and curriculum, often emphasizing “no excuses” approaches and college-prep academics. Highly educated people from outside the city were imported by programs like Teach for America and soon constituted a large share of the city’s teaching workforce. Per-pupil spending increased dramatically, and the district adopted a system of recruiting, regulating, and closing charter schools, altering its approach over time.
Harris is confident he knows what did and did not account for positive results in New Orleans. He writes, “Closing and taking over low-performing schools was the factor, above all others, that explains the improved student outcomes,” adding elsewhere, “There was a lot of good news in New Orleans, to be sure, but competition was not at the root of it.” He repeats variants of these claims more than a dozen times in the book; they constitute the heart of his argument.
How does he know that the state’s charter authorizing and takeovers are responsible for gains in New Orleans, while choice and competition are not? Unlike the educational tourists in Finland, Harris has taken up residence in New Orleans with a research center at Tulane, giving him more intimate knowledge of the city. But like the Finland observers, Harris still focuses on the features he prefers and tells stories to support them, even if his closer view of events makes his account sound more persuasive.
The only empirical evidence he presents to bolster the claim that state takeovers, and not competition, caused improvements in New Orleans can be found in chapter nine, where he describes the results of a 2016 working paper he co- authored, “The Effects of Performance- Based School Closure and Charter Takeover on Student Performance,” published by the Tulane research center that he directs. That study finds that students who attended schools closed by the state and reopened with new operators experienced gains in their academic achievement.
There are good reasons to avoid using this study to claim that state takeovers are the main factor behind gains in New Orleans. Among other limitations, the study finds significant negative effects from a similar school-takeover policy in Baton Rouge at the same time that it finds positive effects in New Orleans. While the book is not explicit about the negative result in Baton Rouge, it suggests that takeovers in places other than New Orleans have shown disappointing results because while closures in New Orleans were “based on performance,” schools in other cities “were apparently taken over for other reasons.” This ad hoc explanation of how the same intervention could have opposite effects in two Louisiana cities is just a story, not science. An alter- native story is that the positive result for takeovers in New Orleans is a fluke or perhaps overstated, which is plausible, given that the difference-in-difference research design his study employs can sometimes produce spurious findings.
But even if one accepts Harris’s interpretation of the study, one is left wondering whether the benefits of school takeovers in New Orleans are large enough to provide the dominant explanation for improvement and to exclude the possibility that other factors, such as charter-school choice and competition, also contributed significantly. Harris writes that “the takeover process was responsible for the vast majority of the measurable improvement in student outcomes in the post-reform period.” Strangely, in the working paper he co-authored, they estimate that the takeover policy in New Orleans “seem[s] to explain 25–40 percent of the academic improvement in the city caused by the post-Katrina school reforms.” If Harris is right in the working paper that takeovers account for about a third of the improvement, that would leave two-thirds to be explained by choice and competition or other factors. It’s unclear whether the “vast majority” claim or the “25–40 percent” claim is right or if these two divergent assertions can somehow be reconciled.
The author similarly calls upon a strained interpretation of the evidence when he inflates the positive role of the state in selecting the right charter operators. The state relied on the National Association of Charter School Authorizers to rate charter applications. Harris writes that there are “some signs that the NACSA ratings were positively associated with growth in student achievement” and identifies “selecting new operators carefully” by relying on “a third party [to] evaluate the applications” as one of the city’s “keys to success.” But only in an endnote does he divulge that the relationship between the NACSA rating of charter applicants and student learning gains is statistically insignificant, which means that it is indistinguishable from zero rather than a positive sign. Harris’s attribution of improvements in New Orleans to the state’s selecting the right charter operators and closing the bad schools is a narrative that lacks scientific backing.
Harris does acknowledge that some bad things also happened in New Orleans, mostly having to do with issues of equity and fairness. But while he credits virtually all of the good things to state takeovers and regulation, he attributes most of the bad things related to equity and fairness to choice and competition. He argues that “school leaders responded to competition in ways that were often superficial or counterproductive. It was a battle for survival. School leaders thought differently about schooling (‘every kid is money’) and acted differently, cutting out pre-K programs, creaming and cropping students, and generally focusing less on the hard-to-measure pursuits of education.” No empirical evidence demonstrates that competition caused these harms. It’s just the story Harris prefers to tell. Another plausible story is that pressure created by the state threat of takeover of poorly performing schools caused schools to select and retain higher-performing students and narrow their focus on measurable outcomes in tested grades to avoid closure. That is, the feature to which Harris attributes the “vast majority” of improvements in the city could actually have been the cause of the vast majority of defects.
As with Finland, it is virtually impossible to know which of the many policies and characteristics operating simultaneously in New Orleans should be credited with having caused positive or negative developments. Rather than focusing on one place, a more scientific approach would systematically compare different places across different time periods to see if the presence or absence of certain policies was associated with better or worse outcomes. Harris does selectively consider evidence from other cities and states, but like a good storyteller he picks and chooses and provides ad hoc explanations to account for inconsistencies in the evidence.
If the author were to undertake a more thorough consideration of evidence from other locations, he would have to account for how Arizona managed to have a thriving charter sector and strong statewide improvement in outcomes without the state doing things like aggressively closing charters with low performance or like using NACSA scores to pick promising charter operators. In Arizona, charters have closed at a comparably high rate as those in New Orleans, but in Arizona, schools were mostly shuttered because of low parental demand and falling revenues rather than by state bureaucratic action. If state takeovers are the key to properly managing charters to produce success, how did Arizona accomplish its growth without this feature, and how did Baton Rouge and several other locations fail despite having state takeovers? Harris’s occasional consideration of evidence outside of New Orleans does not include any mention of Arizona.
By trying to identify the best practices of New Orleans, Harris moves away from scientific inquiry and toward storytelling. The narrow focus on New Orleans is additionally limiting because it seems quite doubtful that the city’s highly unusual arrangements will endure as normalcy and local democratic governance return to the city; nor can these conditions be replicated elsewhere. In New Orleans a hurricane wiped out the traditional school district, the state legislature wrested control of failing schools from the locally elected school board, and large injections of philanthropic and federal dollars and talent helped create and subsidize a new set of schools. As Harris concedes, “the improbability of all of these pieces coming together is hard to overstate.” The usefulness of focusing on New Orleans is further undermined by the inability to adopt or sustain similar reforms in other places, with significant political defeats and educational disappointments in Denver, Tennessee, Michigan, Georgia, and elsewhere.
Harris might bristle at the suggestion that his book is marked more by story-telling than by social science, but there is nothing inherently wrong with closely observing developments in one place and then stringing together bits of facts to create a compelling narrative. Well-done case studies and works of history do this regularly, but to make their accounts compelling they abide by standards of evidence and logical argumentation. In fact, education reform might benefit from less of an obsession with econometric methods for establishing causation and greater acceptance of qualitative and historical work, even if we are less certain about causal effects using those approaches. Harris’s book, however, falls short of the standards for quality historical or case study research while also failing to meet the causal identification standards held by most social scientists.
Jay P. Greene is Distinguished Professor of Education Policy and chair of the Department of Education Reform at the University of Arkansas.
Author Douglas N. Harris’s response to Greene’s review is here: “What Charter School City Actually Says.”
This article appeared in the Fall 2020 issue of Education Next. Suggested citation format:
Greene, J.P. (2020). The Finlandization of New Orleans: Learning from the Big Easy’s success story is not so easy. Education Next, 20(4), 84-86.
