F. Howard Nelson, Bella Rosenberg, and Nancy Van Meter, “Charter School Achievement on the 2003 National Assessment of Educational Progress,” American Federation of Teachers, August 2004
Diana Jean Schemo, “Nation’s Charter Schools Lagging Behind, U.S. Test Scores Reveal,” New York Times, August 17, 2004, page A1
It is not unusual for interest groups to issue reports that further their own political agendas–and to muddle the facts in the process. For this reason, newspapers generally ignore them, treat them with great skepticism, or make sure they properly vet the research with independent observers.
Not so in the case of the study of charter schools leaked by the American Federation of Teachers (AFT) to the New York Times, which then placed it in the right-hand column of the front page of its August 17 edition–a slot typically reserved for the day’s biggest story. Headlined “Nation’s Charter Schools Lagging Behind, U.S. Test Scores Reveal,” the story sent shock waves through the charter school movement and left more than a few education reformers scrambling for cover.
Using the data tool on the National Center for Education Statistics website, the authors of the AFT study called up some basic numbers on the performance of students from a nationally representative sample of charter schools. Their conclusion: “Charter schools are underperforming.” Their evidence: data from the National Assessment of Educational Progress (NAEP), often called the nation’s report card, showing students in charter schools doing less well than students in other public schools nationally, as well as in a small number of more focused comparisons.
The Times had a field day with the news. The AFT’s findings, the paper reported, “dealt a blow to supporters of the charter school movement, including the Bush administration”–a blow made all the more powerful (and credible) by the fact that the AFT had “historically supported charter schools.” Amy Stuart Wells, a sociology professor at Columbia University Teachers College, was quoted as saying the data were “really, really important” as they “confirm what a lot of people who study charter schools have been worried about.” Would that it were that simple.
Where do we begin to sort out the outlandish claims of the AFT study–and those made by others on its behalf? For starters, saying the AFT has historically supported charter schools is like saying that the Chicago Cubs are historically a World Series champion baseball team. While technically true (legendary AFT president Albert Shanker helped introduce the concept in a 1988 speech to the National Press Club), the union’s position on the issue has changed so markedly that it is now one of the staunchest opponents of charter schools around the nation. In recent years the AFT has criticized charter schools in a series of reports, of which August’s was only the latest and best publicized.
But hardly the most sophisticated. Indeed, on a methodological level, the AFT analyses are sufficiently pedestrian to be laughable. And most mainstream newspapers around the country–once the Times had made it the story of the hour–had the good sense to present a more critical view of the study’s import. In the title and lead paragraph of its coverage, USA Today noted that “achievement [is] not so simply measured” and that critics had already pointed out that “the report is hardly a fair look at whether charter schools help kids improve.” The Seattle Times quoted University of Washington researcher Mary Beth Celio’s dismissal of the study as “one of the most unsophisticated, low-level analyses I’ve ever seen.” The editorial board at the Chicago Tribune went further, deeming the AFT findings “about as new as a lava lamp, as revelatory as an old sock, and as significant as a belch.”
A Flawed Report
What’s wrong with the study? The basic problem is straightforward: raw comparisons showing charter school students scoring lower than public school students on standardized tests may simply reflect the fact that charter schools serve students in low-performing districts with high concentrations of poor and minority children. Many states allow charter schools to form only where students are having difficulties, and many charter schools are then asked to accept the most challenging of students. Any credible analysis of their effectiveness must account for these facts on the ground.
Indeed, if the AFT believes its own findings, it must also concede that private religious schools outperform public schools (see Figure 1). According to the same NAEP data that are the basis for the new AFT study, religious private schools outperformed the public schools nationwide by between 9 and 17 points, a gap at least as large as the public school-charter school difference that the AFT–with considerable help from the Times–is trumpeting. On past occasions, the AFT has objected vehemently to interpreting such findings as evidence that religious schools are superior on the grounds that they attract an especially able group of students. But for charter schools, it seems, the problem of selection effects need only be addressed in the most superficial of ways.
The authors’ sole strategy to “enhance the fairness of the analysis” was to look separately at students in 14 categories, including those from six different states, those who qualified for the federal free-lunch program (and those who didn’t), those from different ethnic backgrounds, and those living inside and outside a central city. As a strategy to control for the background characteristics that differentiate students in charter and traditional public schools, this approach is feeble. At best it can eliminate the effects of differences with respect to one background characteristic at a time. But it may not even be effective for that purpose if, for instance, the students eligible for a free lunch who attend charter schools come from even poorer families than eligible students in traditional public schools.
Even so, in most of the comparisons holding just one characteristic constant, the performance differences between charter and traditional public school students attenuate to the point of statistical insignificance. Twenty-one of the 28 comparisons the AFT conducted using 4th-grade average scale scores are statistically insignificant. As previous research has found that ethnic differences in achievement are large, it is especially noteworthy that all comparisons within ethnic groups in the NAEP charter school data cut against the AFT’s overall conclusions. The small differences that remain when looking separately at white, African-American, or Hispanic children are all statistically insignificant–a fact that is not apparent in either the Times story’s text or the tables that accompanied it.
But do any of these comparisons–within ethnic groups or otherwise–tell us anything meaningful about the quality of traditional public, charter, or religious private schools? Not a bit.
Plainly, to account adequately for the influences of a child’s family, home environment, and community on his or her learning capacity, one must do much more than look separately at students grouped by free-lunch status, ethnicity, or school location. At a minimum, it is essential to gather detailed data on students’ background characteristics and to put them to good use. Control variables now standard in education research include parents’ education and marital status, household income, and the quality of learning resources in the home, to name but a few. And rather than using aggregate comparisons within subgroups to eliminate the effects of differences in one background characteristic at a time, as the AFT has done, the influence of all of these factors must be addressed simultaneously.
But all this may just scratch the surface. As schools of choice, charters are likely to attract students who are not doing well in their traditional public schools. Moreover, many charter schools explicitly target “at-risk” students. Both of these facts would lead you to expect students in charter schools to perform at a low level even after taking into account their observable back.ground characteristics.
Ideally, one would therefore study charter schools in the context of a randomized field trial, assigning students randomly to attend either a charter or a traditional public school, gathering data on their performance at baseline, and tracking their progress over time. In the absence of that possibility, it is vital to use data from multiple years to track the learning trajectory of students in both charter and traditional public schools.
Yet another critical flaw in the AFT’s analysis is its failure to account for the length of time that a charter school has been in place–a factor known to affect any school’s performance. Having just hired new staff and teachers, implemented new curricula, and acquired building facilities, new schools often face considerable start-up problems. Almost one-third of the charter schools nationwide were less than two years old when the 2003 NAEP was administered, raising doubts about whether even meaningful findings about charter school performance would apply when more of them are well established.
Encouragingly, research on charter schools using more reliable methods to gauge school quality is under way. Nonetheless, it will be some time before definitive conclusions about the merits of one of the nation’s most prominent, and popular, reform strategies can be drawn. In the meantime, the AFT’s study does not even amount to a good interim report.
Why All the Fuss?
Given all of these problems, why would the Times see fit to bestow instant credibility on the AFT study by granting it glowing, page-one coverage? While we have no special insight into the motives of the newspaper’s editorial staff, the coverage itself suggests two factors that are important.
The first concerns alleged chicanery by the U.S. Department of Education, which, reported the Times, had buried the flawed charter school findings in “mountains of data . . . released without public announcement.” According to the authors of the AFT study, “a combination of intuition, prior knowledge, considerable digging, and luck” was required just to locate the data. Such sleuthing makes for dramatic storytelling–for the next best thing to doing it oneself, in the newspaper business, is reporting (exclusively, one hopes, so you can break the news) on someone else’s discovery of a cover-up.
As Bella Rosenberg, one of the report’s three authors, explained to the press, “Analyses are always welcome, but first things first. . . . Surely the interests of children are better served by timely and straightforward information about whether charter school performance measures up to the claims made for it.” In a letter to the Times, educational psychologist Howard Gardner praised the AFT for its act of public service in issuing the study and then asserted that the Department of Education’s decision not to highlight the findings was ideologically driven: “If the results had been positive, the Education Department would doubtless have heralded them. Across the policy spectrum, the pattern of the administration is all too clear: Call for evidence-based results, tout them when supportive, hide them when not, spin them when possible.”
Perhaps. But we draw a slightly different conclusion. Timeliness and transparency are important, but bad information is worse than none. And uncovering misleading information and presenting it out of context does a greater disservice to the “interests of children” than the Department of Education’s decision not to issue a report that does not control for student background characteristics. From this perspective, the AFT study and the Times‘s breathless coverage of it only made a bad situation worse.
The second probable reason for the prominent attention the Times gave the study stems from the fact that charter schools represent one of several remedies for schools deemed chronically failing under George W. Bush’s No Child Left Behind Act. (Other remedies include replacing much of the school’s staff or turning its operations over to the state or to a private company.) Thus the story’s import was magnified by the politics of education reform: it suggested a flaw in the Bush administration’s game plan. The very next day, the lead Times editorial heralded the report as “a devastating setback” to the Bush administration’s education program.
Ironically, however, it is not at all clear that political cleavages over charter schools follow strictly partisan lines. Indeed, federal financial support for the charter school movement has its origins in the Clinton era. Democratic presidential candidate John Kerry was an enthusiastic supporter of charter schools. And while Secretary of Education Rod Paige was a vocal proponent of charter schools, President Bush said hardly a word about charters on the campaign trail–nor, for that matter, did he say much about them from the White House.
What the NAEP Data Do Tell Us
While the statistics on the nation’s charter schools currently available from the NAEP are not at all useful for assessing these schools’ effectiveness, they do offer, for the first time, a glimpse of the makeup of a nationally representative sample of the students who attend them. As a result, one important fact about charter schools now appears incontrovertible: they are not bastions of wealth and privilege.
As Figure 2 shows, almost 62 percent of the roughly 3,000 4th graders in the NAEP charter school sample attend a school located in a central city, compared with just 32 percent of NAEP 4th graders in traditional public schools. Roughly 33 percent of the charter school students are African-American, compared with only 18 percent of the public school students. Fifty-four percent of elementary charter school students qualify for free or reduced-price lunch programs, compared with 46 percent of public school students. The analogous differences for the 8th graders tested by the NAEP are even more pronounced, perhaps reflecting the fact that a large number of middle and high school charters target at-risk students.
Given the conditions under which states and districts accept charter schools, the language of their mandates, and the characteristics of families most eager for alternatives to traditional public schools, these differences can hardly come as a surprise. For the foreseeable future, charter schools are likely to serve high concentrations of poor and underprivileged students. What remains unclear is how much they can do for this population. Sadly–and despite the impression given by the gray lady of American journalism–the AFT study tells us nothing about that.
William G. Howell is an assistant professor of government at Harvard University. Martin R. West is a research fellow at the Program on Education Policy and Governance at Harvard University and the research editor of Education Next.
