When Minnesota passed the nation’s first charter-school law in 1991, its main purpose was to improve education by allowing for new, autonomous public schools where teachers would have more freedom to innovate and meet students’ needs. Freed from state regulations, district rules, and—in most cases—collective-bargaining constraints, charter schools could develop new models of school management and “serve as laboratories for new educational ideas,” as analyst Brian Hassel observed in an early study of the innovation. In the words of Joe Nathan, a longtime school-choice advocate and former Minnesota teacher, “well-designed public school choice plans provide the freedom educators want and the opportunities students need while encouraging the dynamism our public education system requires.”

Over the next two decades, 45 additional states and Washington, D.C., passed their own laws establishing charter schools. And by 2020–21, nearly 7,800 charter schools enrolled approximately 3.7 million students, or 7.5 percent of all public-school students nationwide. The most recent charter law was passed in 2023 in Montana, though its implementation has so far been blocked by court order; today, only North Dakota, South Dakota, Nebraska, and Vermont have not passed charter legislation.

During those years, advocates have carefully tracked and analyzed state policies and enrollments to compare charter school growth, demand, and access across the United States. But to date, there have been no comparisons of charter school performance across states based on student achievement adjusting for background characteristics on a single set of nationally administered standardized tests. Instead, advocacy organizations routinely rank states based on one or more aspects of their charter school programs—factors such as the degree of autonomy charters are afforded, whether they receive equitable funding, and the share of a state’s students they serve. These rankings are informative, but they do not provide direct information about how much students are learning, which is, ultimately, the general public’s and policymakers’ primary concern.

We provide that information here, based on student performance in reading and math on the National Assessment of Educational Progress, or NAEP, between 2009 and 2019. These rankings, created at the Program on Education Policy and Governance (PEPG) at Harvard University, are adjusted for the age of the charter school and for individual students’ background characteristics. They are based on representative samples of charter-school students in grades 4 and 8 and cover 35 states and Washington, D.C. We also estimate the association between student achievement and various charter laws and characteristics.

Overall, the top-performing states are Alaska, Colorado, Massachusetts, New Hampshire, New York, Oklahoma, and New Jersey. The lowest-ranked charter performance is in Hawaii, followed by Tennessee, Michigan, Oregon, and Pennsylvania. Students in the South tend to perform above average, while students in midwestern Rust Belt states rank at the midpoint or below. We also find that students at schools run by charter networks outperform students at independent charters, on average, while students at schools run by for-profit organizations have lower scores on NAEP, on average. Students at charters authorized by state education agencies have higher scores than students at those authorized by local school districts, non-educational organizations, or universities.

We hope these rankings will spur charter-school improvement in much the same way that NAEP results have stimulated efforts to improve student achievement more generally. Current debates include whether authorizers should regulate schools closely or allow many and diverse flowers to bloom, whether charters should stand alone or be incorporated into charter school networks, and whether for-profit charters should be permitted. A state ranking of charter student performances may not answer such questions, but it can stimulate conversations and foster future research that could.

Assessing State-Level Achievement

We create the PEPG rankings based on NAEP tests in reading and math. The tests, known as the Nation’s Report Card, are administered every two years to representative samples of U.S. students in grades 4 and 8. To obtain a robust sample for each state, each survey wave includes more than 100,000 observations of public-school students in both district and charter schools. The number of tested charter-school students varies between 3,630 and 7,990 per test, depending on the subject, grade, and year.

Our analysis looks at the period between 2009 and 2019, when 24 tests were administered. This yielded 3,732,660 results in all, but we focus on the 145,730 results from charter-school students. We include results from Washington, D.C., and the 35 states with enough tested charter-school students to permit precise estimates. That excludes the five states that do not currently allow charter schools, as well as Alabama, Iowa, Kansas, Kentucky, Maine, Mississippi, Washington, Virginia, West Virginia, and Wyoming. Still, the results in our sample account for more than 99 percent of all charter-school student scores in NAEP.

We also look at anonymized demographic information about test-takers, which was provided by the U.S. Department of Education under a special license. The weighted composition of our sample is 32 percent white, 30 percent Black, 31 percent Hispanic, and 4 percent Asian and Pacific Islanders. Some 58 percent are from a low-income household. Fifty-six percent were tested at a charter school located in a city, 30 percent in a suburb, 5 percent in a small town, and 10 percent in a rural area. Among 8th graders, 45 percent indicate that at least one parent completed college. Another 37 percent report that their parent does not have a college degree, and information is missing for the remaining 18 percent.

In estimating charter performance by state, we place charter scores in each subject on a common scale, adjusting for year of testing, subject, grade level, and the year the charter school opened. NAEP weights test-score observations so they are representative of the true underlying student population. We also adjust scores to take into account the age of the test-taker, parents’ education levels, gender, ethnicity, English proficiency, disability status, eligibility for free and reduced school lunch, student-reported access to books and computers at home, and location.

We then rank states based on the adjusted average scores for their charter students from 2009 to 2019 as compared to the average scores for all charter students nationwide over the same period. We report the size of these differences, whether positive or negative, as a percentage of one standard deviation in student test scores and note here that a full standard deviation is equivalent to roughly three-and-a-half years of learning for students in these grades. Several states have such similar scores they can be considered to be statistically tied, so undue weight should not be placed on any specific rank number. (See the unabridged version of this paper, published in the Journal of School Choice, for information that allows one to calculate whether any two states are statistically tied.)

From a political perspective that values equality and diversity, integrated schools are inherently good. Research also supports the notion that exposure to individuals from a diverse set of backgrounds has positive social and political benefits for a pluralistic society, and an expanding body of research attests to the positive consequences of school integration for academic outcomes.

Yet schools remain highly segregated by race and class, in part because of the segregation of neighborhoods, which largely determine where students enroll. Public charter schools, which have dramatically expanded their reach since they were first established in 1992, now occupy a central role in the public debate over racial isolation in school, with advocates and critics pitching the schools as either a potential cure for, or a key contributor to, segregation.

Charter advocates argue that decoupling school assignment from already intensely segregated residential neighborhoods should lead to more integrated schools. Charter critics, however, allege that these public schools of choice are instead driving resegregation. They worry that if socioeconomically advantaged families take advantage of school-choice opportunities and leave the most disadvantaged students behind in the worst schools, choice could exacerbate segregation.

Which of these camps is correct? How do charter schools affect segregation? The current empirical evidence fails to provide a definitive answer. In this study, we attempt to close that gap with the first nationally comprehensive examination of this question. We use detailed annual records on school enrollment by race spanning a period of 17 years, from 1998 to 2015, and a research design that isolates the causal effect of the charter share of enrollment on the segregation of American school systems.

We find that, on average, an increase in the percentage of students going to charter schools leads to a small increase in the segregation of black and Hispanic students within the school districts in which charters open. Our analysis suggests that an increase of 1 percentage point in the fraction of students attending charter schools in a district causes segregation in that district to increase by 0.11 percentage points. For the average district nationwide, this implies that eliminating charter schools would lead to a modest 5 percent decrease in the segregation of black and Hispanic students.

However, we uncover considerable variation in the size of this effect, particularly depending on how we define a school system. The presence of charter schools slightly increases segregation when it is measured at the level of the town or county, similar to the effect at the school district level. But in metropolitan areas, the net effect is not distinguishable from zero, as the increase in segregation within districts tends to be offset by a decrease in segregation between them. We also find dramatic differences in the charter effect on segregation in different states: in some it is close to zero, and in others it is much larger than the national average.

Taken together, we find compelling evidence that the rise of charter schools over the last 20 years has led to slightly higher levels of racial and ethnic segregation, on average. However, these results need to be interpreted in the context of the purpose of charter schools. A large number of charter schools were founded and specifically tailored to serve students from vulnerable backgrounds, out of which a good number have been successful at improving student outcomes. Patterns resulting from black and Hispanic families choosing schools that they feel meet their children’s needs should not be interpreted with the same lens as the government-mandated segregation that was outlawed by the U.S. Supreme Court in Brown v. Board of Education.

A long history of separate schools

Students in U.S. schools were racially isolated for decades before the Supreme Court declared separate facilities to be constitutional in its 1896 Plessy v. Ferguson ruling, which was not successfully challenged until the court’s Brown decision in 1954. It was only in the mid-1970s that rates of school segregation began to fall substantially, as the result of court-ordered desegregation plans for districts. However, segregation between districts was difficult to address after a 1974 court decision, Milliken v. Bradley, struck down desegregation plans that sought to address segregation across district boundaries.

Hastened by so-called “white flight” and racist housing-market practices, race-based residential patterns at the municipal level continued and segregation between school districts increased. Across the United States, segregation between districts is now higher than segregation within districts, though this trend is somewhat less pronounced in the South and West, where districts tend to be larger and encompass many communities.

Research has found the effects of racial segregation on students are far-reaching, though the precise mechanisms that produce these effects are less clear. Analysis of the desegregation plans that followed the Brown ruling found black students were less likely to drop out of high school or be incarcerated, were more likely to be healthy and employed, and earned better wages. When desegregation orders were terminated, dropout and incarceration rates among students of color increased.

Researchers Sean F. Reardon and Ann Owens suggest that there are two primary ways by which integration might improve student outcomes: by ensuring educational resources are more equitably available to all students, and by increasing the total pool of available resources (because, for example, the political capital of parents in an integrated system may be more directed at acquiring higher total resources for the entire system rather than for specific schools). Thus far, studies have tended to focus on the distribution of available resources, which varies greatly as a function of segregation and seems to be a driving mechanism of the benefits of integration.

Enter public charter schools. Existing research says little about how charter schools affect the distribution of students in school systems. In 2017, for instance, the Associated Press conducted an analysis that compared charter schools to traditional public schools and found that charters were more likely to demonstrate high levels of racial isolation, which was quickly interpreted as more segregated. The reaction to the story exemplified the divisiveness of the issue and the importance of sound measurement.

The president of the American Federation of Teachers, Randi Weingarten, called the data “damning,” and argued that “America’s children deserve better.” The National Education Association announced “Racial Isolation of Charter School Students Exacerbating Resegregation.” Charter proponents pushed back, calling the Associated Press analysis “irresponsible” and asserting that charter schools merely reflect the neighborhoods in which they operate. Charter schools, they argued, were being unfairly criticized for doing exactly what they had set out to do—serve students who are most in need of better education.

What has been lacking is a large-scale study of the effects of charters on school segregation in the United States using a credible research design. Simply comparing the share of charter and traditional public schools that are racially isolated is insufficient, as charter schools are not spread evenly across the educational landscape and their introduction may affect the composition of students in traditional public schools. Rather, what is needed is a strategy to determine how the emergence of charter schools has influenced patterns of school enrollment in the specific systems in which they operate. We provide such an overview here with a longitudinal analysis of the universe of public school systems from 1998 to 2015.

Measuring segregation

Our primary data source is the National Center for Education Statistics’ Common Core of Data, which includes school enrollment counts by grade level, race, and ethnicity, as well as each school’s type (charter or traditional public) and location. Our study period begins in 1998, the first year the charter category was available, but we obtain similar results if we exclude data from 1998 to 2002, when national data on charter schools were of lower quality.

We match school locations to different districts, counties, cities and towns, and metropolitan areas, which we treat as distinct definitions of school systems when computing school segregation. This is particularly important, as it allows us to geo-locate charter schools in the school systems that they affect. For school districts, we use the 2015 definition of school-district boundary maps from the National Center for Education Statistics’ Education Demographic and Geographic Estimates. We also use U.S. Census data to locate all schools within tracts, which provides us with residential population counts by age, race and ethnicity, adult educational attainment, and median household income.

We focus on annual school-district enrollments by grade level: for each year in 1998 through 2015, we observe the racial composition at each grade level from kindergarten to 12th grade for all schools located in U.S. mainland states. We limit our analysis to district-grade combinations with at least two schools throughout this period, so that segregation measures can be computed. Our final sample includes 4,574 school districts, observed for each grade in K–12 across 1998–2015, for a total number of observations that exceeds 500,000.

Nationally, charter schools increased their share of total enrollment from less than 1 percent to nearly 7 percent over this time span (see Figure 1). Among districts that had at least one charter school at some point during this period, the charter enrollment share grew to more than 11 percent.

Charter schools, on average, serve different populations of students from traditional public schools: they enroll higher proportions of black students than white students in elementary and middle schools, and tend to enroll higher proportions of Hispanic students in middle and high schools. These enrollment characteristics largely reflect their locations; charter elementary and middle schools are more likely to be located in census tracts with higher proportions of black residents, while charter middle and high schools are found in areas with higher proportions of Hispanic residents compared to white residents. Charter schools also tend to be located in tracts with lower median income and adult educational attainment.

Determining the effect of charter-school growth on school-system segregation has proved vexing, in part because different methods of measuring segregation can lead to different conclusions. Absolute measures, often referred to as measures of exposure or isolation, determine the extent to which students from one demographic group are exposed to or isolated from another demographic group within individual schools. For example, some researchers have adopted terms such as “hypersegregated” or “intensely segregated” to describe schools that enroll more than 90 percent of students with the same demographic characteristic, and have employed these methods to claim charter schools are more segregated.

While descriptively useful, a drawback of these measures is that they are partly driven by the underlying racial composition of the school system. Schools in high-minority areas may be labeled as segregated simply for reflecting the underlying pool from which they draw students. Recent claims in the media that schools have been resegregating have tended to rely on absolute measures, which do not account for the fact that white students make up a shrinking share of all students in the United States. As a result, it is misleading to compare absolute measures of segregation across time or geography.

Relative measures of imbalance or unevenness adjust for the underlying composition of students, making them comparable across different locations and over time. They are also conceptually different in that they measure how evenly a given population of students is distributed across a school system.

One commonly used metric in this family of relative segregation measures is the variance-ratio index, which we employ here. The variance-ratio index builds from the isolation index but includes a simple adjustment for system-wide composition that allows for accurate comparisons across time or place. It indicates how segregated a system is relative to how segregated it could be, given the demographic mix of students, and can also be interpreted as how predictive a student’s own race is of the racial composition of her school peers. The variance-ratio index ranges from zero (complete integration) to 100 percent (complete segregation).

Still, to be sure that our results are not driven by our choice of segregation measure, we conduct a parallel analysis using another common metric, the index of dissimilarity. The dissimilarity index measures the proportion of a group’s population who would have to change schools to reach an even distribution across each school in the system. We obtain similar results when we use the dissimilarity index instead of the variance-ratio index.

We focus on the segregation of black and Hispanic students from other students, as prior research shows that most segregation occurs between whites and minority groups. In general, we find similar results when we separately measure the segregation of black, Hispanic, and white students from students in all other groups, and we note any exceptions.

We first employ the variance-ratio index to determine trends in average segregation nationally using four different definitions of a “school system”: school districts, cities and towns, counties, and metropolitan areas (see Figure 2). We find the segregation of black and Hispanic students relative to other groups has remained remarkably stable over the last 15 years, and has even declined modestly for metropolitan areas.

These national trends show that the growth of charters has not been accompanied by rising levels of segregation, but they do not indicate whether the presence of charter schools has influenced segregation in specific communities. We thus employ a more localized approach, and compute charter enrollment and segregation for each grade (kindergarten through 12th) in each school system over time. Our analysis identifies the causal effect of the percentage of students attending charter schools by comparing changes in segregation across grade levels within the same system that have experienced differing intensity in charter penetration. For example, if in 2010 the share of 9th-grade charter-school enrollment in Washington, D.C., grew more than in other grades and there was a corresponding increase in 9th-grade segregation relative to other grade levels, our methodology would attribute this to charters having increased segregation. Our national estimate of the effect of charters on segregation is an weighted average of these types of comparisons across all school systems and grade levels over the period 1998–2015.

Results

Charter schools have led to small increases in school-district segregation for each of the racial groups that make up the majority of the student body in most U.S. school systems. An increase of 1 percentage point in the share of enrollment in charter schools leads to an increase of segregation of black and Hispanic students within districts of 0.11 percentage points. Put a different way, if the average district in the sample shut down all of its charter schools, we would expect its overall segregation of black and Hispanic students to decline from 15.0 to 14.2 percent, a decrease of 5 percent. Excluding districts that have never had a charter school, we would expect average segregation to fall from 19.1 to 17.8 percent, a decrease of 7 percent.

Thus, this average effect of charters, while statistically significant, is of modest magnitude—likely due both to charters’ relatively small share of total enrollment and to heterogeneity in the effect of charter schools across different types of districts.

This is important because local school districts are the governing units with the most influence over student-assignment policies. From a geographical perspective, however, charter schools are not constrained from enrolling students from multiple districts. Moreover, the bulk of school segregation in the United States occurs between different districts, not within the same district. Because charter schools can draw students from beyond the school district in which they are geographically located, how we define the geographical school systems may be an important driver of our results.

To test this theory, we group schools into four types of settings—geographic school districts, cities and towns, counties, and metropolitan areas—and examine the effects of charter schools on the segregation of black and Hispanic students in each grouping (see Figure 3). We find that charters have a statistically significant segregative effect at each level of aggregation, with the exception of metropolitan areas. At this broadest level of aggregation, the impact on segregation is positive but neither large nor precisely estimated enough to be statistically significant. When we look at each racial and ethnic group separately, the charter impact on segregation at the metropolitan area level is statistically significant for blacks but not for Hispanics or whites.

We dig deeper into segregation at the metropolitan-area level by dividing it into two components: segregation within districts and between districts (see Figure 4). Between-district segregation reflects differences in the average racial composition of school districts in the same metropolitan area, while within-district segregation is that which occurs due to differences in the composition of schools within the same district. For black and Hispanic students combined, we find that charter schools have counteracting effects on segregation at the metropolitan-area level. As we established before, charters increase segregation inside school districts, but they tend to decrease segregation between districts in the same metropolitan area. When we examine individual racial groups, however, this pattern holds for white and Hispanic students but not for black students.

One interpretation of these results is that charter schools echo the role of magnet schools during the era of court-ordered desegregation plans. Magnet schools were hoped to counteract white flight to suburban school districts by offering programs in urban districts that would attract white families. They were thus meant to partly sacrifice the within-district integration objective in order to limit the more severe problem of growing segregation between districts. Charter schools today appear to have this type of dual effect—but while they alleviate certain demographic imbalances across district lines, this has not resulted in greater school integration overall.

We also compare how charters have affected within-district segregation across the country by looking at enrollment in all U.S. states where at least 1 percent of students attend charter schools (see Figure 5). We find substantial variation, with evidence of notable effects on segregation in Louisiana, New Mexico, North Carolina, Oklahoma, and Rhode Island. And there are several states where charters appear to have little or no effect on segregation, such as Arizona, Florida, Georgia, New Jersey, and Oregon. For a number of other states, the results are too imprecise to come to a definitive conclusion either way.

Implications

Our study shows that critics are incorrect when they say that charters are driving a resegregation of American schools. Their impact on segregation is small, and appears to be somewhat offset by improvements in racial balance across districts in the same metro area. But it also shows that charter proponents are incorrect to assume that freeing public schools from neighborhood boundaries will necessarily enhance racial integration. The evidence in our study shows that charter schools lead to slightly higher levels of racial and ethnic segregation, on average, with wide variation across states.

Is such segregation comparable to the separate and unequal circumstances of the past? Segregation that occurs as a result of a family choosing a charter school designed to meet the particular needs of their children is fundamentally different from the type of school segregation that took place during the pre-Brown era of de jure segregation. Still, there are compelling reasons to enhance school integration, both as an ideal and as a proven path to better outcomes for minority students.

Charters may better serve this purpose through more intentional recruitment policies that are attentive to how relative advantages across families can lead to increased stratification. One promising strategy comes from policies that centralize school-choice options into common enrollment systems, which research has found reduce the burden of choosing a school and increase the proportion of disadvantaged students entering charter schools. Designing these common enrollment systems to intentionally increase diversity (such as by making school-assignment decisions in part based on students’ socioeconomic background) may also be a worthy tactic.

Other promising strategies involve so-called diverse-by-design charter schools—a small but growing trend. Because charter schools are free to target their recruitment strategies from broader geographical areas, such designs have the promise of using charters as agents for integration. While little research has evaluated the effectiveness of such policies, strategies to encourage diversity, such as weighted admission lotteries and targeted recruitment efforts, show promise. For example, San Antonio, Texas, is pursuing a holistic enrollment approach that includes district-authorized charter schools, magnet schools, and traditional public schools in common enrollment systems and weighted admission lotteries, while also strategically locating new schools of choice and increasing funding for transportation for participating students. With the right design features, the promise of school choice as an agent of integration may yet be realized.

Tomas Monarrez is a research associate in the Center on Education Data and Policy at the Urban Institute; Brian Kisida is assistant professor in the Truman School of Public Affairs at the University of Missouri; and Matthew Chingos is vice president for education data and policy at the Urban Institute.

For an unabridged version of this study, visit Urban.org.

Education in the United States has a foundational public purpose: to prepare students for effective citizenship. The idea that an educated and engaged citizenry is essential to the health of a democracy motivated the creation of government-run “common schools” in the early decades of our nation and remains an important value in modern times. Yet adult behavior often falls short of this goal; voter turnout, for example, is relatively low, at about 61 percent in recent presidential elections. And just 22 percent of U.S. 8th graders passed the most recent nationwide civics test, part of the 2014 National Assessment of Educational Progress.

It wasn’t supposed to be this way. Early advocates of common schools theorized that they would naturally inculcate the knowledge, values, and skills needed for effective citizenship, based on their leadership by democratically elected community members. A handful of states have moved beyond this theory to make civics education an explicit part of public-school curricula, but it remains largely overlooked as a field of study. However, the importance of an engaged electorate has resurfaced as a prominent educational issue as of late, and in the past two years, at least 27 states have considered proposals to mandate or expand civics, motivated in part by a divided electorate, fast-paced media landscape, and bruising political discourse.

Amid this activity, some see another potential challenge to that old theory of civics osmosis: public charter schools. Since their debut in the 1990s, charters have represented a new type of public school: they are publicly authorized, publicly funded, publicly regulated, and open to all, but operate autonomously, outside the direct control of elected officials. Will this structure lead charter schools to place less emphasis on the goal of educating students for citizenship? Or might the autonomy afforded to charter schools enable them to find innovative and effective ways to foster civic engagement?

To determine how effective a public charter school might be at encouraging civic engagement, we studied the voter-registration and election-participation rates of former students of a charter network dedicated to encouraging such behavior: Democracy Prep, which has as its mission “to educate responsible citizens for success in the college of their choice and a life of active citizenship.” We find that Democracy Prep has large positive effects on civic participation, increasing its students’ voter-registration rates by about 16 percentage points and their voting rates by about 12 percentage points. Given the low registration and voting rates of young adults nationally, these are substantial impacts. And they provide new evidence that an education focused on preparing students for citizenship can, in fact, boost civic participation in adulthood—even when the school in question is not operated by democratically elected officials.

Educating “Citizen-Scholars”

At Democracy Prep, students are known as “citizen-scholars” and schools follow the motto: “Work Hard. Go to College. Change the World!” The network encourages civic behavior through a variety of curricular and experiential means, including by having students visit legislators, attend public meetings, testify before legislative bodies, and discuss influential essays on civics and government in class. On Election Day each year, students participate in “Get Out the Vote” campaigns and canvass busy street corners wearing “I Can’t Vote, but You Can!” T-shirts. As high-school seniors, they enroll in a capstone course in which they develop a “Change the World” project to investigate a real-world social problem, design a method for addressing the issue, and implement their plan.

The network’s first school, Democracy Prep Charter Middle School, was launched in the Harlem neighborhood of Manhattan in 2006. Today, the network educates more than 6,500 students in 21 elementary, middle, and high-school programs in five cities: New York City, Las Vegas, San Antonio, Baton Rouge, and Camden, New Jersey. Most students are from low-income families of color, characteristics that are associated with lower rates of voter registration and election participation among U.S. adults (see Figure 1). From the 2008–09 to 2013–14 school years, three quarters of students qualified for free or reduced-price school meals, and student enrollment was 69 percent black and 30 percent Hispanic. Among families applying to Democracy Prep for admission, the prior voter-registration rate of parents was 60 percent, about 10 percentage points below the national average in 2017, according to the U.S. Census Bureau.

Is Democracy Prep effective at boosting civic participation among its students after graduation? The existing literature suggests that it is following a sound approach. Both education in general and civics courses in particular have been found to positively affect registration and voting, though most studies have needed to use non-experimental methods that cannot definitively identify a causal relationship. Stanford economist Thomas Dee found that completing additional years of high school and enrolling in college increase voter registration, voting, volunteering, and newspaper readership. Other studies have suggested that achieving high-school graduation increases voting in the United States.

A small number of studies have attempted to measure the effect of attending a private rather than a public school on registration and voting. For example, Dee has presented evidence that students who attended 10th grade at Catholic high schools were more likely to vote as adults, although unmeasured characteristics of the students rather than the schools themselves might have driven those results. Another study, in which Deven Carlson, Matthew Chingos, and David Campbell looked at the voting behavior of students in a randomized voucher lottery in New York City, found that students who won scholarships to attend private schools were no more or less likely to register and vote as adults than those who had not.

The potential link between civics education courses and civic engagement is clearest: education about government and the electoral process specifically aims to increase democratic participation. Rigorous evidence documenting this link is scarce, but a recent non-experimental study by Jennifer Bachner found that students who completed civics coursework were more likely to vote after graduation, a relationship that was amplified among students who reported not discussing politics with their parents at home.

These findings suggest that education may be a lever for enhancing civic engagement, but they have important limitations. Most notably, none of the favorable evidence comes from the sort of studies that permit strong causal inferences. Further, no study has examined the civic effects of charter schools. Our analysis of Democracy Prep provides the first rigorous evidence on whether charters that specifically focus on civic preparation can enhance the public engagement of their graduates.

Data and Research Design

Like many charters, Democracy Prep enrolls students based on a randomized admissions lottery. In such lotteries, offers of admission are determined by chance, and so families who “win” and are offered admission do not differ, on average, from those who “lose” and receive no offer: both groups should be similar in terms of prior achievement, demographic characteristics, and unmeasured factors such as student and parent motivation. The admissions lottery therefore creates a natural experiment that we use to provide the strongest possible evidence about the causal impact of Democracy Prep on voter registration and participation in the 2016 presidential election.

We use data on 1,060 students who were the first applicants in their families entering the lottery to attend any Democracy Prep school in New York City from 2007–08 through 2015–16. We focus on first applicants because they do not benefit from the sibling preferences built into the lottery process. This group of students applied to enroll in grades 6 through 11; younger students were not 18 years old by 2016 and therefore are not part of our study. In all, 35 percent of lottery applicants were offered admission.

However, not all students offered admission went on to attend Democracy Prep. More than half of winners opted to enroll elsewhere, and many lottery losers eventually found a way to enroll (such as through waiting lists). The enrollment rate for lottery winners is 44 percent compared to 19 percent for students who did not win the admissions lottery, a difference of 25 percentage points.

Democracy Prep provided admissions lottery and enrollment data, including applicants’ names, dates of birth, gender, lottery priorities, lottery results, names of parents, and contact information. We matched these lottery and enrollment records to voter registration and participation data provided by Catalist, which maintains a national database with comprehensive information on voting-age individuals. We scrutinized both data sources to ensure that all information matched, and treated records that Catalist could not match as non-registrants and non-voters.

Democracy Prep has tried to promote the civic participation of parents as well as students by, for example, including voter-registration information in enrollment materials. We therefore also use the admissions lotteries to analyze registration and voting among Democracy Prep parents in the 2014 midterm and 2016 presidential elections. The sample for this secondary analysis includes the 5,792 parents of Democracy Prep applicants across all grades, 52 percent of whom had children offered admission through the lottery. None of these estimated impacts on registration or voting by parents are statistically significant. That is, we find no evidence that having a child admitted to one of the schools affects parents’ civic engagement.

Impacts on Registration and Voting

The story for students themselves is quite different. To assess the impact of Democracy Prep on students’ voter registration and election participation, we first use its admissions lotteries to identify two groups of students: the “treatment” group of students offered admission and the “control” group of students not initially offered admission.

Comparing these two groups, we find that winning the admissions lottery for Democracy Prep increases students’ voter-registration and turnout rates in the 2016 election by 6 percentage points (see Figure 2). The estimated impact on voter turnout is statistically significant, while the impact on registration falls just shy of that threshold. Taken together, however, the two results suggest that receiving an offer to enroll in Democracy Prep substantially boosts students’ later involvement in the electoral process.

Moreover, winning the admissions lottery cannot have affected voter registration and turnout among students who chose not to enroll. We can therefore use these comparisons to estimate the impact of actually enrolling in Democracy Prep, as opposed to simply being offered a seat. Because students in the treatment group—that is, those offered admission—were only 25 percentage points more likely to enroll than those in the control group, this adjustment amounts to increasing the raw difference between participation rates across the two groups by a factor of four.

These estimates are dramatically larger than impacts found in previous studies of the effects of education on registration and voting, implying that Democracy Prep more than doubled the expected voting rates of its students. However, due to the limited number of students and schools on which the estimates are based, they also have a considerable degree of statistical uncertainty.

Anticipating this as a potential concern, we decided in advance of conducting our analysis to estimate the likelihood that Democracy Prep does, in fact, have positive effects on students’ registration and voting and to adjust the size of those estimated effects by combining them with results from prior research. To do so, we use what’s known as a Bayesian approach, which allows us to assess the probability of a school’s positive effects on civic engagement in light of the available evidence on similar interventions in the existing literature.

Implementing a Bayesian analysis in our case requires an externally informed understanding of the difficulty of improving civic participation via similar educational interventions. If such interventions had rarely had large impacts on similar outcomes, then we would infer that it is hard to move the needle on registration and voting—which would in turn make a very large impact of Democracy Prep seem less plausible. By contrast, the more often that large effects have been found in the past, the more probable it is that a sizable impact estimate in this study was the result of a true effect rather than random chance.

We therefore gathered information from 29 prior studies that estimated the impacts of eight other educational interventions on civic engagement. Almost all of the prior impact estimates are positive, with average impacts of about 8 percentage points on registration and 6 percentage points on voting. However, because these are estimated impacts, not true effects, it could be unwise to take them entirely at face value. They could be affected by random differences between treatment and control groups, or by systematic errors such as publication bias (that is, the tendency of journals to publish only findings that are statistically significant). To prevent these issues from propagating through to our analysis of Democracy Prep, we assume that the prior estimates are exaggerated by a factor of two.

The bottom line is that it is highly unlikely that our finding from our Bayesian analysis that Democracy Prep has a positive impact on voter registration and turnout is the result of chance. In all, we arrive at a 98 percent probability that enrolling in Democracy Prep increased voter registration and a 98 percent probability that it increased voting in the 2016 election.

Using this same framework, we also generate a complementary set of impact estimates. This analysis suggests that Democracy Prep increases the voter-registration rate of its students by about 16 percentage points and increases the voting rate of its students by about 12 percentage points. In sum, even a conservative analysis, which accounts for possible overestimation of impacts based on the data in our sample, suggests that enrolling in Democracy Prep has large positive effects on students’ democratic participation in adulthood.

Implications

Democracy Prep provides a test case of whether charter schools can successfully serve the foundational purpose of public education—preparation for citizenship—even while operating outside the direct control of elected officials. With respect to the critical civic outcomes of registration and voting, the answer is a clear yes.

There can be little question that increased civic participation overall is an urgent American goal. Indeed, the current fractures in our political and media environments suggest that education for citizenship might be even more important now than in the past. A well-informed, active electorate can counterbalance voter disengagement, the proliferation of misinformation, and political stasis at the party level. In addition, expanding turnout among younger voters is important: in most presidential elections in the past half century, the differential in voter turnout between young adults ages 18 to 29 and older Americans has been 10 to 25 percentage points. Yet decisionmaking at the state and federal level has far-reaching consequences for Americans of all age groups.

Given its explicit mission to educate “citizen-scholars,” Democracy Prep is surely not typical of all charter schools. Nonetheless, its success in raising the registration and voting rates of the low-income minority students it serves provides a proof point for charter schools and conventional public schools alike: an education focused on preparation for citizenship can in fact increase students’ civic participation when they reach adulthood. Renewed attention to the foundational purpose of public schools might broadly increase civic participation across the country.

Brian Gill is a senior fellow at Mathematica Policy Research, where Charles Tilley is a senior analyst, Emilyn Whitesell is a researcher, Mariel Finucane is a senior statistician, and Liz Potamites is a systems analyst. Sean P. Corcoran is associate professor of public policy and education at Vanderbilt University.

Charter schools represent a small share of the national education market: just 6.2 percent of all public schools and 4.6 percent of all students. But their rapid growth over the past two decades has captured an outsized measure of public attention, especially in communities where district and charter schools operate side by side.

Take New York City’s Success Academy, a network of 46 schools led by Eva Moskowitz. Despite long waiting lists and well-documented academic gains for Success students, leaders are in a near-constant battle with city education officials for the space in under-utilized public-school buildings that will allow their programs to continue to grow. Most recently, Moskowitz issued a high-profile rejection of a city plan to house new students from six middle-school programs in two sites, which she and local newspaper editorial boards criticized as an unstable, temporary fix that would force families to travel too far to school.

These pitched battles often follow a similar script about the potential “spillover effects” of public charter schools on non-charter students, one which has informed political campaigns, protests, and even lawsuits. Advocates argue that charter expansion not only meets the needs of students currently on lengthy waiting lists, but also can improve performance at all public schools due to increased competition and opportunities to innovate and share successful strategies. Critics say that charters sap resources and siphon off motivated students from under-resourced district schools, which are often already serving poor and low-performing students. The debate is especially heated in communities that practice co-location, in which charters and district schools operate in the same building and share common spaces like libraries and gymnasiums.

To shed light on the question of spillover effects, I use data from New York City to estimate the effects of charter schools on students in two types of nearby district schools: those in the same neighborhood, and those that are co-located (in the same building). I find that students in district schools do better when charters open nearby: students in these schools earn higher scores on reading and math tests and are less likely to repeat a grade. The closer the schools, the larger the effect: co-location increases test scores by 0.08 standard deviations in math and 0.06 in reading.

These findings show that communities can expand charter schools to meet growing demand without putting district schools at risk of instability or failure. Far from an existential threat to their district-school neighbors, public charter schools can benefit not only their own students but also those in other programs down the street—or hallway.

Data

The impact of public charter schools on their own students’ academic performance is by now well-documented, showing wide variability overall but also clear evidence of large positive effects in many urban centers, including New York City. There is far less research, however, regarding the potential impacts that charters have on the academic performance of neighboring non-charter students.

Prior studies examining this question have focused on the district level or explored the effects of charter schools located within several miles of a traditional public school. But if the spillover effects of urban charter schools on district schools are confined to relatively small neighborhoods, then findings from prior analyses may well be underestimates.

I look at New York City, the nation’s largest school district, where both charter and co-located schools have increased over the past decade. By 2013, charter schools accounted for 11 percent of all city schools, up from 2 percent about a decade earlier. Some 60 percent of all charter schools are in co-located buildings; by contrast, 47 percent of all public schools in New York City were co-located in 2013 (see Figure 1).

My study is based on multiple sources of data from the New York City Department of Education, including student-level administrative data, school report cards, school expenditure reports, and school-environment surveys, as well as school-level data from the federal government’s Common Core of Data. The data span 14 years, from 1996–97 to 2009–10, and include students in grades 3–5 attending a district school located in the same community school district (a sub-unit of a district) where a charter school has at least one overlapping grade. I focus on elementary schools because charter school penetration was (and still is) highest in the elementary grades; I define elementary schools as any school that includes 4th grade.

Public charter schools are not randomly located—neighborhoods where charters have opened serve greater numbers of low-income students and students of color, who have lower average scores on annual statewide tests in reading and math compared to their wealthier, whiter peers in districts without charter schools. I therefore restrict my analysis to students in community school districts that include both district and charter schools.

I also focus on students with at least two years of scores on annual statewide tests in math and reading, to account for past performance and measure their progress over time. This results in a total of 876,731 unique students attending 584 unique elementary schools. The data included students’ race, nativity, immigration history, grade, borough of residence, attendance, eligibility for free and reduced-price school meals, and participation in limited English proficiency (LEP) and special education programs.

I measure whether there is a charter school in a district school’s “neighborhood” by drawing a one-mile radius around each district school. This neighborhood measure meets two key criteria: it is large enough so that it is plausible for other schooling options to exist within its boundaries, and it is small enough so that it excludes schools that a student is not likely to attend. The one-mile radius matches the district’s official definition of walking distance for students in grades 3–6, and city data shows that 75 percent of elementary charter-school students live within one mile of their school.

Finally, due to preferential admissions policies in charter schools, I restrict this neighborhood measure to include only district schools and charter schools located in the same community school district. New York State law has required charter schools to grant admissions preference to students living in the local community school district since 2007–08; prior to that, many charters followed this practice voluntarily.

Methodology

To assess the spillover effect of charter schools on students at district schools, I analyze how individual students’ test scores, attendance, and grade progression change in response to exposure to a charter school. Simply comparing performance based on whether a student’s district school is or is not located within one mile of a charter school could be misleading given differences in where charter schools are located. Instead, I take advantage of changes over time in the presence of nearby charter schools due to new charters opening.

I use the timing of entry of charter schools across the neighborhoods where they open, and their precise locations, to identify their effect using two approaches. I compare the outcomes of students at district schools after a charter school opens nearby to the outcomes of students in the same schools before a charter opened. In addition, I compare the outcomes of students in district schools that are located closer to a charter school with the outcomes of students in the same schools when the nearest charter was farther away.

Limiting comparisons to students who attended the same district schools accounts for all features of schools that do not change over time, such as their spatial attributes (that is, whether nearby buildings are suitable for housing a charter school). I control for other factors that affect all NYC public schools in a given year, such as the appointment of a new chancellor or curriculum changes, and I use prior-year test scores to capture students’ ability and control for previous school and family effects.

I capture individual students’ exposure to charter schools in multiple ways. The most basic is whether there is any charter school located within one mile of a student’s district school. I then analyze results based on the distance between each district school and the nearest charter school, and whether a district school is co-located in the same building as a charter school.

Because students move frequently, including to neighborhoods with higher-performing zoned district schools, I consider each student’s district school to be the first school where they are enrolled. That is, if a student is first observed attending a district school located in a community school district included in this analysis, that student is considered exposed to a charter school in all years after a charter school opens within one mile of that district school, whether or not the student exits to attend another district school not located near a charter school.

However, if a student exits a district school to attend a charter school, he or she is excluded from the analysis. This allows me to address both switching between district schools and switching to a charter school in response to charter entry. It also focuses on changes in individual student performance rather than on school composition over time, since the performance of each student is “fixed” with his or her original school and compared to his or her original classmates throughout their careers at NYC district schools.

Results

Students whose schools are near charters do better, and the closer the charter school is, the better these students do. Students attending a co-located district school perform 0.08 standard deviations better in math and 0.06 standard deviations better in reading. Students in district schools within a half mile of a charter school perform 0.02 standard deviations higher in both math and reading (see Figure 2).

There are no significant spillover effects on students in district schools located more than a half mile away from the nearest charter school. When I look within a three-mile radius, I find no evidence of spillover effects on test scores of students at district schools, positive or negative.

To be sure, scores on standardized tests do not represent the full range of potential charter-school spillover effects. I also find large and meaningful reductions in the percentage of students at district schools who are required to repeat a grade as a result of charter school entry. Students in co-located schools are 1.2 percentage points less likely to be retained, students at schools within a half mile are 1.0 percentage point less likely to be retained, and students at schools between a half mile and one mile from a charter school are 0.6 percentage points less likely to be retained compared to students with no charter school in the neighborhood. Although these effects may seem small in magnitude, they translate into meaningful reductions—between 20 and 40 percent off the baseline grade-retention rate of 3.0 percent among students in this sample.

It also appears that charter schools may have small, negative effects on absenteeism at nearby district schools. In the sample, the average rate of student absences is 8 percent. That rate decreases by 0.3 percentage points in co-located district schools and by roughly half as much in district schools between a half mile and one mile of a charter school.

I next look at whether different types of charter schools generate different spillover effects—and whether the effects of charter school exposure vary across student groups. These analyses examine how the sizes of the overall effects reported above vary based on the density of nearby charter schools, the quality of the charter school (based on test-score performance and charter operator), and a range of student characteristics.

First, I find that as charter density grows, so do the effects. Students in district schools with three or more charter schools within a one-mile radius perform significantly better in math than students with just one charter in the neighborhood. They are also significantly less likely to be retained.

Second, there is suggestive evidence that spillover effects are larger if the charter school appears to be of high quality, which I define as either having high average scores on annual 4th-grade math and reading exams or being operated by an established, respected charter management organization such as KIPP, Success Academy, or Uncommon Schools. Unlike the results for all charter schools, where students in co-located schools experience the most positive effects, the largest effect in reading is for students at traditional public schools located within a half-mile radius of a high-quality charter. This may be because having a charter school in the same building places the same amount of pressure on a district school regardless of charter performance, whereas those district schools located near to but not in the same building as a charter feel stronger pressure from high-performing charters.

Third, I assess the effect of charter proximity on historically underperforming students at district schools, a population often referenced in debates over charter expansion. In math, charter school entry increases performance among all subgroups of students at district schools except Hispanic students and students classified as LEP, who experience no effects; Asian students only experience a significant positive effect in math in district schools located within a half-mile radius. In reading, Hispanic students experience significant gains, whereas most other subgroups show slightly smaller positive effects.

I find that charter schools may be particularly beneficial to students who are poor or eligible for special education services, a finding which is perhaps more striking for these particularly at-risk groups. The results indicate that charter schools tend to increase or, at the very least, do not harm the performance of at-risk student populations in nearby district schools.

I check the validity of my analysis in two ways. First, I investigate the possibility that charters choose where to open based on existing performance trends at district schools, such as opening
near a school where performance is on the decline. Such a pattern would bias my results, but I find no evidence of significantly different performance trends in either math or reading prior to a charter opening in the neighborhood.

Second, I verify that my analysis shows spillover effects of charters, rather than a potential performance bump due to students switching between district and charter schools in their neighborhoods. I do this by looking at the sample of students who are continuously enrolled in a district school between grades 3 and 5 in order to exclude students coming from and going to other schools; their results are nearly identical to, if not slightly larger than, the effects on the full sample. I also find that charter school entry has only a small impact on the probability of students exiting nearby district schools.

How Do Effects Spill Over?

I investigate a number of possible mechanisms by which charter schools might influence students at district schools by examining school-level data and survey results from parents and teachers. Although these estimates provide descriptive rather than causal evidence, they serve as helpful context to educators and policymakers looking to understand these relationships.

One common critique of charter schools is that they attract the best-prepared students from district schools, leaving district schools to serve a higher-need population. My analysis finds no significant changes in school demographics at district schools after charter entry that might explain improved student performance (see Figure 3a). Among district schools within a half-mile and one-mile radius, charter school entry leads to significant decreases in general education enrollment of approximately 16 students per school, on average. In co-located schools, charter school entry leads to a significant decrease of 11.5 special education students; however, there is no change in the overall percentage of special education students at co-located district schools.

Another critique is that charters sap resources from district schools, putting additional stress on neighborhood programs. I find, however, that charter entry leads to a significant increase in instructional spending in district schools that grows with charter school proximity: 8.9 percent for co-located schools, 4.4 percent for schools within a half-mile radius, and 2.0 percent for schools within one mile of a charter school (see Figure 3b). The opening of a charter school leads to small reductions in enrollment at nearby district schools, but does not change the percentage of students from underrepresented minority groups, special education students, or LEP students. This is the case for co-located charter schools as well as for those that open within a half-mile or one-mile radius.

What about adult behavior? Might parents and teachers act differently at district and charter schools, based on varied perceptions of those programs? How might that influence school performance?

To assess parents’ perceptions, I examine data from the NYC school survey, which asks parents about their and their child’s experiences and perceptions of their school. I focus on academic expectations, communication, parental engagement, student engagement, and school safety, finding suggestive evidence that after charter school entry, parents report significantly higher student engagement and parents in co-located schools also report significantly higher perceptions of school safety. Although effects on the other indicators are not statistically significant, in general they are positive and grow in concert with charter school proximity. These results suggest that the improvements in test scores after charter school entry could reflect changes in school practices, such as improving student engagement. Alternatively, higher test scores could reflect a more positive and involved group of parents remaining in district schools.

Finally, I provide the first evidence on how charter school entry may be related to changes in teacher perceptions of practices at district schools along five indexes: academic expectations, communication, engagement, school respect and discipline, and school safety. Similar to those of parents, teacher perceptions are marginally more positive after charter school entry. Teachers in co-located schools report higher levels of academic expectations and more respect and cleanliness, for example. Although there is no significant difference on any of the individual indicators, the sum of responses indicates that teachers’ overall perceptions at a district school improve after a charter school opens nearby. The fact that both parent and teacher perceptions move in the same direction strengthens the case that district schools respond to charter school entry with changes in school practices.

Implications

These findings shed new light on the public debate over the effects of charter schools on non-charter students. Rather than sapping resources and putting students at district schools at a disadvantage, the data in New York City show that students do better when charters open nearby. In particular, students at co-located district schools, where their school shares a building with a charter school, experience the most sharply positive spillover effects. Importantly, the effects of co-location appear to be specific to charter schools, as students in district schools that are co-located with other district schools do not experience similar performance gains.

The survey data suggest that these positive effects may be explained by a combination of increased instructional per-pupil spending and changes in practice, shedding some light on how physical proximity can inspire change. Future research should more fully explore these mechanisms, in particular, the finding of increased per-pupil spending, to determine whether these might be explained by smaller class sizes or changes in the composition of the teaching force at district schools.

But what about students at district schools who are not exposed to charter schools? Do the improvements in performance elsewhere come at a cost to them? My analysis finds no spillover effects on students at district schools within a larger, three-mile radius of a charter school. Further, an examination of trends in citywide performance on math and reading tests between 2000 and 2009, a period of rapid charter expansion, shows that math and reading proficiency continued to increase during those years. Together, this indicates that the positive spillovers of charter schools on nearby students at district schools did not come at the detriment of students across the city.

The implications of this research for policy are twofold. First, charter schools appear to have modest positive effects, or at the very least, no significant negative effects on student performance at district schools nearby. This suggests that rather than capping the number of charter schools, it may be beneficial (and certainly not harmful) to allow for further expansion in NYC. Second, my results indicate that controversial co-location practices may actually be a good policy for both charter and district schools in NYC.

Further research is needed to explore whether performance gains and school-level responses are maintained over the long run and to examine whether charter schools affect students who live nearby in other ways, such as through changes in property values and residential segregation patterns. In addition, the spillover effects of charter schools in NYC found here may reflect particular institutional and contextual factors, such as the relatively small share of city K–8 students attending charters during this period. Future work should examine spillover effects in various institutional contexts and in districts where charter schools have a larger market share, such as New Orleans, Philadelphia, or Denver.

But it is clear that the typical arguments that drive charter-related controversies and public debate fail to capture the ways in which district and charter schools affect one another. More research is needed to better inform the conditions policymakers can set to ensure all schools can operate to the benefit of all students.

Sarah A. Cordes is assistant professor of policy, organizational, and leadership studies at Temple University.

Strict attention to detail, long school days, and a singular focus on college are the hallmarks of “no excuses” charter schools. Families in cities across the United States have flocked to them as academic game changers, and research shows that many of their students beat the odds on standardized tests.

But critics allege that such gains are hollow. The “no excuses” approach, they say, amounts to a paternalistic culture of test preparation that detracts from real learning and comes at a steep cost to social and emotional health. Successfully navigating adult life, including the risks and rigors of college, will take much more.

Do “no excuses” charter high schools merely help students succeed on standardized tests? Or are their students more likely to succeed after they leave school behind? Is it test prep, or true learning? Little prior research is available on this question. And although there is a robust positive correlation between test performance and college enrollment, there is little existing evidence as to whether schools that increase test scores the most also help their students succeed at the next level.

To shed light on these questions, we studied Noble Street College Prep, a high-performing no-excuses charter high school in Chicago where admission is granted via randomized lottery. We analyzed student records to estimate the effect of attending Noble on college enrollment, persistence, and quality, using success in postsecondary studies as a proxy for success in young adulthood.

Overall, our results suggest that the benefits of attending a no-excuses charter high school extend beyond graduation and into early adulthood. Students who attended Noble Street College Prep were not only more likely to enroll in college, but also far more likely to enroll in a competitive four-year school. They were also more likely to persist in college, trends that continued for several years after high school graduation.

We were only able to obtain randomized lottery information from the College Prep campus, but data from a broader group of Noble high schools indicate they have higher college enrollment rates than other schools with similar student populations. This result suggests that the Noble Network of Charter Schools has continued to produce positive results as it has expanded.

These findings provide strong evidence that Noble measurably improves students’ preparation for college as opposed to just pushing marginal students into low-quality institutions. That is in keeping with its stated mission, “to prepare low-income students with the scholarship, dedication, and honor necessary to succeed in college and lead exemplary lives, and be a catalyst for education reform in Chicago.”

A College-Centered Culture

Noble Street College Prep was founded in 1999 as Noble Street Charter High School. In 2006, it was renamed and the network began to expand. The Noble Network currently includes 17 campuses with more than 11,000 students in the west and south sides of Chicago (Figure 1). The schools attract a predominantly poor, minority student body: 98 percent of students are black or Hispanic and 89 percent are eligible for free or reduced-price school meals.

Noble network schools follow key practices and principles typically associated with the no-excuses approach: frequent teacher feedback, data-driven instruction, high-dosage tutoring, increased instructional time, and high expectations. All students are expected to take college entrance exams and win acceptance to college. Acceptances are celebrated publicly, and counselors assist students in applying for grants and scholarships.

On average, students spent 7.5 hours per day and 185 days per year in school, compared to an average of 6.9 hours per day and 170 days per year in Chicago Public Schools (CPS) during the period this analysis covers. This implies that Noble students spent 18 percent more time in school than their peers in Chicago, on average, amounting to 858 hours over four years, or nearly three-quarters of a year of additional instruction.

The school day is structured to ensure that all students receive differentiated instruction in smaller-group settings, organized by performance on interim assessments. During morning and afternoon meetings, teachers track individual academic progress, mark behavioral infractions, and hold students accountable as a group for maintaining academic and behavioral standards. Each afternoon, teachers maintain office hours for optional academic support, which becomes mandatory if a student’s performance falls below a certain threshold. Most campuses also feature some form of afterschool tutoring provided by outside organizations.

Noble aggressively recruits teachers with a demonstrated track record of success and rewards teachers whose students demonstrate above-average academic growth with performance bonuses. Teachers receive regular feedback on their performance and attend campuswide professional development sessions each Friday. They meet regularly to analyze student data and collaboratively plan how to use it to drive instruction.

Looking at a Lottery

We used lottery data to compare Noble students to a comparison group of their peers throughout Chicago in order to estimate the effect of attending Noble on college enrollment and persistence. Our analysis focuses on the three cohorts of Noble students who enrolled between 2003 and 2005, as those were the only years in which students were enrolled via randomized lottery and for which school records were complete.

Our lottery-based results are only for Noble Street College Prep. However, we also report nonexperimental results on a wider group of schools in the Noble network based on school-average data from later years, through 2013.

We gathered data on student characteristics from Noble’s lottery and enrollment files. A typical lottery file included a student’s name, gender, date of birth, address, 8th-grade school, sibling indicators, lottery result (accepted or waitlist), and waitlist position. We reviewed records for both lottery losers and winners, which we defined as students immediately accepted at Noble or offered one of the first 10 waitlist positions.

Noble also provided us with enrollment records, which we merged with the lottery data to identify which students eventually enrolled. To ensure that all students in our analysis had an equal chance of being accepted, we exclude students who were automatically accepted because they had an older sibling enrolled in a Noble school. If students entered multiple lotteries over a series of years, we only included the first entry.

We linked the lottery data to information on college enrollment and persistence from the National Student Clearinghouse (NSC), a nonprofit organization that maintains a database of students’ college enrollment and graduation records. The link was made using each student’s name, date of birth, and expected high-school graduation date. At the time of our submission, the NSC database included enrollment records at more than 90 percent of colleges and universities in the United States. By necessity, we assumed that students who did not match any records in the NSC database never enrolled in college.

Noble also provided us with internally collected data on college enrollment of their graduating seniors, which were consistent with the NSC match for 93 percent of students. We use the NSC data throughout our analysis because they are available for both lottery winners and losers.

We also linked the lottery data to publicly available CPS data on students’ middle schools, including the percentage of 8th graders who scored proficient or better on the math section of the Illinois Standards Achievement Test (ISAT), the percentage scoring proficient or better on the reading section, and the percentage of black or Hispanic students. In total, we were able to match 72 percent of our sample of lottery applicants to their 8th-grade school’s CPS summary records.

Like students in most Chicago schools during the period studied, students who won the lottery and enrolled at Noble were roughly 90 percent black or Hispanic, though the school enrolled a much larger share of Hispanic students than the average charter or traditional public school at that time. In more recent years, the network’s collective student body has been quite similar to the rest of the district on nonracial dimensions. Noble students are marginally less likely to qualify for special education, but slightly more likely to be eligible for a free or subsidized lunch.

Summary data also show that Noble students perform extremely well on standardized tests. More precisely, Noble students enter high school with slightly lower test performance than the average public school student, though significantly higher than the average student at a Chicago charter school. However, by 11th grade, Noble students score markedly higher than the CPS average and the charter average on all sections of the ACT. In the 2013–14 school year, all 10 Noble campuses with enrolled 11th graders ranked in the top quarter of all Chicago high schools for overall ACT performance, scoring in the top 33 out of 156 schools with reported results. In addition, Noble students were also more likely to earn a high school diploma. Students entering 9th grade at a Noble campus were 21 percentage points more likely to graduate within five years of enrollment than their peers in traditional public schools, and 20 percentage points less likely to drop out within five years of enrollment.

The crux of our analysis is a comparison of lottery winners and losers, so it is crucial that the process used to determine who was offered a seat at Noble was truly random. A comparison of students’ data from before the lottery confirms that the two groups are similar with respect to all characteristics we can observe. We found no significant differences between lottery winners and losers on characteristics, including gender, age at high school entry, and the math and reading scores and racial composition of their middle schools.

Effect on College Enrollment

We found that students who won the Noble lottery were far more likely to enroll and persist in college than their peers who lost the lottery and thus attended high school elsewhere. Lottery winners were 10 percentage points more likely to enroll in college than students who did not win the lottery, a 17 percent increase compared to the losers’ college enrollment rate of 59 percent. When we adjusted the results for the fact that not all lottery winners attended Noble, we find that actually enrolling in Noble for any length of time increased college enrollment by 13 percentage points, or 22 percent (Figure 2). Our main analyses control for students’ age, gender, and the average test scores at their middle schools, but we obtain similar results from a simple comparison of lottery winners and losers, as we would expect given the use of the lottery.

A natural concern is that the increase in college enrollment might come at the expense of quality. What if Noble merely pushes students who are on the fence about whether or not to attend college into lower-quality schools? We found that this was not the case by examining the type of degrees offered at the schools students attended, as well as the test scores of incoming students, as a proxy for rigor and quality.

Noble students were 15 percentage points more likely to attend a four-year school and 14 percentage points more likely to attend a college where the median two-subject SAT score was above 1000—increases of 50 and 78 percent, respectively. By comparison, they were 5 percentage points more likely to attend a two-year school after graduation, a difference that was not statistically significant.

Enrolling in school is one thing. But did Noble students stick with their studies? We found that Noble students were 17 percentage points more likely to persist for two semesters or more, and 12 percentage points more likely to persist for four semesters or more. These differences in persistence were driven mainly by students enrolled at four-year colleges.

We reviewed the data to determine if the impacts of attending Noble varied based on students’ gender, middle school quality, and neighborhood poverty rate. We found few consistent patterns in the results to suggest that Noble was most effective for specific groups of students. We also investigated whether college enrollment patterns shifted over time, given the network’s rapid expansion while the students in our sample were enrolled, and found no evidence that Noble’s effectiveness declined appreciably over this period.

Success at Scale?

The experimental results clearly demonstrate that early cohorts attending Noble Street College Prep were more likely to enroll in college, enroll in selective four-year institutions, and remain enrolled for at least four semesters. But those results are for just one school. Can we interpret them to mean that the academic practices and policies implemented throughout the Noble network lead to improved college outcomes? What about at other no-excuses charter high schools with similar practices? Or was there something unique about the original campus that accounts for its students’ success?

To provide evidence on the generalizability of our main results, we carried out a secondary, nonexperimental analysis using school-average data reported by CPS to compare a broader group of Noble high schools to other Chicago high schools in later years. We examined average college enrollment rates for the graduating class of 2013 at 104 schools, seven of which were part of the Noble network. Our analysis takes into account the scores of incoming students on the ACT Explore exam administered to Chicago students in the fall of 9th grade, as well as the racial composition and percentage of students eligible for a subsidized lunch or for special education.

We found that, at the seven Noble high schools with graduating seniors in 2013, students were 19 percentage points more likely to enroll in college than one would predict based on their incoming ability (Figure 3). Noble’s college enrollment rates were among the best in the district, and rates at all seven schools surpassed expectations by a wide margin. Comparing Noble high schools only to other charter schools and adjusting for other differences in students’ background produces an estimated Noble effect of 13 percentage points on college enrollment—a large, significant difference. While complete data were not available for any other year, we repeated this analysis with the Class of 2012 using 10th-grade test scores to control for differences in student ability and found, reassuringly, a similar pattern.

We should be careful when interpreting these nonexperimental results. The research design does not benefit from the random variation used in our earlier analysis, so we cannot rule out the possibility that the students who enrolled in Noble network schools, despite their below-average test scores, would have been more likely to attend college anyway. Nevertheless, we find it reassuring that the best evidence we can muster indicates that Noble students continue to outperform expectations even during the network’s rapid expansion.

Implications

As no-excuses charter schools continue to expand, it is critical to understand whether the short-term academic gains they typically produce translate into long-term improvements in their students’ quality of life. We believe that our findings present the strongest evidence to date of long-lasting academic benefits, and should be a cause for cautious optimism. We see three elements of this analysis that should be of particular interest.

First, Noble’s educational model is broadly consistent with the practices of high-performing charter schools, and our secondary analysis suggests that scaling and reproducing these results is feasible. The effects we estimate are large, persistent, and not driven by any particular subgroup of students. While evidence is strongest in the lottery-based randomized analysis, our estimates for a larger group of students reaffirm those initial estimates.

Second, to our knowledge, our results are the first to demonstrate conclusively that a high school intervention can simultaneously improve overall college enrollment, persistence, and quality. Other studies linking high school quality to college, including evaluations of a public school-choice program in Charlotte-Mecklenburg Schools in North Carolina and of Harlem Children’s Zone Promise Academy in New York City have found impacts that are either transitory or not statistically significant.

Finally, we demonstrate the effectiveness of an intervention that occurs relatively late in students’ academic lives. Much of the public conversation around school improvement focuses on early childhood and the elementary years, in an effort to prevent or lessen inequitable outcomes for poor children. Yet Noble’s intensive academic program does not start until 9th grade, after many of its students have spent their formative years in low-performing schools. Relatively soon after, they are prepared to enroll and succeed in college, a critical step to success in adulthood. It is clear from their experience that such efforts are never too late.

Matthew Davis is a doctoral student in Applied Economics at the University of Pennsylvania. Blake Heller is a doctoral student in Public Policy at Harvard University.

Video: Brian Gill talks with Education Next

---

Charter schools have become a popular alternative to traditional public schools, with some 5,000 schools now serving more than 1.5 million students, and they have received considerable attention among researchers as a result.

Most studies focus on the effects of charter attendance on short-term student achievement (test scores), using either data sets that follow students over time (see “Results from the Tar Heel State,” research, Fall 2005) or random assignment via school admission lotteries (see “New York City Charter Schools,” research, Summer 2008) to control for differences between students in charter and traditional public schools. Beyond measuring achievement effects, however, there has been only limited analysis of the impacts of charters on the students who attend them. Even less research has been conducted on the effects of charter high schools specifically, though a large portion of all charter schools in the U.S. serve some or all of the high school grades.

Developing a high school model suited to the 21st-century student has been the Holy Grail of education reform in recent years, absorbing governors, task forces, and vast sums spent on small schools, university-based schools, and concept schools (see “High School 2.0,” features). With roughly 30 percent of American students dropping out before receiving a diploma—a rate that has been stable for several decades—assessing existing alternatives to the traditional high school is an urgent task.

In this study we use data from Chicago and Florida to estimate the effects of attending a charter high school on the likelihood that a student will complete high school and attend college. Given the impact of educational attainment on a variety of economic and social outcomes, a positive result could have significant implications for the value of school-choice programs that include charter high schools. We find evidence that charter high schools in both locations have substantial positive effects on both high school completion and college attendance. Controlling for key student characteristics (including demographics, prior test scores, and the prior choice to enroll in a charter middle school), students who attend a charter high school are 7 to 15 percentage points more likely to earn a standard diploma than students who attend a traditional public high school. Similarly, those attending a charter high school are 8 to 10 percentage points more likely to attend college (see Figure 1). Results using an alternative method designed to address concerns about unmeasured differences between students attending charter and traditional public high schools suggest even larger positive effects. Our main results are comparable to those of some studies which find that attending a Catholic high school boosts the likelihood of high school graduation and college attendance by 10 to 18 percentage points.

Methods

Determining the influence of charter school attendance on educational attainment is difficult because students who choose to attend charter high schools may be different from students who choose to attend traditional public high schools in ways that are not readily observable. The fact that the charter students and their parents actively sought out an alternative to traditional public schools suggests the students may be more motivated or their parents more involved in their child’s education than is the case for students attending traditional public schools. Since these traits are not easily measured, the estimated impact of charter high schools on educational attainment could be biased.

Our main analysis uses two methods to address students’ self selection into charter schools. First, we control for any observable differences between charter and non-charter high school students prior to high school entry. These include factors such as race/ethnicity, gender, disability status, and family income. The most important characteristic included among our statistical controls is 8th-grade test score, which aims to capture differences in student ability and students’ educational experiences prior to high school.

Second, we limit our analysis to students who attended a charter school in 8th grade, just prior to beginning high school. That is, we compare high school and postsecondary outcomes for 8th-grade charter students who entered charter high schools (the treatment group) with outcomes for 8th-grade charter students who entered conventional public high schools (the comparison group). If there are unmeasured student or family characteristics that lead to the selection of charter schools in general, these unmeasured characteristics should be relatively constant among students and families who choose charter middle schools. Unlike other nonexperimental studies of charter school impacts, our study therefore addresses student self-selection into charter schools directly by ensuring that the comparison students as well as the treatment students were once charter choosers.

Charter school 8th graders who went on to attend a charter high school differed from their peers who subsequently attended a traditional public high school in several respects, particularly in Florida, which suggests the importance of taking such differences into account when assessing the effects of charter attendance (see Figure 2). However, there may still be unmeasured differences that explain why one charter 8th grader attends a charter high school while another charter 8th grader attends a traditional public high school. For this reason, we estimate charter school effects by comparing students who are more likely to attend a charter school because they live closer to one to those less likely to attend a charter school because it is less convenient. For many charter middle-school students, attending a charter high school may be infeasible due to the lack of a charter high school within a reasonable distance. Such students make different choices not because of unmeasured characteristics, but because of a factor out of their control: the distance from home to the nearest charter school.

Data

The data required to analyze the impact of charter high schools on educational attainment are substantial. One must have data on school type (charter or public) and test scores of individual students prior to high school, individual-level high school attendance records and exit information, and college attendance after high school. Finally, the jurisdiction studied must have a sufficient enrollment of students in charter high schools to provide reliable results. The areas we analyze, the state of Florida and the city of Chicago, are two of just a handful of places where all of the necessary data elements are currently in place.

The Florida data, which cover the four cohorts of 8th-grade students from the school years 1997–98 to 2000–01, come from a variety of sources. The primary source for student-level information is the Florida Department of Education’s K-20 Education Data Warehouse (K-20 EDW), an integrated longitudinal database covering all public school students in the state of Florida. The K-20 EDW includes detailed enrollment, demographic, and program participation information for each student, as well as reading and math achievement test scores.

As the name implies, the K-20 EDW includes student records for both K–12 public school students and students enrolled in community colleges or four-year public universities in Florida. The K-20 EDW also contains information that allows us to follow students who attend private institutions of higher education within Florida. Data from the National Student Clearinghouse, a national database that includes enrollment data on 3,300 colleges from throughout the United States, is used to track college attendance outside the state of Florida. Any individual who does not show up as enrolled in a two- or four-year college or university is classified as a non-attendee.

High school graduation is measured using withdrawal information and student award data from the K-20 EDW. Only students who receive a standard high school diploma are considered to be high school graduates. Students earning a GED or special education diploma are counted as not graduating. Similarly, students who withdrew with no intention of returning or left for other reasons, such as nonattendance, court action, joining the military, marriage, pregnancy, and medical problems, but did not later graduate, are counted as not graduating.

The Chicago data, which cover the five cohorts of students who were in 8th grade during the school years 1997–98 to 2001–02, were obtained from the Chicago Public Schools. The data include 8th-grade math and reading test scores and information on student gender, race/ethnicity, bilingual status, free or reduced-price lunch status, and special education status. This data set is also linked to the National Student Clearinghouse. High school graduation is determined by withdrawal information from the Chicago Public Schools data. As in Florida, only students who receive a standard high school diploma are considered to be high school graduates.

Results

The raw data on our study population of students who were in charter schools in 8th grade reveal substantial differences in educational attainment between attendees of charter high schools and those of traditional public high schools. In Florida, 57 percent of students who went from a charter school in 8th grade to a traditional public school in 9th grade received a standard high school diploma within four years, compared to 77 percent of charter 8th graders who attended a charter high school. In Chicago, the corresponding high school graduation rates were 68 and 75 percent. Similar differences are found for college attendance. In Florida, among the study population of charter 8th graders, 57 percent of students attending a charter school in 9th grade went to either a two- or four-year college within five years of starting high school, whereas among students who started high school in a traditional public school the college attendance rate was only 40 percent. In Chicago, the gap in college attendance is smaller but still sizable: among the study population of charter 8th graders, 49 percent of students at charter high schools attended college, compared to 38 percent of students at traditional public high schools.

Controlling for student demographics, 8th-grade test scores, English language skills, special education program participation, free or reduced-price lunch status (a measure of family income), and mobility during middle school does not alter the basic patterns of graduation and college attendance seen in the descriptive comparisons. The estimated impact of attending a charter high school on the probability of obtaining a high school diploma is positive in both Florida and Chicago. In Chicago, students who attended a charter high school were 7 percentage points more likely to earn a regular high school diploma than their counterparts with similar characteristics who attended a traditional public high school. The graduation differential for Florida charter schools was even larger, at 15 percentage points. The findings for college attendance are remarkably similar in Florida and Chicago. Among the study population of charter 8th graders, students who attended a charter high school in 9th grade are 8 to 10 percentage points more likely to attend college than similar students who attended a traditional public high school (see Figure 1).

As discussed above, there remains the possibility that unobserved changes occur between 8th and 9th grade that influence both high school choice and subsequent educational attainment. For example, dissatisfaction with performance in a charter middle school that is not captured by test scores (such as discipline issues or a poor fit between the student’s interests or ability and the curriculum being offered) could lead parents to choose to send their child to a traditional public high school. When we correct for this potential bias by examining students who attended charter or traditional public school based on proximity, we continue to find highly significant positive effects of attending a charter high school on both receipt of a high school diploma and college enrollment. The magnitude of the effects is large, roughly double the size of our main results.

This pattern suggests that, among students enrolled in charter schools as 8th graders, it is those who are less likely to graduate who are choosing to attend charter high schools. We can only speculate as to why this is so. It is possible that parents whose children are at risk of dropping out are more likely to choose charter high schools in a belief that the traditional public school environment would make it more likely that their child leaves school early. Alternatively, although we control for free or reduced-price lunch eligibility, it may be the case that low-income families have a stronger preference for charter schools. If so, families with children in charter high schools would be less likely to be able to afford to send their children to college.

Possible Mechanisms

The analyses reported above cannot explain how or why charter high schools appear to produce positive effects on their students’ educational attainment. Our study lacks data on operations and instruction in the charter schools, so we have little opportunity to explore the mechanisms contributing to their success. Nonetheless, we have a few pieces of information that permit exploratory analyses of factors that might play a role.

First, it is worth considering that charter high schools may raise rates of high school graduation and college enrollment directly, or indirectly through improved academic achievement. We attempt to distinguish between these explanations by controlling in the analysis for math and reading achievement as measured in the 10th grade. Controlling for 10th-grade test scores explains about half the graduation differential for charter high schools in Florida but less than 20 percent of the difference in Chicago. And it has an even smaller effect on the results for college enrollment, reducing the estimated effect of charter school attendance by only about 10 percent in both locations. These patterns suggest that the positive effects of charter school attendance on educational attainment are not due solely to measured differences in the achievement of students in charter and traditional public high schools. This result is similar to those found in some studies of Catholic high schools, which suggest larger benefits for attainment than for test scores.

Second, given that charter high schools tend to be much smaller than traditional public high schools, charter school effects might simply be attributable to their smaller size. In order to assess this possibility, we ran the analyses for high school graduation and college attendance again with an additional control for the total number of students attending the school. The results are comparable to those reported above, indicating that the estimated effects of charter high schools are not due to differences in school size.

Third, we consider the possibility that the charters’ success might be related to grade configurations that often differ from those of traditional public schools. In the traditional public school sector in both Chicago and Florida, high schools are almost always separate from middle schools. This is not the case for charter schools. In 2001–02, about 22 percent of charter schools in Florida offering middle-school grades also offered some or all high-school grades. As a result, about 30 percent of Florida charter 8th-grade students attended schools that also offered at least some high-school grades. In Chicago, 40 percent of charter middle schools offered both middle- and high-school grades, and nearly half of the 8th-grade charter students could attend at least some high-school grades without changing schools. This raises the possibility that the measured effects of attending a charter high school on educational attainment could simply reflect advantages of grouping middle and high school grades together, thereby creating greater continuity for students and eliminating the disruption often associated with changing schools.

In order to examine whether charter-school effects might be attributable to eliminating the transition between middle and high school, we restricted the Florida analysis to those students whose 8th-grade charter school did not offer 9th grade and ran our analyses again. For high school graduation, restricting the sample produces estimates that are nearly identical to the original estimates from our main method. Using the restricted sample and our alternative method, the estimates are about 30 percent smaller than when the full sample is employed, but still large. Meanwhile, estimates of the effect of attending a charter high school on college enrollment are even larger using the restricted sample than with the original sample that includes schools offering both 8th and 9th grade. In Florida, grade configuration is not a primary driver of the estimated positive effects of charter high schools on attainment. In Chicago, however, we could not run similar analyses because grade configuration is too strongly correlated with charter status; we therefore cannot rule out the possibility that positive results in Chicago could be partly attributable to eliminating the transition from middle school to high school.

Finally, we examined an interpretive concern arising from the fact that some charter schools in Florida are former traditional public schools that converted to charter status. If conversion schools were better-than-average traditional public schools to begin with, they may be distorting the estimated impact of charters on educational attainment. We calculated separate effects for Florida conversion and non-conversion (“de novo”) charters in Florida. (In Chicago, virtually all of the charter high schools in our sample were de novo charters). We found that although Florida’s conversion charters have significantly greater effects on high school graduation than do de novo charters, the impact of non-conversion charters is still sizable (nearly equal to the estimate in Chicago). For college attendance, the estimated positive impacts of Florida’s de novo charters are statistically indistinguishable from the estimated positive impacts of Florida’s conversion charters.

Conclusions

Although a number of recent studies analyze the relationship between charter school attendance and student achievement, this is the first analysis of the impacts of charter school attendance on educational attainment. We find that charter schools are associated with an increased likelihood of successful high-school completion and an increased likelihood of enrollment at a two- or four-year college in two disparate jurisdictions, Florida and Chicago. The reasons for these large charter-school effects are not clear. There is certainly room for future work to explore how differences in curricula, expectations, peer characteristics, and other factors may cause charter schools to diminish the high-school dropout rate and ease the transition to postsecondary schooling.

Our findings are consistent with some research on the efficacy of Catholic schools, which finds substantial positive effects of attending a Catholic high school on educational attainment. While just a first step, the results presented here and in the Catholic-school literature suggest that school-choice programs that include alternatives to traditional public high schools may reduce high-school dropout rates and promote college attendance.

Kevin Booker is researcher at Mathematica Policy Research, Inc. Tim R. Sass is professor of economics at Florida State University. Brian Gill is senior social scientist at Mathematica Policy Research, Inc. Ron Zimmer is associate professor at Michigan State University. This article is adapted from research reported in Charter Schools in Eight States (RAND Corporation, 2009).

An unabridged version of this article is available here.

An interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future is available here.

---

School choice supporters, including hundreds of private school students in crisp uniforms, filled Washington, D.C.’s Freedom Plaza last May to protest a congressional decision to eliminate the city’s federally funded school voucher program after the next school year (to see additional images of this event please click here). That afternoon, President Obama announced a compromise proposal to grandfather the more than 1,700 students currently in the District of Columbia Opportunity Scholarship Program, funding their vouchers through high school graduation, but denying entry to additional children. Both program supporters and opponents cite evidence from an ongoing congressionally mandated Institute of Education Sciences (IES) evaluation of the program, for which I am principal investigator, to buttress their positions, rendering the evaluation a Rorschach test for one’s ideological position on this fiercely debated issue.

School vouchers provide funds to parents to enable them to enroll their children in private schools and, as a result, are one of the most controversial education reforms in the United States (to see an interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future please click here). Among the many points of contention is whether voucher programs in fact improve student achievement. Most evaluations of such programs have found at least some positive achievement effects, but not always for all types of participants and not always in both reading and math. This pattern of results has so far failed to generate a scholarly consensus regarding the beneficial effects of school vouchers on student achievement. The policy and academic communities seek more definitive guidance.

The IES released the third-year impact evaluation of the Opportunity Scholarship Program (OSP) in April 2009. The results showed that students who participated in the program performed at significantly higher levels in reading than the students in an experimental control group. Here are the study findings and my own interpretation of what they mean.

Opportunity Scholarships

Currently, 13 directly funded voucher programs operate in four U.S. cities and six states, serving approximately 65,000 students. Another seven programs indirectly fund private K—12 scholarship organizations through government tax credits to individuals or corporations. About 100,000 students receive school vouchers funded through tax credits. All of the directly funded voucher programs are targeted to students with some educational disadvantage, such as low family income, disability, or status as a foster child.

Nineteen of the 20 school voucher programs in the U.S. are funded by state and local governments. The OSP is the only federal voucher initiative. Established in 2004 as part of compromise legislation that also included new spending on charter and traditional public schools in the District of Columbia, the OSP is a means-tested program. Initial eligibility is limited to K—12 students in D.C. with family incomes at or below 185 percent of the poverty line. Congress has appropriated $14 million annually to the program, enough to support about 1,700 students at the maximum voucher amount of $7,500. The voucher covers most or all of the costs of tuition, transportation, and educational fees at any of the 66 D.C. private schools that have participated in the program. By the spring of 2008, a total of 5,331 eligible students had applied for the limited number of Opportunity Scholarships. Recipients are selected by lottery, with priority given to students applying to the program from public schools deemed in need of improvement (SINI) under No Child Left Behind. Scholars and policymakers have since questioned the extent to which SINI designations accurately signal school quality because they are based on levels of achievement instead of the more informative measure of achievement gains over time.

The third-year impact evaluation tracked the experiences of two cohorts of students. All of the students were attending public schools or were rising kindergartners at the time of application to the program. Cohort 1 consisted of 492 students entering grades 6—12 in 2004. Cohort 2 consisted of 1,816 students entering grades K—12 in 2005. The 2,308 students in the study make it the largest school voucher evaluation in the U.S. to employ the “gold standard” method of random assignment.

Voucher Effects
Researchers over the past decade have focused on evaluating voucher programs using experimental research designs called randomized control trials (RCTs). Such experimental designs are widely used to evaluate the efficacy of medical drugs prior to making such treatments available to the public. With an RCT design, a group of students who all qualify for a voucher program and whose parents are equally motivated to exercise private school choice, participate in a lottery. The students who win the lottery become the “treatment” group. The students who lose the lottery become the “control” group. Since only a voucher offer and mere chance distinguish the treatment students from their control group counterparts, any significant difference in student outcomes for the treatment students can be attributed to the program. Although not all students offered a voucher will use it to enroll in a private school, the data from an RCT can also be used to generate a separate estimate of the effect of voucher use (see sidebar, page 50).

Using an RCT research design, the ongoing IES evaluation found no impacts on student math performance but a statistically significant positive impact of the scholarship program on student reading performance, as measured by the Stanford Achievement Test (SAT 9). The estimated impact of using a scholarship to attend a private school for any length of time during the three-year evaluation period was a gain of 5.3 scale points in reading. That estimate provides the impact on all those who ever attended a private school, whether for one month, three years, or any length of time in between (see Figure 1). Consequently, the estimate should be interpreted as a lower-bound estimate of the three-year impact of attending a private school, because many students who used a scholarship during the three-year period did not remain in private school throughout the entire period. The data indicate that members of the treatment group who were attending private schools in the third year of the evaluation gained an average of 7.1 scale score points in reading from the program.

What do these gains mean for students? They mean that the students in the control group would need to remain in school an extra 3.7 months on average to catch up to the level of reading achievement attained by those who used the scholarship opportunity to attend a private school for any period of time. The catch-up time would have been around 5 months for those in the control group as compared to those who were attending a private school in the third year of the evaluation.

Over time, in my opinion, the effects of the program show a trend toward larger reading gains cumulating for students. Especially when one considers that students who used their scholarship in year 1 needed to adjust to a new and different school environment, the reading impacts of using a scholarship of 1.4 scale score points (not significant) in year 1, 4.0 scale score points (not significant) in year 2, and 5.3 scale score points (significant) in year 3 suggest that students are steadily gaining in reading performance relative to their peers in the control group the longer they make use of the scholarship. No trend in program impacts is evident in math.

What explains the fact that positive impacts have been observed as a result of the OSP in reading but not in math? Paul Peterson and Elena Llaudet of Harvard University, in a nonexperimental evaluation of the effects of school sector on student achievement, suggest that private schools may boost reading scores more than math scores for a number of reasons, including a greater content emphasis on reading, the use of phonics instead of whole-language instruction, and the greater availability of well-trained education content specialists in reading than in math. Any or all of these explanations for a voucher advantage in reading but not in math are plausible and could be behind the pattern of results observed for the D.C. Opportunity Scholarships. The experimental design of the D.C. evaluation, while a methodological strength in many ways, makes it difficult to connect the context of students’ educational experiences with specific outcomes in any reliable way. As a result, one can only speculate as to why voucher gains are clear in reading but not observed in math.

Student Characteristics
The OSP serves a highly disadvantaged group of D.C. students. Descriptive information from the first two annual reports indicates that more than 90 percent of students are African American and 9 percent are Hispanic. Their family incomes averaged less than $20,000 in the year in which they applied for the scholarship.

Overall, participating students were performing well below national norms in reading and math when they applied to the program. For example, the Cohort 1 students had initial reading scores on the SAT-9 that averaged below the 24th National Percentile Rank, meaning that 75 percent of students in their respective grades nationally were performing higher than Chart 1 in reading. In my view, these descriptive data show how means tests and other provisions to target school voucher programs to disadvantaged students serve to minimize the threat of cream-skimming. The OSP reached a population of highly disadvantaged students because it was designed by policymakers to do so.

Did Only Some Students Benefit?

Several commentators have sought to minimize the positive findings of the OSP evaluation by suggesting that only certain subgroups of participants benefited from the program. Martin Carnoy states that “the treated students in Cohort 1 were concentrated in middle schools and the effect on their reading score was significantly higher than for treated students in Cohort 2.” Henry Levin likewise asserts that “the evaluators found that receiving a voucher resulted in no advantage in math or reading test scores for either [low achievers or students from SINI schools].”

The actual results of the evaluation provide no scientific basis for claims that some subgroups of students benefited more in reading from the voucher program than other subgroups. The impact of the program on the reading achievement of Cohort 1 students did not differ by a statistically significant amount from the impact of the program on the reading achievement of Cohort 2 students, Carnoy’s claim notwithstanding. Nor did students with low initial levels of achievement and applicants from SINI schools experience significantly different reading gains from the program than high achievers and non-SINI applicants. The mere fact that statistically significant impacts were observed for a particular subgroup does not mean that impacts for that group are significantly different from those not in the subgroup. For example, Group A and Group B may have experienced roughly similar impacts, but the impact for Group A might have been just large enough for it to be significantly different from zero (or no impact at all), while Group B’s quite similar scores fell just below that threshold.

From a scientific standpoint, three conclusions are valid about the achievement results in reading from the year 3 impact evaluation of the OSP:

• The program improved the reading achievement of the treatment group students overall.
• Overall reading gains from the program were not significantly different across the various subgroups examined.
• Three distinct subgroups of students—those who were not from SINI schools, students scheduled to enter grades K-8 in the fall after application to the program, and students in the higher two-thirds of the performance distribution (whose average reading test scores at baseline were at the 37th percentile nationally)—experienced statistically significant reading impacts from the program when their performance was examined separately. Female students and students in Cohort 1 saw reading gains that were statistically significant with reservations due to the possibility of obtaining false positive results when making comparisons across numerous subgroups.
  Why examine and report achievement impacts at the subgroup level, if the evidence indicates only an overall reading gain for the entire sample? The reasons are that Congress mandated an analysis of subgroup impacts, at least for SINI and non-SINI students, and because analyses at the subgroup level might have yielded more conclusive information about disproportionate impacts for certain types of students.

Expanding Choice

The OSP facilitates the enrollment of low-income D.C. students in private schools of their parents’ choosing. It does not guarantee enrollment in a private school, but the $7,500 voucher should make such enrollments relatively common among the students who won the scholarship lottery. The eligible students who lost the scholarship lottery and were assigned to the control group still might attend a private school but they would have to do so by drawing on resources outside of the OSP. At the same time, students in both groups have access to a large number of public charter schools.

The implication is that, for this evaluation of the OSP, winning the lottery does not necessarily mean private schooling, and losing the lottery does not necessarily mean education in a traditional public school. Members of both groups attended all three types of schools—private, public charter, and traditional public—in year 3 of the voucher experiment, although the proportions that attended each type differed markedly based on whether or not they won the scholarship lottery (see Figure 2). In total, about 81 percent of parents placed their child in a private or public school of choice three years after winning the scholarship lottery, as did 46 percent of those who lost the lottery. The desire for an alternative to a neighborhood public school was strong for the families who applied to the OSP in 2004 and 2005.

These enrollment patterns highlight the fact that the effects of voucher use reported above do not amount to a comparison between “school choice” and “no school choice.” Rather, voucher users are exercising private school choice, while control group members are exercising a small amount of private school choice and a substantial amount of public school choice. The positive impacts on reading achievement observed for voucher users therefore reflect the incremental effect of adding private school choice through the OSP to the existing schooling options for low-income D.C. families.

Parent Satisfaction
Another key measure of school reform initiatives is the perception among parents, who see firsthand the effects of changes in their child’s educational environment. Whenever school choice researchers have asked parents about their satisfaction with schools, those who have been given the chance to select their child’s school have reported much higher levels of satisfaction. The OSP study findings fit this pattern. The proportion of parents who assigned a high grade of A or B to their child’s school was 11 percentile points higher if they were offered a voucher, 12 percentile points higher if their child actually used a scholarship, and 21 points higher if their child was attending a private school in year 3, regardless of whether they were in the treatment group. Parents whose children used an Opportunity Scholarship also expressed greater confidence in their children’s safety in school than parents in the control group.

Additional evidence of parental satisfaction with the OSP comes from the series of focus groups conducted independently of the congressionally mandated evaluation. One parent emphasized the expanded freedom inherent in school choice:

“[The OSP] gives me the choice to, freedom to attend other schools than D.C. public schools….I just didn’t feel that I wanted to put him in D.C. public school and I had the opportunity to take one of the scholarships, so, therefore, I can afford it and I’m glad that I did do that.” (Cohort 1 Elementary School Parent, Spring 2008)

Another parent with two children in the OSP may have hinted at a reason achievement impacts were observed specifically in reading:

“They really excel at this program, `cause I know for a fact they would never have received this kind of education at a public school….I listen to them when they talk, and what they are saying, and they articulate better than I do, and I know it’s because of the school, and I like that about them, and I’m proud of them.” (Cohort 1 Elementary School Parent, Spring 2008)

These parents of OSP students clearly see their families as having benefited from this program.

Previous Voucher Research

The IES evaluation of the DC OSP adds to a growing body of research on means-tested school voucher programs in urban districts across the nation. Experimental evaluations of the achievement impacts of publicly funded voucher and privately funded K—12 scholarship programs have been conducted in Milwaukee, New York City, the District of Columbia, Charlotte, North Carolina, and Dayton, Ohio. Different research teams analyzed the data from New York City (three different teams), Milwaukee (two teams), and Charlotte (two teams). The four studies of Milwaukee’s and Charlotte’s programs reported statistically significant achievement gains overall for the members of the treatment group. The individual studies of the privately funded K—12 scholarship programs in the District of Columbia and Dayton reported overall achievement gains only for the large subgroup of African American students in the program. The three different evaluators of the New York City privately funded scholarship program were split in their assessment of achievement impacts, as two research teams reported no overall test-score effects, but did report achievement gains for African Americans; the third team claimed there were no statistically significant test-score impacts overall or for any subgroup of participants.

The specific patterns of achievement impacts vary across these studies, with some gains emerging quickly, but others, like those in the OSP evaluation, taking at least three years to reach a standard level of statistical significance. Earlier experimental evaluations of voucher programs were somewhat more likely to report achievement gains from the programs in math than in reading—the opposite of what was observed for the OSP. Despite these differences, the bulk of the available, high-quality evidence on school voucher programs suggests that they do yield positive achievement effects for participating students.

Conclusions
School voucher initiatives such as the District of Columbia Opportunity Scholarship Program will remain politically controversial in spite of rigorous evaluations such as this one, showing that parents and students benefited in some ways from the program. Critics will continue to point to the fact that no impacts of the program have been observed in math, or that applicants from SINI schools, who were a service priority, have not demonstrated statistically significant achievement gains at the subgroup level, as reasons to characterize these findings as disappointing. Certainly the results would have been even more encouraging if the high-priority SINI students had shown significant reading gains as a distinct subgroup. Still, in my opinion, the bottom line is that the OSP lottery paid off for those students who won it. On average, participating low-income students are performing better in reading because the federal government decided to launch an experimental school choice program in our nation’s capital.

The achievement results from the D.C. voucher evaluation are also striking when compared to the results from other experimental evaluations of education policies. The National Center for Education Evaluation and Regional Assistance (NCEE) at the IES has sponsored and overseen 11 studies that are RCTs, including the OSP evaluation. Only 3 of the 11 education interventions tested, when subjected to such a rigorous evaluation, have demonstrated statistically significant achievement impacts overall in either reading or math. The reading impact of the D.C. voucher program is the largest achievement impact yet reported in an RCT evaluation overseen by the NCEE. A second program was found to increase reading outcomes by about 40 percent less than the reading gain from the DC OSP. The third intervention was reported to have boosted math achievement by less than half the amount of the reading gain from the D.C. voucher program. Of the remaining eight NCEE-sponsored RCTs, six of them found no statistically significant achievement impacts overall and the other two showed a mix of no impacts and actual achievement losses from their programs. Many of these studies are in their early stages and might report more impressive achievement results in the future. Still, the D.C. voucher program has proven to be the most effective education policy evaluated by the federal government’s official education research arm so far.

The experimental evaluation of the District of Columbia Opportunity Scholarship Program is continuing into its fourth and final year of studying the impacts on students and parents. The final evidence collected from the participants may confirm the accumulation of achievement gains in reading and higher levels of parental satisfaction from the program that were evident after three years, or show that those gains have faded. Uncertainty also surrounds the program itself, as the students who gathered on Freedom Plaza in May currently are only guaranteed one final year in their chosen private schools. What will policymakers see as they continue to consider the results of this evaluation? The educational futures of a group of low-income D.C. schoolchildren hinge on the answer.

Patrick J. Wolf is professor of education reform at the University of Arkansas and principal investigator of the D.C. Opportunity Scholarship Program Impact Evaluation. The opinions expressed in this article are his own.

An unabridged version of this article is available here.

---

Methodology Notes
If one’s purpose is to evaluate the effects of a specific public policy, such as the District of Columbia Opportunity Scholarship Program (OSP), then the comparison of the average outcomes of the treatment and control groups, regardless of what proportion attended which types of school, is most appropriate. A school voucher program cannot force scholarship recipients to use a voucher, nor can it prevent control-group students from attending private schools at their own expense. A voucher program can only offer students scholarships that they subsequently may or may not use. Nevertheless, the mere offer of a scholarship, in and of itself, clearly has no impact on the educational outcomes of students. A scholarship could only change the future of a student if it were actually used.

Fortunately, statistical techniques are available that produce reliable estimates of the average effect of using a voucher compared to not being offered one and the average effect of attending private school in year 3 of the study with or without a voucher compared to not attending private school. All three effect estimates—treatment vs. control, effect of voucher use, and impact of private schooling—are provided in the longer version of this article (see “Summary of the OSP Evaluation” at www.educationnext.org), so that individual readers can view those outcomes that are most relevant to their considerations.

I have presented mainly the impacts of scholarship use in this essay. Those impacts are computed by taking the average difference between the out comes of the entire treatment and control groups—the pure experimental impact—and adjusting for the fact that some treatment students never used an Opportunity Scholarship. Since nonusers could not have been affected by the voucher, the impact of scholarship use can be computed easily by dividing the pure experimental impact by the proportion of treatment students who used their scholarships, effectively rescaling the impact across scholarship users instead of all treatment students including nonusers. I focus here on scholarship usage because that specific measure of program impact is easily understood, is relevant to policymakers, and preserves the control group as the natural representation of what would have happened to the treatment group absent the program, including the fact that some of them would have attended private school on their own.