In 2010, the Charlotte-Mecklenburg (North Carolina) school district shuttered four of its eight middle schools, opting to serve students in elementary schools spanning kindergarten through grade 8. In so doing, it followed in the footsteps of urban school districts such as Baltimore, Milwaukee, Philadelphia, and New York City, all of which have in the past decade expanded their reliance on the once ubiquitous K–8 model.

Not all school systems are moving in that direction. In Cambridge, Massachusetts, a district with surprisingly low student performance given the substantial per-pupil resources at its command, the school committee has decided to try to boost student achievement by abandoning its K–8 model in favor of having separate middle schools that serve grades 6 through 8 (though, in an unusual twist, each of the latter will be housed in the same facility as an elementary school).

In short, policymakers nationwide continue to wrestle with a basic question: At what grade level should students move to a new school? In the most common grade configuration in American school districts, public school students make two school transitions, entering a middle school in grade 6 or 7 and a high school in grade 9. This pattern reflects the influence of enrollment pressures and pedagogical theories that, over the past half century, all but eliminated the K–8 school from the American education landscape. A small fraction of students do attend public schools encompassing grades K–8, 6–12, or even K–12, however. We exploit this variation by comparing the achievement trajectories of Florida students entering a middle school or a high school to those of their peers who do not make those transitions.

Our study extends research conducted in New York City (see “Stuck in the Middle,” research, Fall 2010), in which Jonah Rockoff and Benjamin Lockwood found that entering a middle school causes a sharp drop in student achievement relative to the performance of those remaining in K–8 schools. It is hard to know whether one can generalize from results from the nation’s largest city (and school district), however, especially when it employs a complex procedure for assigning students to middle schools. Also, the New York City study was unable to follow students after 8th grade, making it impossible to know whether the negative impacts that were observed were temporary or extended into high school. This is a critical question inasmuch as a key rationale for middle school is its potential for easing the transition to high school. What is lost at the first transition may be more than gained at the second, which is presumably less abrupt for the middle-school child than for the one entering high school directly from an elementary-school environment.

To explore these issues, we use statewide data covering all students in Florida public schools who were in grades 3 to 10 between 2000 and 2009. Although a large majority of Florida students enter a middle school in grade 6, some do so in grade 7. Still others attend K–8 schools and avoid the middle-school transition altogether. To determine whether entering a middle school in grade 6 or grade 7 has any effect on achievement, we examine whether students experience a drop in test scores relative to students in K–8 schools that coincides with their transition to the new school. In the same way, we compare the learning trajectories of students entering high school in grade 9 to those of students who attend K–12, 6–12, or 7–12 schools in order to determine whether high-school transitions affect achievement.

Our results cast serious doubt on the wisdom of the middle-school experiment that has become such a prominent feature of American education. We find that moving to a middle school causes a substantial drop in student test scores (relative to that of students who remain in K–8 schools) the first year in which the transition takes place, not just in New York City but also in the big cities, suburbs, and small-town and rural areas of Florida. Further, we find that the relative achievement of middle-school students continues to decline in the subsequent years they spend in such schools. Nor do we find any sign that the middle-school students catch up with those who remained in the K–8 environment once all of them have entered high school. On the contrary, students entering a middle school in grade 6 are more likely not to be enrolled in any Florida public school as 10th graders (despite having been enrolled in grade 9), a strong indication that they have dropped out of school by that time.

We also find that the transition to high school causes a small drop in student achievement for all students who make this transition (as distinct from those in schools with 6–12 grade configurations). However, this drop holds far less policy significance both because of its size and because the decline does not appear to persist beyond grade 9.

The achievement drops we observe as students move to both middle and high schools suggest that moving from one school to another (or simply being in the youngest grade in a school) adversely affects student performance. The size and persistence of the effect of entering a middle school, however, suggests that such transitions are particularly damaging for adolescent students or that middle schools provide lower-quality education than K–8 schools provide for students at the same point in their education.

Data and Method

We draw the data for our analysis from the Florida Department of Education’s PK-20 Education Data Warehouse. The data contain state math and reading test scores for all Florida students attending public schools in grades 3 to 10 from the 2000–01 through 2008–09 school years. They also include information on the school each student attends and its location as well as student characteristics such as ethnicity, gender, special education classification, and eligibility for a free or reduced-price lunch.

We use different samples of students for different parts of our analysis. First, to estimate the effect of entering a middle school in grade 6 or 7, we examine only students enrolled in grade 3 between 2001 and 2004 who completed the state test in both math and reading in each of the subsequent five years. Second, to investigate whether the effects of middle-school entry persist through grades 9 and 10, we examine only students enrolled in grade 3 in 2001 or 2002 who were tested in both subjects each of the following seven years. Finally, to estimate the effect of entering high school in grade 9, we examine students enrolled in grade 6 between 2001 and 2005 who were tested in both math and reading in the following four years.

Our strategy for identifying the effects of alternative grade configurations on student achievement parallels and extends that of Rockoff and Lockwood’s study of New York City middle schools mentioned above. Specifically, we examine changes in individual students’ achievement over time, focusing on differences in the timing of students’ entry into middle school that result from the grade configuration of the school the student attended in 3rd grade. For example, we are interested in whether students who attended a K–6 school in 3rd grade experience a drop in their achievement in 7th grade relative to students who attended a K–8 school in 3rd grade and thus did not switch schools between grades 6 and 7.

The key assumption of our methodology is that there are no unobserved differences between students who in 3rd grade attended schools that had these different grade configurations that affect achievement precisely in the year when students enter middle school. In other words, we are assuming that the negative effect of a transition is not anticipated by parents and reflected in the choice of a school with a particular grade configuration in grade 3. We conduct an analogous analysis of high-school entry, taking advantage of the different grade configurations of the schools students attended in 6th grade.

Because we compare the achievement of individual students to themselves over time, our analysis takes into account all student characteristics (both observed and unobserved) that do not change over time. In addition, we also control for whether the individual student had been retained in a grade, whether the student had ever been retained, and whether the student attends a charter school (which in Florida are more likely than traditional public schools to have K–8 configurations).

The Middle-School Cliff

We find that students who will enter a middle school in 6th or 7th grade have positive achievement trajectories in math and reading from 3rd grade to 5th, relative to their counterparts who will never enter a middle school because they attend a school that continues through 8th grade. Achievement in both subjects falls dramatically in 6th grade for students who enter middle school in that grade. Students who will enter middle school in grade 7 continue to improve relative to their K–8 peers through grade 6, but experience a sharp drop in achievement upon entering middle school in grade 7.

Specifically, we find math achievement falls by 0.12 standard deviations and reading achievement falls by 0.09 standard deviations for transitions at grade 6 (see Figure 1). Students who make the transition at grade 7 experience even larger drops in their achievement of 0.22 and 0.15 standard deviations in math and reading, respectively. National data indicate that student achievement increases by roughly 0.30 standard deviations in math and 0.25 standard deviations in reading each year for typical 6th- and 7th-grade students. The drops in achievement we observe for students entering middle schools therefore amount to between 3.5 and 7 months of expected learning over the course of a 10-month school year.

Just as troubling is the fact that these students’ relative performance in both subjects continues to decline in subsequent middle-school grades. After three years in a middle school, students who entered in 6th grade score 0.23 standard deviations in math and 0.14 standard deviations in reading worse than we would have expected had they attended a K–8 school. After two years in a middle school, students who entered in 7th grade underperform by 0.31 standard deviations in math and 0.15 standard deviations in reading.

We also find little evidence that students who attend middle school make larger achievement gains than their peers in grades 9 and 10, by which time most Florida students have entered high school. In addressing this issue we must limit our attention to the two cohorts of students entering 3rd grade prior to 2001 or 2002, whose progress we are able to follow through the 10th grade. Although the math achievement of students who entered middle school in 7th grade improves by 0.05 standard deviations in 9th grade relative to students who attended K–8 schools, the same pattern is not evident in reading or in either subject for the much larger group of students who entered middle school in 6th grade (see Figure 2). In other words, we can safely reject the hypothesis that students who attend middle schools benefit at the transition to high school from their previous experience with school transition or from the specific educational programs available in middle schools.

Investigating the transition to high school, we find that students moving to a new high school between grades 8 and 9 suffer a small drop in achievement of 0.03 standard deviations in math and 0.04 standard deviations in reading (relative to those in grade 6–12 schools or schools with another configuration that requires no transition at this point). However, their relative achievement trajectories become positive again after this drop at the transition point.

We supplement our analysis on math and reading achievement with similar analyses of the effects of entering a middle school on the probability of students’ not being enrolled in a Florida public school in 10th grade (a proxy for dropping out of high school by this time) and on being retained in 9th grade (often a strong predictor that a student will leave school prior to graduation). Our results suggest that entering a middle school in 6th grade increases the probability of early dropout by 1.4 percentage points (or 18 percent). Although entering a middle school in 7th grade does not appear to increase early dropout, it increases the probability that a student will be retained in 9th grade by 1 percentage point. Both results provide additional cause for concern with the middle-school model.

Is it possible that our results reflect differences across school districts that employ alternative grade configurations? We explore this question by conducting our test-score analysis separately for schools in Miami-Dade County. With more than 345,000 students, Miami-Dade is the largest district in Florida and offers a wide range of grade configurations for students up through grade 8. We find that the negative effects of entering a middle school for grade 6 or grade 7 are, if anything, even more pronounced in Miami-Dade County than they are statewide.

Not Just an Urban Problem

This result for Miami-Dade County raises the possibility that the negative effects of middle-school entry are only notable in urban settings. We address this issue by looking separately at the effects of entering a middle or high school across communities of varying sizes. Using Census Bureau classifications, we group students into three categories according to the location of the school they attended in 3rd grade: 1) a large or midsize city, 2) suburbia (specifically, the urban fringe of a large or midsize city), and 3) towns and rural areas. The results suggest that the negative effects of entering a middle school are most pronounced in cities, but they remain sizable even in rural areas, confirming that the negative effects of configurations that separate the middle-school grades are by no means limited to urban school districts.

We also examine whether the middle-school effect varies across subgroups of students defined in terms of prior test performance, ethnicity, and gender. Students whose 3rd-grade scores were below the statewide median saw substantially larger declines in math scores at both the middle- and high-school transition points than higher-achieving students. These patterns are consistent with the theory that lower-achieving students have access to fewer educational resources outside of school and may therefore be at higher risk of being adversely affected by school transitions. We find no clear indication that the negative effect differs in size for higher- and lower-achieving students in reading, however.

Results for students of different ethnicities follow a similar pattern. Grade configuration has a larger effect on the math scores of traditionally disadvantaged subgroups than on other students. Black students in particular demonstrate large relative gains in math achievement prior to entering a middle school but then suffer larger drops both at and following the transition. Again, however, we find only small and statistically insignificant differences between the effects estimated for students of different ethnicities in reading. We find no differences in the effects for girls and boys.

Potential Explanations

Our results confirm that transitions into both middle schools and high schools cause drops in student achievement but that these effects are far larger for students entering middle schools. One possible interpretation of this pattern is that school transitions are more disruptive for younger students, perhaps because they are more susceptible to the negative influence of older students. Yet our estimates suggest that the effect of middle-school entry on student achievement is larger for students entering in grade 7 than for students entering in grade 6. Moreover, the fact that relative achievement continues to decline after students’ initial entry into middle schools suggests that average educational quality in Florida is lower in stand-alone middle schools than in schools serving grades K–8.

To explore why this might be the case, we first examine several characteristics of Florida elementary, middle, and K–8 schools. The most striking difference across school types involves cohort sizes (the average number of students in each grade). Although middle schools offer far fewer grades than K–8 schools, Florida middle schools on average enroll 146 more students than their K–8 counterparts; as a result, typical grade cohorts are almost three times as large. Florida middle schools also spend 11 percent less per student and have higher student-teacher ratios than K–8 schools, suggesting a potential role for differences in available resources. In contrast, we find no evidence that differences in observed teacher characteristics could explain our findings. Average teacher experience and average teacher salaries are similar across school types, while the share of the school’s instructional staff without prior experience is modestly higher in K–8 schools.

We conduct two analyses to shed light on whether these observed differences between middle schools and K–8 schools are likely to contribute to differences in school quality. First, we rerun our test-score analysis while controlling for these differences and find a similar pattern of results. Second, we examine whether the size of the drop in relative achievement suffered by students entering middle school in grade 6 varied with the characteristics of the middle school they attended. The results of this analysis again provide little evidence that low middle-school quality stems from differences in the school characteristics we can observe.

Middle schools could also differ from K–8 schools in their educational practices in ways that lead to lower student-achievement gains. To explore this possibility, we draw on a unique survey of Florida school principals conducted in 2003–04 to document responses to the state’s high-stakes accountability system. Confidentiality requirements preclude us from linking survey responses to specific schools, but we can document any differences in the average responses offered by principals of different school types.

We find few significant differences in the educational practices of the two groups of schools in our study. In particular, we observe no differences in the length of the school day or in measures of the extent to which schools had adopted specific policies to help low-performing students, policies to improve the performance of ineffective teachers, and incentives to reward highly effective teachers. If anything, these measures suggest that middle schools are more likely to have policies aimed at improving student achievement. We also find no differences across school types when we measure the degree of teacher autonomy.

A final set of survey items asked not about specific policies or practices but about the school’s overall climate. On these items, middle-school principals expressed significantly lower levels of agreement with statements indicating that their new and veteran teachers were excellent. This suggests that teachers in these schools may be less well equipped to deal with the challenges presented by their students. More middle-school principals also agreed with the statement that parents are worried about violence in the school. Although differences on the remaining items were statistically insignificant, they consistently point in the direction of middle schools having less-favorable school climates than K–8 schools.

In short, we find little evidence that the negative effects of attending a middle school are attributable to differences in resources, cohort sizes, or educational practices. We do, however, find suggestive evidence that the overall climate for student learning is worse in middle schools than in schools that serve students from elementary school through the 8th grade. This suggests a final potential interpretation of our results that is directly related to the choice of grade configuration: students may benefit from being among the oldest students in a school setting that includes very young students, perhaps because they have greater opportunity to take on leadership roles. This interpretation could account for both the gains in relative achievement made by students in K–5 and K–6 schools prior to entering middle schools and the superior performance of K–8 students relative to their peers in middle schools. A possible, if unlikely, alternative explanation is that students entering schools with different grade configurations have different growth trajectories for reasons having nothing to do with their schooling environment.

Taken as a whole, our results suggest that school transitions lower student achievement but that attending middle schools in particular has adverse consequences for American students. Especially when considered along with those of other recent studies, our findings clearly support ongoing efforts in urban school districts to convert stand-alone elementary and middle schools into schools with K–8 configurations. They are also relevant to the expanding charter-school sector, which has the opportunity to choose grade configurations without the disruption caused by school closures. More research is needed to see whether policy or pedagogical innovations can mitigate the effects of middle school. In the meantime, policymakers should exercise caution before extending the middle-school experiment to school districts that still enjoy the K–8 configuration.

Martin R. West is assistant professor of education at the Harvard Graduate School of Education and deputy director of the Program on Education Policy and Governance (PEPG) at Harvard’s Kennedy School. Guido Schwerdt is a researcher at the Ifo Institute for Economic Research in Munich, Germany.

Class Warfare: Inside the Fight to Fix America’s Schools
By Steven Brill
Simon & Schuster, 2011, $28.00; 496 pages.

 

As reviewed by Nathan Glazer

 

Steven Brill’s Class Warfare must be the most prominently reviewed book on education in decades: a lengthy front-page review by Sara Mosle in the New York Times Book Review, a lead review by Joel Klein in the Wall Street Journal, a critical follow-up piece on Brill on the news pages of the Times by Michael Winerip. Brill has had a varied career, founding The American Lawyer magazine and Court TV, writing books on the teamsters and on the effects of 9/11, but he had not dealt with American schools until he wrote a sensational article for The New Yorker on the “rubber rooms,” the rooms in which New York City teachers whom the administration believes should be fired spend their days and years—an average of three—getting their salaries, accruing their benefits, and doing nothing while the arbitration procedures dictated by the union contract grind on.

Class Warfare takes a wider view but one clearly influenced by the experience of the irrationality and inefficiency imposed by lengthy union contracts, which dictate in detail what can and cannot be done in disciplining or, indeed, leading and guiding teachers. For Brill and the reformers in his book, the unions are the enemy, with their defense of incompetent teachers, their hostility to charters, and their resistance to efforts to judge teachers by the achievement of their students on tests. Those who are trying to reform American schools are to Brill defined by their embrace of these measures. Brill tells the story of reform, particularly during the brief years of the Obama administration, beginning with the Race to the Top, through the experience of a varied group of reformers. He begins with a number of individuals shaped by their early experience in Teach For America (TFA), and he follows them and their careers—in New Orleans, Colorado, Washington, and New York—episodically through the book. They are supplemented in his account by political insiders; by Wall Streeters who have developed an interest in education reform; by vigorous administrators trying to implement reform measures, such as Joel Klein, Michelle Rhee, Mark Roosevelt of Pittsburgh, and others; and by billionaire philanthropists such as Eli Broad and Bill Gates. Interestingly, almost all his reformers are Democrats, who face the problem of reconciling measures opposed by the teachers unions with the reality that these are the solidest supporters of Democratic legislators, governors, and presidents. Along the way, Brill gives background on Albert Shanker and the rise of the American Federation of Teachers (AFT), the 1983 report of President Reagan’s National Commission on Excellence in Education and what presidents have done since, the creation of TFA by Wendy Kopp, and on David Levin and KIPP (Knowledge Is Power Program) and other successful efforts to create charter schools.

New York City, about whose public schools, teachers, and unions Brill learned a great deal in his research on the “rubber rooms,” looms large in the accounts, and recurrently we are introduced to Joel Klein struggling with the union on a new contract. Brill generally presents him announcing some victory, which on close examination seems less like a concession than it did at first glance. In his review of Brill’s book, Klein defends himself: the contract of 2005 resulted in “ending forced placement of teachers in schools based on seniority, recapturing a 45-minute period that had previously been given over to teachers…, eliminating certain grievance procedures, and extending the school day for some 300,000 struggling students by 150 minutes a week.” Of course, the perspectives of writer and of administrator, hemmed in by a legislature bound to the union, in a state in which almost everything needs legislative approval, must differ.

And yet in the book itself we learn the limitation of, for example, the ending of “forced placement.” Yes, teachers could not on the basis of their seniority impose themselves on a principal who did not want them, but neither could they be assigned to a school they did not want to teach in. Brill writes, “If either the principal or the teacher did not agree on a placement, then the teacher would stay in limbo as long as it took… but still on the payroll….” The result: “Within five years there would be more than one thousand teachers sitting on a list called the Absent Teacher Reserve. These were the teachers who had been excessed but had not taken positions elsewhere.”

One would have thought the scandal exposed by the rubber-room article would have led to correction, and indeed it did—to some extent. The rubber room had been radically reduced by the time of Brill’s book, from 744 to 83 teachers. But only 33 had been terminated as a result of arbitration. “Another 154 had been allowed to resign (typically in return for receiving some kind of severance payment and being able to keep their pensions)…474…had been…‘returned to service.’ Some went back into classrooms. But in a deft bureaucratic shell game, most of them—272 of these 474 cases—were simply added to the Absent Teacher Reserve list, where they were still paid to do nothing.”

Race to the Top, launched by a Democratic administration, did propose to give large sums to states (but still a pittance of their huge expenditures on public education), which, as part of a general plan for improvement, adopted certain of the favored reform measures, allowing the formation of charter schools and introducing evaluation and compensation of teachers based on the results achieved by their pupils. The necessary legislation has followed in many states, but Brill is not persuaded: in New York State, there is the proviso that “nothing in the law could override existing union contracts.” The procedures prescribed in union contracts remain valid. And, indeed, despite the law, the union is still disputing in the courts the degree to which tested student progress can be taken into account in evaluating teachers.

Despite 420 pages of what amounts to a brief against the unions, there is a surprising about-face in the last 20.

In conversation with David Levin at a New York KIPP school, Brill faces up to the enormous strain on teachers in KIPP and other achieving charter schools and in TFA, a strain that they can take for a few years but will not choose for a lifetime. Substantial improvement in the education of American schoolchildren has to be based not on those rare individuals who are willing to do this, but on raising the level of competence of a workforce of millions. Can one believe that the practices of those millions can be changed for the better by the competition of charter schools (1.5 million children versus 50 million in district schools), by promotion and compensation and dismissal based on test scores of their classes, by the elimination of “last in, first out” layoff rules? All would do some good. Could they amount to a revolution?

Brill proposes a radical step for New York City mayor Michael Bloomberg: appoint Randi Weingarten, president of the AFT, to the post of chancellor of the New York City schools. (“Never in a million years,” says the mayor.) As concerned about her place in history as Al Shanker was, Weingarten would find ways to do the right thing. The teachers and the unions have to be brought into the reform movement. Brill titles this last, surprising chapter “A Marathon, Not a Sprint.” Sara Mosle, in her judicious review in the New York Times, notes that although 1 percent of the New York City teachers may have been in the rubber room, 20 percent of teachers quit after the first year, and 40 percent have left after three years. Is the pay too low, the job too hard, are the wrong people recruited, and the wrong people staying? Dealing effectively with any one of these questions seems beyond the reach of the favored reforms that are at the heart of Brill’s account, worthy as they are.

I should note that Brill concentrates exclusively on those measures around which battles with the unions have been fought. There is almost nothing on possible changes in pedagogy, in school organization and structure, in curriculum: E. D. Hirsch is not mentioned, the push for a national curriculum gets only a few paragraphs. All these have to be part of the marathon.

Nathan Glazer is professor emeritus of education and sociology at Harvard University.

Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies
By Howard Wainer
Princeton University Press, 2011, $24.95; 200 pages.

 

As reviewed by Mark Bauerlein

 

What caring educator would not favor tests that allow students a choice in what they must answer?

What responsible college admissions officer wouldn’t grant applicants the right to withhold their SAT scores?

What committed Advanced Placement (AP) teacher wouldn’t expand access to as many students as possible?

What enlightened test developer wouldn’t prefer tests that identify each test-taker’s actual knowledge and skill levels and not those that just deliver a numerical score?

These aren’t just personal attitudes. They sway large organizations as well. The College Board repeatedly talks about “equitable access” to AP courses, and a 2008 report by the National Association for College Admission Counseling solemnly states, “there may be more colleges and universities that could make appropriate admissions decisions without requiring standardized admission tests such as the ACT and SAT,” and further, “some control clearly rests in the hands of postsecondary institutions to account for inequities that are reflected in test scores.”

But what happens when SAT scores are optional in college applications? When Bowdoin College allowed it, two results emerged, both predictable. One, applicants who withheld their numbers scored on average 120 points lower than did those who submitted their scores. Their withholding hence improved their applications, and it also boosted Bowdoin in the all-important U.S. News & World Report rankings (by making the average SAT score of the entering class look higher). But, two, the “withholders” hurt Bowdoin, for they performed 0.2 grade points worse than “submitters” did in first-year courses.

Or, what happens when tests allow students to choose the questions they answer, for instance, presenting a pool of essay questions from which test-takers choose two? First of all, you end up with inconsistencies: some questions are harder than others. And second, students often choose poorly, selecting the harder questions. Indeed, one study discovered, “the more that examinees liked a particular topic, the lower they scored on an essay they subsequently wrote on that topic!”

These outcomes belie the policies behind them, and they frustrate the generous souls who crafted the plans. Students end up performing worse than officials expected. Who wants to hear the bad news, though? Not many, and that’s precisely the complaint of distinguished statistician Howard Wainer in his book Uneducated Guesses, in which the preceding quotation and the Bowdoin case appear. Most educators stick to their faiths rather than follow the evidence, Wainer complains, and their stubbornness necessitates this blunt retort to education policies founded on bad evidence and good intentions. The volume’s subtitle, Using Evidence to Uncover Misguided Education Policies, describes the method. In 11 curt chapters, Wainer analyzes actual data and uncovers glitches, quirks, misconceptions, and unintended consequences of one practice after another, particularly those related to tests.

Each practice, from Computerized Adaptive Testing (CAT) to coscaling achievement tests, aims to solve a problem or address a need, but under Wainer’s withering assembly of numbers (scores, dollars, demographics), they collapse. He notes the discomfort people feel with the exclusive nature of AP courses, but wonders if it’s right to open them to students who have little chance of passing the exam. On principle, many would answer, “Give everyone a chance!” But, Wainer replies, such principles aren’t free. He takes the case of AP Calculus results in Detroit and estimates that if the city were to restrict the course to students who score 66 or above on the PSAT Math test, then the resulting cost per passing score on the AP test would be $1,167. If the city set the eligible score much lower, at 31, the cost per passing score would reach $4,513. “Would it be a better use of resources to provide a more suitable course for the students who do not show the necessary aptitude?” Wainer suggests.

In the case of CAT, educators favor the format because it calibrates questions to a student’s ability. If a test-taker misses a question, the next question shifts downward in difficulty. If he aces a question, the next one shifts upward. After a few dozen questions, the test identifies the competency level of the student—a better diagnostic than a simple percentage score. But it doesn’t allow test-takers to review and change an answer, which assessment experts consider important to accurate testing. If CAT does incorporate question review, Wainer warns, then when a subject finds an easy question pop up, he assumes he got the previous one wrong and backtracks to change it. Or worse, he deliberately answers every question wrong, ensuring easy questions all the way through. At the end, he returns to the beginning and answers every question correctly, yielding a near-perfect score. In other words, the very customization that educators praise allows savvy students to game the test. Wainer issued that caution in 1993, and he advises that we keep the original CAT because the benefits of item review don’t outweigh the risks of its abuse.  Nevertheless, he notes, test specialists have pressed forward with item review since then—another case of hope overriding evidence.

However sharp and persuasive these exposés, though, they stand at a disadvantage, and Wainer knows it. This idea sounded so right, that innovation so sensible and fair, and watching them fail is depressing. Wainer summons evidence and reality against the modifications, but at stake is not  just this and that policy but avid social hopes, sympathy for students, and feelings of injustice, too. One advocate for question review on CAT tests asserts that students “feel at a disadvantage when they cannot review and alter their responses,” their feelings apparently forcing a change in format. Another proponent begins with a basic condition of test-taking, namely, stress, leading the authors to craft methods that allow students more control over the test but that identify cheating (one recommendation they make is to limit changed answers to 15 percent of the total number of answers). Wainer cites both, but has only a dry reply taken from Albert Einstein: “Old theories never die, just the people who believe in them.”

Mark Bauerlein is professor of English at Emory University.

We are now entering the fourth and final year of the first term of the Obama administration. Enough time has elapsed to provide an opportunity for at least an interim assessment, even though anything more definitive must await the voters’ judgment as to whether a second term is warranted.

At first glance, it looks as if President Obama and his secretary of education, Arne Duncan, have made surprisingly deft moves, both in terms of policy and politics. Even while Republicans are whacking the president “like a piñata,” as one pundit put it, they are treating his K–12 education record with kid gloves.

Senator Lamar Alexander has commented that he has “a lot of admiration” for Obama’s education secretary and “respect” for the president’s “positions on kindergarten through 12th-grade education.” Former House Speaker and presidential candidate Newt Gingrich admitted that this is “the one area where I very much agree” with him. New Jersey governor Chris Christie exclaims that the president has been a “great ally” on education reform. Former Massachusetts governor and presidential candidate Mitt Romney acknowledges that “some of his education policies” have been “positive.”

Is that for good reason? Is President Obama as strong on education reform as these comments suggest? On the surface, at least, the president has a compelling record. His Race to the Top (RttT) initiative catalyzed a chain reaction of legislative action at the state level, securing key reforms on issues ranging from charter schools to teacher evaluations to rigorous standards. His stimulus and “edujobs” bills seemed to maintain a critical level of investment in the public schools during a time of difficult budget cuts and financial strain. His administrative action to provide flexibility on No Child Left Behind’s most onerous provisions bypassed a paralyzed Congress and partially fulfilled his campaign promise to lift the law’s yoke off the backs of decent but maligned schools. And in Arne Duncan he’s got a popular, attractive education secretary to boot, one of the leading stars of his cabinet.

Plenty of these accomplishments are more than skin-deep. For example, both the Common Core State Standards effort and the move toward rigorous teacher evaluations could lead to dramatic increases in student achievement, if implemented faithfully by states and school districts. Neither of these reforms would have been adopted so quickly, in so many places, were it not for the president’s leadership.

Beyond these success stories, however, lie some very real weaknesses—soft spots in Obama’s education record—that raise doubts about the long-term impact of the administration’s efforts.

Wasteful Spending

There’s little reason to doubt that the American Recovery and Reinvestment Act—the 2009 stimulus bill—will long be remembered, along with health-care reform, as Obama’s signature accomplishment. For Democrats, the law saved the nation from a profound depression. For Republicans, as they made clear in the 2010 midterm elections, it constitutes a massive spending program that contributes to a national debt of historic proportions, with few results to show for it.

Accounting for nearly $100 billion (or about double the typical annual federal appropriation for education), the education portion of the stimulus package was one of its central components. In fact, setting aside the bill’s tax cuts, education spending represented the largest piece of the stimulus pie. These dollars were split into a few large categories: supersized spending for the Title I and special-education formula programs, and a “state stabilization fund” that essentially amounted to revenue sharing. (It also included funds for the $4 billion Race to the Top program, discussed separately below.)

Ostensibly, the intent of the education stimulus was to keep teachers from losing their jobs. The macroeconomic argument was that the last thing a damaged economy needed after the 2008 shock was to have hundreds of thousands of public school teachers getting pink slips, going on unemployment, and defaulting on their mortgages. And the nation’s schoolchildren would benefit as well. Protecting education jobs would keep good teachers from getting laid off and class sizes from skyrocketing. In February 2009, Secretary Duncan warned U.S. News & World Report about the consequences if the stimulus bill were not enacted. “My concern is that hundreds of thousands of good teachers, not just bad teachers, are going to go, and that would be devastating. It is to no one’s advantage if class size skyrockets or librarians get eliminated or school counselors disappear.”

This line of reasoning has two problems, as Duncan himself later admitted. First, good teachers were laid off because union protections required districts to implement reductions in force via “last in, first out.” If schools could have used the recession and budget crisis as an opportunity to cut their least-effective teachers, student achievement would actually have risen. As Stanford economist Eric Hanushek has shown, there is no quicker way to lift student improvement than to encourage the lowest-performing teachers to pursue other lines of work. Duncan himself does not disagree. In March 2011 he said, “Layoffs based only on seniority don’t help kids. We have to minimize the negative impact on students.’’

Second, there is little, if any, evidence that a modest increase in class size would have devastating consequences. Class size has fallen markedly over the past few decades. The year Obama was elected, the average number of pupils per professional in the public schools was 15, down from 19 in 1980 and 26 in 1960. In fact, even major layoffs would only return schools to the staffing ratios of the late 1990s, not exactly the Dark Ages, and a time of great progress in raising student achievement nationally. And again, even Duncan admitted as much when he later said that “class size has been a sacred cow and we need to take it on.”

Even when we evaluate the stimulus package on its own terms, protecting teachers’ jobs and keeping classes small, the costs seem wildly in excess of any benefits obtained. According to the Obama administration’s calculations, the stimulus package and edujobs bill kept about 400,000 teachers on the payroll who would have otherwise been terminated. That works out to approximately $150,000 per job, an exceptionally bad deal for taxpayers considering that the average new teacher (who would have been first in line for a pink slip) makes considerably less than half that in salary and benefits. Even if we accept the estimate of teachers’ jobs saved, we have to ask, where did the rest of the money go?

There is evidence that a significant portion of the funds did not go to stemming layoffs. Media reports indicate that some districts used the money for teachers’ raises and bonuses. The Government Accountability Office cited one North Carolina district for using edujobs dollars to pay for movie tickets, fast food, and a water park visit for students. This was in the midst of the worst economic downturn in six decades, when most Americans were either losing their jobs or barely treading water.

The design of the laws may, in fact, have aggravated the funding crisis at the local level. Forced to spend the funds relatively quickly, districts added staff, made new investments, and otherwise increased their costs, which will make the coming “funding cliff” that much more painful. At a time when tough-minded superintendents should have been preparing for leaner times by negotiating concessions from their bargaining units on salaries and benefits, federal policy cut them off at the knees.

Lackluster Results

Race to the Top is President Obama’s most vaunted win. The name itself connotes progress, forward movement, even competition. And there’s plenty of substance for the president to brag about: more than 45 states signing onto rigorous common standards; dozens of states getting serious about teacher evaluations; key jurisdictions removing caps on charter school expansion. This is what New Yorker contributor Steven Brill called “a sweeping overhaul” of the system. Look closer, put the Race to the Top’s results into context, and the scorecard changes considerably.

Secretary Duncan likes to say that RttT is part of a “quiet revolution” in education, with states creating “bold blueprints for reform [that] bear the signatures of many key players at the state and local level who drive change in our schools.” He’s right that the program led to a flurry of reform-friendly legislation. But did the 2009–10 period, when states were competing for RttT funds, see the most reforms ever enacted? No. That distinction belongs to 2011, after the 2010 midterm elections swept historic Republican majorities into office in state after state. While a similar number of states (5) made sizable progress on charter school caps in 2011 as in the previous two years, the number of states that moved forward on teacher evaluations, layoff policies, and vouchers increased significantly (see Figure 1).

Race to the Top wasn’t meant just to catalyze legislative changes. Winning states made bold promises about implementing their proposed reforms, and Obama and Duncan issued stern statements about their intention to pull dollars away from jurisdictions that fell short. How has that effort fared?

In short: not so well. Eleven states and the District of Columbia won first-round grants of up to $700 million from the $4 billion RttT pot in 2010, promising to deliver a range of ambitious programs and results. A little more than a year later, every one of those grantees has amended its plans at least once, with the Department of Education approving a grand total of forty-seven amendments to date. Maryland asked for another year to finish its teacher evaluation system, while North Carolina opted for a more modest teacher-retention bonus program. Time and again goals have been lowered and timelines extended. When in late 2011, in response to Hawaii’s stalling Duncan finally threatened to cut off the Aloha State’s funding, it marked a sharp and belated shift from the dozens of accommodating letters of approval that states wavering on their commitments have received from Washington.

Scaled-back ambitions are only half the problem: many states seem to have barely started putting their plans in motion. As of May 2011, a year after the first RttT awards, just over $80 million of the $4 billion in funding had actually been spent. While it’s at least reassuring that states haven’t been burning through the money, the urgency of the “Race” petered out once the awards were made. With the latest round of RttT grants awarded with little fanfare, the Obama administration’s signature effort is losing steam.

Federal Micromanagement

Complaining about an overbearing federal role in education is a mainstay of Republican campaigns, particularly during primary season, when the battle cry of “local control” most resonates with likely voters. The current nomination contest is no exception, with all of the GOP candidates calling for a smaller federal footprint, if not the outright closure of the Department of Education.

This message is more problematic during general elections, when voters (especially all-important independents) can easily equate a conservative’s plea to “pull back” as an indication of disinterest.

But in skillful hands, painting Uncle Sam as school-yard bully could work.

“We’re going to let states, schools and teachers come up with innovative ways to give our children the skills they need to compete for the jobs of the future,” promised Obama when announcing his NCLB waiver plan. “Because what works in Rhode Island may not be the same thing that works in Tennessee—but every student should have the same opportunity to learn and grow, no matter what state they live in.” Duncan echoed, “instead of being tight on the goals and loose on the means of achieving them, [NCLB] is loose on the goals but tight on the means. We need to flip that and states are already leading the way.”

But for all the talk of state discretion, the Washington screws are actually being tightened. Take the Race to the Top, which one of us once characterized as “a carrot that feels like a stick.” Rather than invite states to present their own compelling reform plans, Obama and Duncan asked governors and state superintendents to develop plans that complied with federal guidelines set forth in excruciating detail. Or take their approach to NCLB waivers, in which they set constitutionally suspect conditions on the flexibility craved by the states (see “Obama’s NCLB Waivers,” forum, page 56). As Senator Alexander remarked, the Obama administration had states “over a barrel.”

And when it comes to federal control, nothing is more troubling than the declaration that a disproportionate percentage of white students in Advanced Placement (AP) classes constitutes evidence of racial discrimination. That’s the administration’s stance, thanks to the Department of Education’s civil rights branch, led by poverty warrior Russlynn Ali. At the very time Duncan was espousing the virtues of state and local flexibility, he and Ali were doubling down on 1960s-style top-down regulations. One stated objective was to address the “disparate impact” of policies that might lead to racial minorities taking fewer challenging classes than their peers, totally ignoring the obvious fact that African American and Hispanic students are, on average, much less prepared for AP courses by the time they reach 11th and 12th grade. Never mind that closing this preparation gap requires a long-term effort starting in elementary school, if not before. The federal government put districts on notice that if they had a disproportionate number of white students in AP classes, they could be immediately subject to civil rights enforcement. This is tight-loose?

Obama and Duncan have been good on education reform, certainly better than any of their Democratic predecessors. But to ignore the shortcomings of the president’s K–12 education-reform record entirely would be a mistake, we think. And it would also be bad for the country. The administration deserves to be pressed on the cost-effectiveness of its education system bailouts, on the results of its Race to the Top initiative, and on the wisdom of its approach to federalism and separation of powers. Education may not play a major role in the 2012 election, but that doesn’t mean that Obama’s education policies should be given a pass.

Michael J. Petrilli is research fellow at Stanford University’s Hoover Institution and executive vice president of the Thomas B. Fordham Institute, where Tyson Eberhardt is a research fellow.

I am one of the lucky ones. I teach in a school that has many excellent teachers. But for nine years, I’ve observed the larger public-school system in crisis and the contentious debate that surrounds it: Why is it failing? How can it be fixed?

Education—or, as economists refer to it, “investment in human capital”—is a cornerstone of every model of economic growth: if our children are not well educated, innovation and productivity will wither away. In other words, our long-term collective quality of life depends on the quality of our schools. And, by most measures, they are inadequate to the task. What are we doing wrong?

My view is that many problems in education are economic in nature. By this I don’t mean that more funding will solve the problem. I mean that the way we run public education violates virtually every basic tenet of economics. We have constructed a public school system that seems intentionally designed to provide the wrong incentives to administrators, teachers, and students.

Take teacher tenure. Job security with few conditions allows teachers to settle, to become lazy and professionally static. Tenure creates a strong disincentive to innovate or work harder. Tenure attracts to the profession security-seekers rather than risk-takers and provides no upward mobility for the ambitious few.

Now imagine a job where one not only cannot get fired, but where one receives automatic raises simply by being there. Even for the most conscientious teachers, there is no incentive to do more than the minimum, because no matter how hard those teachers work, they cannot be paid more. There are no cash bonuses, no rewards for performance. After working many (truly exhausting) years, few teachers could be faulted for either shifting into a lower gear or moving on.

These are hardly the only disincentives to becoming a teacher. When, at age 37, I started teaching high school, I began, in both salary and rank, as a “first-year” teacher. Despite my having worked in intelligence, diplomacy, and business, I was treated like, and earned essentially the same salary as, a 21-year-old teaching second grade. And, like my first-year peers, I was subject to the seniority system’s stubborn adherence to a last-in, first-out policy.

It doesn’t take the sharpest imagination to understand why this would be a disastrous way to run an organization. If I were, say, managing a pharmaceutical company, would I pay someone with 15 years of experience in pharmacological research the same salary as the new undergraduate intern simply because they were both new hires? Could I expect the same outcomes from both? The same productivity? Of course not. Then why would I pay them identically or fire the last one hired, regardless of performance? This is nonetheless the norm in public education.

When I first thought about teaching, I called my county school system. I explained my professional background, including graduate degrees in international affairs and, later, in international economics. They told me to apply immediately. Then I found out that, despite my background, according to the state of Maryland I was not qualified to teach history, political science, or economics. Until I completed 29 credit hours of teacher training and became certified, I would be employed as a “long-term substitute,” a job with full hours, low pay, no benefits, and the real possibility of my being released at the end of the year.

State-mandated teacher certifications (backed by No Child Left Behind–based rules) are preventing highly qualified candidates from becoming teachers. I was an all-too-rare exception. At the time, my wife and I were in a secure enough financial position that I could take two years off without any income to become a teacher and then earn around $45,000 a year once employed. How many experienced professionals, especially those with families, could do that? Why should they have to? I had the academic background and pedagogical skills I needed to be a teacher before expending all that time, money, and effort on a graduate degree in education.

Teacher quality is the key to improving public education in the United States. Nonetheless, we systematically dissuade highly capable people from becoming teachers. If we are to improve our educational system, we must instead create economic incentives that draw the best people to the profession and keep them there.

Vann Prime teaches Advanced Placement (AP) economics, AP European history, and international relations at Mount Hebron High School in Howard County, Maryland.

Evaluations of school-reform measures typically focus on the outcomes that are most easily quantified, namely, test scores, as a proxy for long-term societal benefit. But there are at least two reasons we might want to look beyond test scores and other school-based outcome measures. First, there is evidence that schools facing accountability pressures may be able to raise student test scores through methods that do not translate into long-term improvements in skills or educational attainment, by engaging in test-prep activities or by cheating, for example. Second, even in the absence of such behaviors, the correlation between test-score gains and improvements in long-term outcomes has not been conclusively established. Studies of early-childhood and school-age interventions often find long-term impacts on such outcomes as educational attainment, earnings, and criminal activity despite nonexistence or “fade-out” of test-score gains. In other words, programs can yield long-term benefits without raising test scores, and test-score gains are no guarantee that impacts will persist over time.

In this study, I investigate whether the opportunity to attend a school other than a student’s assigned neighborhood school reduces criminal activity, especially among disadvantaged youth. Many of the schools chosen by the students were “better” on traditional indicators, such as student test scores and teacher characteristics. All of them, however, were preferred by the applicant over the default option. The analysis therefore sheds light on whether efforts to expand school choice can be an effective crime-prevention strategy, particularly when disadvantaged students can gain access to “better” schools.

We know that criminal offenders often have low levels of education: only 35 percent of inmates in U.S. correctional facilities have earned a high school diploma, compared to 82 percent of the general population. Criminal activity is concentrated among minority males; it begins in early adolescence and peaks when most youth should still be enrolled in secondary school. The schools these young men would attend are typically in high-poverty urban neighborhoods, have high rates of violence and school dropout, and struggle to retain effective teachers. Such schools may be a particularly fertile environment for the onset of criminal behavior. Yet little research has been conducted to determine the effect of school quality on crime.

In this study I explore this question using data from the Charlotte-Mecklenburg (North Carolina) school district (CMS) to measure the impact of school quality on arrest and incarceration rates. I take advantage of the CMS districtwide open-enrollment school-choice plan, which until recently let students choose where they wanted to go to school and employed lotteries to admit students to oversubscribed schools. I compare the criminal activity of students who won the lottery to attend their first-choice school to that of students who lost the lottery.

I find consistent evidence that attending a better school reduces crime among those age 16 and older, across various schools, and for both middle and high school students. The effect is largest for African American males and youth who are at highest risk for criminal involvement. In general, high-risk male youth commit about 50 percent less crime as a result of winning the school-choice lottery. They are also more likely to remain enrolled in school, and they show modest improvements on measures of behavior such as absences and suspensions. Yet there is no detectable impact on test scores for any youth in the sample.

School Choice in CMS

With more than 150,000 students enrolled in 2008–09, Charlotte-Mecklenburg is the 20th largest school district in the nation. The CMS attendance area encompasses all of Mecklenburg County, including Charlotte and several surrounding cities. Overall, CMS is racially and demographically diverse. About 45 percent of the students in CMS middle and high schools in 2003 were African American, less than 10 percent were Hispanic (although the Hispanic population was growing rapidly over this period), and about 50 percent were eligible for free or reduced-price lunches. Individual CMS schools vary widely in demographic composition: CMS high schools in 2003 ranged from less than 10 percent to close to 90 percent nonwhite, and were also dissimilar in average test scores and rates of high school graduation.

From 1971 until 2001, CMS schools were forcibly desegregated under a court order. Students were bused all around the district to preserve racial balance in schools. After several years of legal challenges, the court order was overturned, and CMS was instructed that it could no longer determine student assignments based on race. In December 2001, the CMS school board instituted a policy of districtwide open enrollment for the 2002–03 school year. School boundaries were redrawn as contiguous neighborhood zones, and children who lived in each zone were guaranteed access to their neighborhood school. Under busing, schools were racially balanced, but the surrounding neighborhoods remained highly segregated. Thus the redrawing of school boundaries led to concentrations of minority students in some schools.

The first open-enrollment lottery took place in the spring of 2002. CMS conducted an extensive outreach campaign to ensure that choice was broad-based, and 95 percent of parents submitted at least one preferred school; parents could submit up to three (not including their neighborhood school). Admission for all students from outside the neighborhood zone was subject to grade-specific limits. The lottery process for oversubscribed grades gave preference first to students who previously attended the school and their siblings, then to low-income students applying to schools that previously did not have a majority of low-income students, and finally to students applying to a school within their “choice zone” (which would guarantee them access to district-provided transportation). I study the effects of winning a seat at a preferred school in the 2002 lotteries on student outcomes through 2009, seven years after the lotteries were conducted.

Because nearly all rising 12th graders received their first choice, I restrict my study to students in grades 6 through 11. I also exclude the 5 percent of students who were not enrolled in any CMS school in the previous year. About 60 percent of the remaining students chose (and were automatically admitted to) their neighborhood school. About 75 percent of applicants to nonguaranteed schools were in lottery priority groups in which the probability of admission was either zero or one. Even though these students chose a nonguaranteed school, there is no randomness in whether they were admitted, so I do not use them in the study. The resulting sample consists of 1,891 high-school students (grades 9–11) and 2,320 middle-school students (grades 6–8). Compared to all students in CMS, these students were more likely to be African American and eligible for free lunch; they also had lower test scores and higher rates of absence and out-of-school suspensions (see Figure 1).

Data

Since the mid-1990s, the North Carolina Department of Public Instruction (NCDPI) has required all districts to submit data that include demographic information, attendance rates, and behavioral outcomes, yearly test scores in math and reading for grades 3 through 8, and subject-specific tests for higher grades. I used these data, along with internal CMS files that contain student-identifying information such as name, date of birth, and exact address in every year. This information enabled me to match CMS students to arrest records from the Mecklenburg County Sheriff’s Office, which include all arrests of adults (age 16 and over in North Carolina) that occurred in the county.

I measure crime severity in two ways, both of which are intended to capture the idea that not all crimes are equal. First, I use estimates that economists have developed of the social cost of crimes, which include tangible costs, such as lost productivity and medical care, as well as intangible costs, such as impact on quality of life; these estimates are extremely high for fatal crimes. (The estimated social cost of murder is $4.3 million in 2009 dollars. The next costliest crime is rape, which is estimated at $125,000.) To avoid the results being driven entirely by a few murders, in my main analysis I limit the cost of murder to twice the cost of rape. The second measure of severity weighs crimes by the expected punishment resulting from a successful conviction. Neither measure accounts for justice system costs such as police or prisons.

Methodology

If the school lottery is truly random, the winners and losers will on average have identical observed and unobserved characteristics. With a large enough sample, a simple comparison of outcomes between winners and losers would identify the causal effect of winning the lottery. In reality, CMS conducted many lotteries (for each school and grade). The number of students in each lottery is relatively small, so my analysis combines data from all of the middle-school and all of the high-school lotteries. My results reflect the average difference in outcomes between winners and losers across all of the lotteries conducted at each level.

The result is the “intent-to-treat” effect of winning a lottery; it is an intent because students offered a place in their first-choice school did not always take it (for example, they may have moved out of the district). Students who won the lottery are more than 55 percentage points more likely than losers to attend their first-choice school in the first year, and on average spend an additional 1 to 1.5 years enrolled in that school overall. One can therefore obtain a rough estimate of the effect of actually attending the first-choice school (as a result of winning the lottery) by doubling the results presented below.

I examine the impact of winning the lottery on crime separately for groups of students with different propensities to commit crimes, with a focus on the highest-risk group. Because students with adult arrest records can be tracked all the way back to kindergarten in some cases, I use all of the potential predictors of criminal behavior—test scores, demographics, behavior, and neighborhood characteristics—to calculate an index of crime risk. The students in the top 20 percent of this crime-risk index are disproportionately African American males and eligible for free lunch (see Figure 1a). Their test scores are on average one standard deviation below the North Carolina state average, and they are absent and suspended many more days than the average student (see Figure 1b). Because high-risk students are overwhelmingly male, I exclude females from all of the analyses. The results comprise a final sample of 1,014 high-school students and 1,081 middle-school students.

High-school lottery winners attend schools that are demographically very similar to the schools attended by lottery losers, while middle-school winners attend schools that are less African American and higher income on average. All lottery winners travel farther to attend their first-choice school, but the distance is greater for high school students than for middle school students.

High-school lottery winners in the high-risk group and all middle-school lottery winners experience modest increases in standard measures of school quality. Their peers’ average test scores are about 0.15 standard deviations higher, and the new schools have higher-quality teachers, measured in terms of the fraction of teachers with less than three years’ experience, the fraction that are new to the school that year, the percentage of teachers with an advanced degree, and the share of teachers who attended a “highly competitive” college as defined by the Barron’s rankings. For youth in the high-risk group, the gain as measured by these quality indicators is roughly equivalent to moving from one of the lowest-ranked schools to one around the district average.

Results

I find that winning a lottery for admission to a preferred school at the high school level reduces the total number of felony arrests and the social cost of crime. Among middle school students, winning a school-choice lottery reduces the social cost of crime and the number of days incarcerated. Importantly, I find that these overall reductions in criminal activity are concentrated among students in the highest-risk group. Indeed, I find little impact either positive or negative of winning a school-choice lottery on criminal activity for the 80 percent of students outside of this group.

Consider first the results for high school students in the high-risk group. Among these students, winning admission to a preferred school reduces the average number of felony arrests over the study period from 0.77 to 0.43, a pattern driven largely by a reduction of 0.23 in the average number of arrests for drug felonies (see Figure 2). The average social cost of the crimes committed by high-risk lottery winners (after adjusting the cost of murders downward) is $3,916 lower than for lottery losers, a decrease of more than 35 percent. (Without adjusting for the cost of murder, I estimate the reduction in the social cost of crimes committed by lottery winners at $14,106.) High-risk lottery winners on average commit crimes with a total expected sentence of 35 months, compared to 59 months among lottery losers.

Among high-risk middle-school students, I find no effect of winning a school-choice lottery on the average number of felony arrests. Although the number arrests for violent felonies falls, this is offset by an increase in the number of property arrests. Because violent crimes carry greater social costs, however, winning a school-choice lottery reduces the average social cost of the crimes committed by middle school students by $7,843, or 63 percent. It also reduces the total expected sentence of crimes committed by each student by 31 months (64 percent).

An important limitation of this analysis is that I do not have access to data on juvenile crime. Especially for students in the middle school sample, this could mask big differences in juvenile offending in the early years after the lotteries were conducted. As an alternative, I examine the effect of winning the lottery on school disciplinary outcomes such as absences and suspensions, as well as on test scores. Among the high-risk group, lottery winners are absent slightly less than the lottery losers are. The effect on high school suspensions in 2003 is relatively large, but the other school discipline effects are small and statistically insignificant.

In contrast to the results for crime and disciplinary outcomes, I find no evidence that winning admission to a preferred school leads to test-score gains. But I do find some impacts on enrollment, grade progression, and grade attainment for high-risk youth. For example, high-risk middle-school lottery winners are 18 percentage points more likely than lottery losers to be enrolled in CMS in their 10th-grade year. The effect on 11th-grade enrollment is about half the size (9 percentage points), and there is no impact on persistence into 12th grade.

Despite the impacts on enrollment and progression, there is no detectable increase in high school graduation rates. Because I am limited to CMS administrative data, it is difficult to distinguish dropouts from subsequent GED recipients or transfers who may have graduated elsewhere. Administrative records are particularly problematic for high-risk youth, who sometimes disappear from CMS well before they are old enough to do so legally. The graduation rate is only about 25 percent among high-risk high-school students, and currently only about 10 percent among high-risk middle-school students, although some who are still enrolled may yet graduate. Additionally, a bit less than 10 percent of the high-risk middle-school sample never appears in any high school grade but subsequently appears in the arrest data. Because any intervention aimed at high school students would miss this group altogether, this suggests that high school might be too late for the youth at highest risk of criminal activity.

Explanations and Policy Implications

Overall, I find that winning the lottery to attend a first-choice school has a large impact on crime for high-risk youth. High-risk lottery winners experienced roughly a 50 percent reduction in the measures of criminal activity that weight crimes by their severity.

I consider four possible explanations for the reduction in crime among high-risk lottery winners. The first is incapacitation, which advances that winning the lottery entails longer bus rides to and from school, thus occupying youth during high-crime hours. The second is contagion, in which winning the lottery prevents crime by removing high-risk youth from crime-prone peers or neighborhoods, thereby reducing contemporaneous exposure of high-risk youth to criminogenic influences. These first two explanations would predict a strong initial effect that fades over time. If, for example, drug-market activity is concentrated within a few schools, we might expect large differences in criminality in the high school years that diminish as enrollment in the chosen school ends and lottery winners and losers return to the same neighborhoods. When I examine the effect of winning a school lottery separately at different points in time after the lotteries were conducted, however, I find larger effects in later years. I therefore conclude that there is little support for the incapacitation and contagion explanations since they do not fit the pattern of results over time.

A third possibility is that the reduction in crime comes from the skills students gain by attending a higher-quality school. If the schools attended by lottery winners do a better of job of teaching skills that increase students’ ability to find employment, they will stay enrolled in school longer, delaying the onset of criminality through the peak period of offending behaviors. Moreover, youth with more and better schooling will gain access to more and better opportunities for paid work, making crime less attractive. Based on a back-of-the-envelope calculation of the relationship between enrollment and criminal activity in my sample, I estimate that the effects of winning a school lottery on enrollment could potentially explain about 45 percent of the impact on criminal activity in the high school sample, but only about 10 percent in the middle school sample.

Alternatively, peer networks formed in middle or high school could have a persistent influence on adult criminality without affecting skills directly. In my own data, I find relatively little evidence that the propensity of a student’s peers to engage in criminal activity influences the degree to which he commits violent crimes. This may be due in part to the high rate of early dropout among violent felons. However, having crime-prone peers in middle school substantially increases the likelihood of committing a violent crime, especially for youth in the high-risk group. Based on this relationship, I estimate that changes in peers can explain roughly 9 percent of the impact on violent arrests in the middle school sample.

Regardless of the mechanisms by which admittance to a preferred school influences criminal activity, the fact that these impacts are concentrated among high-risk students has important implications for the design of school-choice programs. It may make sense for oversubscribed schools of choice to give preferential admission to students at greatest risk of criminal activity. To illustrate this point, I use my results to evaluate the consequences of two different types of lotteries: 1) those giving priority to the highest-risk students and 2) a simple lottery similar to those virtually all charter schools nationwide are required to use to admit students when the schools are oversubscribed. The actual CMS lottery system gave preferences to low-income students who applied to schools with a low fraction of low-income students. As a consequence, many poor (and high-crime risk) students were automatically admitted to schools while other students had to win the lottery.

If slots in oversubscribed schools were systematically allocated to the highest-risk students, the social cost of crime would fall by an additional 27 percent relative to the actual CMS assignment mechanism. A more realistic form of targeting is the method actually pursued by CMS, giving preference to low-income students within the lottery system. I estimate that this policy choice lowered the social cost of crime by about 12 percent, relative to a simple charter-style lottery with no preferential treatment. Although this analysis does not consider the possibility that a greater concentration of high-risk students could have adverse effects on other students, it nonetheless highlights the likely beneficial consequences of giving preference to disadvantaged students in the admissions process for oversubscribed schools.

Conclusion

In this study, I find that winning a lottery for admission to the school of choice greatly reduces criminal activity, and that the greatest reduction occurs among youth at the highest risk for committing crimes. The impacts persist beyond the initial years of school enrollment, seven years after the school-choice lottery was held. The findings suggest that schools may be an opportune setting for the prevention of future crime. Many high-risk youth drop out of school at a young age and are incarcerated for serious crimes prior to the age of high school graduation. For these youth, who are on the margins of society, public schools may present the best opportunity for intervention.

The end of busing and the implementation of open enrollment in CMS was a significant policy change. The four neighborhood high schools to which most of the lottery applicants were assigned lost more than 20 percent of their enrollment in a single year. In subsequent years, two of these schools were restructured as magnet schools offering specialized programs in a small school setting. Two middle schools that lost significant numbers of students were subsequently closed. The open enrollment policy thus sent a strong signal of parental demand to CMS that may have resulted in the shutting down or restructuring of low-performing schools. The No Child Left Behind Act of 2001 included a provision that allowed parents to transfer students from “persistently dangerous” public schools, but many states have set the legal threshold so high that very few schools qualify. The results here suggest that, to the extent that low-quality schools are also persistently dangerous, allowing students to leave them might benefit individual students as well as society as a whole.

David J. Deming is assistant professor of education at the Harvard Graduate School of Education. This article is adapted from a study in the November 2011 issue of the Quarterly Journal of Economics.

When school districts are failing, what should the federal government do?

A) give districts money?
B) deny districts funds?
C) subject districts to tight regulations?
D) force districts to compete for federal dollars by promis­ing to improve?
E) tell the truth while insisting parents be given a choice of school?

Policymakers have responded to this, the nation’s most challenging multiple-choice education quiz, with four different wrong answers. Now, with the release of the Koret Task Force report, policymakers have a chance to get it right, as they consider the reauthorization of the federal education law, No Child Left Behind (NCLB).

President Jimmy Carter chose the first answer, swelling the federal share of education spending to an all-time high. Yet according to the National Assessment of Educational Progress, high-school seniors perform no better today in math, reading, or science than they did when Carter held office.

President Ronald Reagan curtailed the share of K–12 education spending paid out of the federal treasury. That did not lift student performance either.

With the passage of NCLB, the George W. Bush administration subjected failing schools to sanctions if test performance did not improve. Notable gains were made, as Eric Hanushek points out in his provocative analysis of the benefits of the school accountability law. But NCLB’s complicated regulations proved to be unworkable and ineffectual.

Now, the Obama administration has sought to boost school improvement through Race to the Top by getting states and districts to compete for some federal dollars with promises to execute needed reforms. Not surprisingly, state and district promises are more easily made than kept.

Four strategies. Four failures. What should the federal government try next?

Why not do what the federal government has always done well? Collect the facts about schools and student performance and let the data speak for themselves. When the original Department of Education was founded in 1867, its main task was to collect school statistics on such fundamentals as student enrollment, dollars spent, and numbers of teachers hired. Gradually, the federal government acquired the capacity to compile a sophisticated battery of information on the state of American education. Indeed, the only reason we know that America’s schools have not improved much over the past 50 years is that the federal government has collected the information.

So why not use the power of the federal government to collect even more specific information on student learning? A giant step in the right direction was taken with NCLB’s original passage. When it is reauthorized, further steps need to be made so that accurate information on knowledge gained each year in each classroom is available to every parent.

And to receive federal dollars, districts must give parents the freedom to use this information to select the school of their choice—traditional public, charter, or private.

That is what the Koret Task Force has recommended. It’s the right answer to the nation’s multiple-choice education quiz.

– Paul E. Peterson

Forty-two states and the District of Columbia have charter school laws today, and the nation’s first such law celebrated its 20th anniversary in Minnesota this year. Charters, publicly funded schools formed by parents and community leaders, are expected to provide alternatives to traditional public schools. Yet despite the proliferation of charter laws and new schools around the country, charters and their authorizers still spend their first several years in a fight for survival. Nowhere is this more true than in South Carolina, which was among the first states to adopt a charter statute.

Founders of charter schools sign contracts (or “charters”) with an authorizer, such as a school district or higher-education institution, that stipulate the rules and regulations from which charters are exempted in exchange for accountability for results. In other words, a charter school can be closed if it does not meet certain reporting requirements and student achievement goals.

For years, South Carolina charters struggled mightily after their launch. Far fewer charters are now in operation in South Carolina than in some of the other states that were early adopters (South Carolina has 44, while California, Arizona, and Florida each has hundreds of charters), and charter students make up only 2 percent of the state’s public-school enrollment. Undoubtedly, some of these differences can be attributed to geography and population, but a recurring set of obstacles has also plagued the movement in South Carolina since its inception.

In 1996, then governor David Beasley signed South Carolina’s charter law, but few schools had opened by the turn of the century. This is surprising, considering the state’s record of low student achievement. According to commonly accepted performance indicators, South Carolina’s public schools are among the nation’s worst. Persistently low graduation rates, dismal SAT results, and low NAEP (National Assessment of Educational Progress) scores, especially in reading, have long been the norm. In a historically red state with low-performing schools, a free-market education reform such as charters should be in demand and find strong support from lawmakers. What happened?

Fits and Starts

While South Carolina did indeed pass a charter school law in 1996, a combination of public-school establishment resistance and legislative reticence delayed the law’s maturation. As originally enacted, the law only allowed local school districts to authorize charters. When the first decade of South Carolina’s charter history concluded in 2006, there was little to show. Twenty-nine charter schools were operating, and few of these had a track record of success. Some 14 others had opened and closed. The average life span of the closed schools was 2.7 years, with most not even completing a second year. As is the case nationally, many of the closures were the result of financial problems or poor planning at the outset. While the state board of education addressed the planning concerns through regulation, other policy issues emerged, as certain districts developed a reputation for stonewalling reform efforts. For example, Greenville and Charleston, home districts for two early charter success stories, Greenville Tech and James Island, respectively, are the two largest districts in South Carolina, and each developed an adversarial stance toward charters.

The prolonged period of fits and starts forced charter advocates and their allies in the statehouse to seek a separate peace with their opponents in well-entrenched teacher, superintendent, and school-board associations. In responding, legislators created an alternative authorizer, the South Carolina Public Charter School District (SCPCSD, here CSD), with a plan to commence operations in 2008 under the leadership of an appointed board representing the governor’s office, House and Senate leadership, and various state associations. The new authorizing district proposed to relieve pressure on local districts as the only avenue for a charter. This, plus the authorizing district’s spartan funding provision, helped quell opposition—for a time.

Allison Reaves, principal at South Carolina Connections Academy, a virtual school and one of the first the CSD authorized, was surprised that so little effort had been made to prepare the public system for the new district. “I realized [charters] were still such a novel idea in South Carolina. Local districts have had little to no education on the charter movement,” she says.

With the creation of the CSD, charters could be authorized to operate anywhere in the state, under the auspices of an agency that had no responsibility for traditional public schools. This new state agency/school district hybrid would be a logical alternative for charter hopefuls, especially those in local districts with an anticharter reputation.

The CSD opened in 2008 with five schools, including Connections Academy and two other virtual charter schools, the first of their kind in the state. By the end of the 2008–09 school year, though, one school’s charter had been revoked, two others had asked for loans to make payroll, the district office was operating with barely enough on the balance sheet to make it month to month, and the hybrid administrative concept had been abandoned in favor of a more traditional district model. Further complicating matters, leadership changed, as the inaugural superintendent, Tim Daniels, and board chair, Terrye Seckinger, were replaced at the end of the year. What began as a hopeful new charter authorizer for South Carolina teetered on the brink of oblivion after only one year.

Money Problems

CSD schools immediately found themselves forced to defend their very existence, a common position for charters. Nationally, charters embrace this challenge by vowing to do more with less, but there is a distinct difference between whether a school can stock an additional computer lab or barely pay the electric bill. From the beginning, the South Carolina Charter School Act provided CSD schools with little more than the Base Student Cost (BSE), which varies from year to year depending on the state budget. The most significant source of funds for South Carolina’s traditional public schools—as well as for charters authorized by a local district—is the municipality in which the school is located. CSD schools do not have a local tax base and thus must operate without these funds. “The funding part was totally misleading—there was no way. Anybody with any understanding of finance and schools would realize that the bill created a situation that was not going to be long term,” says current CSD superintendent Wayne Brazell. Principal Reaves says the charter management company behind her school knew the difficulties it would face in South Carolina, but pressed on. “Connections realized they were taking a risk,” she says, “but they also knew there was a need for us in the state.”

In 2008–09, the BSE was $2,476 per student, while the average per-pupil expenditure for traditional public schools in South Carolina totaled $9,162. Some other state funding was available to CSD schools, and they relied substantially on Title I dollars in the district’s first year. But even when federal Title I funding was added to the mix, the CSD per-pupil average was below $4,000, less than half the state average for traditional schools. And this figure varied according to grade level, as high school students and disabled students are weighted more heavily by the state finance office.

“It [the charter school allotment] was certainly inadequate,” says current CSD board chair Don McLaurin, an entrepreneur whose private-sector experience enabled him to recognize immediately the CSD’s precarious financial situation. McLaurin joined the board halfway through the 2008–09 school year and has already been voted chairman twice. “It just wasn’t enough money to run a school. I think we can do things at a more reasonable price than traditional public schools, but the mechanism that was in place in the beginning didn’t allow for the realities of the world.”

Understanding the policy shortcomings in CSD’s creation, legislators added a $700 per-student proviso to the 2009–10 state budget to aid the district. But the proviso, Title I funding, and federal IDEA (Individuals with Disabilities Education Act) funding failed to offset worsening financial conditions in 2009–10. The $700 proviso survived reauthorization in spring 2010 and was available to the district for the 2010–11 school year, but its benefit evaporated when BSE was cut to $1,757 per pupil. Statewide, general-fund revenue collections—for all state services, including education—dropped by nearly 25 percent between 2007 and 2010. The state faced a budget shortfall of $560.9 million in 2010–11, projected to reach $1.4 billion in 2012–13 unless spending was cut. In July 2010, midyear cuts slashed the BSE even further, to $1,630. Superintendent Brazell knew the proviso could only be considered a short-term solution. “My thoughts were that this was done just to get our foot in the door and other funding would become available later,” he says.

Having built annual budgets on significantly higher per-pupil allotments than they were receiving, CSD schools struggled to survive, and the threat of closure loomed. Compounding the problem, CSD schools experienced significant student turnover in their first two years, making enrollment unstable. “The funding level was so low and the opposition from so many traditional public-school groups was so fierce that many potential parents took a ‘wait and see’ stance. The growth in the district was mainly in the virtual schools and that student population was very transient,” says Brazell.

In an effort to save newly opened charter schools, the CSD extended loans to two of its five schools in 2008, but only one of the loans was repaid. This caused consternation among the authorizing district’s board members, especially as new requests for loans came in, and led to a swift reversal of district policy. “We violated what many of us thought we should have been doing as an authorizer, but we had to either help the schools or watch them all close,” says Brazell. With the damage done, the district and its schools were convinced that a funding scheme relying on BSE and Title I funding was untenable. For the next fiscal year (2009–10) the CSD aggressively cut costs, trimming office accounting fees and downgrading budget lines set aside for a legal retainer. As the district rebuilt its depleted reserves, schools again asked for short-term loans. Having learned a hard lesson, the district helped schools make payroll by advancing funds equal in amount to dollars due from the state. When the state funds arrived (typically at the end of the month), the district simply deducted monies already provided to the schools.

These actions were difficult for those board members with a background in education to come to grips with, says board chair McLaurin. But just as a start-up business has to be creative, he knew the new district had to be so as well. “The district either had to be flexible or not survive,” he says. “It was more difficult for educators than entrepreneurs to understand this—and that’s not a slight to educators, it’s just a different perspective.”

These unorthodox measures kept the district afloat while legislators moved to revise the funding scheme. Rep. Phil Owens of Pickens County, chair of the House Education and Public Works Committee, introduced a bill in 2010 aimed at establishing a more sustainable funding scheme for the district, but opposition from members of the education establishment stalled the legislation in committee. McLaurin says, “That we did become one of the largest districts in the state [after two years] was proof-of-concept to the legislature.” District enrollment more than doubled from 2,464 students in 2008–09 to 6,086 in 2009–10. “We proved that people want this; they signed up in droves, and that put a lot of pressure on the legislature to find more money for us,” he says.

Nevertheless, as the 2010 session ended, the CSD anticipated another year of uncertainty and prepared for more legislative battles in 2011.

The Schizophrenic District

From the outset, the CSD encountered another obstacle to progress, further exposing its policy-created vulnerability: it served as both authorizer and support office. Since charters struggled for more than a decade prior to the creation of the CSD, the district was not going to win public or legislative support by allowing new schools to evaporate into the ether due to lack of funds or ignorance of procedures, such as how to report accurate enrollment counts. CSD school officials labored to navigate the state reporting system, as the state shifted software providers between 2009 and 2011. At the same time, the CSD needed to uphold its mission of accountability to create, sustain, and retain high-quality charter schools.

“There is a lot of confusion, and there are a lot of roles and responsibilities [for the CSD],” says Reaves. “The district has to change personalities based on what it encounters in any given day or even any given hour.” Simple operational procedures, which existing traditional schools had mastered, were an enigma to CSD charters. How do they order textbooks? How can they order diplomas? “Student attendance and discipline questions were very common those first two years,” says Brazell, a 30-year veteran of public-school leadership in South Carolina. “I answered the same type of questions as when I was a superintendent in a traditional district.” The CSD desperately wanted to prove that charters could succeed under its auspices, so the district stretched beyond its authorizing role to help the new schools navigate the system.

In fall 2009, the CSD added three new schools, including a virtual high school that enrolled more than 1,000 students, and these schools needed the same guidance and services as the schools that had opened one year earlier. Two challenges faced the district office as it tried to distinguish itself as a charter authorizer and not just a traditional school district.

First, the CSD struggled to implement a comprehensive accountability scheme based on student performance on state assessments. The state department embargoes test scores for months after receipt, so the public does not have access to the results. District staff, parents, and teachers knew test scores, but schoolwide and districtwide averages could not be reported to the CSD board or its school boards because that would become public information. Without these data, school leaders did not have the current achievement information necessary to isolate areas of need and propose interventions. For charter schools, accountability for results is critical. By the time results were made public, the next school year had already begun.

Second, CSD staff continued to provide guidance to existing schools while simultaneously helping to launch new ones. With high staff turnover at existing schools (two of the five principals were replaced between the first and second year, not to mention numerous changes among assistant principals and teachers) and the addition of new schools, school officials needed training in critical procedures. Student information-system management and reports to the state, along with the means for implementing new curricula and distinguishing which state policies charters were exempt from and which they were not, were a mystery to many.

All of these issues converged as schools performed their primary purpose of educating students, frustrating progress on both fronts (operations and accountability). The financial circus kept school budgets in flux, making it difficult to prepare for additional student services, hire teachers, and develop strategic plans.

To make matters worse, the S. C. Charter School Act requires board elections at each charter school annually, resulting in the loss of institutional knowledge every year. “Folks operating charter schools were very naive for the most part, simply because they had never done it before,” says Brazell. Sometimes these new boards wanted to change course and replaced the principal, even after a school’s first year. In other cases, a principal was hired and then replaced before a new school ever opened its doors. At every turn, the CSD was forced to use hasty, temporary measures to help resolve problems that could be traced to the state policies in place. What resulted was a haphazard set of practices, inconsistently applied, with plenty of doubt to go around. “All of this put the district in a really compromising position,” says Connections Academy principal Reaves.

Hope for the Future

Fortunately, the story does not end there. Today, the CSD oversees 13 schools that serve more than 10,000 students. In 2010–11, Superintendent Brazell finished his second school year with the district and has filled key staff positions with knowledgeable personnel, many with a history at the South Carolina Department of Education. CSD staff experience has proved invaluable to the new charter operators. “Staff with experience in operations has helped to get a school’s questions answered quickly, and it’s allowed us to start talking about what a good district should be,” says Brazell. Only one charter has been revoked, so despite a roller coaster of financial adjustments and procedural changes, the schools are stabilizing. In addition, the 2011–12 state budget included a funding increase for CSD schools. Virtual schools received an additional $1,750 per student, while brick-and-mortar charters received an additional $3,250. Although the amount depends on a student’s category (grade level, special needs, etc.), the average CSD student is funded at approximately $5,000, still much lower than the average traditional school student but better than prior levels.

What took South Carolina’s charter movement so long? First, advocacy from key leadership positions had been missing. Brazell had no choice but to handle operational and administrative duties while also explaining the charter concept to legislators in the statehouse. The 2010 elections propelled a strong charter supporter into the state superintendent’s office. Dr. Mick Zais expressed support for charters in his campaign and made the 2011 charter legislation one of his first priorities. “That was a game changer,” acknowledged board chair McLaurin. “I’ve got to believe that we are creating a change in the culture, and he bought into that. He’s genuinely a believer in competition. Our whole relationship with the state department [of education] has changed.”

Second, authorizers with varied commitments to the reform effort slowed the growth of new schools. The statewide authorizer allowed a set of schools located in different areas around the state to coalesce as a group with a common outlook on education reform. All agreed that charters can succeed only if the initial political and administrative obstacles are overcome.

South Carolina’s statewide authorizer is less schizophrenic these days, though the concern coming to the fore is greater focus on support and administration than was intended. Brazell is looking to change that. With less uncertainty as to whether the schools will actually survive, the CSD can concentrate more on school quality and achievement. “The district board is freeing our office to concentrate on oversight and accountability instead of authorization,” says Brazell, which helps to narrow the focus for district staff. “I’ve told the schools that the expectations are higher now, and we are going to be focusing efforts on compliance. We’ve come a long way.”

“In any start-up that I’ve ever seen succeed, five years out from the start the business is never exactly like the business plan said it would be,” says McLaurin. “You’ve got this view of how the world is, but then you get out there and start interacting with the world and things change. I think that process was inevitable.”

Should these problems be solved, the fact remains that so long as the CSD continues to authorize schools, the district will have to train new school leadership and staff on compliance with state standards, while also holding all schools accountable for performance. In August 2011, the CSD approved seven charter-school applicants to open in the 2012–13 school year. Perhaps the strong leadership in the state and district superintendents’ offices, along with more experience among district and school staff, will result in more effective operations and better student outcomes in the future.

Jonathan Butcher is education director for the Goldwater Institute and served as the CSD’s director of accountability from 2009 to April 2011. Joel Medley is the director of the North Carolina Office of Charter Schools and was the director of the Charter School Office at the South Carolina Department of Education from 2008 to 2010.

It’s no surprise that, 28 years after the publication of A Nation at Risk, school-reform efforts have generated so little effect. Our schools have proven, over the past century, quite adept at resisting change.

Recent attempts to inject accountability and innovation have brought us to an important opportunity. No Child Left Behind helped add transparency, and Race to the Top (RttT) motivated states to rethink teacher evaluation, charter limits, and more. The Investing in Innovation fund (i3) has seeded some promising innovations and helped attract more private investment to public education.

But none of these initiatives hits at the reasons that education has proven itself so innovation-resistant: governance and compensation. Further, there is good reason to believe a third impediment—the absence of useful data—will persist even through the Common Core State Standards initiatives.

Finland serves as a model for many reformers. There is a single curriculum; teachers are well educated and well respected. Their system reflects Finnish ideals and builds on Finnish strengths, and their students score at the top of international tests like PISA (Program for International Student Assessment) and TIMSS (Trends in International Mathematics and Science Study).

But a top-down system will continue to be the wrong approach in this country, whether on a national or state level. It doesn’t reflect American values or culture, nor does it address the size, diversity, or income disparity of the United States. (Finland has half as many students as New York City, and only 13 percent live in poverty.) In a country of 300 million people, a top-down approach makes substantive change virtually impossible. To fix our schools, states have to stop trying to fix them; the quickest way to raise performance is command and control, but over the long run martial law does not even work well for generals.

States can create a more agile, more American, system of governance that eliminates impediments to improvement, empowers schools to innovate, and uses data to help families find the right schools for their children. The federal government should encourage them to do so.

None of the proposals below address the role of profitmaking companies in K‒12 education (though my bias might be clear, as I have run education companies for 30 years). It is important not to conflate marketplace with for-profit. It is also important to recognize that it takes time for deregulation and a newly formed marketplace to work. The breakup of AT&T and the telecommunications bill of 1986 did little to help consumers in the very short term, but they cleared the path for lower costs and technologies including the Internet and the cellphone. Occasionally efforts to create a marketplace don’t work at all, as happened with banking deregulation. As education is a public good and requires public funding, proposed structures should be measured by the incentives they will create for schools, districts, and teachers to produce great student outcomes at reasonable expense.

Empower Schools

Although our ultimate goal is a system of schooling that naturally evolves and improves, it’s important to keep in mind that the capacity for experimenting and innovating resides in individual schools, not in central offices. Under the current system of governance and funding, schools have too few resources and too little discretion for experimentation. Without the dollars to implement novel ideas and to discover what works and what doesn’t, most schools look for, at most, incremental improvement.

Right now, every state distributes state and federal funds to districts; in turn, the districts distribute funds to schools. Imagine that states instead channel funds directly to schools and require that the schools contract with a school support organization (SSO) for an array of services similar to what its district’s central office now provides (see Figure 1). There are many ways to implement such a plan, but the recent transition of New York City schools to its empowerment model might serve as a useful example, even though the city may be losing its resolve to change.

Ideally, existing school districts would be spun off as independent nonprofits and freed to compete with other districts, as well as with the new SSOs in the nonprofit and for-profit sectors, for schools and dollars. University of Washington research professor Paul Hill and others have proposed variants of this concept.

Since most schools (especially those in small and wealthy districts) would probably keep their existing districts as their service providers at first, the initial shift would be subtle. But before long the roles and behavior of schools and districts would begin to change. Freed to choose a district or other SSO based on service, cost, and philosophy, schools would demand more for less, and SSOs would step up to pull schools away from their local districts and compete by differentiating themselves from their competitors. Perhaps they would charge less for similar services; perhaps they would deconstruct the services, providing only busing, technology, or financial/purchasing support. Eventually, districts and SSOs would also vie for schools based on their track records of learning outcomes.

Under this proposal, districts would become providers of services rather than owners of geographic zones. With their schools acting as clients rather than dependents, districts would be forced to compete for them, thereby becoming more innovative and cost-effective.

Concrete results would take a while to materialize, but they would come. The current system of big-district purchasing, for example, favors large textbook publishers, which play it safe. School-level purchasing—with proper financial controls—would allow smaller, more responsive companies to compete for business.

Charter schools are the one reform initiative of the past three decades that has addressed the issue of K–12 governance and gained some traction (some 5 percent of public schools are now charters). This proposal builds on some of the lessons learned from the charter school movement and would allow effective charter networks like Green Dot, KIPP, and North Star to operate as school support organizations on a level playing field with districts, with equal funding and authority. A great deal of innovation today is coming from charter networks; this change would encourage districts to match them.

Most states would need to implement significant initiatives to prepare school principals for their new role, and to recruit new principals with the right skills; education schools and programs like New Leaders for New Schools could participate in this effort. Further, states would need to balance power between districts and schools; for example, districts should have the power to reject association with a poorly performing school. Both schools and districts should be pushed to improve themselves and their products and services.

Accountability would become simple (and imperative) under this model. The newly empowered schools should live or die by their performance; similarly, SSOs would lose their customers if they proved unable to support high achievement (which is how the stock of K12, Inc., lost 40 percent of its value following a single critical article in the New York Times). Accountability goes hand in hand with empowerment; promoting one without the other will not succeed.

Empowering schools would also mean encouraging parental choice. After the district’s monopoly is broken up, it would be critical that states create intelligent, consumer-friendly systems to support parents in choosing their children’s schools. Any number of successful models exist, all of which would provide transparency and could be used to balance families’ desire for schools within reasonable distance with their desire for the right outcome.

This is not an easy change; further, many districts are already well run and don’t need change at all. But this proposal would remake the relationships between schools, districts, and states into a far more efficient and effective model, one that would increase agility and remove regulations that limit the autonomy of school leaders. (As Arizona congressman Jeff Flake once asked, “Who out there can sing their district fight song?”)

Offer Teachers a New Deal

Once we’ve empowered schools, we’re ready to address teacher compensation. Many people believe that teachers unions are a major cause of whatever they think is wrong with our schools. It’s not that simple; plenty of research suggests that districts without unions do not perform better than those that have unions, and are only slightly less expensive.

To be sure, pensions and tenure are huge impediments to organic change. But two parties signed the contracts putting them in place: the union, whose job is to get its members more pay for less work, and the district. It was the side representing kids—the districts and state legislatures—that failed. Demonizing unions and teachers is unfair and counterproductive.

The problem isn’t the total compensation; if anything, teachers are underpaid. It’s the structure of that compensation, a series of long-term obligations that severely limit agility while creating off–balance sheet debt that would make Wall Street blush. (According to district budget figures, New York City, for example, spends as much on teachers who no longer teach as on those who still do.)

Ending tenure without ending the current pension system would create some impossible pressures; teachers nearing certain vesting thresholds, for instance, would have a target on their backs. To create an agile system, states must end both tenure and pensions. We can take a big step down this road without reneging on commitments made to a generation of teachers who have accepted lower base salaries for long-term benefits. The starting point, in fact, is something many teachers would embrace.

States should give each teacher the right to choose an alternative contract that contains terms and benefits consistent with those in the private sector (e.g., an at-will contract with standard health-care benefits, 401k, etc.), and sits outside of the existing teacher pension system. Choosing this alternative contract would convert any existing pension to a lump-sum 401k contribution. In return, the new contract would have a far higher base salary; in fairness, states should require districts to hire an auditor to determine the savings that can be expected from each alternative contract teacher, and give that savings to the teacher as increased pay.

Under this plan, no current teachers would be forced to change their contracts. If a state chooses to implement this policy change on a school-by-school basis, teachers who choose the current traditional contract might be offered a transfer or be grandfathered, that is, allowed to continue under their current contract. But the alternative contract could be attractive: depending on the state or district, the expected pension-related savings over a standard contract could be as much as $25,000 per year per teacher. In New York City, for example, a teacher might choose her current contract and a $65,000 salary, or the alternative employment terms with a $90,000 salary but with no tenure guarantees. This change would not reduce costs overall, but it would begin to curb the practice of paying operating expenses with long-term, off–balance sheet debt.

Conversion specifics will vary by state; obviously, those with huge unfunded liabilities will have a tougher time finding an elegant solution to converting past pension obligations for teachers nearing vesting milestones. Some percentage of teachers will refuse to switch; every teacher who does switch, though, will reduce the scope of the long-term problem. Many teachers will prefer to have their retirement funds fully in their control, along with a higher base salary, over a pension subject to fierce political pressure.

So which teachers might choose the alternative contract? My hunch is that newer teachers, who would appreciate the extra cash, and high-performing teachers, who would be unconcerned about the decreased job security, would be likely converts. If that’s true, it’s probable that schools with the highest-need students (who traditionally have the least-experienced faculty) would be most likely to convert over to the new contract, and might thereby be able to attract higher-performing teachers.

Schools operating under the alternative contract would be free to evaluate teachers based on student performance and evaluation, as well as classroom observation and other evidence. These teachers could be empowered to shape their schools, by taking part in choosing the curricula they use in their classrooms and the formative assessments they use to measure student progress, for example. Giving teachers a voice in decisions that affect their work is a logical complement of recognizing and compensating them as professionals rather than as assembly-line workers.

Does this proposal solve the compensation problem? Not entirely, though it would take us halfway there. If we also clean up our accountability systems, we could compare the performance of teachers under each contract and adjust the compensation system to include performance metrics as appropriate.

Align Assessment to Curricula

For all their deficiencies, assessments of student learning are an indispensable component of an evolving school system. Without accurate assessments aligned with curricula and standards, education innovators would be flying blind.

The multistate Common Core State Standards project is an improvement over the patchwork of past state standards. But the standards are not the source of flaws in state accountability systems; the culprits are the state tests.

Tests used by international organizations, like TIMSS and PISA, and also our own NAEP (National Assessment of Educational Progress), can measure performance because they’re both broad and deep; they use a reasonable number of items (many of which are constructed responses) against a large number of standards. But that design makes those tests too long to give to one student. Instead, they’re matrixed; 10 students might each take one-tenth of the test. A few thousand well-selected subjects might give us an accurate picture of 4th graders in a state, but these types of tests cannot be used to measure the performance of a student or school.

A state or national test, on the other hand, can only last an hour or two in each subject. Because such tests must contain several items per standard to be accurate, it will measure only a fraction of the standards. And since a test must be reliable from year to year, it will measure that same subset every year. This limitation encourages schools to narrow their curricula to only those standards likely to be measured and gives rise to illusory performance gains. At present, various groups of states are trying to work out this problem. In the end, they’ll trust that the testing companies will solve this problem, and once again, they’ll be disappointed. There’s a better path.

Imagine if states stopped commissioning their own tests and instead created a small set of requirements for each curriculum provider:

• Adopt or create a secure summative test for each grade level. This test should align closely to the curriculum, and every school using that curriculum would use that test to measure student performance.

• Work with client schools to administer NAEP (or some other matrix-based test aligned to the standards) to 2,000 students each year in key grade levels; use their performance to set the curve for the summative test (think of this as “Curriculum NAEP,” the equivalent of the current state NAEP testing).

• Set the curve for tests on a standard score range that facilitates value-added analysis.

This new way of thinking about summative testing would retain the advantages of the Common Core project and the best state tests while eliminating most of the disadvantages. States would retain the authority to determine the curricula they might subsidize or even allow; they might adopt only one for some subjects and grades (say, for K–6 math); in this case, the world would look a lot like it does now. States would be better off, however, allowing schools to adopt curricula, along with the corresponding summative tests, that best fit their students’ needs. Again, it makes sense to empower schools at the same time that we hold them accountable for student performance. Either way, states could continue to compare schools, since each curriculum would be scored on the same curve and the scores equated through Curriculum NAEP.

This proposal would eliminate most gaming around test scores. There would be no incentive for a provider to dumb down its test, since Curriculum NAEP scores (and therefore the curve) would leave scaled scores unchanged. Moreover, the proposal would create a true alignment between curricula and tests, by removing the state as intermediary. Rather than teach to the state test, schools would teach a curriculum, and then test students accordingly.

Best of all, this regimen would encourage differentiation and competition among curriculum providers. In the end, the curriculum generating the best results for a particular cohort (say, middle-school Latina students) would likely be adopted by schools with large groups of those students.

That competition would extend to the tests themselves. A test should be judged not only by its accuracy, precision, and reliability, but also by its ability to promote learning. Many educators believe that authentic assessment (asking students to perform complex tasks rather than answer multiple-choice questions) encourages better teaching and learning; if this proves true, then curriculum providers using authentic assessments would dominate the market, despite their higher costs.

Finally, this approach would save money. Curriculum providers will find much more agile ways to connect to assessment providers than any state consortium has found so far.

Let the Data Flow

If our schools are to continually improve, we need to gather data and make it available not just to schools, school districts, and parents, but also to independent researchers, who can comb the databases for correlations and any underlying causal connections. Our goal should be to create a veritable education genome project open to all appropriate parties, with proper security measures to address privacy concerns.

We currently gather data through a 1970s-era approach that is slow and expensive. As data move from classroom to school to district to state to the federal government, the details that would allow us to draw meaningful conclusions about things like the effectiveness of a textbook, a supplemental services provider, or an afterschool program are lost. Meanwhile, Google and others manage much more data with far less cost and difficulty. We need to adapt their processes to education data.

The testing companies already collect data from individual schools, as they send and collect test booklets either directly or through the district. These vendors are technically savvy and have the incentive to maintain participation in a lucrative assessment market. States should require their testing vendors to collect data from each school in a standard format, including at least the curricular materials used in each classroom, the calendar and schedule in use at that grade level, the background of the teachers, and any academic interventions used for particular students. The companies should be required to then forward these instructional data, along with test scores, subscores on specific components of the test, and student demographic information, to the state in a standardized format. The state, in turn, should publish a database with accounts allowing schools, districts, education consumers, and (in a privacy-ensured format) researchers to access at will.

There are obvious privacy concerns about publishing personal data in a state database. However, these data are far less sensitive than other data that are commonly secured and made widely available. (Just what would someone do with your son’s 5th-grade math grades?)

Thousands of researchers would surely exploit the resulting database. Curriculum providers would look for evidence of their (or their competitors’) effectiveness. Policymakers would examine the results of various interventions, including afterschool programs, changes in class and day length, or class-size reductions. Teacher preparation and in-service training programs would know whether and where they were having an impact. Parents would be able to make informed choices about where to send their children to school.

Most states would save money by making use of this more efficient way to collect data. At the same time, it would spawn a wave of innovation, as various players start using the data.

Innovation and the New ESEA

All four of these proposals would move us away from a command-and-control education system, and toward an agile education marketplace that encourages innovation and excellence. But even if these proposals sound reasonable to you, you’re probably still wondering how and when they might ever come to pass.

The answer is through the next iteration of the Elementary and Secondary Education Act (ESEA); by attaching the mind-set of RttT and i3 to the billions of dollars of annual education aid to states, we can use incentives to encourage the right behaviors quickly and inexpensively. Title I channels $14 billion per year to states, which pass it along to districts along with their own funding. Imagine if the new law leads states to channel that money, along with their own funds, directly to schools, and discourages them from holding to the status quo. With a small tweak (for example, an increase or decrease in funding of 10 percent), the feds would give states a $3 billion push in the right direction.

The language enabling schools to choose a district or SSO should be simple. Each state should find its own path to empowering schools. Perhaps some states would empower high-performing schools first, while others might put failing schools into governors’ districts like the one currently proposed in New Jersey. Perhaps states with higher population density would create statewide choice systems, while others would favor parents who sought short travel times. There are many mechanisms imaginable for allowing a school community to vote on its district or SSO affiliation and for states to license and monitor school support organizations.

Similarly, Title II provides roughly $3 billion per year for professional development. The federal government could limit those funds to states that give teachers the right to choose the alternative contract. Again, though, the new ESEA should allow states great latitude in structuring that right (for instance, they could give that choice to individual teachers, or allow a school-by-school vote); regardless, each state will have to figure out what to do with its pension obligations to teachers who switch to the new contract.

The process by which Common Core states are creating math and English tests is well under way; it may result in top-notch exams that lead to dramatic performance increases. The easiest place to implement an assessment marketplace, then, is in science, history, and language courses. ESEA should establish a group that registers curricula in those areas; if this marketplace proves effective and states struggle with the Common Core tests, this marketplace can easily expand to incorporate math and English.

The accountability provisions of ESEA should require testing companies to phase in collection of school-level instructional and background data. Initially, the testing companies could provide the data to client states for analysis; perhaps down the road, states or foundations will find it useful to run studies across multiple data sets.

None of these proposals is expensive; in fact, most will save money in the short and long term. And although some might be politically inexpedient, none would have the natural and well-funded opponents of other commonsense reforms. Further, this is not an exhaustive list. Every reader of this article could probably come up with additional reforms that would create a more responsive education system.

This plan places a great deal of faith in competition and innovation, though within the construct of a robust public school system. As I’ve noted, this faith could be misplaced: perhaps education truly is different, and there simply is one immutable right way to run schools. But there is something to be said for empowering our schools with transparency, choice, and agility. American ideals shouldn’t just be taught in the classroom; they should shape that classroom.

John Katzman is the executive chairman of 2tor, Inc.

 

It was goal-setting day in Rich Julian’s 5th-grade class at Covington Elementary School in Los Altos, California, when I visited last fall, and Julian was asking each of his 29 students to list three math goals for the week.

To become proficient at dividing a one-place number into a three-place number, a girl with blue-painted fingernails wrote in her math journal.

To become proficient in multiplying decimals, wrote a dark-haired boy. To become proficient at subtracting one four-place number from another. To become proficient in arithmetic word problems. To complete an exercise in the properties of numbers, like (4 + 9) + 5 = ? + (9 + 5).

No two youngsters seemed to have quite the same math goals because, of course, no two youngsters are quite alike when it comes to learning. That’s why Los Altos is betting the future on an online math program from Khan Academy, and why scores of other schools and districts are clamoring to include Khan Academy in their math curriculum.

For the next 45 minutes, Julian met individually with his 5th graders to refine their goals. (In November, Julian left Los Altos to become assistant principal in the Milpitas Unified School District.) Everyone else logged onto the free Khan Academy web site and called up the “module,” or math concept, that fit their goals. Some watched short video lectures embedded in the module; others worked their way through sets of practice problems. I noticed that one youngster had completed 23 modules five weeks into the school year, one had finished 30, and another was working on his 45th.

As youngsters completed one lesson, an online “knowledge map” helped them plot their next step: finish the module on adding decimals, for example, and the map suggests moving next to place values, or to rounding whole numbers, or to any of four other options.

Julian, meanwhile, tracked everyone’s progress on a computer dashboard that offers him mounds of data and alerts him when someone needs his attention. He showed me, for example, the data for a child who had been working that day on multiplying decimals. The child had watched the Khan video before answering the 1st practice problem correctly, needed a “hint” from the program on the 3rd question, got the 7th wrong after struggling with it for 350 seconds—the problem was 69.0 x 0.524—and got the 18th correct in under a minute.

But just as powerful are the data kids have on themselves. The Covington youngsters regularly pulled up an array of charts that showed them which math concepts they had mastered and which they were working on, needed to review, or were stumbling over.

The classroom buzzed with activity, and amazingly, all the buzz was about math.

Khan’s Rise

By now, more than 1 million people have watched the online video in which Salman Khan—a charming MIT math whiz, Harvard Business School graduate, and former Boston hedge-fund analyst—explains how he began tutoring his New Orleans cousins in math by posting short lessons for them on YouTube. Other people began watching the lessons and sending Khan adulatory notes (“First time I smiled doing a derivative,” wrote one) or thanking him for explaining fractions to an autistic son.

Khan quit the hedge fund, moved to Silicon Valley, and in 2009, with funding from a constellation of technology stars (Bill Gates’s children were using the videos), launched the nonprofit Khan Academy. A year later, Mark Goines, a member of the Los Altos school board and a legendary Silicon Valley investor, introduced Khan to the district’s new superintendent. Los Altos already ranked among the best-performing districts in the state, but it had set itself a goal of improving individual achievement, and “capturing data at a granular level” on each student was proving difficult, Goines told me.

A few weeks later, in November 2010, Los Altos agreed to pilot Khan Academy with two classes of 5th graders and two classes of 7th graders and provide Khan with feedback to refine the web site and tools. By summer 2011, some 250 school districts, charter schools, and independent schools were asking to be part of the pilot—Khan chose only a dozen—and have Khan staff work with them to integrate the videos, data dashboard, and other tools into their curriculum.

Salman Khan’s short videos remain the centerpiece of Khan Academy (there already are 2,576 of them and counting). In each one, Khan’s voice describes a discrete math concept, such as solving a quadratic by factoring or interpreting inequalities, while only his hand-scribbled formulas appear on-screen. Khan’s idea was that youngsters would watch the videos at home and work on problems in class, essentially “flipping” the classroom (see “The Flipped Classroom,” What Next, Winter 2012). But teachers told me that youngsters also are using the videos as a just-in-time solution when they’re stumped on a problem in class, or to move ahead when they feel ready.

The data that the web site churns out and the site’s gaming features seem to be the real learning motivators. Youngsters become “proficient” in a concept by answering a “streak” of 10 consecutive computer-generated questions: miss one and the computer sends you back to the start. Youngsters earn “energy points” for correct answers, and badges for accomplishments as diverse as working speedily (that’s a meteorite badge) or becoming proficient in the Pythagorean theorem (that’s a moon badge).

Ted Mitchell, president of the NewSchools Venture Fund and a Khan Academy board member, told me that Khan developers “were blown away by how important” the games and badges seem to be in giving kids a sense of accomplishment and progress. Even older kids, for whom badges are ho-hum, “are instantly motivated” when they complete a streak, and the program acknowledges their accomplishment, says Brian Greenberg, who until recently was chief academic officer of Envision Schools. “What’s brilliant about Khan Academy is the instant feedback,” Greenberg told me.

Envision runs four charters in Northern California, including one that piloted Khan Academy with a small program for remedial-algebra students last summer.

The Teaching Curve

From Covington Elementary, I dropped in on Courtney Cadwell’s 7th-grade pre-algebra class at Egan Junior High. She, like Julian, piloted Khan Academy last year. Based on that first-year success, Los Altos extended the program to all of its 5th- and 6th-grade classes, and to its 7th graders who were achieving at grade level and below.

Cadwell, a 17-year teacher who was wearing University of Texas orange for her alma mater, calls Khan just “one resource we use.” The previous night, she had assigned worksheet homework; she began the class with a textbook lesson. Math projects ringed the classroom, a reminder that Khan Academy doesn’t include project-based lessons. That night’s homework included a reading on the origin of zero: Cadwell, among others I spoke with, said Khan’s weakness is that it “is not great at helping kids conceptualize math.”

Khan’s strength became clear a few minutes later when the students opened their laptops. Cadwell strolled the room with an iPad in hand, tracking the youngsters as they moved through problems and modules, and intervening with a quick one-on-one when the data identified a student who was stumped. “I’m getting data in real time about each student instead of assuming the entire class needs intervention,” she explained afterward. Khan “lets me use my class time more wisely.”

It also means that teachers have to figure out new ways to work. “Teachers have to be willing to escape from the role of standing in front of the class” and flexible enough to group kids based on need, said Julian, who was a math coach in New York for 20 years and retains his big-city bustle.

As I watched Julian, Cadwell, and later Ruth Negash at Oakland’s Envision Academy of Arts and Technology, they seemed to be always on the move—meeting individually with children, tutoring small groups, and occasionally addressing the whole class. “I actually work harder” with Khan Academy, Julian said. “I’m up and around more, meeting with kids more.” That gives time back to students and, as Cadwell said, makes them “take ownership of their learning” by setting their own goals.

It also means a new level of classroom collaboration: youngsters can look at each other’s data and identify “coaches” among their classmates. Julian urged his 5th graders to ask the Khan program for a hint, watch a video, or ask a coach for help before coming to him. “Show him how to do it, don’t walk around the class giving answers,” he admonished would-be coaches. Pretty soon, a girl in a pink T-shirt turned to a girl in purple for coaching, and the two worked meticulously at solving 1.94 x 5.52.

Making It Work in Oakland

Los Altos is an affluent, tech-savvy community; I next wanted to see how Khan Academy could work in an inner-city classroom. So two days later, I visited Envision Academy, a downtown Oakland charter school, and Ruth Negash, an intense 4th-year teacher with wild, curly hair and two education degrees from San Francisco State University.

In 2011, Negash taught two summer-school classes of 9th, 10th, and 11th graders who had failed Algebra I. One randomly assigned class used Khan Academy; the other was a traditional math class. The results were promising enough that Negash now is using Khan in all of her 9th-grade algebra classes.

On the day I visited, Negash started both of her classes with a minilecture on linear equations, and then had her students solve for x in 7x + 4 = 18. The classes quickly became fidgety, first as Negash explained the problem, and then as youngsters finished at different speeds. Negash had to urge them to “respect the community of learning.”

But that changed a few minutes later when the youngsters opened their computers—I had noticed the same change in Cadwell’s class—and worked on Khan Academy for the next 75 minutes. I heard an occasional groan of exasperation. “They threw a trick question at me and sent me back to the beginning,” one boy moaned when his streak was broken. But the energy now was directed toward everyone’s screen.

Although everyone in Negash’s classes had taken, and presumably passed, algebra in 8th grade, their math competence ranged from marginal to impressive. In both periods, three or four youngsters claimed a table in the hallway, where they worked silently at lessons on quadrilaterals and complementary and supplementary angles, typical geometry exercises. But other students struggled with addition and subtraction, and one quarter don’t know their multiplication tables, Negash told me. (To keep those youngsters from falling even further behind, she gives them a reference sheet with the multiplication tables on it.) Negash told both classes to work on the Khan module on solving for a variable—a continuation of her minilecture—but Khan’s online prompts were urging most youngsters to first review lessons on lower-level skills.

Some of these youngsters simply “feel safer” doing arithmetic and will move on when they’ve experienced some math “success,” Negash predicted. Other educators had similar takes: Khan “takes away a lot of the fear about math” by letting kids backfill their gaps and then move ahead at their own pace, said Sandra McGonagle, the principal of Santa Rita Elementary in Los Altos, which also is using Khan Academy in its 5th and 6th grades.

“You don’t have to worry about getting something wrong in front of the whole class,” one of Julian’s 5th graders, the girl with blue nail polish, told me.

But in Negash’s classes, the wide range of math abilities is clearly a challenge. Negash sat with one low-performing student for much of the first-period class and with three others in the second period, hoping to encourage some of that “success.” Meanwhile, other students were calling for her help. Two boys were stumped by “adjacent” in a word problem; language issues crop up “every day,” Negash said.

When Negash finally had a moment to consult her Khan dashboard at the end of second-period class, she saw that one youngster had spent 62 minutes solidly working on math, but another had spent only 14 minutes. “It’s hard to figure out a different plan for 25 kids every day,” she sighed.

Gia Truong, superintendent of Envision Schools, said Khan Academy developers had urged her to let Negash’s students “start where they were” in math and move forward. But that’s creating a conflict when some kids are so far behind, she told me: “If you do that, you might never get to the algebra standards” that California students must pass in order to graduate.

“You’re in the new paradigm, but the grading standards are in the old paradigm,” she added.

Getting to Results

Test results at both Los Altos and Envision—the only two pilots to have any results so far—suggest that Khan Academy is working. Los Altos says that among the 7th graders who used the program in 2010–11—all remedial students—41 percent scored “proficient” or “advanced” on the California Standards Test compared to 23 percent the year before. Among 5th graders, 96 percent using Khan were proficient or advanced compared to 91 percent in the rest of the district.

At Envision’s summer-school program, the youngsters in the Khan Academy class spent only half their time on algebra—the rest of their time was on lower-level math skills—and yet still slightly outscored the traditional class, which spent all of its time on algebra.

Both districts are quick to say that it’s far too early to claim success: there were only 115 youngsters in the Los Altos pilot and just 20 at Envision. “It’s enough to say this is promising; it’s not enough to say this is the future,” former Envision Schools officer Brian Greenberg said.

Most observers of the Khan experiment agree that the measure of success must be student achievement. Otherwise, “I’m not very sympathetic,” said Michael Horn of Innosight Institute. As teaching is increasingly differentiated, however, schools may need a different kind of assessment. California’s year-end test can tell which 5th graders meet the state’s math standards; it can’t tell if some of those 5th graders have progressed to trigonometry or pre-calculus, as two Los Altos kids did last year.

But several experts also suggested measuring Khan’s impact by also looking at changes in the distribution of test scores. Khan Academy isn’t likely to close the learning gap because some kids, freed from the teach-to-the-middle plod of the usual classroom, gallop ahead. But Khan would be a success if low-performing kids move ahead too and “shift the bell curve to the right,” said the NewSchool Venture Fund’s Ted Mitchell.

Some other Khan watchers gave a surprisingly strong endorsement to such measures as student engagement and self-confidence, and to soft skills like goal setting and teamwork. “I don’t look at it as just based on the data,” said Mark Goines, the Los Altos school board member whose high-tech background (he helped develop and run TurboTax for Intuit, Inc.) suggests a fine reading of the data. “The kids seem to be happy about learning. That makes me excited,” he said.

What about increasing class size, I asked: Should Khan’s success be measured in part on its ability to increase teacher productivity? In elementary schools, where students generally spend the day with one teacher, increasing class size because of Khan would mean bigger classes in every other subject, too. And Goines, who said he has viewed “hundreds” of online programs, cautioned that there aren’t any comparable products in other subjects, especially in writing.

A fear among advocates of online learning is that slow learners will be abandoned in front of a computer, and a large classroom increases those chances. “It would then become a babysitting tool,” said McGonagle, Santa Rita’s principal.

Blending Khan

Finally, I asked for “takeaways” from the Khan Academy experience. Greenberg told me that it’s more important that teachers be “nimble” and “entrepreneurial” than that they be tech wizards. All three teachers said they felt comfortable with technology, but that, more importantly, they were risk-takers. Even before she began piloting Khan Academy, Cadwell asked her PTA to buy classroom laptops for the youngsters in her remedial math class. “I figured if I could get them onto some practice sites, I’d figure things out from there,” she said.

Santa Rita’s McGonagle said it was “crucial” to have pilot teachers like Cadwell who can act as avatars for the rest of the district as it expands its blended learning. Cadwell is mentoring other Los Altos teachers this year. They “don’t need training as much as they need time” with the program, she told me (the data are fairly easy to use, but she and Julian asked Khan’s engineers for so much of it that both say they don’t always use it all).

The schools, meanwhile, are holding rollout meetings for parents and are urging parents to join the web site, where they can see the same data as the teachers, including whether little Bobby is really working on math up in his bedroom as he says he is. “It’s not just training the teachers; it’s training the community,” Goines said.

That training shouldn’t end with just learning to manipulate the data, though. It also means learning how teachers can use their time differently, how to work with youngsters who have different abilities, and how to blend Khan into the curriculum, not substitute for it, everyone told me. Cadwell and Negash said that they find gaps in the Khan curriculum, and that it isn’t completely aligned with either California or core-curriculum standards, although Khan is adding lessons to fill the holes.

“You can’t just put a kid down in front of a computer,” Goines said, although the kids I saw in Julian’s, Cadwell’s, and Negash’s classes sure seemed to enjoy it.

June Kronholz is an Education Next contributing editor.