Like the makers of hot dogs, psychometricians, economists, and other testing experts know too well what goes into the creation of achievement tests. Their intimate knowledge of the technical difficulties involved in measuring student achievement makes a number of these testing experts some of the most vocal (and persuasive) opponents of testing. But the flaws in techniques like value-added assessment do not automatically lead to the conclusion that those techniques shouldn’t be used to hold educators accountable. Testing may be imperfect, but the alternative–the old system, which allowed us to know very little about the performance of educators–is far, far worse.

To be sure, many of the technical criticisms of value-added testing are correct. It’s true that there is more random error in measuring gains in test scores than in measuring the level of test scores. It’s true that there is some uncertainty as to whether gains in one area of the test scale are equal to gains at another point in the test scale. And it’s true that factors besides the quality of the schools can influence the gains that students achieve. But, on balance, these downsides hardly outweigh the benefits to be reaped from being able to measure and reward productivity in education.

Consider what is likely to continue to happen in education without high-stakes value-added assessment. Unless productivity is measured, however imperfectly, it is not possible to reward teachers, administrators, and schools that contribute most to student learning. If we do not reward productivity, we are unlikely to encourage it. If we do not encourage it, we should not expect more of it.

In fact, this is precisely what has been happening in U.S. education during the past few decades. Between 1961 and 2000, spending on education tripled after adjusting for inflation, from $2,360 to $7,086 per pupil. During that time, student performance, as measured by scores on the National Assessment of Educational Progress (NAEP) and high-school graduation rates, has remained basically unchanged. Whenever spending triples without any significant improvement in outcomes, there is a serious productivity crisis. Yet U.S. public schools just keep chugging along, resisting serious attempts at reform.

Meanwhile, private firms in the United States have been able to achieve steady gains in productivity because the discipline of competition has forced them to adopt systems for measuring and rewarding productivity. Firms that fail to measure and reward productivity lose out to their competitors who do.

Moreover, the systems that private companies use to measure and reward productivity are far from flawless. In fact, the challenge of measuring productivity in the private sector is often as great as or greater than in education. Imagine a soft-drink company that wishes to measure and reward the productivity of its sales force. The company might determine bonuses (and even decisions on layoffs) based on its salespeople’s success at increasing soda sales in their sales area. Like measuring gains in test scores, measuring increases in soda sales is fraught with potential error. Changes in soda sales could be influenced by a variety of factors other than the sales acumen of an employee. Unusually cold weather in an area, a local economic downturn, or exceptional promotional efforts by competitors could all suppress the soda sales of even a very good salesperson. If data on sales are collected using survey techniques, there is also the possibility of random error attributable to the survey method, just as testing has random error. Moreover, if we are comparing sales increases across geographic areas, it is unclear whether it takes more skill to sell soda in an area where the market is already saturated than in an area that initially consumes less soda.

In short, many of the same technical flaws that critics find in value-added testing also exist in the measurement of increases in soda sales. Changes in outcomes may be attributable to factors other than the efforts of the employee. There is random error in collecting the data. And the effort required to produce gains at one level may not be the same as at another level. The only difference is that private firms have rightly not let their inability to achieve the best deter them from pursuing the good.

In the private sector, companies have realized that even flawed evaluation systems nevertheless encourage improvements in productivity. This is because employees cannot be sure that a flawed system will completely obscure the picture of how hard they’re working. Employees therefore act as if their productivity were being measured accurately; the chance that slacking will be detected inspires employees to avoid slacking. In fact, evaluation systems with a fairly large amount of error in measuring productivity can still be effective at motivating improvement–if the errors are mostly random, or at the very least do not create perverse incentives, such as encouraging teachers to focus on improving the achievement of one group of students to the exclusion of others.

None of the technical concerns with value-added testing involve perverse incentives. For the most part, the criticisms have to do with random noise in measuring gain scores. Even the nonrandom errors that worry testing critics, such as unevenness in the testing scale or the possible influence of factors outside the school’s control, do not create perverse incentives because there are no strong theories about the kinds of behaviors those errors would encourage.

If no one knows what is being mistakenly rewarded, no one has an incentive to engage in that perverse behavior. As long as educators are aware of what the value-added system is supposed to be rewarding, and as long as that system rewards the desired outcomes more than it erroneously rewards something else, the system will help to elicit more of the desired outcomes–namely, improvements in student achievement.

The Uses of Data

The development of even an imperfect value-added testing system would revolutionize the systems for hiring, promoting, and compensating teachers. Our current methods provide teachers with little incentive to improve achievement. Promotions and salary increases are based on teachers’ seniority and their acquisition of advanced degrees. These characteristics are, at best, weakly related to student achievement. Excellent teachers who possess a master’s degree and a few years’ experience receive exactly the same salary as lousy teachers with the same formal credentials. Under the current system, we have turned the keys over to educators and trusted that their professionalism will yield improvements in student achievement. Education’s productivity crisis in the past four decades should be evidence enough that simple trust is not sufficient.

The development of value-added assessment would also revolutionize how we govern schools and hold them accountable. We currently have little rational basis for saying that a particular school is a good school or that a particular superintendent is a good superintendent. Value-added testing would at least give voters some idea of whether they are getting their tax money’s worth out of the school system by giving them at least some information on how the schools are doing. The fact that voters would have better information on achievement provides the school board with incentives to hire and retain a superintendent who can elicit improvement in student learning. The superintendent, in turn, has an incentive to hire and retain principals who will use the value-added results to hire and promote the best teachers.

Critics of value-added assessment don’t necessarily object to using value-added assessment. They object to using the data gleaned from it for high-stakes purposes, such as rewarding or punishing individual schools and teachers. Instead, they suggest that value-added results be provided to administrators so that they can make informed decisions about their employees. This is, to some extent, what happens in the private sector; most private firms do not use the crude techniques exemplified by the real-estate company in David Mamet’s Glengarry Glen Ross, where the salesperson with the fewest sales was fired. Most companies use productivity measures to inform the subjective assessments of supervisors, with some companies permitting less subjective judgment than others for fear of bias or favoritism.

But here is where the parallel between measuring productivity in public education and private industry ends. Supervisors in private companies have incentives to use the information provided by productivity measures properly, because their companies face the discipline of competition from other companies. If supervisors fail to put data on costs, sales, and revenue to good use, their companies will lose out to competitors who do.

In public education, by contrast, local decisionmakers have few or no incentives to make good use of data in assessing their employees because public schools face no meaningful competition. There are basically no consequences for principals who disregard the results of value-added assessments in making decisions about employees, and they’re more likely to disregard those results if they consider value-added assessment an unreliable analytical technique. Superintendents will not be able to judge whether principals have used their discretion properly, because they will be told that the value-added test results are not proper grounds for assessing the decisions of principals. And school boards and voters, in turn, will all be stymied in making independent judgments because they will be told that the professional decisions of educators are more reliable than value-added test results.

So there is good reason to fear that principals in public schools, if given discretion to reward teachers as they please, will base their decisions on personal relationships rather than on the results of value-added assessments. In both private industry and public education, a balance must be struck between the mechanical use of productivity measures in assessing employees and relying on the subjective judgments of supervisors. But, in public education, the balance needs to tilt more toward the mechanical application of results, because supervisors have fewer incentives to make appropriate subjective judgments. This means that value-added assessment needs to be high stakes to have the desired positive effect on student learning.

Injustices are unavoidable if value-added assessments are used more mechanically. Some educators will be improperly punished for eliciting what appear to be low gains because of measurement error. Conversely, some educators will be rewarded for improvements for which they were not actually responsible. That said, some of the flaws in value-added assessment have potential technical solutions. For example, if judging a teacher based on his classroom’s test scores contains too much error because the sample is too small, we might decide to rate teachers based on a moving average of multiple years of results, thereby increasing the sample size and reducing the random error. But even with technical fixes, some injustices will still occur.

This is not a good reason to abandon the idea, however. After all, some educators will be treated unjustly under any evaluation system. Under the current system, excellent, hard-working teachers who put in tons of overtime receive the same salary as mediocre teachers. What’s fair about that? The Buffalo, New York, district recently announced layoffs as a result of a budget deficit. Who were the first to receive pink slips? Not the worst teachers in the district, but the most recently hired. Surely some excellent teachers lost their jobs, while the district retained its burned-out veterans. A peer- or supervisor-review system may reward teachers who are popular among their peers rather than effective with students. Attempting to measure and reward successful educators, with all of its imperfections, is likely to create fewer injustices than any other arrangement. At least using value-added assessments increases the chance that good teachers are rewarded and bad teachers are sanctioned.

Besides, ensuring that every single educator receives justice is at most a secondary concern. An obvious but infrequently recognized truth is that the primary purpose of the education system is to provide a quality education to all students. If high-stakes value-added assessment can help motivate public schools to provide students with a better education, it’s a promising reform even if it has some cost to school employees. In no other industry would we even entertain the notion that the interests of employees trump those of customers. Only the political dominance of teacher unions makes us consider the question. It is true that customers usually receive the best service when employees are treated well and fairly. Happily, high-stakes value-added assessment is likely to achieve both ends.

The productivity crisis in public education has certainly created injustices for students and taxpayers during the past few decades. Students have failed to receive high-quality instruction while taxpayers have been paying more and more in return for stagnant test scores. Again, what’s fair about that?

-Jay P. Greene is a senior fellow at the Manhattan Institute for Policy Research.

Revolution at the Margins: The Impact of Competition on Urban School Systems
By Frederick M. Hess
Brookings Institution, 2002, $45.95; 268 pages.

As reviewed by Edward B. Fiske

For the most part, the language of economics has informed the public debate over school choice. Free-market economist Milton Friedman was the first to develop the concept of school vouchers, and some of the most enthusiastic supporters of charter schools and vouchers are businessmen who view the delivery of public education through the lens of their own experiences in creating and marketing goods and services. It is thus important to know that Frederick Hess is a political scientist, not an economist. What interests him is the way in which the political and organizational realities of urban schools influence their responses to competition and thus help determine how competition will affect the schools that more than 90 percent of students still attend.

The underlying agendas of choice advocates vary widely. Some view school choice as a social good in and of itself, while others may have indirect objectives, such as funneling public funds to religious schools or privatizing public education. Whatever their agendas, however, most supporters of school choice build their political case on the virtues of competition for public education as a whole. “Perhaps the most commonly advanced argument for school choice,” writes Hess, “is the notion that markets will force the nation’s public schools to improve, particularly in those urban areas where improvement has proved so elusive.” It is this argument-that a rising tide of choice and competition will raise all boats-that Hess explores in this thoughtful volume.

Hess offers a litany of ways in which school systems differ from private enterprises. Whereas a business can focus mainly on serving its customers, public schools serve a multiplicity of stakeholders, of which parents and students are only two. Also, Hess observes that educators tend to be driven by intrinsic motives, such as a “sense of calling,” that lessen the ability of supervisors to force them into a competitive mode. Losing students is not always a threat to public school teachers and managers. In fact, it can be downright attractive when enrollments are rising or when choice relieves them of disgruntled parents or low-performing students. When competition becomes bothersome, it can be handled with political responses, such as running advertising campaigns or creating new high-profile programs, that scarcely relate to educational performance. Moreover, school systems tend to reward the following of rules and procedures, which makes life difficult for those with an entrepreneurial spirit. Finally, education is an “unwieldy market good,” where it is difficult to define quality-or even to know it when you see it.

This discussion is informed by Hess’s case studies of three cities where the conflict of economic theory and political-organizational reality could be observed firsthand. He concedes that these case studies are not intended to be definitive, suggesting that they may be thought of as “theoretically directed journalism” rather than “conventional social-science scholarship.”

Hess devotes two quite thorough chapters to describing the decade-long voucher experiment in Milwaukee. There, choice did not compel the school system to change its routines, but it did lead to a loosening of “bureaucratic procedures and organizational routines” that allowed some educational entrepreneurs to do their thing. Hess calls this the “pickax” response to choice-poking some holes in the system but not making any major changes.

Hess’s second case study focuses on the five-year-old voucher plan in Cleveland, where he finds that the potential benefits of choice and competition were neutralized by multiple factors, including frequent changes in leadership, the state’s move to take over the city’s schools, the modest size of the vouchers (only $2,250), and the existence of strong unions. As a result, he concludes that school officials saw vouchers as “a largely symbolic threat” and felt no need to respond.

Finally, Hess examines the three-year-old Horizon scholarship program in the Edgewood Independent School District of San Antonio. The program offers privately financed scholarships to low-income Hispanic students to attend private schools. Edgewood shows, Hess writes, “how the districts may respond in ways that have little, if anything, to do with educational quality,” including a public-relations campaign, a management study, and the opening of an already planned Fine Arts Academy.

Hess uses these case studies to speculate on how choice might be introduced in ways that both respect the built-in political and organizational constraints of urban school districts and lead to school improvement. “The lesson is not that markets cannot drive more profound change in education,” he writes, “but that such effects will require changing the institutional and organizational context of urban schooling. . . . In short, making competition work as intended will require much more than the simple introduction of market mechanisms.”

Hess questions the pickax approach as a half-measure that cannot compel change or “alter the incentives that drive educational performance.” He likewise questions the workability and desireability of the alternative approach of “unchaining the bulldozer,” or moving to a ruthless competitive system oriented toward coercive accountability and other extrinsic incentives. (I find the metaphor a bit curious, since I have trouble picturing a bulldozer-not a bulldog-on a chain.)

On the matter of starting new schools, Hess suggests that, given the financial and other obstacles involved, the most likely source of choice entrepreneurs is people “operating out of religious obligation or a desire to reap a profit.” This observation fits the current scene. Channeling public funds to parochial schools was reportedly part of the political deal that established the Cleveland voucher plan (an observation that Hess curiously underplays in his narrative), and two-thirds of charter schools in Michigan are now run by education management organizations such as the National Heritage Academies.

Hess states at the outset that he is “not seeking to provide a definitive account of educational markets, but to launch a more useful conversation on the topic,” and he has achieved this goal. Economists have always understood that a variety of factors can interfere with the smooth functioning of markets, and Hess reminds us that the same principle applies to education markets as well.

Hess succeeds in posing a challenge to those who see choice and competition-the manipulation of incentives, if you will-as a way of improving schools without getting bogged down in the nitty-gritty issues of providing a quality education. “In fact,” Hess concludes, “educational competition cannot be divorced from discussions about testing, teacher certification, school district governance, educational administration, or other frustrating conversations that many school choice proponents have long wished to avoid. In the end, the fate of educational markets, for good or ill, is intertwined with broader issues of educational politics and policy.”

Edward B. Fiske, a former education editor at the New York Times, is co-author, with Helen F. Ladd, of When Schools Compete: A Cautionary Tale (Brookings, 2000).

 

---

 

As reviewed by John Gardner

As a member of the Milwaukee school board, I am one of the officials responsible for the school district’s response to the city’s voucher program. I have helped to manage the changes necessary to make the district’s schools educationally competitive, and I have witnessed the ebb and flow of competitive responses and reactive managerial and political opposition. The Milwaukee school choice program and the response of Milwaukee Public Schools are especially significant in light of Frederick M. Hess’s study of the effects of competition on large urban school districts. One of Hess’s conclusions is that among the three American cities where vouchers have been tried, only Milwaukee has a program with the size, duration, and per-pupil payments to test the hypothesis that market competition can improve schools.

Hess’s other case studies include the state-funded voucher program in Cleveland and a privately funded scholarship program in the Edgewood district of San Antonio, Texas. He argues that the choice programs in Edgewood and Cleveland never grew large enough for the local school districts to really notice any loss of students or funding. The publicly funded Milwaukee Parental Choice Program, by contrast, had grown to almost 7,000 students by the 1998-99 school year, the final year of Hess’s study. This was at least a sign that a substantial number of students and parents were unhappy with the quality of the district’s schools. Yet Hess finds that even in Milwaukee the response of the school district was focused on buffing its image much more than on actually improving the system.

In my opinion, Hess’s only limitation is his stopping point, the end of the 1999 school year. Up until then he has the story right. He argues that the leadership of the Milwaukee schools, the school board and the central office, attempted to respond with systemic and school-based reforms, but were perpetually thwarted by managerial and political constraints that he describes with chilling insight and accuracy. They-we-managed to liberate a handful of schools from managerial, contractual, and bureaucratic constraints that had stifled improvement and reform. Expanding these reforms to all schools was constrained by political and managerial constraints at all levels. The board was unwilling to take on the powerful teacher union in many instances when it most mattered. There were not enough effective, entrepreneurial principals to turn around more schools. And the central office remained unresponsive and territorial, more threatened by innovative schools than supportive of them.

What the study’s stopping point leaves out are the revolutionary changes that occurred from 1999 to 2001. Despite a partial counterrevolution, these changes have been sustained. Schools now control 95 percent of the district’s funding, select their own teachers, and develop curricula to meet district and state standards. School budgets are based almost entirely on student enrollment, providing vigorous rewards and penalties for success or failure to attract students. The district’s central office is reorganizing to serve, rather than control, Milwaukee’s schools, in large part because the central office’s budget is increasingly based on funds that schools choose to allocate to it.

Standards now include annual testing in basic subjects in grades 3 through 8. Students must now pass proficiency exams in order to enter and graduate from high school, replacing the system of social promotion. Incompetent teachers and administrators are now terminated through procedures instituted in, of all places, collective-bargaining agreements.

Before 1999, Milwaukee authorized one charter school, with only 70 students. Since then, the district has authorized 14, with more than 9,000 students attending. The number of previously oversubscribed specialty magnet schools has more than doubled. The number of popular K-8 schools has more than tripled. Before- and after-school childcare and youth recreation programs, once oddities within the system, are now in place in every elementary- and middle-school attendance area.

As a result of these and related changes, Milwaukee public school enrollment, parental satisfaction, funding, and academic achievement have all improved. Some of these changes may well have happened without the competitive challenge of school choice, but even choice’s most vigorous opponents concede that at least some of the district’s responses were produced by the threat of external competition.
Despite missing the revolution, Hess’s analysis remains sound and moves the voucher debate helpfully away from the rigidities of the state-vs.-market debate. Education markets in large urban settings, he argues, differ so much from market analysis that one must question whether they are, in fact, even markets.

Hess’s dispassionate, balanced analysis will disappoint both opponents and advocates of market-based education reforms. Hess is hardly a defender of urban public education’s failures. His previous book, Spinning Wheels, documented with depressing consistency the cycle by which urban districts adopt reforms, begin implementing them, encounter roadblocks, abandon reforms before their failure becomes apparent, and develop new reform agendas before the deadline to evaluate previous failures. In some districts the cycle has become a ritual procession of superintendent firings and curriculum changes, endlessly deflecting attention from managerial competence, school performance, and public accountability.

At the same time, Revolution at the Margins will offer little comfort to promoters of market-based education reforms who believe that markets, by themselves, are the answer. Milwaukee demonstrates that markets alone did little to change the public schools. It was the combination of a new market environment and effective responses from the public schools that simultaneously expanded choices for poor families and improved both choices and performance within the Milwaukee public schools.

What remains constant, whether urban districts respond effectively or respond at all, are the constraints and skewed semi-market within which they operate. School districts are, after all, governments, not profit-making enterprises. Like all governments, they respond to political pressures and public opinion more than to market signals and competitive incentives. School board members, central administrators, school principals, and teachers rarely have any significant stake in whether their schools lose some students. Even teachers, in some ways the most vulnerable to perceived or actual reductions in funding, are buffered from market pressures because of the national shortage of teachers, especially in major cities.

School officials and employees are, however, intensely interested in the funds that state legislators and municipal governments authorize. Rather than take on the difficult and challenging work of winning loyalty and enrollment from low-income parents, school districts face much more powerful incentives to fight legislative, legal, and public-relations wars. These offer the simultaneous advantages of familiarity, expectation, and shifting the blame for failure elsewhere.

Also diluting the competitive responses of school districts are the limitations placed on the choice programs themselves. Capping the number of students who can participate in choice programs has kept school districts from suffering any severe drops in enrollment. Similarly, in all three cities it was never clear that enough money left the school districts as a result of student defections to generate any significant economic distress among central administrators, principals, or teachers.

Finally, there is the thorny issue of what the goal of competition is. Do schools exist to teach specific skills or to serve a broad range of social functions, including socialization, assimilation, acculturation, and the provision of a panoply of community services? Has a school successfully responded to competition if it drops its arts program in order to raise test scores in math? In this context, it’s hard to tell who won the competition, or if the competition resulted in better outcomes for everyone.

Hess’s most important contribution is clarifying and redefining the debate. School choice will ultimately prevail or disappear based on how it affects entire urban populations, not just the small group of students who benefit directly from being able to attend private schools tuition-free. Like all markets, public education operates within a framework of governmental mandates, regulations, and finance, both direct and indirect. Like all governments, public education ultimately depends on private market choices and very public political decisions. Hess persuasively shows that if markets are to have significant and positive effects on public education in major American cities, two difficult political revolutions must occur. First, markets must be created intentionally, not simply by introducing external competition, but by changing the internal rules and operations of the public district. Second, districts must develop the political will and managerial competence to respond-transformations that will, in most American districts, require their own revolutions.

John Gardner is an at-large member of the Milwaukee school board. He was elected in 1995 and reelected in 1999.

In one suburban school district, teachers across the system were ranked and evaluated according to the contribution they had made to student learning–based on a value-added analysis of state test results. When they were ranked again the next year, the results were very similar except for one teacher who moved from a top rank to a very low rank. When the school superintendent looked at the results, he immediately identified the teacher–her husband had died during the second year of rankings.

What does this anecdote illustrate? For starters, that value-added assessments can misidentify good and bad teachers–and that the discrepancies can be cleared up by local administrators. More important, however, was the fact that the findings were robust; teachers’ rankings were similar across three years of analysis. Moreover, these rankings were used to grant a standard reward to a top group of teachers, not to make fine distinctions in the amount of compensation that teachers received. In other words, the results of value-added analysis, when examined over a period of two or three years, were stable and were used in a way that respected the margin of error involved in using these statistical techniques.

Critics of value-added assessment tend to embrace the concept but don’t want the results gleaned from such analysis to be used for accountability purposes–and especially don’t want to use the results to reward or sanction teachers. But teachers are the dominant school input, in terms of both spending and impact on student learning. Excluding them essentially leaves the education system without accountability.

The main concern with value-added assessment is that the technique exacerbates the amount of random error involved in measuring student performance. The risk is that teachers and schools may be wrongfully rewarded or punished because value-added techniques either over- or underestimated their students’ learning gains. However, no intelligent users of standardized testing would make policy choices based on a single year’s result or small differences among schools and teachers. Texas, for example, rewards schools based on value-added achievement gains calculated using two-stage regression analysis. The state does not hand out awards based on decimal-point differences among schools; officials reward a previously set percentage of top-ranking schools. The point here is that most of the statistical objections to value-added measurements assume a misuse of the analysis. The statistical “noise” involved in measuring value-added should preclude decisions that are based on small, unreplicated analysis; it should not preclude decisions that are based on gross findings.

Confidence in gross findings can be developed by replication, by averaging results over several time periods, and by using several measures of the development of human capital–not tests alone, but also attendance rates, dropout rates, and promotion rates (a very high-quality assessment will track indicators of human capital such as post-secondary school earnings and higher-education outcomes as well). The richer the measures used, the less weight there is on the psychometric concerns involving test scores. The alternative is to rest teacher compensation on factors that have little to do with student learning. It is now well established, for example, that the number of degrees teachers possess and the number of hours teachers spend in education courses are unrelated to student learning. Put another way, an important criterion in the determination of a teacher’s salary does not have any bearing on the ability of the teacher to develop human capital. We know that because it has been replicated in many studies in many school districts, even though salary schedules have yet to reflect this information.

Critics often cite the difficulty of comparing the results of large and small schools and comparing one subject with another. It is essential in the debate over the usefulness of the value-added assessment approach that the unit of observation and the comparison group be specified. If, as has been the case in a number of places, the comparison group is all the teachers in a given grade in the school district–with, say, the top 15 percent of the 4th grade teachers receiving an award–what is the significance of a big or small school? The class is the unit, and class size tends to be uniform within a school district. At the high-school level, the comparison group is likely to be, for example, all history teachers or all science teachers across the district. The award-receiving group is a percentage of that comparison group, and it is not affected by the test scores in another subject.

If multiple factors are used for assessment, efforts to check on robustness are made, only extreme performances are rewarded or sanctioned, and comparison groups are selected carefully, the technical psychometric points raised by critics should not swamp the incentive and information benefits of performance-based compensation plans for teachers. If we are ready to determine reading programs, language labs, class sizes, and the use of computerized learning on the basis of value-added assessments, we should be ready to reward teachers using the same techniques.

There is already considerable evidence from several places–such as Tennessee and Florida, where value-added analysis has been used for accountability purposes–that low-achieving students are the main beneficiaries of the changes that occur when these techniques are implemented. When low-achieving students are taught the same body of knowledge over and over again, and when they are taught how to work under a time constraint, they benefit. Value-added assessment techniques reveal that information.

The Illusion of Transparency

Critics of value-added assessment say that it is just too complicated for teachers and the public to understand the results these systems will generate. In other words, the results will not be transparent. But if transparency is the criterion that trumps all other criteria, we would be compelled to use an assessment method that we know to be wrong. For example, there is no satisfactory way to make judgments about which method of teaching reading is superior–whole language or phonics–without factoring in the socioeconomic, school, and teacher characteristics of each of the groups of students in the experiment. Statistical controls must be used if the assessments of teachers, schools, or programs are to be accurate, even though very few educators understand the statistical principles and methods involved. I do not require a transparent understanding of the efficacy of the flu shot I take, nor do I require a transparent understanding of the operating characteristics of my car; I trust the experts on the techniques. So must it be in educational evaluation.

The problems with the use of value-added assessments, even for teachers, are greatly exaggerated, and the alternatives are simply untenable.

One proposed alternative is the use of one or more methods of subjective evaluation. Other teachers, students, and/or parents can be surveyed to make the judgments. Most of those surveys focus on whether respondents like the teacher, are happy in the classroom, or equally soft attributes. They do not really determine what kind of learning gains a certain teacher is eliciting relative to other teachers. Most people look back at their primary and secondary schooling and identify an extremely demanding teacher (whom they disliked at the time) as the one who made the biggest contribution to their educational development.

A second possibility is to set up a standard–some threshold–of student achievement as an absolute hurdle to define adequate performance. The problems with this are legion. How should the threshold be determined? Those who want to be rewarded set lower thresholds than those who watch budgets. The threshold becomes a major bargaining tool, quite divorced from the informed decisionmaking objective. What incentives for improvement are there for those who crossed the threshold?

Basically, subjective evaluation allows the information essential to the rational allocation of educational resources to be derived politically rather than scientifically.

The state of American public education has been deplored by critics for many years. The notion that student learning has remained somewhat stagnant over the past century, though the country has spent steadily more real resources on education, is a matter of profound concern. This surely indicates that we do not know a lot about what works and what doesn’t.

We need to know if we are to change the pattern. We must have calibrated results; we must use sophisticated statistical methods to interpret the data; we need to use multiple measures of performance; and we need to implement the analysis in ways that are appropriate to the quality of the information. And teachers, the most important school-controlled input into the educational process, cannot be exempt from this information-gathering activity. Our current unhappy results are consistent with the subjective and/or unsophisticated tools we use for assessing effectiveness and creating incentives.

-Anita A. Summers is a professor emeritus of public policy and management at the Wharton School at the University of Pennsylvania.

Legal experts are already drawing analogies between Zelman v. Simmons-Harris and landmark rulings like Brown v. Board of Education and the seminal First Amendment decisions that have shaped American jurisprudence over the past half century. How the U.S. Supreme Court rules on the constitutionality of Cleveland’s school-voucher program will not only define the legal boundary between church and state more clearly; it could also help redefine the meaning of public education and expand the range of opportunities available to poor children.

Enacted by the Ohio legislature in 1995, the Cleveland Scholarship and Tutoring Program allows 4,000 low-income children to attend private religious and secular schools with up to $2,250 in public support. Participating schools must cap their tuition at $2,500 a year; the state pays up to 90 percent of whatever the school charges, depending on family income. Following a high-profile legal battle, the program was upheld by the Ohio Supreme Court in 1999, prompting opponents to take their case into federal court. On the day before school was to open that year, federal district court judge Solomon Oliver struck down the program, ruling that the use of tax dollars to pay for children to attend religious schools offends the First Amendment’s Establishment Clause. Judge Oliver halted the acceptance of new students to the program while the case was being appealed. Soon thereafter, a sharply divided (5-4) U.S. Supreme Court took an unusual move to vacate Judge Oliver’s injunction and allow the program to continue unaltered while the case is in litigation.

In December 2000, the U.S. Court of Appeals for the 6th Circuit affirmed Judge Oliver’s ruling by a 2-1 vote. The appeals court relied heavily on legal precedents set down by the Supreme Court in 1973 in Committee for Public Education v. Nyquist. At issue in Nyquist was a New York State program that gave low-income parents a partial tuition reimbursement for private-school tuition. The Nyquist Court found that the tuition-grant program had the “impermissible effect of advancing religion.” It concluded that direct or indirect aid to sectarian schools is essentially a government-subsidized incentive to practice religion.

The thinking in Nyquist was remarkable on several counts. Inherent in the incentive concept is the assumption that parochial schools are so superior to public schools that the opportunity to attend the former is irresistible, even to those parents who do not want their children educated in a religious environment. Reasonable people can conclude that the lure of a safe and sound education is an argument for choice rather than against it. Preoccupied with the religious character of parochial schools, the majority also presumed that the court is capable of looking into the minds of legislators to determine their motivations. Using effect to derive intent, the court concluded that incidental aid to religious institutions in the form of tuition relief to parents is tantamount to a purposeful government act to promote religion.

Challengers pointed out that most of the schools involved in the voucher program (46 of 56, accounting for 96 percent of the students) were religious.

There were specific facts pertinent to the Zelman case that the appellate panel deemed relevant to the incentive argument. Challengers pointed out that most of the schools involved in the voucher program (46 of 56, accounting for 96 percent of the students) were religious, leaving few secular options available for participating families. The appeals court accepted this argument even though the Supreme Court had acknowledged similar circumstances in 1983 (Mueller v. Allen) when it upheld a Minnesota program that gave a tax deduction to parents for tuition and other education expenses. While recognizing that most of the deductions were used for parochial school tuition, the Mueller Court found that because parents could deduct expenses for public, private, or religious schools, the deduction was neutral toward religion.

In Zelman, the Ohio attorney general further pointed out that schools participating in the Cleveland voucher program represent only a small portion of the range of choices available outside the regular public schools. In 1999 Cleveland had 23 magnet schools with 13,000 students in attendance and eight charter schools with 1,600 students in attendance, compared with the 3,800 in the voucher program. The two-person majority refused to accept the range-of-choice argument, however, because the magnet and the charter programs were not enacted under the auspices of the voucher law that was being reviewed. Legally speaking, these other choices did not exist. Under the rules of evidence defined by these judges, the same court that could peek into the minds of legislators to determine intent could not recognize hard evidence crucial to determining whether dissatisfied parents at regular public schools had choices beyond religious schools.

Both sides introduced evidence concerning the amount of the voucher. The state of Ohio, in an effort to rebut the incentive argument, explained that the amount ($2,250 maximum) was small in comparison with the per-pupil spending in regular public ($7,746), magnet ($7,746), and charter schools ($4,518). Looking at things strictly from a resource perspective, parents had a disincentive to send their children to schools participating in the voucher program. Opponents argued that since parochial schools were the only nonpublic schools with tuition rates low enough to be covered by the voucher, the program was indeed an incentive to attend these schools. To the extent that the latter argument has merit, the remedy seems obvious: amend the voucher law to make the amount higher, let’s say equal to the per capita amount spent in regular public schools. This would have to be done by the Ohio legislature.

There would be a delicious irony to such a resolution. It would certainly appear equitable, in light of the tortured history of school-finance litigation in Ohio. Carried to its logical conclusion, such a resolution might also add charter schools to the mix of institutions eligible for equal funding. Opponents of choice who raise the funding issue as a means of striking down vouchers would not welcome such a remedy, however. It is not financial equity they seek, but the defeat of the voucher law. And school-finance reformers who have spent oodles of time and money in litigation are not likely to receive such a remedy kindly either. For the most part, their sense of fairness applies only to children in public schools.

What Will the Court Do?

Of course, predicting what the Supreme Court is going to do in a particular case can be more difficult than calling the World Series in the middle of May. We begin with certain general expectations based on past performance, all the while knowing that anything is possible. At least baseball has winners and losers; legal contests are more complicated. To say that the Supreme Court will rule one way or another oversimplifies the process of judicial decisionmaking. Crafted to accommodate the philosophies and styles of the individual justices needed to assemble a majority, legal opinions are written with great nuance. Their outcome depends on what question or questions the judges agree to address, and with what level of specificity. As a rule, the Court tends toward more narrow rulings, with deference paid to precedent. But precedents are rarely consistent; and First Amendment case law is among the most inconsonant.

The First Amendment jurisprudence that has unfolded over the past two decades, however, seems to favor the program in question. In the aforementioned Mueller case, the Court approved a tuition-tax deduction program in Minnesota. More recently, the Rehnquist Court has overturned longstanding precedents in order to allow public school teachers to provide remedial services to children on the premises of religious schools (Agostini v. Felton, 1997) and parochial schools to receive direct aid in the form of computers and other instructional equipment (Mitchell v. Helms, 2000). In 1998, it refused to hear a challenge to a voucher program in Milwaukee that was approved by the Wisconsin Supreme Court.

Still, there is a distinct, though improbable, possibility that the Supreme Court will rule against the voucher program on fundamental First Amendment grounds. This is improbable because since 1986 (Witters v. Washington) the Court has adopted guidelines that allow indirect aid to parochial schools as long as the aid is appropriated neutrally and results from independent decisions by parents who select those schools. These more permissive guidelines were drafted in a concurring opinion by Justice Lewis F. Powell Jr., the author of Nyquist. The original plaintiffs in Zelman argued that parental independence is compromised by an administrative process that sends the voucher check directly to a religious school to be signed by the parent. Again the solution to this problem, if it is really a problem, is rather easy: just send the check to the parent, and let the parent pay the school. The existing procedure was implemented for the sake of administrative expediency. Either way, the money reaches the school because a parent chose that school for her child. But in some courts procedure trumps principle. Legal reasoning would require that the voucher program, operating as it does, be struck down.

Let’s imagine, for a moment, what would happen if it were. In Ohio the issue would be turned over to the state legislature, and once again political irony would be the order of the day. The same people who raised procedure as a point of contention in court would do everything possible to preserve the procedure in the law in order to maintain that the program is void. If choice supporters succeeded in changing the procedure, the issue of constitutionality would be resolved. Realistically speaking, however, the fate of the Cleveland program is unlikely to turn on the payment question. There is a bigger question.

While unlikely, it is conceivable that a majority of the justices hearing Zelman could agree with the argument that providing unrestricted aid to children attending sectarian schools allows the state to endorse, subsidize, and advance religion. The impact of this ruling would be substantial. Not only would it terminate the voucher program for 4,000 children in Cleveland; it would open to challenge the Milwaukee program through which 10,000 low-income students receive up to $5,553 in tuition relief for private and religious schools. Also likely to fall would be the Florida A+ program, which provides up to $3,472 for children who attend chronically failing public schools. It enrolls only a few dozen students in Pensacola, but has the potential to expand statewide. One might say that for all practical purposes, vouchers would be dead. But the same is not necessarily true of school choice.

If the Court strikes down the Ohio law, and by implication those in Wisconsin and Florida, choice supporters will probably pin their hopes on funding schemes involving tax relief, like the tax-deduction plan in Minnesota and less ambitious programs in Illinois and Iowa. By providing a more direct benefit to families, tax-relief programs add a level of separation between the state and the school and are generally deemed to be less vulnerable to legal challenge. Activists will also focus their energies on tax-relief programs for third parties that provide scholarships for poor children to attend religious and private schools. Such programs already exist in Arizona, Florida, and Pennsylvania. In fact, more children (60,000) participate in privately funded voucher programs than in publicly funded programs. Private initiatives such as the Children’s Scholarship Fund and Children First America are likely to grow no matter what the outcome of the Ohio case.

Federally funded Pell grants may be used to pay tuition at religious colleges. Are they a model for school vouchers?

It’s Constitutional, but . . .

Suppose the Supreme Court upholds the Cleveland program. Again the impact of the decision depends on the specific questions the majority chooses to address. Will it deem the distinction between direct and indirect aid significant? Will it finally dispose of the legal quibbling over form versus substance regarding the payment procedure? Will it adopt the broader standard of neutrality suggested by Justice Clarence Thomas in Mitchell v. Helms? Under this criterion, aid is permissible when it “is offered to a broad range of groups or persons without regard to religion” and “results from the genuinely independent and private choices of individual parents.” Or will it decide the case on more narrow grounds? It is quite possible for the case to result in a split majority, as in Helms, where four justices accepted the neutrality standard, while Justices Sandra Day O’Connor and Stephen Breyer permitted the aid on the basis of more narrowly defined criteria. Either way there is bound to be another round of litigation.

One possible site for a new legal battle is Maine. Maine has a 130-year-old voucher law that once allowed children living in towns without high schools to attend private or parochial schools with state support. In 1981 the law was changed to exclude religious schools. The shift in policy was upheld by the Maine Supreme Court on First Amendment grounds in 1999 and subsequently confirmed by a federal appellate court, with the Supreme Court refusing to hear an appeal. If the U.S. Supreme Court approves the Cleveland program, choice supporters in Maine could have their case reheard to reinstate the eligibility of religious schools.

It is no accident that most of the recent legal challenges to existing voucher laws began in the state courts. Opponents have based their litigation strategies on “Blaine Amendment” provisions in state constitutions that set a more rigid standard for church-state separation than that enshrined in the First Amendment-at least as the U.S. Supreme Court has interpreted it over the past two decades. Blaine amendments, a remnant of the 19th-century battles over public aid to parochial schools, are named for James G. Blaine, a presidential aspirant and congressman from Maine, who in 1875 tried unsuccessfully to enact a federal constitutional amendment prohibiting such aid. Although Blaine failed to assemble the supermajority of votes needed to pass a federal constitutional amendment, his proposal became a model for state legislators who shared his separationist and anti-Catholic sentiments, then widespread in the nation. By the end of the 19th century, 29 states had written similar amendments into their constitutions. These provisions, as well as others that were added later, could have a major impact on the future viability of voucher programs-but not without opening the door to yet another wave of litigation. Although opponents lost their challenges in the state courts of Ohio and Wisconsin, they prevailed in Vermont, which, like Maine, has a century-old voucher law that disqualified religious schools from participation in 1995. On appeal the Supreme Court refused to review that case also.

The specific exclusion of religious schools from state voucher programs, as in Vermont, raises federal questions beyond the Establishment Clause. Choice supporters claim that such discriminatory exclusion violates the Fourteenth Amendment’s equal protection clause and the First Amendment’s free exercise clause. Since the state constitutional issue has already been addressed by the Ohio Supreme Court, there is no reason to expect the question to arise in Zelman. But if the U.S. Supreme Court determines that vouchers are allowed under the Establishment Clause, it is only a matter of time before the Court will be asked to settle these larger questions. The same Court that set guidelines for permissible aid in Witters also left the door open for states to set their own standards for church-state separation. While adopting a more accommodating approach to the First Amendment than its predecessor, the Rehnquist Court has also shown a strong sympathy for state prerogatives on matters of federalism. Inevitably, voucher proponents will insist that the constitutional rights secured by a victory in Zelman would prove hollow if the states were permitted to undermine choice on their own. What the Court does to resolve the inevitable clash between these claims and its notion of federalism remains to be seen.

Back to Politics

Even if the Supreme Court were to resolve the federal and state legal questions in favor of vouchers, it would only be setting the stage for the next arena of conflict. Courts only review laws; they do not make them. The most generous judicial interpretation of the voucher question could at most require that states not exclude religious schools from choice programs that are open to other private schools. States would be allowed to continue restricting public funding to government-run public schools, as most do now. Battles over school vouchers have already taken place in more than half the state legislatures, and they will go on.

No doubt a ruling in their favor from the Supreme Court would reinvigorate voucher proponents. It might motivate President George W. Bush-whose solicitor general gave oral argument in Zelman in support of the Cleveland program-to revisit his controversial proposal for federal vouchers. (He has already endorsed tax credits in his current budget proposal.) But the more significant political battles will be fought in the state legislatures, where most education policy is made. Once again the structure of the alliances that form will be filled with political irony. As was so in Cleveland and Milwaukee, the most consistent advocates for school vouchers in America are low-income black and Hispanic parents who live in central cities where the public schools have a history of poor performance. Some black leaders-such as the Reverend Floyd Flake, a former congressman from New York; city councilman (and mayoral candidate) Cory Booker of Newark; and Howard Fuller of the Black Alliance for Educational Options-see choice as a civil-rights issue, a mechanism to provide poor families with the same opportunities enjoyed by the middle class-indeed, as a fulfillment of the promise articulated in Brown v. Board of Education: to make education available to all “on equal terms.”

However, the majority of black and Hispanic political leaders oppose vouchers. Their position is supported by mainline organizations like the NAACP, the American Civil Liberties Union, and the National Urban League, all of which have a long history of advocacy on behalf of disadvantaged populations. While reaching out to fellow Democrats, choice proponents in the minority community have sought to build alliances with Republicans and with libertarian organizations like the Institute for Justice, which has represented poor parents in every voucher case that has come before the courts in the past dozen years, including Zelman. These are not always easy partnerships. People on the left side of the political spectrum favor targeted choice aimed at the poor, while those on the right prefer universal vouchers made available to all parents. Nonetheless, these alliances have managed to move choice along in places like Wisconsin, Ohio, and Florida.

The Democratic Party has its own tensions to resolve under an ideological tent that tries to accommodate both old-line labor unions, which instinctively oppose school choice, and a younger generation of black and Hispanic activists who demand it. Party leaders have failed to respond adequately to the question of why poor minority parents should be required to send their children to failing public schools when luminaries like Bill Clinton, Al Gore, and Ted Kennedy saw fit to send their own children to private schools.

To a large extent, the choice cat is already out of the political bag. The development of voucher programs in Wisconsin, Ohio, and Florida has fostered a serious national debate over a question that once could be discussed only on the outer margins of politics. More important, the existence of private voucher programs in nearly every state has introduced poor parents to the idea that there is an alternative to failing inner-city schools, and it is winning more converts every day. That being said, polls indicate that the nation as a whole is at best ambivalent about using tax money to send children to religious schools. And voucher proposals are consistently rejected in popular referendums, as in Michigan and California during the 2000 election, where vouchers were defeated by a 2-1 margin. Furthermore, teacher unions that vehemently oppose vouchers are a powerful force within most state legislatures, almost assuring rejection in most places. Yes, the political debate is very much alive, but it remains tilted against choice.

The most consistent advocates for school vouchers in America are low-income black and Hispanic parents who live in central cities.

When the Court Speaks

What the Supreme Court says in Zelman could have a marked effect in structuring the terms of the political debate-not just in determining who wins the legal argument, but in explaining its broader implications in a way that only the Supreme Court can. In Zelman, the Court is being asked to weigh two competing political values: strict church-state separation on the one hand and the right of poor families to choose the education their children receive on the other. If the majority settles on a strict interpretation of the Establishment Clause, it will add an air of legitimacy to an already dominant political coalition that opposes school vouchers and other forms of private-school choice. It will raise the wall of separation between church and state to a level it has not seen since the Burger Court.

The political impact of the court’s decision could be even greater if it approves the Cleveland program. At a minimum it would lift the constitutional cover from those political actors who hide behind the First Amendment as a reason to oppose choice. It would lay bare a fundamental struggle over who controls the education of children, the parents or the providers. It also might help to reverse the prevailing political dynamic. Depending on the wording of the opinion, the decision could add a moral dimension to the pleas of poor parents who want educational choices similar to those enjoyed by the middle class. It is difficult to overestimate the power of moral argument in American politics. History has shown it to be an essential ingredient for reversing dominant political patterns in response to demands by weaker parties in pursuit of social justice. The Court provided such a platform in the Brown decision in 1954. While the immediate impact of the ruling was to prohibit de jure school segregation, Brown breathed life into the political struggle that, against all political odds, brought about a revolution in public policy, affecting every branch of government at the federal, state, and local levels.

Whether the Supreme Court perceives school choice as a fulfillment of the promise articulated in Brown remains to be seen. Judicial majorities do tend toward more narrow rulings, except of course when they have something more significant to say beyond the particular legal questions set before them. This may be one of those extraordinary times. If so, such an opinion would be especially compelling coming from the pen of Justice Thomas, the lone black member of the Court, who has written for the majority in a number of important cases involving religion and education. It would remind the nation that there is more than one voice in the black community, each driven by a vision of educational equality, following different paths to get there.

But even a more narrow decision may prove to be pathbreaking. After all, Brown itself was a cautious decision, declaring unconstitutional only segregation in schools, not segregation at train stations, parks, or other public facilities. And Brown only required the states to implement school desegregation “with all deliberate speed,” something less than a clarion call for immediately rectifying the effects of racial injustice. Yet 50 years later, the spirit of Brown is vastly more important than its wording. So it may be with Zelman.

–Joseph P. Viteritti is a research professor of public policy at New York University and the author of Choosing Equality: School Choice, the Constitution, and Civil Society (Brookings).