Promoters of school accountability systems based on rigorous testing often point to the high achievement of secondary-school students in those European and East Asian countries that use curriculum-based exit exams such as France’s Baccalaureate and England’s GCSE and A-level exams. Such exams can carry extremely high stakes. In England, for instance, they effectively determine whether students are eligible to enroll at a university and to which university and field of study they are admitted. In the United States, the only worthy comparisons are New York’s famed Regents exams and a more recently developed system in North Carolina. Traditionally, students in New York who passed the Regents exams in various subjects, all tied to the state’s curricula, received a more prestigious diploma that signaled their mastery.

While a number of states, including Texas, Tennessee, Virginia, Maryland, and possibly California, are phasing in or plan to develop a system of rigorous curriculum-based external exams, most states have chosen instead to administer minimum-competency exams. Known generally as exit exams, students must pass minimum-competency exams like New Jersey’s High School Proficiency Test (HSPT) to graduate from high school. These exams usually test basic skills in English and mathematics, rarely testing students’ knowledge of science, history, or other subjects. Eighteen states required the graduating class of 2000 to pass minimum-competency exams; another 11 states are developing or phasing in such exams. Five states–Connecticut, Illinois, Michigan, Oregon, and Pennsylvania–put on transcripts and honors diplomas students’ scores on state tests taken in the 10th and 11th grades, but they do not use them as a prerequisite for graduation.

In reality, the United States may never move to a system of truly high-stakes exams along the lines of France’s. The ideal of equal opportunity, and education’s unique role in advancing that ideal, is deeply grooved into America’s national ideology. Sorting and sifting is certainly a part of the U.S. education system (witness the SAT), but the system is also endlessly forgiving; any student of any age, so long as the money or loans are available, can find a university to attend, unlike in some European and East Asian countries. So reformers who routinely invoke the academic excellence of Europe and East Asia should understand the differences between European and East Asian-style curriculum-based exams and the minimum-competency exams being used by many states. Do they both raise student achievement? By how much? What kinds of positive incentives do they create? And what are the negative repercussions, if any?

Curriculum-Based Exams

My analysis of data collected by the 1995 Third International Mathematics and Science Study (TIMSS) of students in 40 countries shows that curriculum-based exit exams do raise achievement. The study found that students from countries with medium- and high-stakes exit examination systems outperform students from other countries at a comparable level of economic development by 1.3 U.S. grade-level equivalents in science and by 1.0 U.S. grade-level equivalents in mathematics. A similar analysis of 1991 International Assessment of Educational Progress data on 13-year-olds in 15 nations found that students from countries with curriculum-based exit exams outperformed their peers in other countries by about 2.0 U.S. grade-level equivalents in math and about two-thirds of a U.S. grade-level equivalent in science and geography. Analysis of data from the International Association for the Evaluation of Educational Achievement’s study of the reading literacy of 14-year-olds in 24 countries found that students in countries with rigorous, curriculum-based exams were about 1.0 U.S. grade-level equivalents ahead of students in nations at comparable levels of development but lacking such exams. The final study of the effects of curriculum-based exams compared students living in different Canadian provinces. Students attending school in provinces with rigorous exam systems were a statistically significant one-half of a U.S. grade-level equivalent ahead of comparable students living in provinces without such exams in math and science. Other estimates show similarly positive impacts of curriculum-based exams (see Figure 1).

Why do students in nations and provinces with rigorous exams learn more? How do curriculum-based exams influence school policies and instructional practices? The data show that curriculum-based exams are associated with neither higher teacher-pupil ratios nor greater spending on K-12 education. They are, however, associated with higher standards for entry into the teaching profession, higher teacher salaries (30 to 34 percent higher for secondary-school teachers), and teachers who are more likely to specialize in one subject in middle school and to have majored in the subjects they teach. Teachers appear to be less satisfied with their jobs, possibly due to the increased pressure for accountability under an exam system. Schools, countries, and provinces with rigorous exams devote more hours to math and science instruction, and they build and equip better science labs. The number of computers and library books per student is unaffected by the existence of curriculum-based exams.

Fears that curriculum-based exams have caused the quality of instruction to deteriorate appear to be unfounded. Students in nations with rigorous exam systems were less likely to report that memorization is the best way to learn and more likely to report that they conducted experiments in science class. Apparently, teachers subject to the subtle pressure of an external exam four years into the future adopted strategies that are conventionally viewed as best practices, not strategies designed to maximize scores on multiple-choice tests. Quizzes and tests were more common; otherwise, a variety of pedagogical indicators showed no differences in regions with rigorous exams. Students were also more likely to get tutoring assistance from teachers after school. They were no less likely to like the subject and they were more likely to agree with the statement that science is useful in everyday life. Students also talked more with their parents about schoolwork and reported that their parents had more positive attitudes about the subject.

Similarities

Should we expect similar gains in achievement from the use of minimum-competency exams? Not necessarily. There are important differences between curriculum-based exit exams and tests of minimum competency that may lead to different results. There are important similarities as well. For instance, they both:

• Elicit signals of accomplishment that have real consequences for students. In many education systems, exam results are averaged with teacher assessments to generate final grades for certain courses. In some cases, passing the exam is necessary to graduate from high school. In other cases, passing the exam confers eligibility for a more prestigious diploma or the right to enroll in university. In Europe and East Asia, exam grades also influence the hiring decisions of employers and limit access to oversubscribed lines of study at universities.

• Define achievement relative to an external standard, not relative to other students in the classroom or the school. Exams, whether curriculum-based or minimum-competency, make possible comparisons across schools and among students taught by different teachers. In a theoretical analysis published in the American Economic Review, Robert Costrell of the University of Massachusetts concluded that more centralized standard setting (state or national achievement exams) results in higher standards, higher achievement, and higher social welfare than decentralized standard setting (in other words, teachers’ grades or local schools’ graduation requirements).

• Are controlled by the education authority that establishes the curriculum for and funds K-12 education. When a state department of education sponsors an external exam, it is more likely to be aligned with the state’s curriculum. It is, consequently, more likely to be used for school accountability, not just as an instrument of student accountability. It makes coordinated changes in curricula and exams feasible. Tests established and mandated by other organizations serve the interests of other masters. America’s most influential high-stakes exams–the SAT-I and the ACT–serve higher education’s need to sort students by aptitude, not the needs of high schools that are trying to reward students who have learned what the school is trying to teach.

• Cover the vast majority of secondary-school students. Exams intended for a set of elite schools or advanced courses influence standards at the top, but they have little effect on the rest of the students. A single exam taken by all is not essential. Many nations allow students to select the subjects they will be examined on; some, such as Ireland, the Netherlands, Scotland, and England, offer both high-level and intermediate-level exams for some subjects.

• Assess a major portion of what students are expected to know and be able to do. Studying to prepare for an exam should mean learning important material and developing valued skills.

Differences

Three critical characteristics distinguish curriculum-based exams from minimum-competency exams. Curriculum-based exit exams:

• Are collections of end-of-course exams. Since curriculum-based exams assess student performance in specific courses, the teachers of those courses (or course sequences) will inevitably feel responsible for how well their students do on the exams. Teachers will not only want to set higher standards, they will also find their students more attentive in class and more likely to complete demanding homework assignments. They become coaches helping their team to do battle with the state exam. Grades on the external exam are typically part of the overall course grade, further integrating the external exam into the classroom culture.

• Report multiple levels of achievement in the subject. If students can only pass or fail an exam, as is the case with almost all minimum-competency exams, the standard will, for political reasons, have to be set low enough so that almost everyone will pass. This will not stimulate most students to put in more effort. End-of-course exams measure a student’s achievement level in the subject, not just whether the student exceeds or falls below a specific cutoff point. Consequently, all students, not just those at the bottom, have an incentive to study hard.

• Assess more difficult material. Since curriculum-based end-of-course exams are supposed to measure the full range of achievement in the subject, they contain more difficult questions and problems. This induces teachers to spend more time on cognitively demanding skills and topics. Minimum-competency exams, by contrast, are designed to identify students who have failed to pass a rather low minimum standard. As a result, they tend not to ask questions or pose problems that students near the borderline are unlikely to be able to answer or solve. The likely result is that too much class time will be devoted to practicing low-level skills.

Culture of the Classroom

Curriculum-based exit exams often have profound effects on the relationships between teachers and students and among the students themselves. Consider what happened when a proposal was put forward in Ireland to drop the nation’s system of external assessments in favor of having teachers assess their students. The union representing Ireland’s secondary-school teachers reacted with a statement saying that a major strength of the Irish education system has been students’ perception of their teachers as “an advocate in terms of nationally certified examinations rather than as a judge.” Asking teachers to assess their students, the union wrote, would “automatically result in a distancing between the teacher, the pupil, and the parent. It also opens the door to possible distortion of the results in response to either parental pressure or to pressure emanating from competition among local schools for pupils.”

Note how Irish teachers feared that doing away with external assessments would result in their being under pressure to lower standards. For American teachers such pressure is a daily reality. According to an American Federation of Teachers survey, 30 percent of American teachers say they “feel pressure to give higher grades than students’ work deserves.” Thirty percent also feel pressure to reduce the difficulty and amount of work they assign. Curriculum-based end-of-course exams are likely to alleviate such pressures.

End-of-course examinations may also ameliorate another scourge of contemporary classroom culture: nerd harassment. In Beyond the Classroom, Laurence Steinberg, Bradford Brown, and Sanford Dornbusch’s recent study of nine high schools in California and Wisconsin, the authors concluded that “less than 5 percent of all students are members of a high-achieving crowd that defines itself mainly on the basis of academic excellence…. Of all the crowds the ‘brains’ were the least happy with who they are–nearly half wished they were in a different crowd.”

Why are the studious called suck ups, dorks, and nerds or accused of “acting white”? In part, it is because many teachers grade on a curve. This means that doing well in a class makes it more difficult for others to get top grades. When exams are graded on a curve or college admissions are based on class rank, students can maximize their joint welfare if no one puts in extra effort. In the game that results, rewards, such as friendship and respect, and punishments, such as ridicule, harassment, and ostracism, enforce the cooperative solution: “don’t study much.” If, by contrast, students are gauged by an outside standard, they no longer have a personal interest in getting teachers off track or persuading one another to refrain from studying. Peers should become less supportive of students who joke around in class and more supportive of those who cooperate with the teacher.

Evidence from the United States

In the United States, states with minimum-competency exams tend also to have adopted school accountability systems that reward high-achieving schools or sanction failing schools. Therefore, in trying to isolate the effects of minimum-competency exams and end-of-course exams on achievement in the United States, it is necessary to account for the presence or absence of other standards-based reforms. In a study of states’ 8th-grade reading, math, and science scores on the 1996 and 1998 National Assessments of Educational Progress, my colleagues and I studied the effect of five different standards-based reform strategies:

• School-by-school reporting of the results of statewide testing

• Rewards for schools that improve on statewide tests or exceed targets set for them

• Sanctions for failing schools, such as closure, reconstitution, or loss of accreditation

• Minimum-competency exams

• Voluntary end-of-course exams combined with minimum-competency exams, à la New York and North Carolina’s policy mix during the 1990s.

We also controlled for the following demographic characteristics of the students in each state: the share of children living in poverty, parental education, and the percentages of public school students who are African-American, Hispanic, or Asian-American.

The hybrid end-of-course/minimum-competency exam systems that have been in place in New York State since the early 1980s and in North Carolina since about 1990 clearly had the largest effects on test scores. In science and math, 8th graders in New York and North Carolina were approximately 45 percent of a grade-level equivalent ahead of comparable students in states without such exams. They were also 65 percent of a grade-level equivalent ahead in reading (see Figure 1). This confirms my earlier findings that New York State did significantly better on SAT tests and on the 1992 8th-grade NAEP math tests than other states with demographically similar populations.

High stakes for teachers and schools had significant effects on all three measures of 8th-grade achievement. Students living in states that, during the 1996-97 school year, both rewarded successful schools and threatened to sanction failing schools scored about 28 percent of a grade-level equivalent higher in all three subjects than students in states that did neither. Public reporting is necessary for the execution of these other policies, but on its own it had no discernable effect on student achievement.

The effects of minimum-competency exams on average 8th grade NAEP test scores were positive but small and mainly insignificant. For students who were approaching graduation, however, the effects grew. Analysis of longitudinal data found that students with C- grade-point averages in 8th grade learned about 16 percent of a grade-level equivalent more when they lived in states requiring minimum-competency exams before graduation. Students with higher GPAs were unaffected by minimum-competency exams.

After High School

Studies of 1990 census data at the state level show that increasing the number of courses required to graduate raised dropout rates and reduced graduation rates. Minimum-competency exams had no such effect. When, however, my colleagues and I analyzed longitudinal data that adjusted for the grades and test scores of students in 8th grade, we found that students at schools with minimum-competency exams with C- grades in 8th grade, while not more likely to drop out, were about 7 percentage points less likely to get a high-school diploma or a General Education Diploma (GED) within six years. Minimum-competency exams had no significant effect on the graduation rates of students with A or B/B- averages.

The study of longitudinal data also found that college attendance rates were reduced by higher course graduation requirements, but increased by minimum-competency exams. Eighth graders living in states with a minimum-competency exam were 2 to 4 percentage points more likely to be attending college six years later than were comparable students from states without such exams. Curriculum-based exit exams substantially increased the college-attendance rates of students with low GPAs in 8th grade, but had no effect on students with high GPAs.

Students who grew up in states with minimum-competency exams earned significantly more in the years immediately after graduating than students growing up in other states. Growing up in a state with minimum-competency exams raises by about 11 percent the earnings of those who had low GPAs in 8th grade. Students with high grades in 8th grade earn about 7.5 percent extra when they grow up in a state with minimum-competency exams.

Policy Implications

Our analysis showed that states that reward schools for success and sanction schools that are failing had significantly higher achievement levels than states without these incentives. We also found that they had lower dropout rates. State requirements that students pass minimum-competency exams in order to graduate had both positive and negative effects on students. While students with average or above-average grades were unaffected, students with low grades in 8th grade were less likely to graduate during the next six years. The effects of minimum-competency exams on achievement in 8th grade and test score gains during high school were small and often not statistically significant. But students at both ends of the spectrum–that is, with either high or low grades–were significantly (about 2 to 4 percentage points) more likely to attend college in 1993-94 when they lived in a state with minimum-competency exams. In addition, employers responded to the enhanced reputation of recent high-school graduates by paying them about 9 percent more immediately after high school.

Curriculum-based external exit exam systems had by far the greatest effects on test scores. On the negative side, New York students of the early 1990s were more likely to get GEDs and tended to take longer to get their diplomas. They were not, however, less likely to graduate, and students with low grade-point averages were significantly more likely to go to college. Achievement levels at the end of high school were roughly one grade-level equivalent ahead of comparable states. These are the effects of a voluntary Regents examination system with moderate stakes, not the compulsory high-stakes exam system that New York is now phasing in. States that are reluctant to implement a high-stakes high school graduation test might want to look at the old Regents end-of-course exam system as a possible model for a moderate-stakes student accountability system.

John H. Bishop is a professor of human resource studies at Cornell University.

For competition to fulfill its promise as a reform strategy, traditional public schools must feel challenged. The officials who run the schools and the teachers who helm the classrooms must feel as if their jobs and per-quisites are in jeopardy if they fail to stem enrollment losses to independent charter and private schools. So far, few states and cities have nurtured such a vigorous, elastic competitive environment. States often cap the number of charter schools allowed to open each year. Those that do open are rarely funded at a level that would allow them to provide the sorts of facilities, like up-to-date science labs and sports fields, that public schools regularly supply. Voucher programs that enable students to attend private schools in Milwaukee and Cleveland are severely underfunded and under constant attack in the courts and state legislatures. Yet, in some school districts, signs of a public school renewal are beginning to appear. To attract students, public schools are promoting their wares, altering their curriculum, and producing higher test scores. Even a watered-down version of competition seems to encourage some public schools to improve.

• Frederick M. Hess wonders whether policymakers have the will to incite real competition
• David Arsen, David N. Plank, and Gary Sykes find a market emerging in Michigan
• Robert Maranto visits Arizona, the epicenter of the charter school movement

Federal funding and direction routinely lead to research and development breakthroughs in science, medicine, and defense, among other fields. Witness the recent debate over federal funding of stem-cell research, during which supporters of stem-cell research emphasized that federal funding tends to draw the best scholars to a field. So why do research programs funded by the U.S. Department of Education continue to disseminate research that is barely respected by serious scholars? Many academics and policymakers consider the education research sponsored by the National Science Foundation or the National Institutes of Health to be more rigorous and scientifically sound than the research emanating from the Department of Education’s Office of Educational Research and Improvement (OERI). Likewise, the department’s Planning and Evaluation Service (PES), tasked with evaluating the performance of federal education programs like Title I, has a mixed record of developing sound and timely information on program quality.

As policymakers look to redesign or replace many of the basic federal education programs established during the 1960s, they face mounds of Department-produced research to guide them, but much of it is of limited value. This is a problem that has cut across the decades, during both Republican and Democratic administrations. With OERI up for reauthorization by Congress, this seems like the perfect time to revisit the federal role in research and development.

The Centers and Labs

Federal education research and development has been fragmented and disorganized for most of its history. In an effort to address these problems, in the mid-1960s the government created two complementary sets of institutions intended to promote more coordinated, long-term, large-scale education research and development: the Research and Development Centers and the Regional Educational Laboratories. However, most of their work turned out to be (and continues to be) short term and small scale. They tend to focus on small, individual research projects rather than supporting large-scale development and evaluations of alternative techniques and programs for helping all children to succeed in school.

The value of their research and development efforts has been routinely criticized, yet the share of research funding devoted to the centers and laboratories rose under both the National Institute of Education and its successor, OERI. The labs used their lobbying arm, the Council for Educational Development and Research, to protect their budgets even as funding for research proposed by individual scholars and other initiatives was disappearing or was dramatically curtailed during the Reagan years. As funding for OERI grew through the 1990s, a large share continued to go to the centers and labs or to activities only loosely related to research or development, such as distributing funds for introducing more technology in the classrooms (see Figure 1).

 

Neither the recent OERI third-year assessment of the centers nor the PES third-year review of the labs was a thorough or totally objective evaluation of the R&D contributions of these institutions. The results were also not made available to the public or to policymakers in a timely or convenient manner. Yet in the closing months of the Clinton administration, OERI made a last-minute decision to fund some of the existing centers for another five years and offered new five-year contracts to the labs-when the agency hasn’t even been reauthorized by Congress yet. It might make some sense to extend existing center or lab grants for another year while awaiting OERI reauthorization, but the decision to make long-term commitments to these institutions, based on inadequate evaluations and before Congress has even signaled its willingness to continue to fund them, seems quite misguided.

To be sure, the sheer paucity of funding places a serious limit on the quality of federally funded education research and evaluation, yet this hardly explains all the problems of education research and evaluation today. OERI has frequently suffered from unstable and intellectually weak leadership and the absence of a distinguished research staff. After Sharon Robinson completed three-and-a-half years as head of OERI, the agency went through four different heads in less than a year. During the Clinton administration, OERI personnel were cut by 25 percent, worsening the agency’s shortage of capable, innovative researchers. Furthermore, the agency has not taken advantage of its opportunities to hire qualified researchers. In the waning days of the Clinton administration, OERI appointed full-time directors for its five research institutes, but only one of these individuals has extensive research training and scholarly experience.

As a result, neither OERI nor Planning and Evaluation Service is providing a significant number of scientifically sound and educationally relevant program evaluations. The education marketplace is crowded with competing “whole school reform” models, such as Success for All, Accelerated Schools, and Core Knowledge, all promising remarkable gains in student achievement, especially among disadvantaged students. If one were to define the federal government’s role in education, wouldn’t one of its core functions be studying whether any of these programs actually live up to their claims? OERI has yet to systematically examine which models of education interventions are especially effective at improving student achievement in different settings. Nor has PES, which has the primary responsibility for conducting program evaluations, been able to accomplish this important task. Consequently, school districts lack the kind of research-based evidence necessary to choose among such programs; often the only information they have comes from the program developers themselves. Even worse, the federal government, through the Obey-Porter legislation, is giving aid to districts to implement many of these programs even though few of them are backed by any solid research.

Likewise, a variety of proposed reforms-charter schools, expanded standardized testing programs, class-size reduction, and school vouchers, to name a few-are currently being considered by legislators nationwide. Yet researchers all too often must turn to private foundations to secure enough funding to evaluate these reforms; OERI has almost no presence in many of these debates. What is the purpose of the Department of Education if not to provide scientifically reliable research information to guide legislators on the merits of various policy choices?

Protecting Research Independence

Federally funded education research needs to be objective and trustworthy, yet periodic political intrusions by Congress and the White House continue to muddy some of the key operations of OERI and of Planning and Evaluation Service. For example, congressional mandates specifying how OERI must spend its R&D funds hamper the agency’s ability to operate efficiently and effectively. Since the mid-1970s a few members of Congress from both parties have allied themselves with the largest beneficiaries of federal research contracts (the labs and centers) and ensured that these organizations receive a substantial portion of federal education funding for R&D.

The executive branch is equally guilty of periodically interfering with the research process. For instance, Pascal “Pat” Forgione was widely regarded as a conscientious and effective commissioner of the National Center for Education Statistics (NCES), the statistical branch of OERI. The national center’s Advisory Council on Education Statistics had strongly urged the secretary of education to reappoint him. Yet he was not renominated, and some speculate that the real cause of Forgione’s departure was his public protest of then-vice president Al Gore’s inappropriate politicization of the Department’s release of reading scores from the National Assessment of Educational Progress (NAEP). Rather than wait for NCES to release the NAEP results objectively, as was the normal and expected procedure, Gore decided to use the occasion to provide his own interpretation of the data-an interpretation that many observers thought was too slanted toward the administration’s own policy interests. Likewise, the Department of Education’s attempts during the late 1990s to use OERI staff and resources in developing the controversial voluntary national tests in reading and math, without congressional authorization, led some legislators to question the research agency’s independence.

At times the Department of Education has delayed much too long in releasing unfavorable results that might have challenged current initiatives and priorities.

The Department of Education itself is no stranger to politicking. When it suits its policy purposes, Planning and Evaluation Service has publicized evaluation results early-before the actual reports have even been made public. In 1993, PES published selected excerpts from the preliminary results from Prospects, the national evaluation of Chapter 1 (later renamed Title I), before the full results of the preliminary assessment had been made public. At other times, the agency has delayed much too long in releasing unfavorable results that might have challenged current initiatives and priorities. For instance, the $9 million Longitudinal Evaluation of School Change and Performance was designed to evaluate the effectiveness of the current Title I program. Though they could have, the Department of Education and its contractors did not provide those results in time for deliberations during the 106th Congress, when Title I was up for reauthorization.

The political independence of OERI needs to be safeguarded. Research and evaluation need to be carried out objectively and reported accurately-even if the results challenge some of the policies of the current administration. One solution many analysts have suggested is to create an altogether independent agency. This would be a wise first step. There are some drawbacks to having the agency responsible for education research located outside the Department of Education, but they are outweighed by the benefits of freeing the research unit from political interference and enabling it to develop more rigorous and scientifically sound research practices. Therefore, an independent agency, reporting to the President, should be created to handle education research, development, evaluation, and statistics-something similar to the National Endowment for the Humanities or the National Science Foundation. The proposed organization would house a series of quasi-independent and separate agencies, such as the National Center for Education Research, the National Center for Development and Evaluation, the National Center for Education Statistics, and the National Assessment Governing Board.

The National Center for Education Research would be led by a commissioner of research appointed to a six-year term. It is imperative that the commissioner of research have a distinguished background in research, development, or evaluation. The size of the center’s staff should be restored to at least the 1992 levels of OERI staff, and additional distinguished research and development professionals should be recruited. The R&D centers, located in the new agency, need to grow and become more focused if they are to be relevant sources of research. Some centers survive on an annual budget of only $1.5 million to $2.5 million, hardly enough to support a legitimate large-scale program of research. An individual R&D center’s budget should be at least $5.0 million annually. Moreover, the centers need to develop a coherent, focused, coordinated five-year program of research. They should not, as is now the case, have 20 to 30 different small-scale, uncoordinated projects scattered among a half-dozen different centers across the nation.

Another independent agency needs to be created to initiate and oversee serious efforts to develop effective educational strategies and evaluate federal education programs-a National Center for Development and Evaluation, overseen by a commissioner appointed for a six-year term. Large-scale, rigorous development programs have not been a part of the department’s agenda. Likewise, the Department of Education has not conducted many scientifically sound, nonpartisan, objective program evaluations during the past two decades.

The program development and evaluation effort should be overseen by an independent, objective group of experts who will not only provide technical support, but will also ensure that the design, implementation, and interpretation of the work is scientifically sound as well as useful to educators and policymakers. Moreover, they will help to ensure that the national center’s development projects and evaluations will be readily and equally available to everyone-and not just to those who control the Department of Education at the time.

The new National Center for Development and Evaluation should solicit and implement large-scale, systematic development projects-a role that has been largely missing from the Department of Education’s agenda in recent decades. This initiative might focus its energies initially on three to five long-term projects in areas such as developing reading improvement programs or helping at-risk children make successful transitions from early-childhood programs into regular classrooms. Any research outfit, including the federally funded centers and labs, could compete for these demonstration projects. The open competition would not only spur existing education research and development providers to develop better proposals, but it might also attract interest from other major social science research organizations, such as the Manpower Development Research Corporation, the RAND Corporation, or the Urban Institute.

1 The Regional Educational Laboratory at AEL Inc. 2 Northeast and Islands Regional Educational Laboratory at Brown University 3 Laboratory for Student Success 4 Mid-continent Research for Education and Learning 5 North Central Regional Educational Laboratory 6 Northwest Regional Educational Laboratory 7 Pacific Resources for Education and Learning 8 Southwest Educational Development Laboratory 9 The Regional Educational Laboratory at SERVE 10 Western Regional Education Laboratory at WestED 11 National Research and Development Centers Primary Site Location Center for Research on Education, Diversity, and Excellence-University of California, Santa Cruz 12 Center for Research on the Education of Students Placed At-Risk-Johns Hopkins University 13 Center for Research on Evaluation, Standards, and Student Testing-University of California, Los Angeles 14 Center for the Improvement of Early Reading Achievement-University of Michigan 15 Center for the Study of Teaching and Policy-University of Washington 16 National Center for Early Development and Learning-University of North Carolina at Chapel Hill 17 National Center for Improving Student Learning and Achievement in Mathematics and Science-University of Wisconsin 18 National Center for Postsecondary Improvement-Stanford University 19 National Center for the Study of Adult Learning and Literacy-Harvard University 20 National Center on Increasing the Effectiveness of State and Local Education Reform Efforts-University of Pennsylvania 21 National Research and Development Center on English Learning and Achievement-University of Albany 22 National Research Center on the Gifted and Talented-University of Connecticut

The new center would also be responsible for interpreting the effectiveness of the various strategies. The program evaluations will vary according to the types of information analysts are looking for. The most rigorous and statistically reliable studies will use randomized-assignment control groups-though the much higher costs associated with randomized experiments will limit the number of these studies.

The collection, analysis, and distribution of education statistics also need to be improved and protected from undue political interference. The National Center for Education Statistics should be continued, protected, and expanded as an independent entity with a commissioner of education statistics appointed for a six-year term. And the National Assessment Governing Board (NAGB), which now shares responsibility for overseeing NAEP, should be continued as an independent group and given additional resources to improve the quality of its work. In the past there has been some tension and confusion between NAGB and NCES over who administers the NAEP contract, how the NAEP scores are presented in terms of achievement levels, as well as when and how the results are disseminated to the public. The National Assessment Governing Board should be given the responsibility not only for setting NAEP policies, but also for administering and monitoring the NAEP contract, and releasing the results to the public. Currently, the National Assessment Governing Board sets NAEP policies, but does not monitor the contract for the program, and it should be given more control over the administration of NAEP so that the oversight and coordination of these vital assessments can be improved.

The Department of Education should continue to provide information and technical assistance to states, school districts, and local schools by funding education experts to help educators improve their overall operations or classroom teaching. But this important technical assistance work of OERI’s labs and the comparable and overlapping work of the recently created comprehensive regional assistance centers (currently overseen by the Office of Elementary and Secondary Education) should be merged and transferred to a new office of reform assistance and dissemination under an assistant secretary for planning and dissemination in the Department of Education. Moreover, states and local districts ought to be given more flexibility in deciding how to spend federal funds designated for technical assistance. They should be able to choose from among a variety of providers rather than being forced to receive those services from either the labs or the comprehensive regional assistance centers.

Same Old Song?

There has been no shortage of thoughtful suggestions for reforming federal R&D over the past three decades. Almost all reformers seem to call for more research funding; researchers with more training; more permanent and distinguished OERI leaders; more strategic planning to meet the needs of classroom teachers and students; more long-term, scientifically sound R&D projects; and preservation of the intellectual and political independence of the agency. Sound familiar? Indeed, most of these recommendations have made their way into the language of OERI’s periodic congressional reauthorizations.

Yet little seems to have changed. Structural weaknesses in the design of OERI, inadequate funding, and excessive congressional micromanagement all contribute to the inability of OERI to transform itself. But the agency’s leadership over the past 25 years must share some of the responsibility for its shortcomings; OERI directors have not always tried to recruit distinguished researchers or insisted on high-quality work from all of the agency’s grantees and contractors. Nor have all members of the education research community been sufficiently committed to making OERI a distinguished agency-especially when it meant subjecting their own federally sponsored work to rigorous evaluations or facing more frequent competitions for their funding.

Many in the education community are becoming impatient with the recitation of familiar promises to improve research and development in the near future. Unless policymakers transform OERI into a first-rate, high-quality R&D operation, some members of Congress might consider shifting some of the funding and responsibilities currently under OERI’s watch to other research and statistical agencies.

There are promising success stories in the remaking of federal efforts in education research. In the mid-1980s, the Department of Education commissioned the National Academy of Sciences to undertake a thorough review of the National Center for Education Statistics. The review panel was so disappointed with the NCES’s work that it recommended the center’s dissolution if corrective measures were not taken immediately. A few dedicated and talented individuals rose to the challenge. Working closely with the appropriate OERI staff as well as with several influential members of Congress, they managed within the space of only a few years to create an organization that is now acknowledged as a distinguished and effective federal statistical agency. Revamping OERI will take no less commitment and effort, but the result could be a rigorous and effective research agency that is able to really improve education for all of America’s children.

Maris A. Vinovskis is a professor of history at the University of Michigan and the author of Revitalizing Federal Education Research: Improving the Regional Educational Laboratories, the R&D Centers, and the “New” OERI (University of Michigan Press, 2001).

The most scathing critique of voucher programs and charter schools is that they may bleed traditional public schools of their best students and most active parents, leaving the children who are left behind even worse off. Moreover, as the students leave, taking their per-pupil funding with them, the public schools will find themselves stripped of the human and monetary resources necessary to answer the call of competition. “Skimming,” the term of art for this hypothetical phenomenon, may lower overall achievement, as the downward spiral of the public schools swamps any gains made by the students who take advantage of school choice.

Market enthusiasts have always argued the very opposite: that competition will improve the public schools, just as the entry of Federal Express and DHL into the package-delivery market forced the U.S. Postal Service to lower its costs and offer new services, such as Express Mail. Few analysts expected the Postal Service to be able to compete with its new rivals, yet several decades later it is a worthy opponent. Supporters of school choice believe that public school administrators and teachers would respond with equal vigor to the prospect of seeing their students and funding walk out the front door. Their professional pride and livelihood in jeopardy, they would work harder, adopt more effective curricula, hire more talented staff, and turn the district office into more of a support center than a maker and enforcer of rules. They would be spurred to innovate in ways that improve student achievement and parental satisfaction. Competition would be the proverbial rising tide that lifts all boats.

For the most part, the research is in on the question of whether students in private schools, using a publicly funded voucher or paying tuition, perform better than their peers in public schools after adjusting for all the background characteristics that affect achievement. Studies comparing students in private Catholic schools with students in public school, and students who receive a voucher to attend a private school with those who don’t, show substantial achievement gains as a result of attending a private school. How competition affects the students who remain in public schools, however, is a relatively unstudied question. In the vast majority of cities and states, charter schools and voucher programs are either too young or too limited for the public schools to have responded in any significant way.

Only in one city, Milwaukee, and in two states, Arizona and Michigan, have the new choice reforms created truly fluid education marketplaces for a sustained period. Students in Milwaukee have been using vouchers to attend private schools since the 1990-91 school year, though only in 1998-99 was the cap on the number of voucher students raised from 1 percent of the district’s enrollment to 15 percent. The Milwaukee district loses a significant amount of state aid to the voucher program, enough at least to notice if not to elicit some kind of competitive response. This study examines the trend in student achievement in Milwaukee schools where large shares of the student body are eligible for vouchers.

Both Arizona and Michigan have generous charter school laws, approving their applications more easily and funding them more fully than other states. They were both early converts to the charter movement, and some of their public schools are now suffering noticeable enrollment and funding losses as a result of competition from the charter school sector. This study examines achievement trends in districts and municipalities in both states where charters have captured significant market share. Taken together, the findings presented here, from Milwaukee, Arizona, and Michigan, offer a first glimpse at how public schools are responding to these new forms of school choice. They suggest that the fears of a downward spiral aren’t merely overblown. They’re simply wrong.

Time-Tested Choices

It is important to recognize, before discussing the effects of charters and vouchers on public schools, that these new forms of choice simply add to the varieties of de facto choice already available in many parts of the country. It is not at all unusual for parents in a metropolitan urban area to be able to choose from among many schools and many school districts and from a variety of low-cost private school options. These kinds of choice have been around for so long and are so universally taken for granted that we tend not to think of them as choice. But take a family living in the Boston area. They can choose among 70 independent districts located within a 30-minute drive of downtown and many more in the metropolitan area. Towns looking to attract families and raise property values face clear incentives to safeguard the quality of their schools.

Other metropolitan areas offer far less competition among local districts. A family living in Las Vegas, Miami (where one school district, Dade County, covers the entire metropolitan area), or Hawaii (where the entire state is one school district) will be served by the same school district no matter where they choose to reside; the district has a virtual monopoly over public schooling in the area. Comparing public school performance in highly competitive metropolitan areas with performance in less competitive areas is one way of discovering how competition affects public schools.

My research shows that metropolitan areas with maximum interdistrict choice elicit consistently higher test scores than do areas with zero interdistrict choice. The 8th grade reading scores of students in highly competitive areas are 3.8 national percentile points higher than those of students in areas with no competition; their 10th grade math scores are 3.1 national percentile points higher; and their 12th grade reading scores are 5.8 national percentile points higher. Moreover, highly competitive districts spend 7.6 percent less than do districts with no competition. In other words, interdistrict competition appears to raise performance while lowering costs–the result predicted by market enthusiasts.

School districts face competition not only from other school districts but also from private schools. Metropolitan areas vary in this regard as well. Areas with long-standing religious populations, such as Catholics in the Northeast and Lutherans in the Midwest, tend to have developed a large market for private schooling, with many options whose tuition is substantially subsidized by donations of land, buildings, and money given primarily during the first half of the 20th century. Private schools in areas whose religious populations are relatively young or supportive of public schools tend to be less competitive; they simply don’t have the endowments or contributions to keep their tuition low enough to make them accessible to most parents. With strategies similar to those used to analyze the effects of interdistrict choice on public schools, I compared public school performance in areas where public schools face strong competition from private schools with public school performance in areas where little competition between public and private schools exists.

My comparison showed that all schools perform better in areas where there is vigorous competition among public and private schools. Areas with many low-cost private school choices score 2.7 national percentile points higher in 8th grade reading; 2.5 national percentile points higher in 8th grade math; 3.4 national percentile points higher in 12th grade reading; and 3.7 national percentile points higher in 12th grade math.

In short, both traditional forms of choice–choice among school districts and between public and private schools–influence public schools in a positive manner. To place the influence of competition on school performance in perspective, if every school in the nation were to face a high level of competition both from other districts and from private schools, the productivity of America’s schools, in terms of students’ level of learning at a given level of spending, would be 28 percent higher than it is now. And that is with a relatively diluted form of competition; traditional forms of choice do not provide strong competition because money does not follow students in a direct way. Furthermore, traditional forms of choice are not available to many families, either because they live in an uncompetitive area or because they are too poor to move to another district or pay private school tuition.

New Competitors

One advantage of studying traditional forms of school choice is the insight they give into how competition unfolds over the long run–over the many decades traditional forms of choice have been in place. Only the short-term effects of competition can be seen through the study of vouchers, charter schools, and their impact on the public schools against which they compete. Therefore, for this study it was crucial to isolate those instances where competition was lively and long-standing enough to potentially provoke a competitive response from the public schools. As noted earlier, only charter schools in Arizona and Michigan and the voucher program in Milwaukee met this basic criterion.

School vouchers in Milwaukee. In Milwaukee, students from families with incomes at or below 175 percent of the poverty line are eligible for vouchers to attend a private school. For every student using a voucher to leave the Milwaukee public schools, the school loses state aid equal to half the value of the voucher. During the 1999–2000 school year, the year studied here, the voucher was the lesser of $5,106 or the cost of tuition at the private school the student chose. So the district lost $2,553 of its $8,752 in per-pupil spending, or 29 percent, for every student who used a voucher. More than 90 percent of the vouchers went to students in grades 1-7 during 1999-2000 because the vouchers were sufficient to cover tuition only at private elementary schools; high schools tend to charge more. Thus the only public schools in Milwaukee that faced serious competition from the voucher program were elementary schools, so I focused on students’ scores in 4th grade, the only elementary grade in which all Wisconsin students take a statewide exam.

It was not until the cap on the voucher program was raised from 1 percent of the district’s enrollment to 15 percent in the 1998-99 school year that vouchers generated real competition for the public schools. Therefore, it made sense to compare public school performance during the 1996-97 (before significant competition) and 1999-2000 (after significant competition) school years. I divided Milwaukee schools into those that were more “treated” to competition because at least two-thirds of their students were eligible for vouchers, and those that were less treated because less than two-thirds were eligible. I expected that the more-treated schools would respond more strongly than the less treated, but the latter is not a true control group. A full 25 percent of the students in the least-treated schools were still eligible for vouchers. The response of these schools to vouchers might be attenuated, but it would still be a response. To find schools to serve as a true control group, I used the following criteria: 1) the schools were not in Milwaukee (and so entirely unaffected by vouchers); 2) the schools were urban; 3) at least 25 percent of the students were eligible for free or reduced-price lunch; and 4) African-Americans composed at least 15 percent of the student body.

Only 12 schools in Wisconsin met all these criteria. Overall, these schools are still richer and have fewer minority students than the Milwaukee schools. The same is true of the less-treated schools in Milwaukee: they are richer and have fewer minorities than the most-treated schools. Research has shown that these richer schools outside Milwaukee and the less-treated schools in Milwaukee are ordinarily likely to improve more rapidly than the most-treated Milwaukee schools. Put another way, the rates of improvement in the schools facing the most competition from vouchers will probably look less impressive than they actually are. It is simply more difficult for performance to improve in these low-income schools than in the less-treated control group schools.

Given this, the results for Milwaukee’s most-treated schools are remarkable. As shown in Figure 1, 4th-grade math scores rose by about 7 percentile points per year in the most-treated schools, 5 percentile points per year in the less-treated schools, and just 4 percentile points in the control schools. Social-studies scores in the most-treated schools rose by 4.2 percentile points per year, while in control schools the scores rose by only 1.5 percentile points per year. The scores of the students in the most-treated schools, the schools facing the most potential competition from vouchers, improved by more in every subject area tested than did the scores of the students facing less or no competition from vouchers. In fact, though reading scores improved only slightly in the most-treated schools (by 0.6 percentile points per year), reading scores actually dropped in the less-treated schools (by -0.4 percentile points per year) and control schools (by -1.4 percentile points per year), perhaps due to Wisconsin’s adoption of a controversial new “whole language” reading curriculum.

Recall that the most-treated schools were those with the most to overcome in terms of raising their scores, yet they bested the less-treated and untreated schools on every measure of improvement. Their mostly poor and minority students experienced an upward spiral in achievement as a result of competition, and the improvement is even more impressive than the mere comparison of the numbers suggests. Of course, it cannot yet be known how long these schools will be able to maintain their rates of improvement. It is possible that improvement will slow after a few more years of competition.

Charter schools in Michigan. Michigan’s charter school program was established in 1994. It was relatively easy to isolate the schools targeted by competition in Wisconsin: no school outside Milwaukee faced competition from vouchers, and the degree of competition faced by Milwaukee schools varied with students’ poverty. In Michigan, I had to choose a threshold level of charter school enrollment, above which I would classify the neighboring public schools as being subject to charter competition. Because enrollment in Michigan schools normally fluctuates by about 5 percent a year, even when there is no competition from alternative schools, I chose a threshold of 6 percent. That is, I defined a treated public school as any school in a district where charter school enrollment was at least 6 percent of regular public school enrollment.

There are advantages to studying Michigan. For one, a Michigan district that loses a student to a charter school loses a substantial amount of money–the state’s minimum level of per-pupil spending, given the characteristics of the school’s student population. During the 1999-2000 school year, the average spending on a charter school student was $6,600, compared with $7,440 for the average public school student. Also, charter schools in Michigan receive their charters from statewide organizations, such as universities. Unlike charter schools in many other states, they do not have to get their charter approved by the very district that would be their competitor. Big-city, small-city, and small-town schools are all well represented among the schools that face charter competition in Michigan.

As in Milwaukee, achievement improved in Michigan public schools faced with significant competition. As shown in Figure 2, their scores climbed by 2.4 scale points more per year in 4th grade reading and 2.5 scale points more per year in 4th grade math (4th grade is, again, the only elementary grade in which Michigan administers a statewide test). These improvements are above and beyond their achievement trends before they were subject to charter competition. Moreover, they are above and beyond the improvements made during the same period in public schools that did not face charter competition. Just to give a sense of the magnitude of these improvements, one can compare Detroit (a district that did face competition) with one of its most affluent suburbs, Grosse Pointe (a district that did not face competition). If Detroit were to maintain its faster rate of improvement, it would close the achievement gap between its students and Grosse Pointe’s students in just under two decades.

Charter schools in Arizona. Arizona’s charter school law, passed in 1994, is widely regarded as the friendliest to charter schools. It gives charter schools considerable financial and legal autonomy and imposes few constraints on their growth. Consequently, about 5 percent of Arizona’s public school enrollment attended charter schools during the 1999-2000 school year, the highest share of any state in the country. Charter schools in Arizona can be state sponsored, in which case they get a fee equal to the state’s share of revenue (45 percent of the total revenue for a regular public school). They can also be district-sponsored, in which case they get a fee equal to local per-pupil revenue but are less able to compete since they must seek renewal of their charters from the very districts with which they compete.

I followed the same strategy in evaluating Arizona that I used with Michigan, with one exception. A single municipality in Arizona may contain several school districts, so rather than associate regular public schools and charter schools with a district, I associate them with a municipality. As in Michigan, I used charter school enrollment of 6 percent of regular public school enrollment as the threshold for classifying a municipality as facing charter competition. Also, as in Michigan and Wisconsin, I focused on 4th grade scores because it is the elementary grade that has been tested statewide for the longest time.

The results in Arizona were similar to those in Michigan and Milwaukee. As can be seen in Figure 3, regular public schools that faced charter school competition improved both their 4th grade reading scores and their 4th grade math scores by 1.4 national percentile points a year. These improvements are above and beyond their achievement trends before charter competition. They are also larger than the improvements made over the same period by public schools that did not face charter competition. Again, for perspective, let’s compare a municipality that did face charter competition, such as Phoenix, with its affluent suburbs. If Phoenix were to maintain its faster rate of improvement, it would close the achievement gap between its students and those in its affluent suburbs in less than ten years.

Conclusion

The findings presented here consistently show public schools’ responding favorably to competition. In Milwaukee, schools facing more competition from vouchers improved at rates faster than schools facing little or no competition from vouchers. Public schools in Michigan and Arizona began improving at faster rates after they lost significant shares of their enrollment to charter schools. It is risky to extrapolate from these short-term results, but the long-term results found in my studies of traditional competition among districts and between public and private schools seem to confirm that competition is in general good for the public schools.

In fact, say the critics’ worst fears were realized: A school in Milwaukee went from being the best school in the city to the worst as students began leaving on vouchers; an extreme creaming effect took place. The top 10 percent of Milwaukee schools performed 32 percentile points better than the worst 10 percent of Milwaukee schools on Wisconsin’s statewide math exam. Consider a student in this hypothetical school: not only the school’s average score but the student’s score on the statewide math exam dropped 32 points. Even in this case, so bad as to be barely plausible, if the student’s rate of growth in achievement were the same as in the Milwaukee schools that were most treated by voucher competition, after five years the student would be achieving at a higher level than he was before vouchers induced some students to choose a new school.

If one critique can be leveled against these findings, it is the very opposite of skimming: that schools subjected to the most competition might have seen their lowest-performing students leave for charter or private schools. In that case, any achievement gains among the public schools would be at least partly an illusion, simply the result of having lost their worst students to schools of choice. This seems somewhat plausible: the worst students might have highly displeased parents who are eager for new options. But the worst students might also have apathetic parents who could care less. There is no way of knowing. Regardless, those who have studied the test scores and family backgrounds of students switching from public to the new choice schools have found repeatedly that the students are about average, not much different from those left behind. Still, if opponents of school choice wish to stipulate that schools of choice actually attract the worst students, leaving the public schools to teach the so-called cream of the crop, so be it. You’ll get no argument here.

Caroline Minter Hoxby is a professor of economics at Harvard University and a visiting fellow at the Hoover Institution, Stanford University.

The Florida A-Plus program is a school accountability system with teeth. Each public school is assigned a grade based on the performance of its students on the Florida Comprehensive Assessment Test (FCAT) in reading, math, and writing. Reading and writing FCATs are administered in the 4th, 8th, and 10th grades; students take the math FCAT in the 5th, 8th, and 10th grades. The scale-score results from these tests are divided into five categories. The letter grade that each school receives is determined by the percentage of its students scoring above the thresholds established by these five categories or levels. If a school receives two F grades in a four-year period, its students are offered vouchers that they can use to attend a private school. They are also offered the opportunity to attend a better-performing public school.

The FCAT was first administered in the spring of 1998. So far, only two schools in the state, both located in Escambia County, have received two failing grades, the second coming during the 1999 round of testing in both cases. Students in both schools were offered vouchers, and nearly 50 students and their families chose to attend one of a handful of nearby private schools, most of which were religiously affiliated. No additional schools were subject to the voucher provision after the 2000 administration of the FCAT because none failed for a second time.

The theory undergirding this system is that schools in danger of failing will improve their academic performance to avoid the political embarrassment and potential loss in revenues from having their students depart with tuition vouchers. Whether the theory accords with the evidence is the issue addressed here. Perhaps the threat of vouchers being offered to students will provide the impetus for reform. But it is also plausible that schools will develop strategies for improving the grade they receive from the state without actually improving the academic performance of students. Perhaps schools will not have the resources or flexibility to adopt necessary reforms even if they have the incentives to do so. Perhaps the incentives of the accountability system interact with the incentives of school politics to produce unintended outcomes.

The evidence suggests that the theory holds true: that the A-Plus program has been successful at motivating failing schools to improve their academic performance. The gains, moreover, seem to reflect real improvement rather than a mere manipulation of the state’s testing and grading system.

The Literature

The question of whether testing and accountability systems are an effective reform tool has seldom been the subject of rigorous research. Most research attention has been devoted to evaluations of the accountability system in Texas. The Texas Assessment of Academic Skills (TAAS) has been in existence for a decade and is the most comprehensive of all state testing systems. Students in Texas are tested in 3rd through 8th grades in math and reading. In addition, students must pass an exit exam first offered in 10th grade in order to graduate. The state is also phasing in requirements that students pass exams in order to be promoted to the next grade.

The comprehensive nature of Texas’s accountability system and the fact that its governor was a candidate for the presidency attracted considerable attention to the TAAS. The most systematic research on TAAS appeared in two somewhat contradictory reports issued by the RAND Corporation (for a critique of both reports, see Eric Hanushek’s “Deconstructing RAND” in the Spring 2001 issue, available on-line at www.educationnext.org). In the first report, released in July of 2000, David Grissmer and his colleagues analyzed scores from the National Assessment of Educational Progress (NAEP), a test administered by the U.S. Department of Education, in order to identify state policies that may contribute to higher academic performance. They found that states like Texas and North Carolina, with extensive accountability systems, were among the highest-scoring and fastest-improving states after demographic factors were controlled for. The report featured a lengthy comparison of student performance in California, which has an underdeveloped accountability system and weak academic performance, and Texas to highlight the importance of TAAS in improving academic achievement, as measured by NAEP.

The second report, released in October of 2000 by Stephen Klein and his colleagues, cast doubt on the validity of TAAS scores by suggesting that the results do not correlate with the test results of other standardized tests. Because the other standardized tests are “low-stakes tests,” without any reward or punishment attached to student or school performance, the authors reason that there are few incentives to manipulate the results or cheat, making the low-stakes test results a reliable measure of student performance (although it is also possible that schools and students won’t prepare enough for a low-stakes test to demonstrate their true abilities). By contrast, schools and students might have incentives and opportunities to manipulate the results of high-stakes tests, like TAAS. The dissonance between the different tests, the authors argue, should at least raise a red flag regarding the gains observed on TAAS. Klein and his colleagues also analyzed NAEP results in Texas, and, contrary to the findings of Grissmer and his colleagues, concluded that Texas’s performance on NAEP was not exceptionally strong.

Klein and his colleagues, however, cannot rule out alternative explanations for the weak correlation between TAAS results and the results of low-stakes standardized tests. It is possible that TAAS, which is based on the mandated Texas curriculum, tests different skills than those tested by the national standardized tests. Both could produce valid results and still be weakly correlated with one another if they are testing different things. It is also possible that the pool of standardized tests that were available to the RAND researchers were not representative of Texas as a whole. The standardized test results that were compared with TAAS results were only from 2,000 non-randomly selected 5th-grade students from one part of Texas. If this limited group of students were not representative of all Texas students, it would be inaccurate to draw any conclusions about TAAS as a whole.

Another examination of NAEP scores in Texas, which I conducted, showed that NAEP improvements were exceptionally strong in Texas while the TAAS accountability system was in place. The disparate findings regarding the relationship between Texas’s scores on TAAS and NAEP can be partially explained by differences in the time periods and grade levels examined, and by the presence or absence of controls for student demographics. For now it is enough to say that there is some ambiguity regarding any conclusions that can be drawn from a comparison of NAEP and TAAS results. This ambiguity is in part a result of the fact that NAEP is administered infrequently and only in certain grade levels.

A more recent collection of studies edited by Martin Carnoy of Stanford University and issued by the Economic Policy Institute finds that the accountability systems in Texas, North Carolina, and Florida (before the adoption of A-Plus) all motivated failing schools to produce significant gains. Unfortunately, none of the studies released by the Economic Policy Institute confirm the validity of the state testing results by comparing them to the results on national exams. It is possible that the critics of testing are right, that some or even all of the gains measured only by state tests are the product of teaching to the test, cheating, or other manipulations of the testing system. In addition, the pre-A-Plus Florida analysis reported by the Economic Policy Institute is plagued by several research design flaws. For example, the study compares results from schools that took several different standardized tests without making any effort to ensure that the results are comparable. And because only pass rates were available, the scale scores analyzed were estimated based on a series of assumptions.

The research presently available on the potential of vouchers to improve achievement in public schools is also less than conclusive. Recent studies by economist Caroline Minter Hoxby, as discussed in this issue, have attempted to address this question by examining the consequences of variation in the extent of choice currently available in the United States. They suggest that areas with more choice and competition experience better academic outcomes than areas with less choice and competition. While these results support the contention that vouchers would improve the quality of education for the entire education system, it remains to be seen whether even the prospect of competition can provoke a public school response.

Studying the A-Plus accountability and choice system in Florida therefore offers the possibility of some important contributions to the existing research literature. An evaluation of A-Plus can reveal whether the prospect of competition, in the form of vouchers offered to students at chronically failing schools, represents an effective incentive for improvement. Unlike other studies of accountability systems, the ability to validate the scores used in the A-Plus system by comparing them with performance on nationally normed exams offers the possibility of dispelling concerns about whether the observed gains are real or the products of teaching to the test, cheating, or manipulation of the testing system.

Validating the FCAT Results

The first section of the analysis addresses the question of whether Florida’s test is a valid test of students’ academic abilities. Given the concerns raised by the Klein study regarding the validity of the TAAS exams in Texas, I decided to use the same analytical technique as Klein: comparing results on the FCAT with results on low-stakes standardized tests given at around the same time and in the same grade.

During the spring of 2000, Florida schools administered both the FCAT and a version of the Stanford 9, which is a widely used and respected nationally normed standardized test. Performance on the FCAT determined a school’s grade from the state and therefore determined whether students would receive vouchers. Performance on the Stanford 9 carried no similar consequences, so schools and students had little reason to manipulate, cheat, or teach to the Stanford 9. If the results of the Stanford 9 are similar to the results of the FCAT, the FCAT is likely to be a valid measure of academic achievement. If the results are not similar, it is possible that the FCAT results are not a valid measure of student performance.

The results of this analysis suggest that the FCAT results are valid measures of student achievement. Schools with the highest scores on the FCAT also had the highest scores on the Stanford 9 tests that were administered around the same time in the spring of 2000. Likewise, schools with the lowest FCAT scores tended to have the lowest Stanford 9 scores. If the correlation were 1.00, the results from the FCAT and Stanford 9 test would be identical. As it turns out, the correlation coefficient was 0.86 between the 4th grade FCAT and Stanford 9 reading test results. In 8th grade the correlation between the high-stakes FCAT and low-stakes standardized reading test was 0.95. In 5th-grade math, the correlation coefficient was 0.90; in 8th-grade math, the correlation was 0.95; and in 10th-grade math, the correlation was 0.91. In other words, the results of the two tests are quite similar. (It was not possible to verify the validity of the FCAT writing test with this technique because no Stanford 9 writing test was administered.)

In the second RAND study of TAAS in Texas, Klein and his colleagues never found a correlation of more than 0.21 between the school-level results from TAAS and the school-level results from low-stakes standardized tests. In this analysis there was never a correlation between FCAT and the Stanford 9 below 0.86.

To exclude the possibility that teaching to the test, cheating, or manipulation occurred only among schools that were previously failing, I also examined the correlations between the FCAT and Stanford 9 results among this subset of schools. This revealed that even among previously failing schools the correlations between the two test results remain very high, ranging from 0.77 to 0.99. It appears as if the pressures placed on previously failing schools did not lead them to distort their test results.

The Prospect of Vouchers

Now that the validity of the FCAT as a measure of student performance has been established, the question of whether vouchers inspired improvement among Florida’s failing schools can be studied. The greatest improvements should be seen among schools that had already received one F grade from the state, since their students would become eligible for vouchers if they received a second F. To test this hypothesis, average FCAT scale-score improvements for schools were broken out by the grade they received the year before.

In fact, the incentives appear to operate as expected. Schools that had received F grades in 1999 experienced the largest gains on the FCAT between 1999 and 2000. The year-to-year changes in school-level FCAT results did not differ systematically according to whether the school had received a grade of A, B, or C from the state. Schools that had received D grades and were close to the failing grade that could precipitate vouchers’ being offered to their students, by contrast, appear to have achieved somewhat greater improvements than those achieved by the schools with higher state grades. Schools that received F grades in 1999 experienced increases in test scores that were more than twice as large as those experienced by schools with higher state-assigned grades.

On the FCAT reading test, which uses a scale with results between 100 and 500, schools that had received an A grade from the state in 1999 improved by an average of 2 points between 1999 and 2000 (see Figure 1). Schools that had received a B grade improved by 5 points. Those earning a C in 1999 increased by 5 points. By contrast, schools with a D grade in 1999 improved by 10 points. Schools with F grades in 1999 showed an average gain of 18 points, equal to 0.8 standard deviations. In other words, the lower the grade in 1999, the greater the improvement in 2000.

A similar pattern emerged in the FCAT math results. Schools earning an A grade experienced an average 11-point gain. Schools with a B gained 9 points. Schools with C grades in 1999 showed gains of 12 points, on average, between 1999 and 2000. Schools earning D grades improved by 16 points, while schools that received F grades in 1999 made gains of 26 points, equal to 1.25 standard deviations.

The FCAT writing exam, whose scores range from 0 to 6, also shows larger gains for schools earning an F grade in 1999. Schools with an A grade in 1999 improved by 0.4 points on the writing test; B schools had an average gain of 0.4 points; and C schools gained 0.5 points. D schools improved 0.5 points, while F schools demonstrated an average gain of .9 points, equal to an astounding 2.2 standard deviations.

Alternative Explanations?

The fact that gains among schools facing the prospect of vouchers were nearly twice as large as the gains achieved by other schools might be at least partially attributable to other factors. One possible factor is regression to the mean, the statistical tendency for very low or very high scores to move closer to the group average when retested. This common dynamic could account for at least some of the extraordinary gains realized by previously failing schools. It is also plausible that the extraordinary gains of failing schools were the result of their being provided with additional resources not available to other schools. And some observers have speculated that the exceptional gains observed in Florida could be explained by a change in rules regarding the test scores of high-mobility students who move in and out of schools and districts often.

To test these alternative explanations I compared the improvements recorded by F-level schools that had above-average initial scores for their category with D-level schools that had below-average initial scores for their category. The intuition here is that high-scoring F schools and low-scoring D schools are very much alike initially, yet one group is subject to the accountability system’s punishments (the F label and the prospect of vouchers), while the other group of schools is not. In many ways this comparison approximates a randomized experiment. Because the two groups were so close to the threshold dividing D and F schools, chance may explain to a fair degree why these schools received one grade or the other. This is not to say that grading systems are inherently arbitrary; it is only recognizing the reality that luck is an important factor at the margins.

The initial similarity between the two groups of schools allows us to be confident that any difference in the gains realized by high-scoring F schools and low-scoring D schools is the result of the accountability system and not other factors. Regression to the mean cannot explain the gains of high-scoring F schools relative to low-scoring D-schools because both groups begin with similarly low scores. In fact, because the letter grade is based on the percentage of students scoring above certain thresholds and not on the average score in each school, the high-scoring F schools actually have slightly higher initial reading and math scores than do the low-scoring D schools. In addition, statistical techniques can control for the influence of differences in the background of students in each group or in the additional resources provided to each group.

Comparing the demographic characteristics of high-scoring F schools and low-scoring D schools confirms that the two groups are quite similar. They also do not differ significantly in their initial per-pupil spending, average class size, percentage of students receiving subsidized school lunches, percentage of students with limited English proficiency or disabilities, and the mobility of their student populations.

Note that the comparison between high-scoring F schools and low-scoring D schools is likely to underreport the true effect of labeling schools as failing and forcing them to face the prospect of vouchers. The comparison only measures the amount by which certain F schools outperform certain D schools, ignoring the possibility that D schools are also inspired to improve for fear of failing for the first time. Indeed, simply assigning grades to schools may inspire them to improve in order to get better grades. All schools face this incentive to some degree.

Nevertheless, high-scoring F schools did experience gains larger than their low-scoring D counterparts. After controlling for average class size, per-pupil spending in 1998-99, the percentage of students with disabilities, the percentage of students receiving a free or reduced- price school lunch, the percentage of students with limited English proficiency, and student mobility rates, high-scoring F schools achieved gains that were 2.5 points greater than their below-average D counterparts in reading (see Figure 2). The math results show that the prospect of vouchers inspired additional gains of 5.2 points. On the writing test, which has a scale of 1 to 6, the effect was 0.2 points, although, since the validity of the FCAT writing test cannot be confirmed, this finding is less definitive. Therefore, schools that received an F grade-and faced the prospect of vouchers should they receive another F-experienced gains superior to those made by schools at a similar level of performance but that did not face the threat of vouchers.

The larger gains made by schools facing the threat of vouchers cannot be explained by spending increases. While F schools did receive additional resources-about $600 per pupil in additional funding, compared with about $200 per pupil in D schools-taking this additional spending into account does not alter the extra gains achieved by schools that faced the prospect of vouchers. This is an especially important finding because the additional resources obtained by F schools may have been at least partially the result of the threat of vouchers. That is, school districts may have allocated more money to failing schools or failing schools may have been more aggressive in their grant writing precisely because they were taking action to avoid receiving a second F. The fact that including additional resources in the analysis does not diminish the magnitude of the motivational effect of vouchers suggests that the results are quite robust. Furthermore, the fact that controlling for the rate of student mobility does not have any effect on the results suggests that the exceptional gains achieved by F schools were not caused by a change in the rules concerning the treatment of high-mobility students.

Conclusion

To put the magnitude of the voucher effect into perspective, the same models can be used to calculate how much additional spending it would take to produce gains as large as those produced by labeling schools and threatening them with vouchers. According to the models comparing high-scoring F schools with low-scoring D schools, to achieve the same 5-point gain in math that the threat of vouchers accomplished, Florida schools would need to increase per-pupil spending by $3,484 at previously failing schools. This would be an increase of more than 60 percent in education spending. To realize the same gain as the A-Plus program accomplished in reading, Florida schools would need to spend $888 more per pupil, more than a 15 percent increase in per-pupil spending. To produce the same gain in writing scores, per-pupil spending would have to be increased by $2,805, more than a 50 percent increase.

For many years policymakers have focused on providing schools with enough resources to educate students. The evidence from the A-Plus accountability and choice program suggests that policymakers must also ensure that schools are provided with the appropriate incentives to use their resources effectively. Grading schools and using vouchers as a sanction for repeated failure inspires improvement at schools in a way that simply providing additional resources cannot. The evidence from Florida also suggests that the gains produced by such an accountability system are real indicators of improvement in learning, and not simply teaching to the test, cheating, or other manipulations of the testing system. Whether the same gains could have been produced using alternative sanctions is unknown. But the fact is that vouchers were used, and they were unquestionably effective.

Jay P. Greene is a senior fellow at the Manhattan Institute for Policy Research.

From the Editors

After the September 11 terrorist attacks, President Bush asked Pennsylvania governor Tom Ridge to devote his energy to finding ways to strengthen the country’s domestic security. Before the tragedy, Ridge had worked equally hard at introducing competition into Pennsylvania’s education system. On the last page of this issue, Ridge explains why.

How will competition affect traditional public schools? Will they increase their productivity? Slide into passivity and bankruptcy? Turn into charity schools for the very poor and downcast? Shed responsibility for teaching challenging children? Or muddle along as usual?

No one can be certain, theorists have said, until a full-scale competitive system is tried in a serious fashion, on a large scale, over a lengthy period. This might be done as a huge experiment or simply as a forthright policy change. Cities might provide fully funded vouchers to everyone, for example. Or states might allow an unlimited number of charter schools to open, with enough funds to cover both operations and capital costs. This would facilitate the establishment of both individualized charter schools and multiple private companies with powerful education brand names.

So we won’t know for sure until we try competition on a large scale. In the meantime, however, should we not at least listen to the first whispers of information emanating from experiments currently under way? Undoubtedly the hints they provide will be subtle and ambiguous. Much as weak signals from the outer realms of the universe are both hard to detect and even more difficult to interpret, so, too, preliminary findings about the ways in which new forms of school choice will shape the public schools are hardly definitive. Yet few scientists would ignore well-researched results.

So it is that we bring together in this issue the best of the new evidence on how choice may be affecting public schools as well as a robust, informed conversation about its longer-term potential. The carefully conducted research by Caroline Hoxby and Jay Greene tells us that choice-even the threat of choice-provokes a detectable response from the public schools. Test scores rose when public schools were placed in more competitive contexts in Milwaukee, Michigan, Arizona, and Florida.

What policy changes actually produced these test-score gains? The participants in our forum take a close look at the competitive environments in Milwaukee, Michigan, and Arizona and find tangible changes hard to come by.

The most visible response to competition has been aggressive advertising campaigns. Perhaps more important changes are occurring at the grassroots. But turning a disheveled mountain of public education into a national schoolhouse on a hill will be a job as daunting as the marvelous, half-century-long, but ongoing rendering of a statue of Crazy Horse, the great Native American warrior, out of a Black Hills mountain by the Korzczak family.

The topic is a large one, and we invite you to tell us about any signs, positive or negative, that you have detected. Send your letters to Education Next, 226 Littauer North Yard, 1875 Cambridge Street, Cambridge, MA 02138, or email to Editor_EM@latte.harvard.edu.

We hope you will also enjoy the remainder of this issue. Home schooling has received a lot of press recently; Christopher Hammons places much of the hysteria in the public media in thoughtful perspective. John Bishop makes the case for requiring high schoolers to pass curriculum-based exams in order to graduate, pointing out that students learn more and schools become more focused whenever these exit exams are in place. Maris Vinovskis calls attention to the misuse of the federal research dollar in education and the need for a basic restructuring of the federal research enterprise. And our forum on urban schools reveals that much of the governmental gyrating that is termed reform simply replaces one batch of ineffective institutions with another. If choice has yet to provide the definitive word on school reform, the governmental rearrangements conventionally attempted hardly seem more attractive.

— The Editors

The Ambiguous Embrace: Government and Faith-Based Schools and Social Agencies
by Charles L. Glenn
Princeton University Press, 2000, $37.50; 304 pages.

Schools, Vouchers, and the American Public
by Terry M. Moe
Brookings Institution, 2001, $29.95; 350 pages.

Choosing Schools: Consumer Choice and the Quality of American Schools
by Mark Schneider, Paul Teske, and Melissa Marschall
Princeton University Press, 2000, $35; 320 pages.

As reviewed by Sally Kilgore

The economist Milton Friedman was the first to propose vouchers as a different way of distributing public funds for the education of all children, rich and poor-with the assumption that parents, as consumers, would act in ways that improved education for all students, not just their own children. Because they were more interested in promoting equality of opportunity than simply consumer choice, sociologist Christopher Jencks and law professors John Coons and Stephen Sugarman proposed placing some constraints on how vouchers could be used: Disadvantaged students would receive larger vouchers, and regulations would prevent any school that accepted vouchers from imposing tuition and fees beyond the value of the voucher.

As the choice movement has moved from theory to practice in recent years, the translation has been much more limited than the theorists could have imagined. Vouchers have come to include the use of private funding as partial tuition support for low-income students to attend private schools (as in Washington, D.C., San Antonio, and New York); the use of public funds to allow a small number of low-income students to attend private schools (as in Milwaukee and Cleveland); or, as in the case of Florida, the provision of public funds for students to attend a private school or another public school if their current public school has a poor aca-demic record. In each case, eligibility has been so limited and the level of funding set so low that vouchers have yet to be fully tested, much less to have demonstrated the potential that their original conceptualizers saw in them.

The main reason there hasn’t been a proper test has been the political influence of critics and opponents. Critics of school choice are concerned about the degree to which a choice regimen will cause schools to become more racially segregated and the degree to which choice will result in creaming-the phenomenon in which only savvy, involved parents exercise their ability to choose, thereby leaving disadvantaged children concentrated in schools that few others would consider attending. Critics also doubt that the criteria that some parents might use in selecting schools (proximity to home, sports programs) would lead to the promised outcome-academically superior schools for all students.

In The Ambiguous Embrace, Charles Glenn wrestles with the question of whether school choice will be good not only for students but also for private religious schools. The paradox for religious schools and other faith-based service providers is that public funding may enable them to expand their services at the cost of compromising the very qualities that made their services effective in the first place. Public funds necessarily and appropriately bring public oversight. Some regulation is needed to ensure, for instance, the safety and well-being of children and the responsible use of public funds. But what if the regulations begin to impinge on a faith-based organization’s core principles or essential service-delivery strategies?

Consider treatments for drug addiction. Whereas faith-based organizations view addiction as a matter of morality and values, more medically oriented programs tend to treat it as a disease. According to Glenn, research comparing these divergent treatment models finds the faith-based approach more successful: Addicts are more apt to be drug- free and gainfully employed if they have participated in a faith-based treatment program. But these programs can pose challenges to government regulators. For example, once an organization accepts public funding and thereby becomes subject to local health and safety codes, it may find itself facing a limit on the number of people that can be housed in a building. This could cause trouble for a faith-based organization that has a policy of never turning away anyone seeking help. Likewise, government rules often require staff members to hold certain certifications. If faith-based organizations are required to hire certified staff without regard to their religious beliefs, the organization’s consensus on treatment strategies is lost. In this scenario, public funding can undermine whatever made the faith-based organization unique.

Glenn shows how some faith-based organizations, such as the Salvation Army, have been able to maintain the core principles of their faith while still accepting public funds. They clearly limit the types of services for which they will accept public funds, depending on their judgment of how strongly public funding will affect their ability to maintain their essential philosophy and mission. Even in these instances, however, tensions emerge between the professionals hired to deliver the publicly supported services and the volunteers (or church members) who staff other services.

Glenn believes that government funds should be made available to faith-based organizations as a demonstration of respect for the freedom required in a pluralistic society and as a way of providing effective service. The challenge is to frame the relationship in ways that protect the essential character of those services. Reviewing the practices of other industrialized countries as well as U.S. case law, Glenn concludes that third-party payments (or vouchers) to individuals seeking treatment or services is the best way of using public funds to provide services while preserving the essential character of religiously affiliated service providers. Indirectly funding these organizations through consumer choice protects them from some of the regulatory intrusions that apply when governments directly fund religious organizations. Even so, such a strategy should guarantee that there is public accountability for outcomes, for staff certification, and for eligibility for services. Faith-based service strategies-such as instructional methods-could be largely protected by allowing such organizations to discriminate in hiring on the basis of religious affiliation. Yet, as Glenn shows, the definition of whether an organization is “pervasively religious”-the condition an organization must meet under U.S. case law in order to be able to discriminate in hiring practices-is hardly consistent.

Glenn’s concern with the regulation of faith-based organizations is in some ways justified by the survey results on which Terry Moe relies in Schools, Vouchers, and the American Public. Moe finds that, by a huge majority, Americans think private schools should be regulated in certain key areas, including curriculum, teacher quality, financial audits, student performance, and the fairness of their admissions policies. These findings differ from Glenn’s recommendations most sharply in terms of curriculum and instruction. Glenn endorses public accountability for the content of instruction (academic standards, in other words), but would want, whenever possible, to allow individual schools or organizations to determine their own instructional strategies.

How religious schools are to be regulated, however, is not Terry Moe’s main line of inquiry. He is more interested in Americans’ attitudes toward school choice and how policymakers can respond to those attitudes. Using a 1995 survey of 4,700 adults (with an over-sample of urban adults), Moe finds that Americans like the public school system but think that private schools are better. They are not satisfied with certain features of the public school system, especially inequities, the absence of voluntary prayer, the level of parental involvement (they’d like more), and school size (they’d like smaller). The surveyed adults are pretty clear that they think market incentives would make schools more productive.

Moe uses these data to address the concern that allowing more children to attend private schools would lead to further social separation-by race, religion, and income. First he selects those parents whose responses signal that they would be likely to enroll their children in private schools should tuition no longer be a barrier (with a predicted probability of entry of 70 percent). Then he compares the current racial and economic composition of the public and private sectors with their hypothetical composition after these parents make their move to private schools. He finds that the social and ethnic disparities between public and private schools would actually be moderated. That said, Moe’s analysis does not, and cannot, address the larger question of how social disparity would be distributed within each sector if all students were given vouchers to attend any school-public or private.

Moe finds the American public most comfortable with incremental changes in funding strategies and the gradual or limited use of vouchers. So, for instance, respondents are quite comfortable providing public funds to enable students in low-performing schools to attend private schools. In fact, Moe concludes that incremental change in school governance and practice is the best option both because of the clear preferences of the American public and the low risk attached to staging change over time. Yet incremental change may be worse than no change at all. For incremental changes in funding strategies to qualify as a move toward equal educational opportunity, they must move beyond partial financial support to low-income students who wish to go to private schools to full financial support of such students in any public or private school. Anything less will fail to touch the truly disadvantaged children of this country.

In Choosing Schools, Mark Schneider, Paul Teske, and Melissa Marschall study the processes and effects of public school choice using a quasi-experimental design in four school districts in New York City and New Jersey. When choosing schools, the authors find, parents of all economic and ethnic groups place academic quality first. The quality of teachers, in fact, was a school’s most important attribute for respondents. Safety was second; test scores a very close third. Moe’s findings are similar: academic performance is clearly most important to parents, regardless of their education level or ethnicity. In fact, in a multivariate analysis Schneider et al. find that black parents, as well as less-educated parents, place a higher priority on the test scores in a school than do other groups of parents.

Schneider, Teske, and Marschall do find, however, that parents are not equally informed about schools. They get their information from different sources, with better-educated parents being able to rely more heavily on informal networks. But they contend that not every parent needs to be highly informed in order for a choice system to function effectively.

Choosing Schools‘ most unique contribution is to evaluate systems of school choice in terms of how they could serve various public interests-namely, the degree to which a system of choice can promote equity, student achievement, and social capital (or social connectedness). Comparing districts with and without systems of choice, Schneider, Teske, and Marschall find little evidence of increasing inequities where choice is available; the academic performance of all schools appears to increase with even limited choice in a district; and parents become more engaged when allowed to choose their schools-thus enhancing the community’s social capital.

Sally Kilgore is president of the Modern Red Schoolhouse Institute, an organization that provides technical assistance to public schools throughout the country.

 

---

 

As reviewed by Michael Mintrom

As an advocate of state funding for religious schools, Charles Glenn supports the use of school vouchers. The complicating detail-or the ambiguity of the embrace, as Glenn terms it-is the danger state funding poses to the schools’ uniquely religious perspective. The danger comes from two sources. First, when government contracts with religious groups for services, those groups become subject to new forms of regulation and accountability. Second, when religious groups build up the service delivery aspects of what they do, the logic of effective management techniques (bureaucratization and professionalization) often serves to diminish the focus on faith that is said to lie at the heart of the organization’s mission. Few observers in the voucher debate recognize the hazards presented to religious schools of mixing church and state. Glenn’s perspective here is thus refreshing. In particular, Glenn’s case study of the Salvation Army-its near loss of religious focus and its religious reinvigoration-makes for compelling reading.

Nevertheless, Glenn too often assumes that the relevant religious organizations are mainstream ones. Since Judeo-Christian principles infuse Western society, Glenn’s assumption may seem innocuous. And it is this assumption that allows him to argue that parents should trump the state in determining how children are to be educated. Thus does Glenn create an escape hatch for himself, one that helps him avoid confronting any tricky questions concerning the kind of socialization that is appropriate for children to receive within the polyglot of contemporary America. Fortunately, the United States has not experienced religious wars. But a major reason why is that the national and state governments have gone to considerable lengths to institutionalize religious tolerance. The system of public schooling has been integral to that project. Would Glenn feel so comfortable if “faith-based” service provision meant Catholic organizations had to battle routinely with religious cults for government money or over what it means to inculcate “American” values in the young?

More also needs to be said about the supposed effectiveness of faith-based service delivery organizations. Glenn’s uncritical discussion of a drug rehabilitation program in Texas, “Teen Challenge,” leaves a serious question lingering: Is Teen Challenge offering an effective rehabilitation program, or are its “successes” predicated on the screening out of people who are not yet willing to accept, among other things, that “sin is the major root problem”? Thankfully, many state and county governments across the country still stand ready to provide drug rehabilitation for sinners. Furthermore, throughout the book Glenn equates government with inefficiency, often in vague ways. Yet by assuming that government is always and everywhere inefficient, Glenn denies the critical fact that the very existence and successful evolution of markets, civil society, and organized religions have been facilitated by effective government.

Shifting the focus to the public sector, Mark Schneider and his colleagues examine the ways in which parents behave when allowed to choose among public schools in Choosing Schools. The study is rooted in analyses of parental behavior in District 4 in Manhattan and in suburban Montclair, New Jersey, with comparisons with neighboring districts that offer limited or no public school choice. Schneider et al. find that parents of lower socioeconomic status exhibit preferences for their children’s schools that are different from their more highly educated counterparts. But the relationship is actually the opposite of what one might expect: while all parents place a high value on teacher quality, low-income parents are more likely to emphasize the importance of school safety, test scores, and discipline. Higher-income parents place more emphasis on things like school values and the diversity of the student population. Given the critics’ claims that school choice will fuel further racial segregation, another surprising finding is that racial preferences apparently play a minor role in shaping how parents choose schools. The authors acknowledge that this finding might be driven by parents’ providing socially acceptable responses to the interviewers. But it might also reflect the reality that, in a diverse society, parents view other attributes of a school as much more important than its racial composition.

How do parents act on their preferences? Schneider et al. note that choosing schools requires a certain level of skill on the part of parents. Even though low-income parents might care deeply about the schools their children attend, they do not always have access to-or the ability to access-information that would help them determine the best schools in which to place their children. When it comes to making use of information about schools, the authors find that education matters a great deal. As parents’ levels of education increase, they rely more on their highly educated friends to supply them with information about schools. Less-educated parents (who are often non-white) are less able to tap into rich, informal information sources. The authors find that parents who are allowed to choose their schools do not always possess more knowledge about the schools they have chosen than do non-choosing parents.

This focus on the demand side of school choice and information use by “consumers” should be of interest to designers of school choice programs. Schneider and his colleagues believe that consumer choice in schooling can have positive effects on education quality, and they support their belief with test score data from New York City. But they also appear sympathetic to the view that school choice can be a double-edged sword, allowing the children of well-educated parents to move to appropriate schools and leaving others in poorly performing schools. In their study, the authors do not find a great deal of evidence that this is the case. Parents might not know a lot about the schools their children attend, but maybe the mechanism of school choice produces good results anyway. That said, the findings presented here suggest that government agencies must stand ready to ensure high-quality information flows to parents if school choice is to generate dynamics that raise the standards of all schools.

Terry M. Moe also focuses on the demand side in Schools, Vouchers, and the American Public. Best known for his and John Chubb’s seminal work on school choice, Politics, Markets, and America’s Schools, Moe here seeks to discover where Americans stand on the voucher issue through public opinion surveys. This is an exceptionally interesting and important contribution to scholarship on school choice, and it is bound to play a key role in shaping future dialogue about vouchers as a reform strategy. The book is built around analyses of data collected in a unique survey conducted in 1995 that yielded 4,700 respondents. As with all of Moe’s work, the writing is clear, the analysis is penetrating, and the arguments are cogently made.

Moe recognizes that public opinion on education represents a powerful resource that savvy advocates of policy reform can potentially tap to give political weight to their arguments. In the tradition of E. E. Schattschneider, who closely studied the dynamics of interest-group politics, Moe emphasizes the importance for reform advocates of widening the scope of conflict, or making winning appeals to the broader citizenry, in ways that help them to achieve their political goals. Of course, to make good use of public opinion, advocates must have a clear sense of what public opinion is. Moe observes that the “reality is that most Americans are uninformed about the voucher issue, more ambivalent than committed in their views, and likely to shift in the presence of new information. Elites are well aware of this and inevitably see it as an opportunity to swing public opinion over to their side.”

Moe finds that a large number of parents are satisfied with the public schools to which they send their children. But they also express a fair amount of dissatisfaction, especially when asked about specific aspects of the public schools, such as their ability to promote quality, equity, and racial diversity. This dissatisfaction, notes Moe, gives voucher advocates quite a lot to work with as they seek to move public opinion in their favor. However, one curiosity Moe observes is that the very people who might benefit most from improvements in their schools often appear to be the most satisfied with them. Moe argues that this may be a result of people’s holding low expectations of schools to begin with: “Parents who are poorly educated and have low expectations are more likely to be satisfied with their local schools, whatever their quality, than other parents are.”

Of course, not all parents are satisfied with their public schools, and parents in school districts with the poorest performing schools often display strong desires to escape them. In asking who might switch from public to private schools if money were not an object, Moe finds that the appeal of private schools is especially strong among parents who are minority, low income, and living in low-performing districts. This, argues Moe, presents political opportunities for voucher proponents. Yet nobody should imagine that such opportunities will translate into easy victories. Voucher advocates will need to do much to educate the public, especially those members of the public who are disadvantaged and who could potentially benefit most from voucher-based education. Moe finds that close to 80 percent of the urban poor in his survey have no clue what vouchers are. This, we are told, is “rational ignorance,” the kind you choose. Calling it plain ignorance would undercut the rational-choice foundations of the voucher idea.

Among the strengths of this book is the painstaking effort to which Moe has gone to extrapolate from his public opinion data to develop a number of “what if?” scenarios concerning people’s behavior if given vouchers. Importantly, Moe finds that “the effect of choice … is to reduce the social differences between public and private” in terms of the educational background, income, race, and religiosity of parents who would place their children in private schools. Still, a couple of issues arise. First, this is a study of attitudes, not actions. Second, reducing the social differences across the public and private sectors doesn’t necessarily mean that we would find more diversity in the classrooms of private schools under a voucher system. Careful regulation might be needed to ensure that voucher systems produce favorable social outcomes. Putting himself somewhat at odds with many voucher proponents, Moe suggests that-based on what the public says-voucher proponents would do well to “get away from free markets and accept an integral role for government regulation in the design of voucher programs.”

Schools, Vouchers, and the American Public offers a gold mine of fascinating information and ideas. For me, it represents something of a change in Terry Moe’s thinking about school choice. In the past his writings have often placed interest-group politics at center stage. Moe now acknowledges that “the opinions of the American people will have a great deal to do with how much power each side can successfully wield in democratic politics-and whether, in the end, vouchers will prove a passing fancy, a revolution, or something in-between.” Here is explicit recognition that the education reform battle is not so much a battle of interests as a battle of ideas about interests and who gets to define the public interest. Rather than viewing democratic politics as something to be side-stepped if we are to have better-performing schools (see Politics, Markets, and America’s Schools), in this book Moe calls for reformers to think more deeply and more democratically about the linkages between elite ideas and public opinion. Of course, Moe’s views on the merits of vouchers haven’t changed. But here he shares his thinking on how a voucher system might be achieved.

Michael Mintrom is an associate professor of political science at Michigan State University.

Growing up in a middle-class family in Erie, Pennsylvania, my brother, sister, and I didn’t have much in the way of luxuries. But we never lacked for love and support. Our parents did everything they could to make sure we would have more opportunities than they had. The key was a quality education. My dad worked two jobs all his life in order to send us to the best schools possible. He never complained. And I was never able to thank him enough.

The best way I know to thank all parents who care about their children’s education is to empower them to make the right choices. Recently Pennsylvania gave parents some powerful new tools. Elementary-school children who are struggling in reading and math can now qualify for $500 grants for after-school instruction. They can use these grants to purchase extra help at the education provider of their choice, public, private, or parochial. There’s no other program like it in the nation-but I predict there will soon be many others.

Another law I signed this year provides a tax credit to groups that donate public- and private-school scholarships for families. We expect this will leverage up to $40 million in school-choice scholarships by the second year.

You see, I believe that parental empowerment is incomplete without school choice. Families struggling to make ends meet should have the right to take their precious education dollars to whichever schools best fit their children’s needs. Wealthy families already have the ability to choose the best schools for their children. It’s time to level the playing field.

The traditional arguments in favor of school choice-that it will allow children to escape failing schools; that it will improve public education through competition-are well known. But consider one more: it will get parents more involved in their children’s education.

Parents have the ultimate responsibility for their children’s education. They’re the ones who check the homework, who attend the PTA meetings, who drive their sons and daughters to music lessons and basketball practice. In a nationwide survey recently conducted by the Educational Testing Service, parents and educators alike cited “lack of parental involvement” as the number one cause of school problems.

The vast majority of parents do not run away from this responsibility; they embrace it. But too often our public school system shuts parents out, sending the message, subtly or otherwise, that they are not qualified to take charge of their children’s education.

Giving parents school choice would be like hanging a sign reading “Moms and Dads Wanted” in the windows of our classrooms. Schools competing for students will communicate with parents more often. Parents will be alerted sooner, rather than later, that their children may be in danger of falling behind.

My own parents and teachers communicated often-too often, I thought at the time. Now that I’m a father, I’ve come to realize what a blessing that was. When I had trouble reading in the 1st and 2nd grades, my parents worked with my teachers to help me solve the problem. It took after-school tutoring and some long hours of help from dedicated educators, but we did it. I want all families to have the same “early warning system” I enjoyed.

School choice is not for parents who are completely satisfied with their schools and their children’s performance in them. It’s for everyone else. Like the 50,000 families in Pittsburgh and Philadelphia who applied for just 2,000 scholarships donated by the Children’s Scholarship Fund in 1999. Or the 12,000 Pennsylvania parents currently on waiting lists to get into charter schools.

Those families should have the right to choose their schools. It’s not the entire solution. But it is an essential first step.

Tom Ridge, former governor of Pennsylvania, is the Director of the U.S. Office for Homeland Security.