The National Council for Accreditation of Teacher Education (known broadly as NCATE, pronounced “en kate”) was launched in 1954 by a coalition of professional organizations from across the education community. Previously, teacher-training programs had been accredited by states, regional accrediting bodies, or an association of teacher colleges, each equipped with its own benchmarks and methods of evaluation. NCATE aimed to professionalize teaching by establishing national standards for accreditation.

Support for NCATE’s mission runs deep within the education establishment. Five of education’s most powerful organizations jointly founded NCATE: the National Education Association and four associations representing education schools, state education officials, and school boards. Today, NCATE has 33 member organizations, including the two national teachers unions; the major subject-based associations in English, math, social studies, and science; and policy and administrator groups (see Figure 1).

Despite NCATE’s having such a strong imprimatur, only 540 of the nation’s roughly 1,300 teacher-preparation programs have received NCATE accreditation. The scene is changing quickly, though. In just the past three years, 100 training programs have become accredited, and another 100 or so are currently under review for initial accreditation. Foundations are also increasingly enthusiastic about NCATE’s mission, having contributed more than $10 million since 1990. NCATE president Arthur E. Wise attributes this growth to “the new climate of accountability and the desire of institutions to be able to demonstrate that they have met rigorous national standards.” NCATE’s legitimacy as an accrediting body is recognized by the U.S. Department of Education and the private Council for Higher Education Accreditation, a nationally recognized advocate for voluntary accreditation and quality assurance.

This only begins to describe NCATE’s growing influence. It has yet to become education’s equivalent of the American Bar Association, which effectively controls entry into the legal profession, but it now accredits institutions that train more than 70 percent of the nation’s teachers. Eight states mandate NCATE accreditation for teacher-training programs, and NCATE has formal partnerships with 46 states for conducting joint reviews of schools of education. All joint reviews are based on NCATE’s accreditation standards, essentially making NCATE’s standards the benchmark for teacher preparation across the nation. The most visible sign of NCATE’s expanding reach may be U.S. News & World Report‘s decision, as of 2002, to list whether a school is accredited by NCATE in its annual ranking of the Best Graduate Schools in Education.

NCATE’s path to prominence reflects an across-the-board push to professionalize teaching along lines similar to medicine and law. Together with sister organizations like the National Board for Professional Teaching Standards and the National Commission on Teaching & America’s Future, NCATE is developing increasingly symbiotic relationships with state-level licensing boards–relationships that allow these organizations to push forward their agenda of increased training requirements, tougher licensing exams, and the higher pay to match. The question is whether the mere trappings of professionalism will actually improve the teaching profession.

The Process

NCATE claims to be improving the quality of teacher education through its stringent standards and rigorous system of evaluation. A look at the process, however, reveals a focus on the attributes of a given program more than on its success in producing good teachers (see Figure 2).

How does a department, college, or school obtain NCATE accreditation? First, two to three years before an accreditation visit, the college becomes a precandidate for NCATE accreditation. Precandidates must pay an annual fee of $1,615 and submit an annual report to NCATE. Three semesters before the accreditation visit, the college must meet nine preconditions. These preconditions include the submission of numerous documents and a “conceptual framework” that specifies the program’s mission, philosophy, performance expectations for candidates, and candidate assessment system. Depending on the nature of NCATE’s partnership with the state, the college may also be required to produce detailed reports (limited to 140 pages each) on each of its education programs. Such reports must be approved by NCATE’s specialty associations. Once all preconditions are met, NCATE establishes formal candidacy for the college, and the accreditation visit is usually conducted one to two years later. Two months before the accreditation visit, the college prepares and delivers an extensive report on how it meets NCATE’s standards.

Three to eight members of the NCATE Board of Examiners then conduct a five-day site visit. These visitors are part of NCATE’s pool of about 450 examiners, comprising teachers, professors, and representatives from NCATE’s policymaking and specialty organizations. They tend to do about two visits a year. Examiners are given a week of training when first hired and are evaluated by their peers and by the institutions at the conclusion of each visit.

The site team interviews faculty, students, staff, graduates, and employers, gathers data, and reports its findings. The college under review is permitted to write a response. All materials are then sent to NCATE’s accreditation board for review. The board meets twice a year for a week at a time, and it rules on about 120 accreditation requests each year. The board is composed of 33 members from NCATE’s constituent organizations–one-third coming from teacher organizations, another third from teacher-education organizations, the rest from policy and school specialist organizations.

Accredited departments or colleges pay an annual accreditation fee of between $1,615 and $3,095. Accredited institutions that are not affiliated with the American Association of Colleges for Teacher Education must pay an additional annual “sustaining fee” of between $745 and $1,245. Accreditation visits occur every five years and cost between $3,000 and $8,000. Other expenses involved in preparing the required materials (for instance, faculty time and clerical support) are borne by the institution.

NCATE’s 33 member organizations also pay annual dues ranging from $12,000 to $250,000. The dues structure is based on the number of members an organization names to the NCATE policy boards. In return for their dues, the organizations are given the opportunity to help develop and implement NCATE’s standards. Revenue is also generated by the recently opened “NCATE Store” on the NCATE website, which sells golf shirts, pens, mugs, mousepads, and lapel pins with the NCATE logo.

The NCATE Standards

NCATE revises its accreditation standards every five years. The most recent revision, completed in the fall of 2000, was promoted by NCATE as having aligned its standards with the broader movement toward accountability and outcome standards. NCATE president Wise has hailed this revision as “a radical reform of our accreditation process. This is a revolutionary approach to accreditation because NCATE used to measure what colleges offered their students. Now we’re focusing on outcomes.”

In the past, NCATE claims, its standards focused on inputs: the quality of a program’s curriculum and resources. Now, NCATE insists, “It is no longer acceptable for the candidates simply to have been exposed to certain topics in the curriculum. . . . Institutions must demonstrate that candidates know their subject and how to teach it effectively.”

Let’s examine these claims through a review of the six standards:

• Candidate Knowledge, Skills, and Dispositions. Graduates are to demonstrate “the content, pedagogical, and professional knowledge, skills, and dispositions necessary to help all students learn” in a manner consistent with professional, state, and institutional standards. Departments or colleges are to document their performance using their own assessment data and external measures such as candidates’ performance on state licensing exams. In addition to the 6 NCATE professional standards, NCATE has approved 18 sets of program standards that have been developed by its constituent specialty associations, such as the National Council of Teachers of Mathematics and the National Council of Teachers of English. NCATE expects programs within a college to be congruent with the associations’ standards. Yet there is widespread concern that these associations have developed standards based on normative conceptions of good pedagogy and curricula that have not necessarily been shown to improve student achievement. For experienced teachers in graduate programs, NCATE suggests that they work to meet the requirements of the National Board for Professional Teaching Standards for advanced certification. However, researchers have yet to determine whether board certification affects student achievement.

• Assessment. Colleges or departments are to engage in extensive efforts to collect and analyze data on applicants’ qualifications, candidate and graduate performance, and unit operations. They are to use this data for continuing program evaluation and improvement. The rigor implied in this standard is admirable. Unfortunately, the professional, state, and institutional standards against which programs are asked to measure themselves have not typically been validated with equal rigor.

• Field Experience. Colleges or departments and the local schools with which they form partnerships are to provide field experiences that help teacher candidates to develop the knowledge, skills, and dispositions needed to help all students learn. NCATE lists many possible measures for this standard and requires that “all candidates participate in field experiences or clinical practice that include students with exceptionalities and students from diverse ethnic, racial, gender, and socioeconomic groups.”

• Diversity. The discussions of “diversity” in four of the other five standards are only a warm-up for this standard. Programs are charged with ensuring that candidates work with “diverse” higher education and school faculty, peers, and students, and must demonstrate “good-faith efforts” to admit candidates and hire faculty from diverse cultural backgrounds. Candidates must learn to develop “a classroom and school climate that values diversity . . . [an awareness] of different teaching and learning styles shaped by cultural influences . . . dispositions that respect and value differences” and “skills for working in diverse settings.” Candidates are also required to learn to teach from “multicultural and global perspectives” and program faculty are to design learning experiences that help candidates to “process diversity concepts.”

• Faculty Qualities. Faculty are to be “qualified,” model “best professional practices,” engage in self-assessment, and collaborate with practitioners and disciplinary colleagues. NCATE requires data on faculty degrees held, research productivity, and teaching evaluations. NCATE standards for faculty include that members “integrate diversity and technology throughout their teaching” and “introduce candidates to research and good practice that counter myths and misperceptions about teaching and learning.”

• Unit Governance and Resources. The last standard requires units to have leadership, authority, budget, personnel, facilities, and resources adequate for preparing candidates to meet professional, state, and institutional standards.

It should be clear that the standards NCATE uses to evaluate teacher-training programs are primarily input-driven: the main measures of a program’s quality are such things as the degrees held by faculty, program resources, and curricular content. Wise describes these criteria as “selected elements of process that are embedded in our standards. They emphasize some things that our committee and community believe are particularly important: like diversity, the use of technology, and the availability of necessary resources.” He argues that an accreditation model based strictly on outcomes is problematic “because, paradoxically, if you only hold institutions accountable for results, you let those with good inputs off the hook.” In other words, focusing solely on outcomes lets the Harvards of the world breeze by on the strength of their students’ natural abilities, while programs with weaker students are punished unfairly regardless of the value they are adding.

Wise’s point is reasonable, but it highlights the ambiguity at the heart of NCATE’s mission. As Wise himself notes, accreditation is a “pass/fail” proposition intended only to ensure that an institution is meeting certain standards–not to guarantee that they train teachers to be effective.

One way in which NCATE attempts to demonstrate its effectiveness is by citing the fact that the three states that required NCATE accreditation for all schools of education during the 1980s–Arkansas, North Carolina, and West Virginia–experienced greater than average increases in student achievement on the NAEP assessments during the 1990s. But many factors can influence a state’s educational performance, from changes in state policies to changes in their demographic makeup, making it extraordinarily difficult for researchers to isolate any one cause for an increase in achievement.

The piece of evidence cited most frequently by NCATE measured how teacher candidates fared between 1995 and 1997 on the Educational Testing Service’s PRAXIS II licensing exams for teachers. NCATE trumpets the fact that graduates of NCATE-accredited schools fared 7 percentage points better than did graduates of non-NCATE programs. Yet economists Dale Ballou and Michael Podgursky identified significant methodological flaws in this study. Moreover, their analyses of graduates of NCATE and of non-NCATE programs found little evidence of differences between the two groups (see Figure 2). School administrators seeking to hire new teachers cannot be confident that graduates of NCATE-accredited institutions are likely to be better teachers than other applicants.

Absent empirical evidence that teachers trained “the NCATE way” serve students more effectively than other teachers, the value of NCATE accreditation is a matter of faith. If one questions the value or objectivity of NCATE’s standards, the process can seem like a mere bureaucratic mechanism that endorses institutions that train teachers in accord with certain pedagogical and curricular precepts. This problem is particularly acute because NCATE’s subject-matter standards were drafted by its constituent specialty associations–and those groups, like the National Council of Teachers of Mathematics and the National Council of Teachers of English, have a track record of promoting methods of instruction whose value has not been well established.

Thus we come to the tension at the heart of teacher preparation. After all, efforts to pin down teacher educators regarding “essential” knowledge or skills illuminate the slipperiness of our conception of the competent teacher. The NCATE standards document declares, “The profession has reached a consensus about the knowledge and skills a teacher needs to help P-12 students learn. That consensus forms the basis for the NCATE standards.” However, this consensus has not been embraced by many other education stakeholders. As noted by Frank B. Murray, president of the Teacher Education Accreditation Council (TEAC), a rival accrediting agency, even among education scholars and practitioners, “there is still no consensus . . . about what would constitute educational malpractice.”

As a result, NCATE’s ostensibly outcome-based standards lean heavily on inputs and on confirming the presence of practices that it deems desirable. However, determining “best practices” in teaching is far from a straightforward process, as even professionals disagree over how an effective teacher approaches bilingual education, homework, student testing, student spelling and grammar, student-directed learning, “multicultural” lessons, discipline, desk arrangement, scientific inquiry, and so on. Moreover, prominent voices in teacher preparation continue to question whether clear-cut measures of student outcomes–such as graduation rates or test performance–are legitimate measures of educational performance.

Without widespread agreement on what good teachers need to know and be able to do, and given the skepticism among professional educators about the fairness or value of standardized assessments, NCATE has little leeway for assessing outcomes without inspiring controversy. As a result, the dean of a nationally ranked school of education notes, “NCATE focuses on things like the number of books in our library and the number of degrees among our faculty rather than what we actually do.”

Some critics have charged that NCATE’s focus on procedures, resources, and extensive documentation is a significant burden for smaller programs. Richard Ekman, president of the Council of Independent Colleges, says, “NCATE calls for lots and lots of methods courses that go well beyond what small colleges can offer. Yet the graduates from these colleges come out with good, solid content majors . . . which is very much in demand by the school districts doing the hiring.” NCATE rejects such criticisms, responding that “all types of institutions can and do meet professional accreditation expectations” and noting that 200 of its 540 accredited institutions are private, independent liberal arts colleges. However, our analysis of NCATE-accredited institutions reveals some significant patterns. Only 23 percent of education programs at private institutions are NCATE-accredited, compared with 69 percent of the education programs at public institutions (see Figure 3). For both public and private institutions, the frequency of NCATE accreditation increases with the size of the institution. Only 22 percent of small private institutions and 63 percent of small public institutions possess NCATE accreditation, compared with 44 percent of large private institutions and 74 percent of large public institutions (see Figure 4).

Even larger institutions report that the NCATE process is burdensome, and administrators at leading universities have questioned the value of participating in the NCATE system. The dean of a nationally ranked, NCATE-accredited school of education told us: “The [NCATE] process was very time-consuming and expensive, to be sure. And I’m not entirely convinced that all that we were asked to do is necessary or really contributes to the quality of the program. We have more resources than most schools, so it was manageable. I’m not sure many schools could manage it.”

Another nationally ranked program stopped participating in the NCATE system because it was, in the words of a university official, “a labor-intensive and therefore expensive process from which we got very little new information. There were also conflicts between a few NCATE policies and those that our faculty had endorsed. A sense of exclusive or proprietary responsibility at NCATE was in conflict with a culture of shared responsibility for teacher education that we value and try to cultivate.”

Fourteen of the top 25 graduate schools of education, as ranked by U.S. News & World Report, are not accredited by NCATE (see Figure 5). Three of the 14 schools, including Columbia University’s Teachers College, are NCATE candidates or applicants. A Teachers College official remarked, “The state is forcing our hand” and indicated that the college would not be seeking NCATE accreditation in the absence of a new state mandate.

Colleges of education may attempt to use NCATE’s requirements as leverage for securing increased resources from university budgets. Karen Zumwalt, the former dean and now a professor at Teachers College, said that while the college’s pursuit of NCATE accreditation will involve “an incredible amount of work, hopefully, it will provide both the impetus for program improvement and for obtaining the needed institutional resources.” Moreover, some colleges that are not required to participate in the NCATE system may do so anyway out of a sense of duty–to offer symbolic support for the objectives of NCATE and its allies. The dean of another top-ranked school explained that “the primary reason” for the school’s participation in the NCATE system “is to support (and participate in) an important effort to professionalize teaching.”
An Alternative to NCATE

The recent emergence of the Teacher Education Accreditation Council (TEAC) has challenged the accreditation monopoly long enjoyed by NCATE. Incorporated in 1997, TEAC was created with help from presidents of small, independent colleges who thought that NCATE’s prescriptive standards favored larger schools and neglected program outputs.

TEAC has been recognized by the Council for Higher Education Accreditation since 2001, and it is seeking approval from the U.S. Department of Education. TEAC membership costs $2,000, and an accreditation audit costs $1,000. TEAC has received more than $2 million in external funding from organizations including the Olin Foundation, the Pew Charitable Trusts, and the Atlantic Philanthropies. Programs accredited by TEAC set their own standards within TEAC guidelines, using an accountability model similar to that used in charter schooling. TEAC accreditation audits focus on three quality principles: evidence of teacher candidate learning, evidence that the assessment of such learning is valid, and evidence of the program’s own continuous improvement and quality control.

Programs seeking accreditation prepare a “research monograph” in which they present evidence that they are meeting the standards of the three principles, using multiple outcome measures such as teacher candidates’ grades, scores on standardized entrance and exit exams and portfolios; community service data; and employer surveys. TEAC then audits the evidence to evaluate whether the program is preparing competent, caring, and qualified educators and decides whether to accredit the program.

Unlike NCATE, TEAC does not require applicants for accreditation to comply with standards developed by external organizations. Rather, TEAC requires that the standards selected by the accreditation candidate be based on research and lead to the preparation of competent, caring, and qualified teachers.

By allowing teacher-education faculty to select the standards and evidence by which they will be judged, TEAC permits a tight alignment of accountability measures and program goals. Such a process can enhance faculty commitment and is well suited to fostering productive self-evaluation and continuous improvement. Most promising, TEAC’s heavy emphasis on the obligation of teacher-education programs to produce convincing evidence in support of the claims they make about their own quality has the potential to enhance accountability in teacher education.

Conclusion

In a way, whether NCATE or another accrediting body attains a position of influence within teacher education is irrelevant; NCATE’s standards reflect the collective judgment of teachers, teacher educators, and their representatives, and at worst they will probably do no harm to the enterprise of teacher training. The real concern is that NCATE and its sister organizations will succeed in tightening their grip on entry to the teaching profession itself. Once NCATE convinces states that all teacher-training programs must be NCATE-accredited, it is easy to imagine states requiring that all teachers pass through NCATE-accredited programs. This is precisely what happened in fields like medicine and law.

But teaching is not medicine or law. While the curricula of accredited medical schools must instruct students in “the fundamental principles of medicine” and accredited law schools must impart the “basic principles of public and private law,” the essential and fundamental principles of teaching have yet to be established. Accredited medical school curricula must include specific content from a range of scientific disciplines. Similarly, in law, there is consensus on a specific body of knowledge that is to be acquired. By contrast, there is no consensus on a necessary skill set and knowledge base for teachers. No one debates whether doctors or lawyers should have to undergo professional training in a university or college environment, but there is still substantial debate over the very value of teacher-training programs.

Indeed, it is questionable whether any form of accreditation is useful or appropriate in a context of widespread disagreement about what skills, dispositions, and methods are essential to good teaching. If teacher preparation were deregulated and current programs stripped of their monopoly, competitive pressure might compel them to document their quality in useful and appropriate ways without the bureaucratic constraints or costs of accreditation.

In a nation marked by disagreement about what teachers need to know, what they ought to be able to do, and what dispositions they should have, the challenges of devising a sound accreditation system are mighty. Compared with NCATE’s standards, TEAC’s lack of direction risks appearing relativistic. Yet, at the very least, NCATE should not be granted de facto status as the accrediting body for teacher education. Whether TEAC or another alternative will prove more effective is not clear. What is clear, however, is that the goal of a quality teacher in every classroom is more likely to be met in a system where various models of quality control are tried, tested, and compared than in a world where NCATE’s constituent organizations are effectively crafting the standards for the pedagogy, curriculum, and practice of teacher education.

Sandra Vergari is an assistant professor of educational administration and policy at the University at Albany, State University of New York. Frederick M. Hess is an assistant professor of education and government at the University of Virginia and executive editor of Education Next.

Catholic Schools: Private and Social Effects
Kluwer Academic Publishers, 2000, $100; 160 pages
By William Sander

The Education Gap: Vouchers and Urban Schools
Brookings Institution, 2002, $28.95; 275 pages
By William G. Howell and Paul Peterson, with Patrick J. Wolf and David E. Campbell

As reviewed by R. Kenneth Godwin

The advantage of reading The Education Gap and Catholic Schools together is in being able to appreciate their use of diverging research strategies. At the heart of The Education Gap is a large-scale study of privately funded school-voucher systems in three cities, New York, Washington, D.C., and Dayton, Ohio. The authors, Paul Peterson [editor-in-chief of Education Next] and William Howell, mounted a randomized field trial like those used in medicine in order to test the effects on achievement of being given a voucher to attend private school. Catholic Schools, by contrast, uses a nonexperimental approach in studying the influence of Catholic schools. The strengths of the experimental approach compensate for the weaknesses of the nonexperimental approach and vice versa. When both types of studies yield similar conclusions, the results inspire greater confidence.
In The Education Gap, Howell and Peterson call randomized field trials the “gold standard” of social-science research. Randomized trials, built on the model of medical experiments, allow researchers to estimate the effects of a policy change by randomly sorting individuals into two comparable groups—in the case of school vouchers, a test group that receives vouchers and a control group that doesn’t. The random allocation ensures that there are no systematic differences—such as income, achievement levels, or parental involvement—between the two groups that might influence the results. Despite their advantages, randomized trials have weaknesses, the most basic being that they do not tell the researcher whether the results may be generalized to other situations, to nonvolunteers, or how the treatment variable will interact with other policy-relevant variables.

Randomized field trials are so expensive that they often must be limited to only a subset of the population of interest. For example, the study might include only a few locations and only a few grades. Even if the outcomes of the treatment and control groups differ significantly, we cannot be sure that the results will generalize to other locations and to other grades. If the experiment was limited to a single income group, we cannot know if the treatment would work equally well for other income groups. Despite these limitations, however, randomized field trials are an extremely useful type of evaluation, and they have an exceptional level of internal validity. We generally can be more confident in conclusions reached by randomized field trials than those produced by nonexperimental research as long as we do not generalize beyond the subgroups studied.

Most social-science research cannot use randomized field trials because of their expense and because it is rarely feasible to assign a sample population to treatment and control groups. Social-science researchers therefore typically observe relationships in nonexperimental settings and attempt to adjust statistically for all the relevant variables. For example, if we are interested in whether private school students and public school students have different outcomes, we try to obtain good measures for all other variables that influence the outcomes and then use regression analysis to discover whether private school students did better than the public school students. In Catholic Schools: Private and Social Effects, William Sander uses this method to study the effects of attending a Catholic school on various academic and nonacademic outcomes.

Nonexperimental research in education has two important methodological problems to overcome: omitted variables and selection bias. Selection bias is a special case of the omitted variable problem. Assume for a moment that two families live next door to each other. The parents have the same occupations, educations, and incomes, and the children have similar abilities. One family pays private school tuition, while the other enrolls its children in public schools. If we observe that the children in the private school learn more, can we conclude that private schools are better? No, because the families obviously have different values and goals. It is likely that the parents paying tuition have other characteristics that encourage their children to value education highly and to work hard in school. Statistical comparisons of public and private school outcomes must find a way to account for these unobserved, but very important, characteristics.

Randomized field trials essentially eliminate the problem of selection bias. At the same time, nonexperimental research can use large sample sizes to test the effects of Catholic schools in a wide variety of situations and to test for interactions between attending a private school and such variables as race, class, religion, and ability level. That these diverging research strategies have common findings lends them serious credibility. The most important common finding was that attending a private school significantly improves the education outcomes of African-American children in the inner cities but not of other students.

A second common finding is that instruction at a religious school has a positive effect on the religiosity of students. A major reason that parents choose to send their children to religious schools is to obtain religious instruction. The Education Gap finds that such instruction increases the religious observance of students. Catholic Schools shows that this increase lasts into adulthood and affects religious behaviors such as the frequency of prayer and the contributions one makes to the church.

Because all of the randomized experiments occurred in urban areas, a question that could not be answered by the Howell and Peterson study is, “Do private schools make a difference in rural areas?” Catholic Schools tells us that they do not. Similarly, because the experiments in The Education Gap included only low-income students, we do not know if children from higher income families who attend private schools learn more than higher income students who attend public schools. Again, Catholic Schools says no.

The Education Gap provides important information concerning the effects of vouchers on political tolerance. Opponents of expanded school choice fear that allowing families to choose private schools, particularly fundamentalist schools, will encourage intolerance among children. Although previous nonexperimental research has found that students in private schools showed greater tolerance and concern for the less fortunate than did public school students, these results may have been caused by selection bias. That the randomized field trials found similar relationships gives us much greater confidence that private schools do not promote intolerance.

Given the weight that the black-white test-score gap carries in the American psyche, the advantage that Catholic schools give black children has substantial policy implications. The most immediate of these is that providing vouchers to low-income children in the inner cities appears to be a cost-effective tool for improving their education outcomes. Such a policy would increase educational equity across racial and ethnic groups. A second implication is that voucher policies that include sectarian schools will not reduce political tolerance in the United States and may increase feelings of civic duty and political participation. This should alleviate the fears expressed by such prominent philosophers of education as Amy Gutmann of Princeton and Eamonn Callan of Stanford. It doesn’t appear that vouchers would reduce the quality of democracy in the United States.

R. Kenneth Godwin is a professor of political science at the University of North Carolina at Charlotte and coauthor, with Frank R. Kemerer, of School Choice Tradeoffs: Liberty, Equity, and Diversity (2002).

---

The Education Gap: Vouchers and Urban Schools

By William G. Howell and Paul Peterson, with Patrick J. Wolf and David E. Campbell

As reviewed by John E. Coons

The “gap” that serves as The Education Gap‘s leitmotif is a catchall for educational inequalities among demographically distinct groups—racial, ethnic, and economic. The scene is painfully familiar, with the “haves,” well-off whites, receiving resources and opportunities not available to the “have-nots.” With little distortion (and some convenience) I will refer to these two groups respectively as the Lucky and the Unlucky. The hypothetical gaps between them lurk in an array of particular inputs and outputs of schooling. On the input side are certain personal and family attributes plus all those features of schools—skilled teachers, computers, updated facilities—that money now buys directly and that those without money could, instead, buy through school vouchers. On the output side are test scores and the various behaviors and attitudes of parents and children who enjoy (or do not) a subsidized choice.

The measured effects of choice on the relative positions of the Lucky and the Unlucky can be surprising and even paradoxical. For example, though today there is no systematic gap between the Lucky and Unlucky in per-pupil expenditures, the existing systems of vouchers, in cities like Milwaukee and Cleveland, automatically create one by setting the value of the voucher lower than the cost of educating a child in the public schools. Now the paradox: at least for black students, the authors of The Education Gap find, this decline in financial support is associated with higher test scores; the notorious gap between white and black students is diminished. Nor does the decline in dollars injure the test scores of whites and Hispanics who also choose the private sector.
The book takes the measure of other alleged gaps, bringing me to my principal criticism. Statistics are always about something that is selected before the counting begins. The Education Gap sometimes gives the impression that it has identified everything still needing to be counted, and, indeed, it does suggest various sensible projects. But these appear to have been selected—and the project thereby limited—largely in response to criticisms from opponents of choice. When Professor X complains that choice for the poor will skim the “best and the brightest,” the authors answer that it did not do so in the programs studied. Fine, but might we also wonder whether, over time, students switching from public to private schools could have a “pioneer” effect, alerting the more apathetic families to their opportunity for responsibility and autonomy?

Or, when Professor Y worries that vouchers will destroy the unity of the curriculum and thereby diminish our “civic” focus, the authors rightly report the superior record of private schools on measures of tolerance. What they fail to question is their opponent’s premise of an existing civic curriculum. Others observe that contradictory moral and civic theories flourish among public schools, making coercive assignment of the poor an intellectual lottery. If they are right, any civic criticism of choice is at best incoherent. If The Education Gap‘s substance and tone throughout is relentlessly fair to its ideological opponents, at some points it seems modest to a fault.

This forensic constriction of the authors’ agenda is illustrated more broadly in its dogged focus on test scores and the empirics of school life. Their questionnaires for parents inquire mainly about their experiences in the chosen school yet only skirt the possibility of deep effects on the family itself: “How often did you . . . help this child with . . . homework; . . . talk . . . about experiences at school; attend school activities; work on school projects?” And “How many parent-teacher conferences . . . . How many hours . . . volunteered . . . . Are you a member of the PTA,” and are you “satisfied” with this school and why?

I would not discourage such inquiries, as long as they do not scant the larger hypothesis that “the problem” to which choice may be an answer lies in the unlucky family itself. The middle-class parent who enjoys choice can function as a full and responsible human being; the child senses this and remains consciously integrated in this little platoon of potent and important people. By contrast, conscriptive school assignment strips the unlucky family of its dignity and power, inviting the parent to accept the passive role of the permanent loser; the child grasps this impotence as the deadly threat of his or her own isolation and vulnerability. Before all else the extension of school choice is an instrument of family policy, as Lyndon Johnson and Patrick Moynihan understood and as free-market theorists tend to forget.

Nearly as fateful is the question of the child’s own autonomy. If freedom is a value of childhood, is it imperiled by our subsidizing parents? Some “liberals” say as much, but all we really know in empirical terms is that small children are inevitably dominated by particular adults. The middle class nevertheless has supposed that such inescapable adult sovereignty can liberate when exercised by the family; autonomy is the product of love harnessed to the parent’s practical capacity to act responsibly over time. I hear that teenagers in the suburbs influence the family’s choice of school. Would it be any different within the unlucky family? The Education Gap tells us only that the effect of parental choice on student self-confidence “appears to be moderately positive.” I should think so. And the unluckier the family, the greater its rejuvenation.

“Family effects” of this sort could explain the grand phenomenon that so puzzles the authors: “How greater choice translates into achievement gains for African-American students remains unknown.” But which among our unlucky families presently have the least coherence simply as families? We know that Hispanics and Asians retain a surprising integrity; for them the pretensions of conscriptive state schools may be simply a huge annoyance and little threat to family identity. But, as Moynihan warned us long ago, the “Negro Family” verges on prostration for want of serious mission. Before all other therapies, it cries out for a devolution of responsibility—for real tasks that are important, dignified, and civic. What could be better for demoralized parents and children than to acquire the last word on school assignment? Perhaps it is deep inside the family that we should look to explain these happy effects of experiments with school choice—higher test scores included.

Chesterton wrote of his friend and opponent Bernard Shaw that he “is like the Venus of Milo; all that there is of him is admirable.” One could say the same of The Education Gap with this twist: what is well presented of the “statue” of choice are her amiable extremities—arms from an otherwise missing figure. It is time to seek the body, the head—and the heart.

John E. Coons is a professor of law at the University of California-Berkeley and coauthor, with Patrick Brennan, of By Nature Equal: The Anatomy of a Western Insight (1999).

Kingdom of Children: Culture and Controversy in the Homeschooling Movement
Princeton University Press, 2001, $24.95; 238 pages.
by Mitchell L. Stevens

Reviewed by Paul T. Hill

Deciding to home school your children is a huge commitment, but in many ways it is simply an extension of the commitments made by many other parents. Parents who send their children to public schools often volunteer in the schools so that they can identify the best teachers and ensure that their children are assigned to their classrooms. Some parents also use grievance procedures and the special-education IEP (individualized education plan) process to make sure their children get exactly the right instructional program. Parents who pay for private schooling or who start new charter schools also make sacrifices in order to give their children a particular kind of education. It is the American way: except for paying taxes and avoiding criminal activity, determined families can influence or opt out of collective activities that conflict with their self-defined values or needs.

Home schoolers alone, however, are forced to face the criticisms that they’ve abandoned the public schools or so cloistered their children that they’re unlikely to develop the skills they need to function in the outside world. Answering the critics, Mitchell Stevens portrays home schoolers as communitarians, not isolationists. Stevens, a sociologist, conducted in-depth interviews of several dozen families and participated in home-school association meetings throughout the country. Though he admits his study is not statistically representative, his findings clearly reflect the experience of home schoolers who cooperate with one another and are open about their practices. These parents, probably the majority of home schoolers, are mainly middle-class parents who believe in prolonged intimate contact between family and child, but who do not mean to impede their children’s access to higher education and jobs or their ability to act as good citizens.

Stevens’s portraits of home-schooling families are sympathetic and vivid. Of the homes studied, the overwhelming majority have intact marriages, and both spouses have the skills required to work successfully for pay. The spouses decide to live on less money and commit themselves to an extended period of very hard work, all because they think it is good for their children. These portraits leave some important questions unanswered, however. Stevens contacted families that joined mutual-help groups, attended meetings, and subscribed to newsletters. Inevitably, he missed the families who hold themselves completely apart even from other home schoolers. Thus there still may be a proportion of home schoolers who fit the stereotype of the isolationist hoping to keep children in the state of nature.

Because his data come directly from the families he met, he also cannot answer the question of whether home-schooled children learn as much as those who attend regular schools. Success stories of children graduating with honors from elite colleges show that home schooling can have excellent results, but they certainly don’t represent the range of actual outcomes.

Christians and “Inclusives”

These limitations aside, Stevens provides much that is new and intriguing. He shows that home schoolers’ ideas about what is worth learning and how to teach it are not far outside the mainstream. Christian home schoolers (members of evangelical and Baptist congregations), whom Stevens estimates make up as much as 80 percent of the home-schooling families, want their children to learn traditional subjects, and they use familiar teacher-centered methods. Christian home-schooling parents organize learning around traditional grade-level expectations. Parents buy (and on occasion publish and sell) instructional materials. Children hear lectures, use published workbooks, and write papers. They often work in spaces modified to look like classrooms.

There is a second group of home schoolers, whom Stevens calls the “inclusives,” a broad category that covers left-wing and counterculture groups as well as Jews, Catholics, and mainstream Protestants who are not comfortable with the Christian home schoolers’ pietistic style. Inclusives, especially the nonreligious majority, tend to emphasize child-directed learning via projects and self-guided work and are much less comfortable with anyone, including a parent, lecturing to students or with students’ learning through textbooks.

These divisions among home schoolers reflect divisions in the larger society. Public school boards struggle over whether instruction should be didactic or discovery-based, and schools of both stripes compose the private school market. Even the charter school movement is split between users of traditional and child-directed pedagogy.

Stevens’s findings also undermine the fear that children in Christian home schools will learn only by rote and be unable to use what they learn. The teacher-directed part of the day lasts only a few hours; many children then apply their reading and arithmetic skills to church activities or to helping their parents at work. Thus the epithet “drill and kill” does not fit home schoolers any better than it does the Direct Instruction and Core Knowledge schools to which it is usually applied.

Stevens provides less detailed information about the methods used by the inclusives. It is possible that some of these families don’t trouble children with the details of reading or counting until they express a desire to learn such things.

As Stevens portrays them, inclusives think the only authentic authority in education comes from the child’s inner goodness. Standard teaching materials and adult-led organizations are suspect even when intended to promote or protect home schooling. Inclusives are so averse to exercising or accepting leadership that they find it difficult even to arrange meetings and assign tasks. Stevens contrasts the Christian home schoolers’ quick action against a congressional proposal that they considered threatening with the inclusives’ much slower and less-effective process.

Christians and inclusives are increasingly separate from and suspicious of one another. Christian home schoolers have created a powerful legal-defense organization, which the inclusives disdain as the work of money-grubbing lawyers tainted by their association with regulation. When threatened by government action, the Christians turn for help to the Republicans, the inclusives to the Democrats.

One fact unites all home schoolers: dependence on the efforts of mothers. Women do the lion’s share of the work-searching for materials, planning lessons, guiding children through their studies, and sustaining mutual-help organizations. Fathers in home-schooling families face pressure as sole wage earners, and many do some tutoring and help with housework. But Stevens shows that home schooling is a woman’s enterprise, in which men are usually junior partners.

Some of Stevens’s most thoughtful passages concern how home schooling coexists with the late-20th century liberation of women. Stevens argues that women’s role in home schooling is more consistent with broader social tendencies than it first appears. Christian home schooling creates a new quasi profession, which allows women both to stay with their children and to do intellectually challenging, nonroutine work. Home-schooling organizations are dominated by women, some of whom become well known as leaders in the movement or as sources of good teaching ideas and materials. Christian home-schooling mothers get emotional support and validation from their churches and from one another.

The more left-wing women among the inclusives are often well educated but dedicated to living counterculture lives. They are self-sufficient mothers who have bonded so deeply with their children that they cannot bear to entrust their education to anyone else. Mothers (and sometimes fathers) find ways to make modest livings, often in counterculture food and crafts organizations.

What is the future of home schooling? More than a million children are taught at home, more than now attend charter schools. But a number of factors will limit home schooling’s growth, not least the difficulty of raising a family on one income. That, and the self-defined fringe character of the bohemian lifestyle, will probably rule out any significant growth of “inclusive” home schooling. However, Christian women face very different rewards and supports. And evangelical Protestant churches can offer a constant flow of potential recruits.

Much depends on the actions of mothers whose children have “aged out” of home schooling. Will they find other work entirely, or will they stay connected to home schooling by running organizations, writing and selling materials, and advising less experienced home schoolers? Stevens’s evidence suggests that there is a future for Christian home-schooling mothers who want to build the movement and draw income from their reputations and expertise. If even a few do this, Christian home schooling might continue to grow. It might also lead to the formation of cooperative enterprises among current and former home-schooling mothers. These might come to be called schools.

One can only hope that there is a new Stevens in the wings, studying the actions of home-schooling mothers whose children are grown, and tracking the growth of the market for home-schooling materials and advice. We also need a nationally representative study of home schooling and its effects. But future work will all build on the foundation Stevens has laid.

Paul T. Hill is a professor of public affairs at the University of Washington and a visiting fellow at the Hoover Institution.

In the late 1950s and early 1960s, if you were a black basketball player in Milwaukee and thought you “had game,” there were two playgrounds to establish your credentials: Franklin Square and Lapham Park. I spent many hours on both courts.

Although there are new playgrounds today, the tradition continues. I see the young men and now young women playing “hustle” and “taking it to the hoop.” I look at them and I see my friends, Dewitt, Harry, Pickens, Marvin, and “Ham,” and myself. I see the young men we were when North Divison High School became the first team from Milwaukee to play in the state tournament. I still feel the elation of our victories and the anguish of our one loss during our senior year—the championship game of the state tournament. The memories are still there, 44 years later.

Basketball is not the only memory of my youthful days. I recall how much my grandmother and my mother pushed the importance of education. In the 1940s my mother was a housekeeper for a white family in Shreveport, Louisiana. They paid my tuition at the Catholic school their children attended. My mother continued to send me to Catholic school when we moved to Milwaukee, where I attended St. Boniface Catholic School until the 8th grade. Because of our income, we were not required to pay tuition. To this day, neither my mother nor I know who paid for my education.

I transferred to public school because of my interest in sports and my desire to be with my friends. I never really thought about the difference between public and private schools. My mother and grandmother clearly thought private schools were better. But they both also fully supported my decision.

I never realized back then, as they obviously did, the importance of having choices for my education. Today I am fighting to make sure that children from low-income and working-class families in Milwaukee, indeed all over this country, have the same opportunity.

The term “choice” is often misunderstood by well-meaning people or distorted purposefully by people who want to discredit it. Choice is often equated only with vouchers. Vouchers are indeed one form of parental choice—a very important form. However, parental choice involves more than just vouchers. It means providing families with the capacity to choose from a wide range of learning environments.

I support a variety of policy initiatives that provide options for parents and students, including charter schools, public/private partnerships, contract schools, home schooling, cyber schools, historically black independent schools like Piney Woods in Jackson, Mississippi, as well as innovative governance arrangements in the existing institutions of public education.

Such choices enhance children’s access to better schools, thereby increasing the likelihood that they will be able to gain the skills necessary for full participation in a democratic society. Without parental choice programs, only well-to-do families have a choice unless they are fortunate enough, like my family, to have benefactors.

We must give low-income and working-class parents the power to choose schools—public or private, nonsectarian or religious—where their children will succeed. And we must give all schools the incentives to work to meet children’s needs. Consider the power of choice in the hands of families who have little or no power because they control no resources. Consider how the absence of choice will continue to consign their children to schools that the affluent parents who oppose choice would never tolerate for their own children.

I still love to see young people “taking it to the hoop.” But I want them to share my passion in the classroom as well. I want them and their parents to have the power to decide which schools can best give them that opportunity and support.

Howard Fuller is the director of the Institute for the Transformation of Learning at Marquette University and chairman of the board of directors of the Black Alliance for Educational Options.

Education is a powerful instrument for reducing poverty and inequality, improving health and social well-being, and laying the basis for sustained economic growth. . . . [We will] ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling.

— Declaration endorsed at the United Nations Monterrey Summit on Financing for Development, March 2002

Illustration by Tom Curry.

The goal of universal elementary education has been enunciated countless times in international conferences on economic development. In fact the language above, from the Monterrey Summit, is essentially unchanged from that of the World Conference on Education for All, held 12 years earlier in Thailand. The Conference members officially declared that schooling ensures “a safer, healthier, more prosperous and environmentally sound world, while simultaneously contributing to social, economic, and cultural progress, tolerance, and international cooperation.” The delegates in Thailand set a goal of universal primary education in every country by the year 2000. Overall, the campaign has had moderate success in increasing schooling worldwide, but each successive conference displays little curiosity about why the target year must be continually pushed back.

Each conference is similarly incurious about what kind of payoff more education yields. It may come as a surprise-as it did to me-to learn that the dramatic education expansion of the past three decades has had a uniquely disappointing effect on economic growth. Education may have a good payoff when combined with high-tech factories and good institutions, but when these are lacking, its payoff is dismal. Thus Eric Hanushek’s emphasis on education as key to labor productivity seems not altogether consistent with recent experience.

The problem is one of context. In wealthy countries, education is associated with prosperity and high-paying jobs. As a consequence, it is believed that transferring education to poor countries will bring prosperity. Alas, “education” is much more complex than that. If aluminum makes airplanes, and airplanes enable you to fly, it doesn’t follow that jumping off the Empire State Building with a sheet of aluminum will enable you to fly.

Creating people with high skill in countries where the only profitable activity is lobbying the government for favors is not a formula for success. Creating skills where there exists no technology to use them is not going to foster economic growth. Parents in such circumstances are understandably unwilling to invest as much in their children’s education as U.N. bureaucrats think they should. Expanding education is worth little more than yesterday’s newspaper if the incentives to invest in the future are not there.

The Education Explosion

Fueled by the emphasis of the World Bank and other donors on basic education, enrollment in primary school reached 100 percent in half the world’s countries by 1990, up from 28 percent of the world’s countries in 1960. The median primary enrollment increased from 80 percent in 1960 to 99 percent in 1990. Behind these figures lie miracles like Nepal, which rose from 10 percent primary enrollment in 1960 to 80 percent in 1990.

How have the economies of the world reacted to the education explosion? Hardly at all, it seems.

The lack of association between growth in schooling and growth in gross domestic product (GDP) has been noted in several studies. Africa’s stagnation in the face of its education explosion provoked economist Lant Pritchett to ask, “Where has all the education gone?” Pritchett’s analyses found no positive association between an expanding education sector and growth in output per worker or in productivity. (He actually found a negative and significant relationship in some statistical exercises.)

African countries with rapid growth in human capital, or the skills and knowledge possessed by laborers, during the 1960 to 1987 period, such as Angola, Mozambique, Ghana, Zambia, Madagascar, Sudan, and Senegal, were nevertheless disasters with regard to economic growth. Meanwhile, countries like Japan became economic miracles despite only modest growth in human capital. Other East Asian miracles like Singapore, Korea, and China did have rapid growth in human capital, but no greater than that of the African growth disasters. For example, Zambia had slightly faster expansion in human capital than Korea, but Zambia’s growth rate was 7 percentage points lower.

Indeed, at the same time that education has experienced a massive expansion in poor countries, the median growth rate of those countries has fallen. Productivity growth was 3 percent in the 1960s, 2.5 percent in the 1970s, -0.5 percent in the 1980s, and 0 percent in the 1990s (see Figure 1). A similar study of how growth responds to the percentage change in the labor force’s average years of schooling found no relationship between growth in years of schooling and growth in GDP per capita.

Another group of economists found that variations in growth across nations have very little to do with variations in the growth of human capital. If a given country’s per-capita growth rate is 1 percentage point faster than average, economists attribute only 0.06 percentage points of this to the fact that human capital has grown faster than average. By comparison, economists found that productivity growth accounts for .91 percentage points of each 1.0 percentage point increase in the speed of output growth.

Alan Krueger and Mikael Lindahl found that changes in years of schooling had a positive effect on economic growth. They argued that Pritchett’s earlier findings of a nonrelationship between education and growth were due to measurement error (although Pritchett addressed measurement error). However, Krueger and Lindahl used the absolute change in years of schooling rather than the percentage growth in human capital as their unit of measurement. This has the effect of treating an increase of one year of schooling in a highly educated country as the equal of an additional year in a poorly educated country. This is counterintuitive, since an additional year of schooling in a poorly educated country implies a much faster rate of growth of human capital. Growth of output should respond to the growth of human capital. Even with this methodology and controlling for measurement error and other variables, Krueger and Lindahl found that the effect of the change in schooling on growth did not always pass standard tests for a significant statistical relationship.

Yet another study pointed out a more subtle problem with the idea that growth in human capital is a major force behind economic growth. If human capital growth is driving GDP growth, then rapidly growing economies will have rapidly growing human capital. This means that young workers will have considerably more human capital than the old fuddy-duddies who were educated during a time of much lower human capital. This factor would tend to give the young workers higher wages than the old workers. But wages worldwide almost always increase with years of experience-the older workers always earn significantly more than the young, even in rapidly growing economies. Even if years of experience count for something, fast-growing countries should have less of a wage increase with experience, because of the human capital advantage of the young. But this hasn’t been found anywhere. Therefore, the growth of human capital cannot be that rapid in a fast-growing economy and so cannot account for its rapid growth.

Zambia had slightly faster expansion in human capital than Korea, but Zambia’s economic growth rate was 7 percentage points lower. Photograph by Paul Kenward/Stone.

Education and Income

The finding that education doesn’t have much effect on economic growth is intensely controversial. Despite the failure of human capital growth to explain variations in economic growth, a number of economists aver that both physical capital and human capital can explain the large international variations in income. These economists, like Gregory Mankiw of Harvard, point out that long-run income in Robert Solow’s growth model is determined by saving in the form of physical capital and by saving in the form of human capital. Mankiw uses the share of children enrolled in secondary school as his measure of saving in human capital. There is indeed a strong relationship between income levels and secondary enrollment ratios. Mankiw shows that his measures of saving in physical capital and human capital can explain as much as 78 percent of the per-capita income differences among nations. How can this finding be reconciled with the finding that growth in output is not related to growth in human capital?

There are three problems with Mankiw’s relationship between secondary enrollment and income.

The first problem is that secondary education is a very narrow measure of education accumulation. What about primary education? The relationship between per-capita income and primary enrollment is considerably less satisfying. There appears to be no strong relationship among nations with primary enrollment of 20 percent to 90 percent: all these countries are poor. The many countries with universal primary enrollment have a higher average income, but they also have an incredible range of incomes, from very poor to very rich. In short, primary education varies much less across countries than secondary education and explains much less of the variation in income. As Peter Klenow and Andrés Rodrí­guez-Clare point out, Mankiw has exaggerated the variation of education in general by concentrating on secondary education alone.

The second problem is with human capital’s earnings under Mankiw’s assumptions. Mankiw assumed that financial capital flows would equalize rates of return on investments in physical capital. That leaves only investments in human capital to have different rates of return across countries. Explaining income differences with human capital alone is explaining big differences in income with a relatively minor ingredient. If a poor country is poor because of lack of skills, as Stanford’s Paul Romer points out, the few skilled workers must be earning very high salaries.

Let’s compare the United States and India. In 1992 per-capita income in the United States was 14 times that of India. This is also the ratio of unskilled wages in the United States to unskilled wages in India. Unskilled labor is abundant in India, while skilled labor is scarce. Mankiw’s assumptions imply that the wage for skilled labor should be three times larger in India than in the United States. Such wage differentials should induce skilled labor to try to move from the United States to India. Instead, we see the reverse: skilled Indians come to America. What’s more, if the predictions of Mankiw’s approach had come true, we would expect that unskilled Indians would want to move to the United States, while skilled Indians would stay put. That didn’t happen: educated Indians were 14.4 times more likely to move to the United States than uneducated Indians.

The propensity of skilled Indians to migrate to the United States is part of the general “brain drain” phenomenon. A recent study of 61 poor countries found that people with secondary education and above were more likely to move to the United States than those with primary education and below in all 61 countries. Those with university education were more likely to migrate than those with secondary education in 51 out of the 61 countries. Some countries are losing most of their skilled workforce to the United States. In Guyana, for example, a conservative estimate is that 77 percent of those with university education have moved to the United States.

Instead of skilled wages being three times higher in India than in the United States, as the Mankiw framework predicted, skilled wages are 24 times higher in the United States than in India. An engineer in Bombay earns $2,300 per year; an engineer in New York earns $55,000 a year. Mankiw’s framework predicts a negative association between skilled wages and per-capita income; instead, there is a very strong positive association.

The third problem is causality (again). What if high-school education is a luxury in which you indulge yourself as you get richer? Then naturally demand for high schools would go up as per-capita income rose, but that would not prove that high schools make you more productive.

Education and Incentives

Why is education worth little more than hula-hoops to a society that wants to grow? One question to ask is what educated people are doing with their skills. In an economy with extensive government intervention, the activity with the highest returns to skills might be lobbying the government for favors. The government creates profit opportunities through its interventions. For example, a government that fixes the exchange rate, prohibits trading of foreign currency, and spurs high inflation has created the opportunity for profitable trading in dollars. Skilled people will want to lobby the government for access to foreign exchange at the low fixed rate and then resell it on the black market for a fat profit. This activity does not contribute to higher GDP; it just redistributes income from the poor exporter who was forced to turn over his dollars at the official exchange rate to the black-market trader. In an economy with many government interventions, skilled people opt for activities that redistribute income rather than activities that create growth. (One somewhat whimsical piece of evidence that supports this story is that economies with lots of lawyers grow more slowly than economies with lots of engineers.)

Another question to ask is what kind of education is being provided. Expansion of education is usually driven by the state. But administrative targets for universal primary education do not in themselves create the incentives for investing in the future that fuel growth. The quality of education will vary depending on the economic, medical, and governmental prognosis for a country. In an economy where the future looks bright, students will apply themselves to their studies, parents will monitor the quality of education, and teachers will face pressure to really teach. In a stagnant economy or a nation riven with strife, students will goof off in the classroom or sometimes not show up at all, parents will often pull their kids away to work on the farm, and teachers will while the time away as overqualified babysitters.

Likewise, providing access to education is not the same as providing access to a good education. Corruption, low salaries for teachers, and inadequate spending on classroom materials all plague schools in poor countries. In Vila Junqueira, Brazil, people told interviewers that the “state school is falling apart, there are whole weeks without a teacher, no director or efficient teachers, no safety, no hygiene.” In Malawi, respondents said:

We hear the government introduced free primary education and provides for all essential requirements, notebooks, pens, and pencils. The pupils have never received these items. We still have to provide them ourselves. We strongly believe it is not the government’s fault, but it is sheer malpractice on the part of the school’s management. We have seen several teachers going around selling notebooks and pens. In addition the teachers are not dedicated to their duty. Often pupils go back home without attending even a single lesson. . . . Only ten pupils have been selected to secondary schools in the last six years.

In Pakistan, politicians dispense teaching positions as patronage. There is large-scale cheating at examinations, which are supervised by unscrupulous or intimidated teachers. Three-quarters of the teachers could not pass the exams they administer to their students. The medium of instruction in the public schools is Urdu, although the working language in this multilingual society is English. Some of the publicly supported schools are Islamic schools where the students mainly learn the Koran. The other public schools are of such poor quality that any parents who can afford to do so send their children to expensive private schools. High-school students from rival religious factions have fought each other with AK-47s. Not much good is going to happen when there are many guns and few textbooks in the schools.

A common pattern is that much more is spent on teacher salaries than on materials like textbooks, paper, and pencils. One study found that spending on school materials has a rate of return 10 to 100 times larger than additional spending on teachers, which means that school materials are very scarce relative to teachers.

These quality-of-education considerations are not the whole story in the failed relationship between education expansion and economic growth. Even high-quality schooling will go to waste if skilled labor and enterprise are not rewarded in the economy or are combined with inferior technology. A sign of this is the frequent mismatches between labor productivity and the test scores presented in Hanushek’s Figure 1 (see p. 15). The most obvious example is the United States itself, which has the highest labor productivity in the world. Meanwhile, at certain points its test scores lag behind those of low labor-productivity countries such as Hungary, the Slovak Republic, and Thailand. The story of an education crisis in the world-beating U.S. economy just doesn’t make sense. Either the quality of education in the United States is being mismeasured, or education quality is not as important for labor productivity as Hanushek makes it out to be.

If school quality is a key to growth, how to explain the case of Africa? Africa has stagnated terribly despite rapid expansion in education. For school quality to be crucial to growth, the quality of African schools would have to have deteriorated dramatically. Anecdotal evidence suggests some deterioration, but not a collapse of the magnitude required to offset the quantitative expansion of schooling. In general, one would think school quality would change rather slowly, thus being a fairly minor influence on quality-adjusted growth in human capital.

Despite all the lofty sentiments about education, the education explosion of the past four decades has been disappointing. I agree with Hanushek that learning under the right circumstances is a very good thing. No country has ever become rich with an illiterate population. But administrative targets for enrollment rates and overwrought rhetoric from international commissions, as well as more measured alarms about school quality, do not in themselves create the incentive to grow. For education to raise earning potential, many supporting conditions and incentives must be present. When they are missing, don’t jump off the building with nothing but a sheet of aluminum.

-William Easterly is a senior fellow at the Washington, D.C.-based Center for Global Development and Institute for International Economics and the author of The Elusive Quest for Growth: Economists’ Adventures and Misadventures in the Tropics (MIT Press, 2001).

Illustration by Dan Vasconcellos.

In their continuing efforts to extract more school spending from state legislatures through the courts, advocacy groups recently acquired a powerful new weapon: the standards movement. Their success provides yet another example of the law of unintended consequences.

Recently, plaintiffs in two prominent cases, in New York and North Carolina, successfully used the states’ standards and performance expectations as evidence of the states’ failure to provide an adequate education to all students, especially poor and minority students. Judges in both states ordered the legislatures to rework the formulas used to distribute state funds to local school districts. The New York case is currently in limbo after a state appeals court reversed the trial court’s decision in June 2002.

This development marks the latest, and perhaps the most ingenious, turn in a run of litigation that spans almost three decades. Ever since the U.S. Supreme Court, in the 1973 San Antonio Independent School District v. Rodriguez case, declared that funding disparities among local school districts were of no federal concern, plaintiffs have pinned their hopes on the education clauses in state constitutions. State constitutions usually require the states to provide their residents with a “thorough and efficient” or “sound basic” education. What this language means-whether it mandates equal funding or simply a minimum level of funding necessary for a “sound basic education”-has been the central issue in the past 12 years in 28 major lawsuits, of which the plaintiffs have won 18.

The chief target of this litigation is the long-standing practice of funding schools mainly with local property taxes. This practice contributes to per-pupil spending differences among school districts that are sometimes stark. Well-to-do neighborhoods with high property values can afford top-notch facilities and salaries high enough to attract good teachers. Low-income areas, meanwhile, often struggle just to put a certified teacher in every classroom. Early state-level lawsuits-the so-called “equity” wave of litigation-asked state courts to mandate that all children have a right to the same level of education, as measured by per-pupil spending. Such lawsuits met with limited success, however. For every lawsuit that succeeded, another failed. Moreover, even successful litigants were rarely able to win substantial increases in spending equity.

In addition, equity lawsuits began to lose support from many of the nation’s large urban school districts, even as they continued to struggle mightily with delivering basic educational services. Many urban districts realized that, despite the obvious challenges they confront in serving children, gaps in educational spending between urban and nonurban districts are not the key problem. Indeed, in most states urban school systems benefit from spending levels that exceed state averages. As a result, in successful equity-based school finance lawsuits, urban schools stood to lose or, at best, not to gain additional resources.

The Emergence of Adequacy

In 1989 the world of school finance litigation changed in a way that welcomed urban districts back into the fold. Many observers point to a 1989 decision by the Kentucky Supreme Court, Rose v. Council for Basic Education, as ushering in the “adequacy” theory of school finance litigation. Unlike suits based on equity, which sought to close spending gaps between high- and low-income districts, lawsuits based on adequacy challenge state school finance systems not because some districts benefit more than others, but because some districts provide education of miserable quality. The question for the courts then became how to define adequacy.

Enter the standards movement. The states themselves have established the levels at which they expect schools and students to perform. Persistently underperforming schools in some states are identified as “low performing” and are subject to additional external supervision. Students who fail to achieve minimum scores on state tests are prevented from graduating from high school with full academic diplomas. Such policy changes seek to shift school regulation away from the traditional focus on inputs-teacher-to-student ratios, per-pupil spending, number of certified teachers-and toward a focus on performance as the basic metric of education quality. However, in an ironic twist, this output-driven movement has made it much easier for activists to appeal to the courts for more inputs. The standards movement enables activists to define adequacy as that level of funding necessary for a school district and its students to meet state education standards.

Thus a new wave of litigation may be upon us, one that turns the states’ efforts to improve achievement through standards against the state and enables school districts to gain financially from their inability to perform at desired levels. These failures are used in court to bolster legal claims that such schools underachieve because their resources are inadequate and, therefore, unconstitutional. Judicial participation in this race to new money uncovers old and new problems that arise when courts are asked to set education policy.

New York

New York’s constitution guarantees its citizens a “sound basic education.” After more than a decade of development and debate, New York adopted the Regents Learning Standards in 1996. The standards articulate expectations at three educational stages (elementary, intermediate, and graduation) in such core subjects as English, math, and science. The standards are aligned with the Regents exams, which, for many New York students, especially the college-bound, have become a familiar rite of passage. Presently, passing the Regents exams is necessary only to earn a special Regents diploma. By 2004, however, the Regents exams will be a graduation requirement for all.

Unlike other urban districts nationwide, New York City’s public school spending falls just below the state average (New York City spent $9,623 per pupil in 1998-99, versus a state average of $10,317). Frustrated with unsuccessful appeals to lawmakers for increased resources, school finance advocates, joined by 14 of New York City’s 32 school districts and numerous New York City public school students and their parents, turned to the courts. In Campaign for Fiscal Equity v. State of New York, a widely covered case decided in 2001, the plaintiffs sought an increased share of the state’s education spending. They argued that the state funding formula worked in a manner that denied city schoolchildren the opportunity to receive a sound basic education.

In defending its school finance system, the state of New York argued that New York City spent enough to provide an adequate education and that the state’s share met constitutional requirements. The city’s poor performance on state tests was a result of mismanagement and a bloated bureaucracy, the state argued, not a lack of resources. Implicit in the state’s argument is the idea that the constitutional command for educational “adequacy” requires only that the state lift all students to a minimal floor. From the state’s perspective, this “floor” meant that it must equip students with the basic tools necessary for active, productive citizenship.

New York City construed educational adequacy differently and drew on the state’s own Regents Learning Standards as a definitional guide. The plaintiffs cited the failure of many city students to earn Regents diplomas as evidence that they are not receiving an adequate education. In 2000, for example, only 27 percent of New York City high-school graduates earned Regents diplomas, versus 49 percent statewide. New York City’s claim pivots partly on the assumption that adequate funding is the amount necessary to ensure that New York City’s students meet the state standards at a level comparable with their counterparts statewide.

Trial court judge Leland DeGrasse bought most of the plaintiffs’ argument and concluded that city students are not receiving a “minimally adequate” education. The trial court construed adequate education as that which would enable city students to compete for jobs with their counterparts statewide. However, the state appeals court reversed the decision in June 2002, holding that the trial court’s understanding of what the New York constitution requires in terms of education was flawed. The appeals court concluded that the state is only required to provide students with a basic education that will enable them to participate in the economy. Equally significant is that the appeals court refused to conflate educational opportunity and student achievement, especially as measured by the Regents exams. That is, the appeals court held that a failure to earn Regents diplomas is no evidence of a constitutionally inadequate education. Indeed, the appeals court went on to note that New York’s Regents Learning Standards exceed any notion of a sound basic education. The plaintiffs have promised to appeal and, consequently, New York State’s highest court will be called upon to resolve the matter.

North Carolina

North Carolina’s constitution requires that the state provide a “general and uniform” education system. To discharge its constitutional obligations, North Carolina lawmakers rewrote the state’s Basic Education Program in 1985. Shortly thereafter, the state board of education developed a Standard Course of Study designed to help all North Carolina students navigate successfully as adults and citizens. North Carolina lawmakers also implemented end-of-grade and end-of-course exams that seek to chart student progress towards mastering the state’s academic goals and to help increase school accountability. High-school graduates in North Carolina must also pass the North Carolina High School Competency test, which is set at approximately an 8th-grade skill level.

In 1994 both low- and high-spending school districts challenged the constitutionality of the state’s school finance system. All plaintiff districts wanted greater resources, but for different reasons. The low-spending (largely rural) districts wanted more funding to close the resource gap. High-spending districts, principally in urban areas, sought increased funding to offset the peculiar challenges incident to the production and delivery of educational services in low-income areas.

In a 1997 decision, Leandro v. State of North Carolina, the North Carolina Supreme Court considered various challenges to the state’s school funding system. While not deciding the substantive claims, the court held that, in construing whether a sound and basic education was being provided, courts could properly consider results from student assessments.

Following the state supreme court’s guidance, a North Carolina trial court, in a series of opinions, decided to prompt a restructuring of the state’s pre-kindergarten education system. The court, noting a yawning gap between the performance of at-risk and not-at-risk students on statewide assessments, concluded that greater attention to the special needs of at-risk students was constitutionally required. Like its counterpart in New York, the North Carolina court implicitly assumed that providing schools with more resources would help close the achievement gap, though researchers have found no such direct link between increased spending and increased performance. Also linking the New York and North Carolina decisions is the key role played in the litigation by results from high-stakes testing.

That is, in construing whether North Carolina lawmakers were providing constitutionally adequate educational services, the court looked to student progress on the state’s learning standards.
Perhaps mindful of the magnitude of the burden its decision placed on the legislative branch, the North Carolina court ordered lawmakers to change the state funding formula “at a reasoned and deliberate pace.” While the court’s language was no doubt intended to allay fears about judicial overreach, the obvious reference to the Brown v. Board of Education opinion (“all deliberate speed”) may have achieved the opposite effect. This legal battle is far from over as North Carolina governor Mike Easley recently urged the state attorney general to appeal the trial court’s final order to the state’s supreme court.

The recent state court decisions in New York and North Carolina portend an emerging trend in school finance litigation. Indeed, these court decisions are already beginning to influence school finance activists nationwide. Litigants in Florida appear poised to join that state’s standards and assessments program with its recently amended constitution in an effort to boost education funding. Recent federal legislation will further fuel this trend. The mandates of the federal No Child Left Behind Act, which requires testing in grades 3 through 8 and further labeling of schools as “failing” for not achieving adequate progress on state tests, will only accelerate the states’ development of standards-based accountability systems. The early successes of standards-based lawsuits ensure that the present efforts to improve achievement will have the unintended consequence of stimulating litigation against the states.

Many observers assumed that the educational standards movement would provoke a wave of lawsuits. High stakes of any kind-students being denied diplomas as a result of failing to pass state tests, schools denied funding for failing to improve performance-were certain to trigger litigation. But careful policy design and implementation along with an almost unlimited supply of second chances for students blunted such legal challenges. Now the worry is that lawsuits will derail the standards movement by taking advantage of its most promising attributes: its setting of clear academic standards and its expectation that schools and students will meet them.

Lessons and Predictions

The New York and North Carolina cases illustrate three critical issues that litigants and lawmakers would be wise to consider as standards-based litigation moves forward.

1) Courts are not particularly good at governing schools. Despite the integral role that the Supreme Court’s Brown v. Board of Education decision played in ensuring equal educational opportunity, courts are notoriously bad at developing and implementing education policy. They are structurally ill-equipped to make the sometimes delicate policy trade-offs incident to the school finance enterprise. Formal litigation, designed to resolve disputes in an adversarial manner, was never meant to serve as a dispassionate, thoughtful, deliberate forum for considering and weighing competing policy and funding objectives and goals. The adversarial setting is not conducive to generating the political consensus necessary to carry out policy decisions. This is not to say that the legislative process is perfect. Clearly, it is not. But despite its defects, the political process remains comparatively better structured than courts to set school finance policy.

Consider New Jersey’s three-decade-long saga with school finance litigation. Despite many judicial victories and an increase in school spending to among the nation’s highest, student achievement in urban areas continues to lag. The main by-product has been to breed resentment in the many suburbs that must endure the higher property taxes needed to finance the judges’ orders. This resentment boiled over when voters denied Governor Jim Florio a second term in office after he pushed for a tax increase incident to the court decisions.

The most enforceable thing a court can do is order a state to spend more money. What courts cannot do is ensure that the money is spent effectively. Courts cannot make a district improve its teacher training, step up its recruitment efforts, or eliminate waste and mismanagement in the central office. Many judges assume that extra funding alone will result in higher achievement. But giving more money to a system rife with patronage, corruption, and mediocrity is unlikely to stimulate meaningful change. The research literature is clear that money can improve achievement only if it is used well. And there is the rub.  The overwhelming majority of judges (and their clerks) are not trained as policy analysts, and school finance litigation frequently forces them into unfamiliar technical and policy terrain. For example, in the latest chapter of the decade-long DeRolph litigation, the Ohio Supreme Court all but admitted that its understanding of the state’s complicated school funding formula in a recent decision was flawed.

Another worrisome aspect of courts’ involvement in education policymaking is their seeming inability to disengage from judicial supervision once begun. The nation’s experience with school desegregation aptly illustrates this point. Almost 50 years have passed since the Brown decision, yet federal courts remain embroiled in many desegregation plans. As litigants continue to squabble over what it means for a school district to be “unitary” or “fully integrated,” the direct and indirect costs associated with school desegregation plans mount. Analogous battles over school finance issues will likely become just as contentious and prolonged.

2) Litigation lets legislators and policymakers off the hook. School finance litigation buffers elected officials and lawmakers from their responsibility to improve schools. By deploying standards and assessments in a manner that recasts school finance questions as legal questions, litigants seek to extract from courts and judges what they cannot get from legislators and governors. In the process, they threaten to cost state governments dearly and further erode lawmakers’ discretion over education policymaking and budgets. Many governors and lawmakers are displeased to find that a school finance court decision has blown a multimillion-dollar hole in a state’s carefully crafted, long-negotiated budget.

Paradoxically, some lawmakers welcome the judicial intrusion and seize upon a chance to point to judges and courts as the culprits when taxes must be increased to comply with school finance decisions. Some lawmakers-especially those who believe that schools merit more funding but are wary of a potential antitax backlash-relish judges’ taking the political heat, even at the cost of giving up some of their own legislative authority. However, such a consequence raises the twin specter of an increasingly politicized judiciary and an increasingly legalized legislature. Both results place additional stress on our traditional notions of separation of powers and the proper structure of government.

3) Vested interests adapt quickly to a changing policy milieu. The initial resistance of school boards and teacher unions toward standards and accountability was easy to predict. Less predictable was their ability to conscript the standards movement in the service of school finance litigation. Bolder still is the way activists managed to transform classroom “failure” into courtroom success and additional taxpayer dollars. Education is a highly labor-intensive activity and, as a result, labor costs consume the bulk of most school district budgets. Thus, victories in court for increased spending on education invariably inure to the benefit of school administrators and teacher unions. Of course, the carcasses of past reform endeavors coopted by the education establishment litter the landscape of education policy. During the past few decades alone, efforts to reform teacher training and selection, to introduce “site-based management” and merit-based compensation programs have largely succumbed to inertia (and worse) and failed to achieve their goals. Such a landscape is a testimonial to the phenomenal ability of organized interest groups to transform well-meaning reforms into vehicles for little more than additional resources, control, and retention of the status quo.

Those seeking more funding for schools that struggle to deliver acceptable educational services find many judges far more receptive than lawmakers to their claims. Reflexive pleas to lawmakers for increased resources are beginning to wear thin on legislators and governors attuned to a constituency anxious to see some reliable, clear returns on their investment. Taxpayer revolts flare up with increasing regularity across the country. Big-city school districts such as New York and Chicago do not dare try to increase education spending through appeals to the ballot box. Instead, they must turn to the more expensive capital markets by floating school bonds. In such a political environment, it is understandable why those seeking additional resources for public schools are eager to steer clear of legislatures and the political process. Courts provide school finance activists an alternative to increasingly skeptical lawmakers and a more demanding political marketplace.

Risks and Rewards

Despite notable successes, school finance activists committed to such a judicial strategy should recall a history replete with profound resistance to judicial intrusions into core education policymaking. Just as various interest groups within the education field have demonstrated, once again, their instincts for self-preservation and further entrenchment within an ever-changing policy environment, competing institutions are capable of similar adaptability. Even successful lawsuits typically fail to achieve litigants’ goals unless they are backed by substantial public and political support.

Recent history evidences this point. With all due respect to the Brown litigation’s seminal accomplishments (and they are both real and plentiful), after almost 50 years of school desegregation litigation, public schools today remain largely segregated by income and race. Moreover, litigation seeking to close gaps in per-pupil spending levels has achieved only marginal success nationwide. One clear consequence has been a shift of funding and control from the local level to the state level. This degree of centralization makes it easier to redistribute school resources from wealthy to poor districts. However, to the extent that this shift in control degrades the bond between local communities and their public schools, it is possible that net public support for public schools might decline. This, of course, will reduce the amount of funding available for redistribution.

The taxpayer revolt in California, prompted partly by that state supreme court’s school finance decision in the famous Serrano case, illustrates how resistance to court decisions can severely blunt the effectiveness of a judicial strategy. Before the successful school finance lawsuit, per-pupil spending in California was among the nation’s highest. Following the Serrano decision, per-pupil spending in California fell to among the nation’s lowest. Among the many factors contributing to this decline were a dramatic shift from local to state resources for school funding and a revolt among California taxpayers. Clearly, a judicial strategy that cleverly seeks to leverage education standards and assessments to bolster school finance lawsuits presents risks of its own.

Michael Heise is a professor of law at Case Western Reserve University.

Illustration by Craig Frazier.

The principle that social interventions ought to be evaluated has a long pedigree. Eager readers of the Muquadimah know that Ibn Khaldun considered competing explanations for the success of Arab regimes in the 13th century. In the 19th century, Florence Nightingale reproved the English Parliament for failing to weigh seriously the consequences of making changes to social programs, charging that “without an inquiry after results, past or present, it is all experiment, see-saw, doctrinaire, a shuttlecock between two battledores.” A 21st-century Nightingale could testify at a congressional hearing with only slight revisions. After a 1995 review of evidence on the performance of federal employment and training programs, the General Accounting Office concluded that “most federal agencies do not know if their programs are working.”

Nevertheless, the 20th century brought great progress in the theory and methods of evaluation as well as more understanding of its necessity. The most promising innovation was the randomized field trial. Such experiments randomly “treat” individuals and even whole institutions, such as hospitals or schools, with different interventions in order to learn which work better. The random allocation ensures that the two groups being compared are not different in ways that would influence their response to a particular treatment. For instance, suppose you wanted to test the quality of a new reading curriculum. You might select some students to be taught using the new curriculum and compare their progress with that of students who stayed with the old curriculum. But the students chosen to receive the new curriculum might differ in a way that influences their academic progress-they might, for instance, have had better reading teachers in the past, be more motivated, or have access to more educational resources at home. Randomly allocating students or multiple classrooms or multiple schools to the new reading program and to control conditions or to an alternative program eliminates the possibility that the two groups will differ in a systematic way and thus compromise the results.

Randomized field trials are a sturdy method of generating defensible evidence about the relative effectiveness of various interventions; nonrandomized trials do so, at times, but unpredictably. For instance, in the Salk polio vaccine studies of the 1950s, randomized trials that were mounted in some states produced estimates of the vaccine’s effect on polio that were appreciably greater than estimates from a parallel series of nonrandomized trials. The factors that might have led to this difference are still not well understood. After the Salk trials, controversy erupted over the use of oxygen-enrichment therapy for premature infants. Early nonrandomized studies suggested that the infant death rate was reduced significantly by the oxygen-enrichment therapy. Subsequent randomized clinical trials helped to reveal that enriched oxygen environments for premature infants caused blindness and did not decrease infant mortality.

Yet this powerful technique of discovering what works has been slow to come to the field of education. Consider the widespread adoption of “whole school reform” models such as “Success for All,” “Accelerated Schools,” and “Expeditionary Learning.” The promise of whole-school reform was to install standardized, high-performing, research-tested curricula in school after school, instead of forcing each school and school district to be its own curriculum developer. Much energy and expense has been devoted to implementing these prepackaged school-reform programs in mainly low-income school districts. Nevertheless, in 1999 the American Institutes of Research reviewed all the studies done on 24 reform models and found that 5 of the programs had no evidence beyond anecdote and personal testimonials to support their claims of raising achievement. The other 19 programs were subject to about 116 independent studies of their effectiveness. As far as can be determined from appendices in the report, only one of these studies, involving a test of a specialized instructional strategy in the context of the Paideia program, depended on a randomized trial.

Photograph by Steve Cole/Photodisc.

Inventive Research

Before chastising education reformers, however, it is crucial to recognize how young an innovation the randomized field trial truly is. Even in fields where the use of randomized trials is now standard practice, such as medical research, their adoption was fairly recent. The earliest randomized trials, to test the effects of a diphtheria serum, were undertaken by the Danes in 1900. But the American and European research communities didn’t regard randomized field trials as essential until the 1950s, when Jonas Salk discovered the polio vaccine and Sen. Estes Kefauver held hearings on the testing of thalidomide and other drugs. Now all pharmaceuticals must be subjected to rigorous clinical trials before being made available to the public. Research on industrial materials and chemical processes now depends in some measure on such trials as well.

Not just education but the social sciences in general lag behind the hard sciences in the use of randomized trials. Started in 1993, the international Cochrane Collaboration’s library of existing randomized and possibly randomized trials on the effects of health-care interventions contains 250,000 entries. By contrast, a sister effort, the international Campbell Collaboration’s Social, Psychological, Educational, and Criminological Trials Registry, begun in 1999, includes about 10,000 randomized and possibly randomized trials. More randomized trials will doubtlessly be unearthed as Campbell Collaboration researchers, mostly volunteers, continue to comb through the professional literature. But even if the number of trials were to increase five-fold, the social and behavioral sciences would remain relatively bereft of precise experimentation.

Two fundamental problems plague the use of randomized trials in the social sciences. First, randomized trials can be an enormous undertaking; testing the effects of a job-training program or a new work incentive in welfare is not the same as giving one set of patients a new drug and another set a placebo. What measurements to use (future earnings? children’s health and welfare? happiness?) and how to isolate the effects of a given program versus all the other influences the world presents remain vexing questions in the design of randomized trials. In education, any number of factors can influence the outcome of a randomized trial. A new curriculum might be highly effective, but teachers may not be trained to use it properly. In order to obtain reliable results, policies must be implemented as they were designed to be. Medical researchers face similar issues, but not necessarily to the same degree as social scientists. The second problem is funding. Research on pharmaceuticals is funded by private firms with one goal in mind: bringing profitable drugs to market. They can pass their research costs onto consumers. Social scientists, by contrast, are most often testing the effects of programs funded by government or by philanthropic organizations.

Not until the 1960s, when Congress mandated that the newly created Head Start program be evaluated, was high-quality research performed in education. During this period, the Ypsilanti (Michigan) Preschool Demonstration Project was distinctive in using a randomized trial to estimate the High/Scope program’s effect on children’s achievement. But few other preschool programs were evaluated using randomized trials in the ’60s; the Cochrane Collaboration database lists only six trials. The record hasn’t been much better in the decades since. A hand search of every article in every issue of the American Educational Research Journal since its inception revealed that of about 1,200 articles, only 35 concern randomized trials in math or science education (see Figure 1). Moreover, there was no obvious increase in their number during the 1964 to 1998 period. This is very embarrassing; the education community is not doing much to generate solid evidence on what interventions work.

Turning the Tide

There are reasons to be optimistic, though. In the current data-driven reform environment, more and more education policymakers are asking for reliable evidence of the effectiveness of new and existing policies and programs. At the same time, researchers have shown that high-quality randomized trials can be mounted in the education field. Some of the best trials lie at the intersection of education, social services, and juvenile-justice systems. Anthony Petrosino’s systematic reviews of randomized trials on the “Scared Straight” program, which involves prisoners lecturing at-risk youth about the consequences of crime, showed that its effects are at worst negative, at best negligible. Randomized trials on the D.A.R.E. program, which encourages students to stay away from drugs, showed the program was worthless in its mature multimillion-dollar form, all the while enjoying mostly uncritical support from police, teachers, and journalists. By 2001, D.A.R.E. sponsors and staffs had decided to modify the program in light of the findings.

Perhaps the most promising sign is that two of today’s liveliest debates-over the merits of class-size reduction and school choice-are informed by results from well-run randomized trials. Tennessee’s STAR experiment, which reduced class sizes in the early grades, established that a substantial reduction in class size yields significant gains in achievement. Independent reanalyses of the data by Frederick Mosteller, Alan Krueger, and others helped to verify the study’s original conclusions. These findings were largely ignored for five or more years after the results were published in the American Education Research Journal and assorted trade journals or reports. But by the late 1990s, at least a dozen state governors had built their education reform platforms around the idea, and it had become a federal priority under the Clinton administration. The STAR experiment shows how a single randomized field trial can begin to clarify the effect of a particular intervention against a backdrop of many nonrandomized trials. However, quasi-experimental evidence from Connecticut and other locations suggests that the Tennessee effects may not extend to all situations.

Findings from randomized trials of the privately funded voucher programs in New York City, Washington, D.C., and Dayton, Ohio, have drawn considerable press attention, and in spring 2002 they were mentioned in oral arguments before the U.S. Supreme Court. All these data have a way of elevating the discussion immeasurably. Once it is possible to estimate reliably the gains elicited by an intervention, the debate becomes more sophisticated and grounded than just Nightingale’s “shuttlecock between two battledores.”

The Bush administration’s current focus on reading was informed by a series of studies conducted under the aegis of the National Institutes of Child Health and Human Development. The work of Jack Fletcher, Barbara Foorman, and others depends on randomized trials, high-end nonrandomized trials, and scientifically conscientious looking around. The vicious debate over the merits of various methods of teaching reading was essentially settled when randomized trials showed that phonetic understanding was a necessary but not sufficient component of learning to read.

The U.S. Department of Education is increasingly recognizing the value of randomized trials. From 1995 to 1997 its Planning and Evaluation Service awarded 51 contracts to study the effects of federally sponsored programs, of which 5 involved randomized field trials on programs like Upward Bound, the Even Start Family Literacy Program, and the School Dropout Demonstration Assistance Program. These involve large commitments of personnel and funds over the course of several years. In 1996 the total amount awarded in contracts was about $18.6 million, of which $1.4 million was devoted to these randomized trials. This amount may seem small, but the trials are multiyear. Thus the total commitment from 1991 to 1995 for the School Dropout Demonstration Assistance Program, involving a randomized trial in each of 16 sites, was $7.3 million. For Upward Bound trials, which involved 67 sites, the commitment exceeded $5.4 million from 1992 to 1996.

Many private foundations in the United States have no explicit policy on the evaluation of projects that they sponsor. As a consequence, relatively few foundations have funded randomized trials that generate evidence about whether the programs that they support actually work. Nonetheless, a few foundations seem admirable in subsidizing field tests that 1) a national, state, or local government is unable or unwilling to mount; and 2) build on government investments in randomized field trials to generate new knowledge. In the education arena, the Rockefeller Foundation’s support of randomized trials on the Minority Female Single Parent Program and of the program itself was remarkable. The William T. Grant Foundation has supported trials on mentoring, nurse visitation programs, and New Chance, a teen-parent demonstration program. The Smith Richardson Foundation has supported the previously mentioned randomized trials on voucher-based school choice as well as Krueger’s reanalysis of data from the Tennessee class-size trials.

However, foundations are still more willing to fund randomized experiments in medicine than in education. The Gates Foundation, for instance, funded by the multibillionaire founder of Microsoft, has admirably committed $50 million for a carefully designed randomized study of an AIDS prevention initiative in Africa. At the same time, none of the foundation’s investments in education, though similar in scale, use this powerful scientific technique. Given the foundation’s commitment to scientific research, the policy will undoubtedly evolve during the coming years.

The School as Subject

Most randomized field trials involve measuring the effects of an intervention on individual research participants. But in the education field especially, society is interested not only in individual effects but in the effects on schools and school districts of policy changes. A new kind of trial is randomly assigning whole organizations, including schools and school districts, to alternatives in order to test their effectiveness. This sounds impossible to some people, but there are more than a few good precedents. They anticipate the way in which trials on “whole school reform” programs, among other schoolwide or districtwide interventions, might be run.

In the health education arena, for example, researchers have contributed substantially to understanding when and how to run randomized trials using schools as the units of allocation and analysis to test schoolwide risk-reduction programs. Entire hospitals have been randomized in tests of ways to educate staff, including physicians, and to change the handling of certain illnesses. Housing projects and factories have been the units in trials on AIDS prevention initiatives that had education as a major component. The Rockefeller Foundation is sponsoring trials that involve randomizing housing projects to test the effects of integrated programs to enhance social, economic, and other kinds of capital in the projects.

The Campbell Collaboration’s registry contains at least 50 entries on analogous trials in which conventional measures of academic achievement were the main outcome variable. Classrooms or schools have been used by Len Jason and his colleagues on tests of programs for transfer students, in Leanard Bickman’s admirable but aborted trials on teacher-incentive programs in Tennessee, and in Sheppard Kellam’s Baltimore studies on mental health and children’s achievement. At least one sizable trial has been mounted to test a violence prevention program based on a set of elementary schools that were randomly assigned to the program and to a control condition. Thomas Cook and his colleagues set a precedent for testing a schoolwide reform program with his trials on James Comer’s vaunted School Development Program. Two of these studies succeeded in randomly allocating eligible and willing schools to the Comer program and to control conditions and in producing unbiased estimates of the program’s effect. The Chicago trial provided evidence that the program affected perceptions of school and academic climate and had a small effect on academic achievement over the period covered by the study. In Prince George’s County, Maryland, no effects on beliefs or performance could be detected at the study’s completion.

Dissemination

A rigorous, well-run randomized trial is useless if its results aren’t made accessible to policymakers and the public. Two major efforts are under way to fill the gap between research and practice. In the health-care arena, the Cochrane Collaboration has created an international system to identify the best trials and to carry out an exquisitely conscientious review of them. Trustworthy studies are identified as such, while untrustworthy ones, despite their bright lights, are not. Since 1993 the Cochrane Collaboration has produced about 1,000 systematic reviews of studies on the effects of diverse health-care interventions. Hit the button on the web site, ask about your illness, and find out what the reviews say based on good trials. Roughly 2,000 people from 40 countries are involved in this effort. It is a good sign of people’s interest in evidence about what works in this arena.

In the late 1990s, the Cochrane people debated whether to enlarge the group’s scope. The conversations led to the creation of the international Campbell Collaboration, the aim of which is to produce systematic reviews of high-quality studies of the effects of new approaches to education, crime and justice, and social welfare. The Campbell Collaboration focuses first on randomized trials; its secondary focus is nonrandomized trials. This organization transcends disciplinary boundaries and geopolitical limits and promises enhanced attention to what works and what does not. Both collaborations have depended heavily on voluntary efforts in their development, a good sign of common interest in their aims. Moreover, collaborators hail from different disciplines. For education researchers, this makes it easier to learn about studies of crime interventions in which education outcomes were measured.

Randomized trials will only become more frequent in the education arena as time goes on. Today’s reform environment demands hard evidence on which programs and policies are effective at raising student achievement. So far, randomized trials provide the best way of obtaining such evidence, and their techniques become more sophisticated by the day. The question is whether these results will be used to effect change or be held hostage to the ideological posturing that so often substitutes for evidence in the education world.

Robert Boruch is a professor of education in the Graduate School of Education and of statistics in the Wharton School of Business at the University of Pennsylvania.

Illustration by Greg Clarke.

The No Child Left Behind Act, signed by President Bush in January 2002, builds squarely on the framework of standards-based reform established in Goals 2000 and the 1994 reauthorization of the Elementary and Secondary Education Act (ESEA). These two pieces of legislation, proposed by President Clinton, forged a new federal-state partnership to implement in every state an accountability system based on challenging standards and aligned assessments for all students. They also targeted federal resources to high poverty schools and provided states, districts, and schools with greater flexibility in the use of federal resources in order to achieve results.

While there are substantial continuities between the Clinton and Bush reforms, there are also some important differences that will bear significantly on the implementation challenges now facing states and the federal government. The 1994 legislation recognized that states had been leading the education reform movement for more than a decade, and it sought to bring the federal government into line with the standards-based reforms being formulated at the state level. Together Goals 2000 and the 1994 ESEA reauthorization provided a broad and flexible framework for state action, and deliberately placed considerable trust in states to work out the details themselves.

In contrast, the No Child Left Behind Act reflects significant impatience in Washington with the pace of state-led improvement and, in particular, with the slow pace at which states have instituted tough accountability systems. The legislation contains new and highly prescriptive testing and accountability requirements for states. It requires more testing and more frequent testing, sets a 12-year timetable by which every state and every school must bring every student and every demographic subgroup of students up to proficiency, and spells out in more detail than previously the consequences for schools that fail to make adequate progress toward this goal.

These provisions substantially shift the historic balance of federal vs. state control over education. They combine new federal dictates over the timing of improvement with continued state control over the standards for improvement, perhaps providing perverse incentives for states to lower the academic standards they recently adopted. In addition, most analysts believe that the new definition of “adequate yearly progress” included in the statute will result in a substantial proportion of the schools in each state immediately being identified as failing, due as much to the statistical properties of the definition as to the quality of the schools in question.

Implementation of the new law will therefore pose significant challenges to the states and the federal government. States will be looking for ways to meet new federal requirements without seriously disrupting their own standards-based reforms. The U.S. Department of Education will be working to ensure full compliance with the new requirements, but realistically must also provide states with enough flexibility to adapt the requirements to the states’ varying circumstances.

Requirements for states to implement systems of standards, assessments, and accountability have been the central feature of federal elementary and secondary education programs since 1994. They have also been the most politically difficult to craft and implement. Their successful implementation this time will depend on the willingness and ability of federal and state officials to negotiate a complex set of technical, political, and organizational challenges-and a good deal of luck.

The actions that state and federal officials take to address these requirements will determine whether they provide the right pressure to drive needed changes in state and local practice, prove to be utterly unworkable, or are relegated to the margins of state and local improvement strategies. The experience of implementing the 1994 requirements can shed some light on the choices facing federal and state officials this time around.

The Limits of Enforcement

It is often said that the federal government has little power when it comes to education reform because it kicks in less than 10 percent of the total cost of K-12 education. The 1994 reforms belie this notion; in fact, federal legislation can move the states quite far, even if their actions don’t all comply with the letter of the law. In 1993, at the onset of the Clinton administration, only a handful of states were developing standards and aligned assessments and preparing to use them as the cornerstone of their education reform strategy. Now every state is organizing its K-12 system around standards-based reform. The point here is simple but important: much of the federal impact lies in the overall direction it provides; it is not solely a result of the specific strings tied to federal funds. Significant changes in existing federal programs frame the terms of the deliberations that go on in every state and community. They mobilize supporters and create an expectation that states will act in a manner consistent with the new law.

Still, that doesn’t mean states will walk in lockstep with federal mandates. States won’t implement requirements that are unworkable or meet deadlines that can’t be met, no matter what the law says. The 1994 law required states to establish content and performance standards (the cut scores that indicate different levels of mastery on the assessments) in reading and math by the 1997-98 school year, and final assessments aligned with the standards by the 2000-01 school year. While most states met the deadline for content standards, almost none met the deadline for performance standards. States found it almost impossible to develop, define, and describe performance standards in the absence of the assessments that made them real and concrete. As a result, the Department of Education agreed to waive the deadline for performance standards.

This was the right decision, but it wasn’t without consequence. Freely granted waivers contributed to a belief that the department would not strictly enforce any of the law’s provisions and may therefore have undermined state compliance over time.

The 2000-01 deadline for having a testing system up and running was workable if a state started development work soon after the law was enacted. However, many states delayed the development of assessments for several years. Once that delay occurred, and once a state began good-faith efforts to develop the assessments, there was little either the state or the federal government could do to speed up the process. Creating and field-testing questions, developing scoring procedures, and conducting validity and reliability studies all take time. Moreover, full compliance required significant changes in test design, administration, and reporting practices. States had to shift from norm-referenced to standards-based assessments and to end longstanding practices of excluding students with disabilities and students with limited English proficiency from the state testing, reporting, and accountability programs. As the deadline approached, a number of states clearly weren’t going to meet it despite working in good faith to complete the development process. No sanctions could speed the process at that point. The department’s response was to waive the statutory deadline and hold the state to its planned timetable. Some view these decisions as evidence of lax enforcement, but I see it as evidence of the limits of the department’s enforcement ability.

These limits will become apparent again soon because some states will not be able to meet the deadlines in the new law. For example, the results from tests administered this spring are to form the baseline for defining adequate yearly progress. However, those states that still do not have final assessments in place or that plan necessary changes in their current assessments will be unable to use this year’s results for that purpose, since they will be using different tests in a year or so.

Knowledge Counts

States failed to meet some requirements not for lack of trying, but because they didn’t have a clear sense of how to do certain things well. The best example is their setting of standards for school improvement-known in the law as “adequate yearly progress.” States vary widely in the annual progress-in terms of increases in test scores-they expect from schools. As a result, they also vary widely in the percentage of schools identified as “needing improvement.” Texas and North Carolina labeled only 1 percent and 5 percent, respectively, of their Title I schools as “needing improvement.” Kentucky and Arkansas, by contrast, applied that label to 71 and 64 percent, respectively, of their Title I schools.

One of the reasons for this is that states, and the education community overall, do not have a clear, research-based idea of how to effectively set performance or progress targets for individual schools or school districts. Some states set absolute standards for schools, others looked at improvement over time, and others used a combination of the two. Few states or educators had a clear understanding of the behaviors they were rewarding or the incentives they were providing as a result of the design choices they made. And no state had sufficient research or experience to guide it in understanding the rate of progress it could reasonably or even ideally expect a school to make.

By tightly prescribing the definition of adequate yearly progress that states must adopt, the new law eliminates most of the difficult design decisions states must make. However, states and the U.S. Department of Education must together confront substantial implementation problems in this area, because the new requirements are fraught with technical complications, will lead to dramatic increases in the number of schools identified as failing, set improvement targets that many see as unrealistic, and are no more grounded in research or experience than the standards set by most states under the old law.

Going Their Own Way

While state education agencies are responsible for implementing federal programs, in many states the governor and the legislature, not the chief state school officer and the state education agency, are in charge of testing and accountability policy. In general, legislators and governors don’t pay attention to Title I requirements; they may not even be aware of their existence. While the relevant aspects of the No Child Left Behind Act generated a fair amount of media attention, few state policymakers will give them much thought six months to a year from now-let alone by 2005 and beyond, when the new testing requirements must be implemented. Odds are that governors and legislatures in most states will continue to think they have a free hand on these issues.

For instance, the Department of Education found that the assessment systems in California, Wisconsin, West Virginia, and Alabama were substantially out of compliance with the requirements of the 1994 legislation. In each of these cases the main problem was state-level decisions by the legislatures to mandate norm-referenced tests that were not aligned with state standards. Chief state school officers in those states made clear that the legislature acted without much knowledge of or attention to the Title I requirements. Now consider that during the debate over the No Child Left Behind Act, the National Conference of State Legislatures sent members of Congress a letter criticizing the testing requirements as “an egregious example of a top-down, one-size-fits-all federal reform.” Clearly, given the deep ambivalence about the federal role in education among many elected state officials, no one should assume that state compliance will be voluntary or automatic.

Hints of future problems have already surfaced. In Michigan, for instance, the new chief state school officer recently replaced the test-focused accountability system with a new grading system that relies on a broader set of measures of school quality, including family involvement, the quality of professional development, attendance, and dropout rates, among others. Schools will receive their first “grades” in 2003, but sanctions cannot be applied until 2005. Almost none of the accountability features appear to comport with the requirements of the No Child Left Behind Act, the basic provisions of which have been known since the summer of 2001. The federal law permits multiple measures as long as those added to the state assessment program don’t reduce or change the schools identified for improvement. Interventions in low-performing schools must begin immediately, at least for those already identified under the 1994 provisions. Michigan has already identified some 80 percent of its Title I schools as needing improvement, so widespread interventions should be occurring now, not delayed until 2005. In brief, it appears that, at least in Michigan, the state’s accountability system and the Title I accountability system operate in parallel universes.

Lax Track Record

The Department of Education does not have a strong track record of monitoring compliance in ESEA programs, a problem that has spanned administrations of both parties. The department is widely viewed as impotent, since no one believes that it will ultimately withhold funds from states or local districts.

The weakness of this longstanding track record was surely compounded by both the intent of Goals 2000 and the politics surrounding its enactment and implementation. Goals 2000 was intended to help states jump-start standards-based reform, while deliberately providing states with a great deal of flexibility in the design of reform strategies and the use of federal funds. While the secretary of education was required to review and approve each state’s education reform plan, we were keenly aware that Goals 2000 provided less than 1 percent of total state education expenditures and worked hard to ensure that the peer review of a state’s plans recognized the limits this imposed. Many states understood and appreciated this approach, but others saw it as another indication that the department lacked the will for tough-minded compliance monitoring.

The political assault on Goals 2000 in a number of states (the governors of Virginia, California, Alabama, and New Hampshire refused to accept Goals 2000 funds because of the “federal intrusion” and “strings” that accompanied it), coupled with simultaneous efforts in Congress to abolish both Goals 2000 and the Department of Education, also contributed to a widespread view that tough enforcement would be a particularly hazardous course of action for the department to pursue. Ironically, many of those who pressured the Education Department to give states a free rein during the 1990s have now championed a much more intrusive set of requirements a decade later.

Laboratories of Democracy

Still, in its effort to ensure compliance the Department of Education must avoid choking off creativity. As states work to address these issues, they need the flexibility to take advantage of emerging solutions and opportunities and to address problems not foreseen in, or perhaps created by, the legislation. Here are a few areas where the department should not let compliance get in the way of quality and innovation:

• The requirements for adequate yearly progress and school improvement apply equally to high schools and elementary schools, yet it is not clear that they make as much sense at the secondary level. For example, states with high-stakes high-school graduation requirements must find effective ways to intervene in high schools with high failure and/or dropout rates, even if the percentage of students passing the test increases significantly each year. A school where the pass rate goes from 65 to 75 percent in a year may be making exceptional progress, but it is hardly adequate if a quarter of the students can’t meet the graduation requirements. Consequently, many states will need to find different yardsticks for judging the performance of high schools and more powerful and swift intervention strategies than the graduated series of steps provided for in the statute.

• Online assessment appears to offer many advantages for states, teachers, and students. It holds the promise of immediate results and feedback so that the tests can be used to improve teaching and learning for the students who take them. They can be administered at different times, enabling students to take them when they are ready to demonstrate that they have met the standards, rather than on a single “one-size-fits-all” testing date. They may be customized for individual students, enabling them to take fewer questions that are better geared to their level of performance, potentially increasing both the efficiency and the diagnostic value of the tests. Yet Education Week reported in February that department officials have indicated that Idaho’s approach to online testing may not meet Title I requirements. How this pans out will depend on the particulars of the Idaho situation, but in general the Title I requirements must not become a barrier to the necessary development and experimentation of online testing in the states.

• The annual testing in grades 3 through 8 required by the federal law will make it possible for states and districts to use “value added” approaches to measuring the performance of schools. It isn’t clear whether this approach is necessarily superior to the cohort approach the law builds in to the definition of adequate yearly progress, but it certainly deserves serious consideration. In any event, value-added analysis would almost certainly identify as low performing a set of schools different from those the prescribed approach might identify, and thus it may not be permissible under the statute. The Title I requirements should not be a barrier to sorting out the most appropriate approaches to identifying low performing schools.

The bottom line here is simple. We don’t yet know all that we must in order to translate the principles guiding No Child Left Behind into effective policy. Yet the specific requirements appear to leave some approaches off the table, even if they may turn out to be more promising. While the Education Department has a clear responsibility to ensure that every state complies with the new requirements, it also has a responsibility to help states find the most effective approaches to meet the overall purpose of improving achievement.

In the end, even 100 percent compliance will not be enough to bring about adequate gains in student achievement. Translating tougher accountability measures into large-scale achievement gains for all students will require substantial investments at the federal, state, and local levels. States and districts will need to recruit, prepare, and retain talented teachers and principals and support them with high-quality professional development, curriculum and instructional materials aligned with standards, and the tools to support data-based decisions. It will also require substantial investments to give students the opportunities to learn, including smaller classes, modern buildings, 21st-century technology, and extended learning opportunities through after-school and summer programs. The key here is that compliance is a means, not an end. The end is excellence.

Michael Cohen is a senior fellow at the Aspen Institute and the former assistant secretary for elementary and secondary education in the Clinton administration.