Stuyvesant. Boston Latin. Bronx Science. Thomas Jefferson. Lowell. Illinois Math and Science Academy. These are some of the highest-achieving high schools in the United States. In contrast to elite boarding and day schools such as Andover and Sidwell Friends, however, they are public. And unlike the comprehensive taxpayer-funded options in affluent suburbs such as Palo Alto and Winnetka, they don’t admit everyone who lives in their attendance area.

Sometimes called “exam schools,” these academically selective institutions have long been a part of the American secondary-education landscape. The schools are diverse in origin and purpose. No single catalyst describes why or how they began as or morphed into academically selective institutions. Some arose from a desire (among parents, superintendents, school boards, governors, legislators) to provide a self-contained, high-powered college-prep education for able youngsters in a community, region, or state. Others started through philanthropic ventures or as university initiatives. A number of them were products of the country’s efforts to desegregate—and integrate—its public-education system, prompted by court orders, civil rights enforcers and activists, or federal “magnet school” dollars.

Exam schools are sometimes controversial because “selectivity” is hard to reconcile with the mission of “public” education. Even school-choice advocates typically assert that, while families should be free to choose their children’s schools, schools have no business selecting their pupils. Other people are troubled by reports of insufficient “diversity” among the youngsters admitted to such schools.

With such criticisms in mind, we set out to explore this unique and little-understood sector of the education landscape. Wanting first to determine how many there are and where they are located, we also wondered whether the “exam school” could be a worthy response to the dilemma of how best to develop the talents of our nation’s high-performing and high-potential youth in a climate consumed with gap closing and leaving no child behind. Could the selective public high school play a larger role in educating our country’s high-achieving pupils?

Who Goes There?

Almost all the schools have far more applicants than they can accommodate. Nearly two-thirds of those surveyed accept fewer than half of their applicants. About one-quarter also reported rising numbers of applications in recent years, perhaps due to media attention, awards, school performance, population growth, and the closing of underperforming schools in the area. Respondents also noted changes in the composition of their applicant pools, mainly increases in the number who are female, Asian, or Hispanic. Several schools reported a decrease in the number of white applicants in recent years. Nearly all schools we surveyed engaged in earnest, wide-ranging outreach to expand or diversify their applicant pools. A few also engage in “affirmative action” within the selection process.

The schools’ actual admission criteria and procedures are interesting, variegated, and somewhat sensitive. Some school officials are uneasy about the practice of selectivity, given possible allegations of “elitism” and anxiety over pupil diversity. Still, most rely primarily on applicants’ prior school performance and scores on various tests.

Viewed as a whole, selective public high schools have a surprising demographic profile. Their overall student body is only slightly less poor than the universe of U.S. public school students. Some schools, we expected, would enroll many Asian American youngsters, but we were struck when they turned out to comprise 21 percent of the schools’ total enrollment, though they make up only 5 percent of students in all public high schools. More striking still: African Americans are also “overrepresented” in these schools, comprising 30 percent of enrollments versus 17 percent in the larger high-school population. Hispanic students are correspondingly underrepresented, but so are white youngsters. Individual exam schools often qualify as racially “imbalanced”: in nearly 70 percent of them, half or more of the students are of one race.

Inside the Schools

The schools we visited were serious, purposeful places: competitive but supportive, energized yet calm. Behavior problems (save for cheating and plagiarism) were minimal and students attended regularly, often even when ill. The kids wanted to be there, and were motivated to succeed. (Bear in mind that many of the schools seek such qualities in their applicants.)

In general, the schools structured their schedules in ways that facilitate in-depth learning and prepare students for the typical college schedule: staggered start times, eight-hour days, class periods of varying lengths, fewer class meeting days per week, and dedicated time for collaborative and independent research projects. Most classrooms we observed were alive, engaged places in which teachers appeared to have high expectations for their pupils and planned their instruction around the assumption that students can and want to learn.

Most schools offered Advanced Placement (AP) courses or the International Baccalaureate (IB) program. Several noted that they “only offer honors and AP courses.” A few schools noted that students do not take AP courses per se, either because they take actual college classes (at host colleges or through dual-enrollment arrangements) or because they earn college credit for advanced courses taught within the school itself.

We also came upon other kinds of specialized and advanced courses, in addition to or in lieu of AP and IB. Schools with a STEM focus or university affiliations, for example, reported an array of upper-level science and math courses that few ordinary high schools—even very large ones—could offer. Among them were Human Infectious Diseases, Chemical Pharmacology, Logic and Game Theory, and Vector Calculus.

There’s lots of homework but ample extracurricular opportunities, too. We encountered literary magazines, robotics competitions, sophisticated music and theater offerings, most of the usual clubs and organizations, plenty of field trips, and no dearth of sports—though champion football and basketball teams were rare!

Our site visits revealed faculties consisting mostly of intelligent, dedicated individuals, well grounded in their fields. Turnover was low. Most teachers belong to unions and are paid on the “contract scale,” but many receive additional compensation for longer days and extra duties. They tended to come early, stay late, and design complex assignments and lesson plans that may take as much time for them to formulate and grade as for their students to complete.

One assumption about selective public schools is that they have more and “better” teachers. It turns out, however, that their pupil-teacher ratio is actually a bit higher (17:1) than in all public high schools (15:1). (One likely reason: not much “special ed.”) The percentage with doctoral degrees is higher, too (11 vs. 1.5 percent), as is the percentage with master’s degrees (66 vs. 46 percent.) Nontrivial numbers of teachers also have experience in industry, science, and universities.

Nearly two-thirds of survey respondents indicated that teacher-hiring decisions are made at the school level. As for the criteria they employ in selecting faculty, of greatest importance are subject-matter knowledge, pedagogical knowledge and expertise, and the ability to engage adolescent learners. Many schools also seek proven classroom-management strategies, compatible teaching philosophies, technology prowess, and collegiality. Some require demonstration lessons and interviews by current teachers (and sometimes students). And some criteria are clearly aligned with the schools’ singular missions and student bodies (e.g., PhD in biology, training in AP instruction, ability to work with gifted pupils).

The schools’ principals hailed from various backgrounds. As a group, however, they exhibited traits that one would expect of leaders of successful high schools that in some cases are the pride of their communities and in every case are closely watched: extraordinarily dedicated and hard-working individuals who are also politically astute.

Governance and Finance

The schools are remarkably varied when it comes to history, mission, structure, and organizational arrangements. The oldest among them—New York’s Townsend Harris High School, Boston Latin School—have been around in one form or another for centuries, while half the schools for which we have such information are creations of the past two decades. With rare exceptions (mainly in Louisiana), however, the schools are not charters. Although they’re “schools of choice,” they are operated in more top-down fashion by districts, states, or sometimes universities rather than as freestanding and self-propelled institutions under their states’ charter laws.

We asked survey respondents about waivers and exemptions from the customary rules and regulations within which public schools operate. Because many of the schools on our list occupy distinctive niches within their local communities, districts, or states, we were also curious whether their teachers are fully subject to the provisions of collective-bargaining contracts. Most certainly are, but almost one in five is not (or not fully) subject to seniority-based staffing decisions.

A handful of responding schools said either that they are not required to hire teachers with state certification, or that other credentials (e.g., PhD in relevant field) preempt certification, at least for several years. In general, however, routine regulations and contract provisions prevail. We were struck by how few schools reported explicit freedom from them. Principals did say, however, that they could usually “work things out” as needed.

The schools vary widely in funding levels and other resources, from those that can barely make ends meet on per-pupil allotments that are lower than other high schools in the area to a few schools that amass large budgets from multiple sources and boast extraordinary technology and staffing. But all the schools we visited were worried about budget cuts associated with economic distress and pressure on state and local resources.

Leaders of these schools felt doubly vulnerable as attention—and resources—were concentrated on low-performing schools and students. (“Smart kids will do fine, regardless, and in any case are not today’s priority” was the undertone they picked up.) Many had become accustomed to having at least some extra resources, often for transportation or smaller classes. While some schools benefit from certain categorical funds (e.g., magnet dollars, STEM, or tech-voc dollars), many don’t qualify for other state and federal programs, such as Title I, bilingual education, and special education. Most engage in supplementary private fundraising to sustain resources for transportation, smaller classes, or other school features to which they and their students, parents, and teachers are accustomed.

Despite such challenges, the schools seem to enjoy levels of support that mitigate the budgetary distress and bolster their resilience. Most, for example, benefit—politically and in other ways, such as fundraising—from exceptionally devoted friends, sometimes in high places, including alums, local politicians, business and university leaders, even journalists. Many have ties with outside organizations, including universities, labs, and businesses, which bring expertise and some resources into the school, afford it some political protection, and supply it with venues for student internships and independent projects.

Some schools are also viewed as magnets for economic development and talent recruitment for their community or state. School-board members and district leaders believe that the presence of the school encourages middle- and upper-middle-class families to stay in town and stick with public education.

Perhaps most importantly, the schools are blessed with overwhelming advocacy from alumni and the parents of their students, many of whom feel that their children are receiving a private school–quality education at public expense. That parents strongly believe the schools provide safety (physical, emotional, intellectual), short- and long-term academic and career opportunities, and social benefits for their children will likely go a long way toward ensuring the survival of the schools, if not their expansion or replication.

The AP Quandary

Nearly every school on our list offers a host of AP courses and has a huge number of students enrolling in them (either by requirement or by choice) and racking up solid scores on the AP exams. At northern Virginia’s celebrated Thomas Jefferson High School for Science and Technology, for example, students take an average of seven AP tests—four are all but universal—and do extremely well, earning scores of 3 or better on a mind-blowing 98 percent of the 3,357 AP exams that they sat for in 2010. Here and at many (though not all) schools on our list, students compete—and are pressed by parents—to rack up as many AP credits as possible.

Yet today’s scramble for entry into top-tier colleges plus the premium placed (by multiple players) on taking and passing AP exams plus standardized-test-based accountability pressures emanating from government do not add up to an optimal environment for these high schools. Here they don’t raise standards as much as they standardize. They press on students, parents, and teachers in ways that are plausibly said to discourage experimentation, risk-taking, unconventional thinking, unique courses, and individualized research, as well as pedagogical creativity and curricular innovation.

We spoke with frustrated teachers and exasperated administrators, well aware that they’re riding the back of an AP tiger from which it’s hard to dismount, especially for a public school that must weigh the priorities of parents, taxpayers, and voters. We talked with highly motivated students, too, who were (as one young man put it) “exhausted” from carrying course loads that included as many as six AP classes a semester in pursuit of a high school transcript that would wow the admissions committees of elite universities.

Some school leaders are pushing back, encouraging teachers to develop challenging courses that don’t fit the AP mold, or offering college-level courses shorn of the AP label. But only a few—such as the statewide, residential Illinois Math and Science Academy—have succeeded in putting their own stamp on the entire curriculum and withstanding the AP tsunami.

Are Exam Schools Effective?

The selection criteria employed by these schools all but guarantee students who are likely to do well academically, which raises the question of whether the schools’ generally impressive outcomes are caused by what happens inside them—their standards, curricula, teachers, homework—or are largely a function of what the kids bring with them. The schools’ peer culture likely has some influence on their pupils, too, as do high teacher expectations. Much like private schools, which are more apt to trade on their reputations and college-placement records than on hard evidence of what students learn in their classrooms, the schools on our list generally don’t know—in any rigorous, formal sense—how much their students learn or how much difference the school itself makes. As one puzzled principal put it, “Do the kids do well because of us or in spite of us? We’re not sure.”

The schools themselves are only partly culpable, however. They’ve seldom been asked to justify themselves in terms of learning gains. They’re flooded with eager applicants, media attention, and accolades. They can proudly demonstrate intricate research projects, cases full of academic prizes, science-fair and robotics-competition ribbons, National Merit lists, and messages from grateful alums. But they have access to little “value-added” data. Nearly all the tests their pupils take show “mastery”—like earning a 5 on an AP exam or racking up a lofty SAT score—rather than serving as before-and-after assessments. And insofar as their states impose graduation tests as prerequisites for receiving diplomas, the passing score is generally a cinch for these students.

The research community has mostly ignored these schools, too. One recent study by Duke economist Atila Abdulkadiroglu and Joshua D. Angrist and Parag A. Pathak of MIT—the first of its kind, say the authors—set out to explore this territory. Using a sophisticated methodology to look for value-added effects (gauged by scores on state tests and SAT and AP exams) in six prominent “exam schools” in Boston and New York City, they didn’t find much to applaud:

Our results offer little evidence of an achievement gain for those admitted to an exam school…. In spite of their exposure to much higher-achieving peers and a more challenging curriculum, marginal students admitted to exam schools generally do no better on a variety of standardized tests.

A similar study by Roland Fryer and Will Dobbie was confined to the three oldest and most famous of New York’s “exam schools” and used similar methods. It found that “attending an exam school increases the rigor of high school courses taken and the probability that a student graduates with an advanced high school degree” but “has little impact on Scholastic Aptitude Test scores, college enrollment, or college graduation.”

These pioneering studies are sobering, albeit limited both by their focus on “marginal” students (those barely over and just under the schools’ entry-score cutoffs) and by their reliance on short-term measures of effectiveness. The schools’ effects on other kinds of outcomes and over the longer haul are simply unknown, as are their effects on youngsters whose exam scores were well above the cutoff. This is obviously a ripe area for further investigation and analysis, but today it’s legitimate to observe, even on the basis of this limited research, that the burden is shifting to the schools and their supporters to measure and make public whatever academic benefit they do bestow on their students versus what similar young people learn in other settings. The marketplace signals, however, are undeniable: far more youngsters want to attend these schools than they can accommodate. Many applicants go to exceptional lengths to prepare for the admissions gauntlet, which may well lead to more learning in earlier grades than the same youngsters might have absorbed without this incentive. And we also know that most of those who are admitted stick with it through graduation; an average graduation rate of 91 percent was reported by the schools responding to our survey.

Would America Benefit from More Exam Schools?

At a time when American education is striving to customize its offerings to students’ interests and needs, and to afford families more choices among schools and education programs, the market is pointing to the skimpy supply of schools of this kind. Moreover, if the best of such schools are hothouses for incubating a disproportionate share of tomorrow’s leaders in science, technology, entrepreneurship, and other sectors that bear on society’s long-term prosperity and well-being, we’d be better off as a country if we had more of them.

This challenge, however, goes far beyond the specialized world of selective high schools. It’s evident from multiple studies that our K–12 education system overall is doing a mediocre job of serving its “gifted and talented” youngsters and is paying too little attention to creating appealing and viable opportunities for advanced learning. What policymakers have seen as more urgent needs (for basic literacy, adequate teachers, sufficient skills to earn a living, for example) have generally prevailed. The argument for across-the-board talent development has been trumped by “closing the achievement gap” and focusing on test scores at the low end.

American education could and should be doing much more to help every youngster achieve all that he or she is capable of. A major push to strengthen the cultivation of future leaders is overdue, and any such push should include careful attention to the “whole school” model. Such institutions can develop a critical mass of instructional tools and equipment, financial resources, reputations, alumni/ae, and outside supporters that is hard to assemble for a smallish program within a comprehensive school. And the critical-mass effect is visible in the curriculum, too. Instead of isolated honors and AP classes, single-purpose schools can amass entire sequences at that level. They can also develop courses that go beyond AP offerings, do more with individual student projects, concentrate their counseling efforts on college placement, and muster teams of eager students (and teachers) for science competitions and the like.

Insofar as students benefit from peer effects in classrooms, corridors, and clubs, and insofar as being surrounded by other smart kids challenges these students (and wards off allegations of “nerdiness”), schools with overall cultures of high academic attainment are apt to yield more such benefits.

Finally, viewed as a community asset, having an entire school of this sort to show parents, colleges, employers, firms looking to relocate, real estate agents, and others can bring a kind of élan or appeal to a place that may also help with economic development, the retention of middle-class families, and more. It’s also a fact, however, that in times when resources are tight, communities and states are unlikely to hasten to create many more selective high schools, even where the reasons for doing so may be compelling.

Whether we deploy many more “whole schools” of this kind or opt mainly for specialized courses and programs within ordinary schools, the kinds of rigorous and advanced education that selective-admission schools seek to provide, and the youngsters that they serve, need to rise higher in our national consciousness and our policy priorities.

Chester E. Finn, Jr. is president of the Thomas B. Fordham Institute and senior fellow at Stanford University’s Hoover Institution. Jessica Hockett is an educational consultant specializing in differentiated instruction, curriculum design, and teacher professional development. This article is based on the authors’ forthcoming book, Exam Schools: Inside America’s Most Selective Public High Schools (Princeton University Press), a joint undertaking of the Hoover Institution’s Koret Task Force on K‒12 Education and the Thomas B. Fordham Institute.

The modernization of teacher evaluation systems, an increasingly common component of school reform efforts, promises to reveal new, systematic information about the performance of individual classroom teachers. Yet while states and districts race to design new systems, most discussion of how the information might be used has focused on traditional human resource–management tasks, namely, hiring, firing, and compensation. By contrast, very little is known about how the availability of new information, or the experience of being evaluated, might change teacher effort and effectiveness.

In the research reported here, we study one approach to teacher evaluation: practice-based assessment that relies on multiple, highly structured classroom observations conducted by experienced peer teachers and administrators. While this approach contrasts starkly with status quo “principal walk-through” styles of class observation, its use is on the rise in new and proposed evaluation systems in which rigorous classroom observation is often combined with other measures, such as teacher value-added based on student test scores.

Proponents of evaluation systems that include high-quality classroom observations point to their potential value for improving instruction (see “Capturing the Dimensions of Effective Teaching,” Features, Fall 2o12). Individualized, specific information about performance is especially scarce in the teaching profession, suggesting that a lack of information on how to improve could be a substantial barrier to individual improvement among teachers. Well-designed evaluation might fill that knowledge gap in several ways. First, teachers could gain information through the formal scoring and feedback routines of an evaluation program. Second, evaluation could encourage teachers to be generally more self-reflective, regardless of the evaluative criteria. Third, the evaluation process could create more opportunities for conversations with other teachers and administrators about effective practices.

In short, there are good reasons to expect that well-designed teacher-evaluation programs could have a direct and lasting effect on individual teacher performance. To our knowledge, however, ours is the first study to test this hypothesis directly. We study a sample of midcareer elementary and middle school teachers in the Cincinnati Public Schools, all of whom were evaluated in a yearlong program, based largely on classroom observation, sometime between the 2003–04 and 2009–10 school years. The specific school year of each teacher’s evaluation was determined years earlier by a district planning process. This policy-based assignment of when evaluation occurred permits a quasi-experimental analysis. We compare the achievement of individual teachers’ students before, during, and after the teacher’s evaluation year.

We find that teachers are more effective at raising student achievement during the school year when they are being evaluated than they were previously, and even more effective in the years after evaluation. A student instructed by a teacher after that teacher has been through the Cincinnati evaluation will score about 11 percent of a standard deviation (4.5 percentile points for a median student) higher in math than a similar student taught by the same teacher before the teacher was evaluated.

Our data do not allow us to identify the exact mechanisms driving these improvements. Nevertheless, the results contrast sharply with the view that the effectiveness of individual teachers is essentially fixed after the first few years on the job. Indeed, we find that postevaluation improvements in performance were largest for teachers whose performance was weakest prior to evaluation, suggesting that rigorous teacher evaluation may offer a new way to think about teacher professional development.

Evaluation in Cincinnati

The data for our analysis come from the Cincinnati Public Schools. In the 2000–01 school year, Cincinnati launched the Teacher Evaluation System (TES) in which teachers’ performance in and out of the classroom is assessed through classroom observations and a review of work products. During the yearlong TES process, teachers are typically observed in the classroom and scored four times: three times by an assigned peer evaluator—a high-performing, experienced teacher who previously taught in a different school in the district—and once by the principal or another school administrator. Teachers are informed of the week during which the first observation will occur, with all other observations unannounced. Owing mostly to cost, tenured teachers are typically evaluated only once every five years.

The evaluation measures dozens of specific skills and practices covering classroom management, instruction, content knowledge, and planning, among other topics. Evaluators use a scoring rubric based on Charlotte Danielson’s Enhancing Professional Practice: A Framework for Teaching, which describes performance of each skill and practice at four levels: “Distinguished,” “Proficient,” “Basic,” and “Unsatisfactory.” (See Table 1 for a sample standard.)

Both the peer evaluators and administrators complete an intensive TES training course and must accurately score videotaped teaching examples. After each classroom observation, peer evaluators and administrators provide written feedback to the teacher and meet with the teacher at least once to discuss the results. At the end of the evaluation school year, a final summative score in each of four domains of practice is calculated and presented to the evaluated teacher. Only these final scores carry explicit consequences. For beginning teachers (those evaluated in their first and fourth years), a poor evaluation could result in nonrenewal of their contract, while a successful evaluation is required before receiving tenure. For tenured teachers, evaluation scores determine eligibility for some promotions or additional tenure protection, or, in the case of very low scores, placement in a peer assistance program with a small risk of termination.

Despite the training and detailed rubric provided to evaluators, the TES program experiences some of the leniency bias typical of other teacher-evaluation programs. More than 90 percent of teachers receive final overall TES scores in the highest two categories. Leniency is much less frequent in the individual rubric items and individual observations. We hypothesize that this microlevel evaluation feedback is more important to lasting performance improvements than the final, overall TES scores.

Previous research has found that the scores produced by TES predict student achievement gains (see “Evaluating Teacher Effectiveness,” research, Summer 2011). Student math achievement was 0.09 standard deviations higher for teachers whose overall evaluation score was 1 standard deviation higher (the estimate for reading was 0.08). This relationship suggests that Cincinnati’s evaluation program provides feedback on teaching skills that are associated with larger gains in student achievement.

As mentioned above, teachers only undergo comprehensive evaluation periodically. All teachers newly hired by the district, regardless of experience, are evaluated during their first year working in Cincinnati schools. Teachers are also evaluated just prior to receiving tenure, typically their fourth year after being hired, and every fifth year after achieving tenure.

Teachers hired before the TES program began in 2000–01 were not initially evaluated until some years into the life of the program. Our analysis only includes these pre-TES hires: specifically, teachers hired by the district in the school years from 1993–94 through 1999–2000. We further focus, given available data, on those who were teaching 4th through 8th grade in the years 2003–04 through 2009–10. We limit our analysis to this sample of midcareer teachers for three reasons. First, for teachers hired before the new TES program began in 2000–01, the timing of their first TES review was determined largely by a “phase-in” schedule devised during the program’s planning stages. This schedule set the year of first evaluation based on a teacher’s year of hire, thus reducing the potential for bias that would arise if the timing of evaluation coincided with, for example, a favorable class assignment. Second, because the timing of evaluation was determined by year of hire, and not experience level, teachers in our sample were evaluated at different points in their careers. This allows us to measure the effect of evaluation on performance separate from any gains that come from increased experience. Third, the delay in first evaluation allows us to observe the achievement gains of these teachers’ students in classes the teachers taught before the TES assessment so that we can make before-and-after comparisons of the same teacher.

Additionally, our study focuses on math test scores in grades 4–8. For most other subjects and grades, student achievement measures are simply not available. Students are tested in reading, but empirical research frequently finds less teacher-driven variation in reading achievement than in math, and ultimately this is the case for the present analysis as well. While not the focus of our research, we briefly discuss reading results below.

Data provided by the Cincinnati Public Schools identify the year(s) in which a teacher was evaluated by TES, the dates when each observation occurred, and the scores. We combine these TES data with additional administrative data provided by the district that allow us to match teachers to students and student test scores. As we would expect, the 105 teachers in our analysis sample are a highly experienced group: 66.5 percent have 10 to 19 years of experience, compared to 29.3 percent for the rest of the district. Teachers in our analysis are also more likely to have a graduate degree and be certified by the National Board for Professional Teaching Standards, two characteristics correlated with experience.

Methodology

Our objective is to measure the impact of practice-based performance evaluation on teacher effectiveness. Simply comparing the test scores of students whose teachers are evaluated in a given year to the scores of other teachers’ students would produce misleading results because, among other methodological issues, less-experienced teachers are more likely to be evaluated than more-experienced teachers.

Instead, we compare the achievement of a teacher’s students during the year that she is evaluated to the achievement of the same teacher’s students in the years before and after the evaluation year. As a result, we effectively control for any characteristics of the teacher that do not change over time. In addition, we control for determinants of student achievement that may change over time, such as a teacher’s experience level, as well as for student characteristics, such as prior-year test scores, gender, racial/ethnic subgroup, special education classification, gifted classification, English proficiency classification, and whether the student was retained in the same grade.

Our approach will correctly measure the effect of evaluation on teacher effectiveness as long as the timing of a teacher’s evaluation is unrelated to any student characteristics that we have not controlled for in the analysis but that affect achievement growth. This key condition would be violated, for example, if during an evaluation year or in the years after, teachers were systematically assigned students who were better (or worse) in ways we cannot determine and control for using the available data. It would also be violated if evaluation coincided with a change in individual teacher performance unrelated to evaluation per se. Below, we discuss evidence that our results are not affected by these kinds of issues. We also find no evidence that teachers are systematically assigned students with better (or worse) observable characteristics in their evaluation year compared to prior and subsequent years.

Results

We find suggestive evidence that the effectiveness of individual teachers improves during the school year when they are evaluated. Specifically, the average teacher’s students score 0.05 standard deviations higher on end-of-year math tests during the evaluation year than in previous years, although this result is not consistently statistically significant across our different specifications.

These improvements persist and, in fact, increase in the years after evaluation (see Figure 1). We estimate that the average teacher’s students score 0.11 standard deviations higher in years after the teacher has undergone an evaluation compared to how her students scored in the years before her evaluation. To get a sense of the magnitude of this impact, consider two students taught by the same teacher in different years who both begin the year at the 50th percentile of math achievement. The student taught after the teacher went through the TES process would score about 4.5 percentile points higher at the end of the year than the student taught before the teacher went through the evaluation.

We also find evidence that the effects of going through evaluation in the TES system are not the same for all teachers. The improvement in teacher performance from before to after evaluation is larger for teachers who received relatively low TES scores, teachers whose TES scores improved the most during the TES year, and especially for teachers who were relatively ineffective in raising student test scores prior to TES. The fact that the effects were largest for teachers who, presumably, received more critical feedback and for those with the most room for improvement strengthens our confidence in the causal interpretation of the overall results.

Our findings remain similar when we make changes to our methodological choices, such as varying the way we control for teacher experience, not controlling for teacher experience, and not controlling for student characteristics. We also examine whether our results could be biased by a preexisting upward trend in each teacher’s performance unrelated to experience or evaluation, and find no evidence of such a trend. Finally, we find no evidence that our results reflect teacher turnover from school to school or from grade to grade that causes them not to appear in our data in later years (for example, by moving to a nontested grade or leaving the Cincinnati Public Schools).

In contrast to the results for math achievement, we do not find any evidence that being evaluated increases the impact that teachers have on their students’ reading achievement. Many studies find less variation in teachers’ effect on reading achievement compared to teachers’ effect on math achievement, a pattern that is also evident in our data from Cincinnati. Some have hypothesized that the smaller differences in effectiveness among reading teachers could arise because students learn reading in many in- and out-of-school settings (e.g., reading with family at home) that are outside of a formal reading class. If teachers have less influence on reading achievement, then even if evaluation induces changes in teacher practices, those changes would have smaller effects on achievement growth.

Discussion

The results presented here—greater teacher performance as measured by student achievement gains in years following TES review—strongly suggest that teachers develop skills or otherwise change their behavior in a lasting manner as a result of undergoing subjective performance evaluation in the TES process. A potential explanation for these results is that teachers learn new information about their own performance during the evaluation and subsequently develop new skills. New information is potentially created by the formal scoring and feedback routines of TES, as well as increased opportunities for self-reflection and for conversations regarding effective teaching practice in the TES environment.

Moreover, two features of this study—the analysis sample of experienced teachers and Cincinnati’s use of peer evaluators—may increase the saliency of these hypothesized mechanisms. First, the teachers we study experienced their first rigorous evaluation after 8 to 17 years on the job. Thus they may have been particularly receptive to and in need of information on their performance. If, by contrast, teachers were evaluated every school year (as they are in a new but similar program in Washington, D.C.), the effect resulting from each subsequent year’s evaluation might well be smaller. Second, Cincinnati’s use of peer evaluators may result in teachers being more receptive to feedback from their subjective evaluation than they would be were the feedback to come solely from their supervising principals.

Teachers also appear to generate higher test-score gains during the year they are being evaluated, though these estimates, while consistently positive, are smaller. These improvements during the evaluation could represent the beginning of the changes seen in years following the review, or they could be the result of simple incentives to try harder during the year of evaluation, or some combination of the two.

A remaining question is whether the effects we find are small or large. A natural comparison would be to the estimated effects of different teacher professional-development programs (in-service training often delivered in formal classroom settings). Unfortunately, despite the substantial budgets allocated to such programs, there is little rigorous evidence on their effects. There are, however, other results from research on teacher effectiveness that can be used for comparison. First, the largest gains in teacher effectiveness appear to occur as teachers gain on-the-job experience in the first three to five years. Jonah Rockoff reports gains of about 0.10 student standard deviations over the first two years of teaching when effectiveness is measured by improvements in math computation skills; when using an alternative student math test measuring conceptual understanding, the gains are about half as large. Second, Kirabo Jackson and Elias Bruegmann find that having more effective teacher peers improves a teacher’s own performance; a 1-standard-deviation increase in teacher-peer quality is associated with a 0.04-standard-deviation increase in student math achievement. Compared to these two findings, the sustained effect of TES assessment is large.

But are these benefits worth the costs? The direct expenditures for the TES program are substantial, which is not surprising given its atypically intensive approach. From 2004–05 to 2009–10, the Cincinnati district budget directly allocated between $1.8 and $2.1 million per year to the TES program, or about $7,500 per teacher evaluated. More than 90 percent of this cost is associated with evaluator salaries.

A second, potentially larger “cost” of the program is the departure from the classroom of the experienced and presumably highly effective teachers selected to be peer evaluators. The students who would otherwise have been taught by the peer evaluators will likely be taught by less-effective, less-experienced teachers; in those classrooms, the students’ achievement gains will be smaller on average. (The peer evaluator may in practice be replaced by an equally effective or more effective teacher, but that teacher must herself be replaced in the classroom she left.)

While this second cost is more difficult to calculate, it is certainly offset by the larger gains made by students in the evaluated teachers’ classrooms. Those students are scoring, on average, 10 percent of a standard deviation better than they would have otherwise, and since each peer evaluator evaluates 10 to 15 teachers each year, those gains are occurring in multiple teachers’ classrooms for a number of years.

The results of our study provide evidence that subjective evaluation can improve employee performance, even after the evaluation period ends. This is particularly encouraging for the education sector. In recent years, the consensus among policymakers and researchers has been that after the first few years on the job, teacher performance, at least as measured by student test-score growth, cannot be improved. In contrast, we demonstrate that, at least in this setting, experienced teachers provided with unusually detailed information on their performance improved substantially.

American public schools have been under new pressure from regulators and constituents to improve teacher performance. To date, the discussion has focused primarily on evaluation systems as sorting mechanisms, a means to identify the lowest-performing teachers for selective termination. Our work suggests optimism that, while costly, well-structured evaluation systems can not only serve this sorting purpose but can also enhance education through improvements in teacher effectiveness. In other words, if done well, performance evaluation can be an effective form of teacher professional development.

Eric S. Taylor is a doctoral student at Stanford University. John H. Tyler is professor of education, economics, and public policy at Brown University. This article is based in part on a forthcoming study in the American Economic Review.

In the past, preparing for the SAT meant reading through a test-prep book, hiring a private tutor, or attending test-prep classes. Grockit, an education start-up, is trying to change all that by taking test prep—and studying in general—online to make it convenient and affordable, more effective, and more engaging and social.

Students sign up through Facebook or Grockit’s web site. Some dive into solo mode, working through questions and receiving relevant tips from tutor-recorded videos on “Grockit TV.” But most enter group mode, studying with friends on Facebook or in virtual study groups that Grockit sets up.

Throughout the experience, students receive real-time feedback on how they are doing, the chance to review old concepts, and score predictions that show their improvement based on their learning progressions.

At the third annual Education Innovation Summit at Arizona State University (ASU), the 800 attendees—among them leading education entrepreneurs, venture capitalists, private-equity funders, foundation officers, policymakers, and others—were abuzz, or perhaps “atwitter,” about this hot and emerging space called social learning into which Grockit, among others, has blazed a trail over the last few years.

Conversations at the event and across the broader education sector reveal a mixture of excitement and nervousness as to whether the purveyors of tech-enabled social-learning experiences will develop sustainable business models and whether the experiences they create will bolster student learning or be just another education fad.

Either way, social learning is sweeping through education circles. To be fair though, despite Facebook’s recent stumble into public ownership, if you place the word “social” in front of nearly anything these days, you can get a meeting in Silicon Valley. It’s not just educators who are excited about the possibilities.

Social learning, if understood as people gleaning information from one another, has of course been alive and well for as long as humanity itself. As Farb Nivi, founder of Grockit, said at the ASU event, it was the dominant way learning occurred before the industrialization of education in the mid-to-late 1800s. Social learning as a distinct sector in the world of education technology is a more recent phenomenon. Here, technology brings teachers and learners together in a vast network to create and share information online.

Building off the craze around social media—and social networking giant Facebook’s success in generating high user engagement—education entrepreneurs are increasingly weaving social components into their online learning innovations. For educators, the enticement is the opportunity to increase student engagement and enable students to learn from other students, teachers from other teachers, and students from teachers around the world. Of course, just as many people worry about the accuracy of articles on Wikipedia, there are concerns about the quality of the information students might receive from their peers. But the goal is clear: for every online learner to have access to personalized, tutorial-like experiences on demand.

How these trends will develop is anyone’s guess, but the reigning vision of online learning as a solitary experience will likely be quickly replaced. Social is hot, and just as it has radically altered how we operate in our personal lives, it will transform how we learn as well.

Grockit arrived on the scene toward the end of 2007 and is now expanding from test prep into K–12 and higher education. As it does so, it is bringing in motivational elements from other fields. For example, students earn badges—a staple in the “gaming” world—as they make progress in Grockit.

The company’s solution is built around its central belief that group study is the most effective way for students to learn. Some outside research hints that group study can be fruitful. Richard Light, professor at the Harvard Graduate School of Education, has reported that students’ ability to form and participate in small study groups influences their success in college more than multiple other factors, although many of us also know that study groups are not always helpful, and the research remains anything but conclusive.

Grockit reports that students engaged in its social learning experience study longer, answer more questions, and get more questions correct. The company employs several researchers who conduct studies on everything from learning outcomes to time spent on task.

When in 2010 Facebook unveiled its “Open Graph,” Grockit was one of two education companies among the 60 initial partners. Open Graph extends the “social graph” (individual members and their connections) to include among members’ visible Facebook activities the various interests they pursue online via third-party applications (reading articles, listening to music, and recipe browsing, for example), and allows students to study with their friends and to show off their learning progress.

Obstacles remain, however. Many schools bar their students from using Facebook in school, and some bar teachers from communicating with students through the social networking site.

Enter Edmodo. Founded in 2008 by two technologists working at schools in Chicago, the company aims to help educators harness the power of social media to customize the classroom for every learner. Edmodo provides teachers and students with a secure place to connect and collaborate, share content and educational applications, and access homework, grades, class discussions, and notifications.

Edmodo looks a lot like Facebook, complete with third-party applications. And it has grown virally like Facebook, too, at least by education standards; more than 8 million teachers and students around the world use the platform. It doesn’t hurt that it is free to use—just like Facebook. Teachers have flocked to it to engage students, connect with their peers, share and store content, track and measure success, and access online professional development.

Open questions include whether Edmodo changes the instructional paradigm in a meaningful way. With its focus on the teacher, Edmodo to some extent reinforces the traditional one-teacher-to-many-students classroom structure. What is promising, however, is that Edmodo appears “disruptive” relative to many other learning management systems: initially it was limited in its ability to help a teacher organize a full course, but because it is free and cloud-based, and therefore more convenient and simpler to use than most such systems, it gained rapid adoption and continues to improve. All this suggests that Edmodo could play a role in a student-centric digital learning ecosystem. Whether the company will be able to find enough consistent revenue sources to stay viable remains to be seen.

Another company to offer social learning opportunities free of charge is Sophia Learning, which was founded in 2009 and incubated out of—and recently acquired by—Capella University, one of the nation’s online universities. Sophia is a social teaching and learning network. In the company’s words, “It’s where you can teach what you know and learn what you don’t.”

Sophia Learning looks far more like the Khan Academy than Facebook, as it provides thousands of academic tutorials for students. The critical difference is that these tutorials are taught in a variety of ways by a variety of people, and the site offers math, English, science, and more.

These are far from the only players. Among competitors in the K–12 market are Schoology, a learning management system and social network that makes it easy to create and share academic content and recently hit 1 million users; ePals, a social network optimized for K–12 learning with more than half a million classrooms in 200 countries and territories signed up; Remix Learning, a customizable social-learning network for primary and secondary education that employs a subscription model for schools, nonprofit organizations, museums, libraries, and other cultural institutions; and Sokikom, which provides a social learning environment for web-based math games for primary-school students. A slew of other companies are engaged in helping students and teachers from around the world collaborate.

Will greater engagement, personalized learning, and peer collaboration give a much-needed boost to student achievement? That’s the hope among educators—and the challenge to social learning entrepreneurs.

Michael Horn is executive director of the education practice of Innosight Institute and executive editor at Education Next.

I was observing a class called Designing Assessments at the new Relay Graduate School of Education when a student asked if it was OK to rework questions from a teachers’ guide to fit the English lesson she was teaching in a Brooklyn middle school that week. Sure, said Mayme Hostetter, Relay’s dean: “No need to totally invent the wheel. Just make the wheel amazing.”

Hostetter might just as surely have been talking about Relay, which aims to transform teacher education to fit the needs of urban schools. The amazing—or at least attention-getting—improvement on the wheel is that New York–based Relay is linking the success of its students to the success of their students.

During their second year in Relay’s two-year masters-degree program, elementary-school teachers are asked to show that their own students averaged a full year’s reading growth during the school year. They must also set a reading goal for each child, perhaps two years’ growth for a child who is three years behind, for example. Students can earn credit toward an honors degree if 80 percent of the children they teach meet their individual reading goals.

To earn their degrees, elementary-school teachers are also asked to show that their students earned, on average, 70 percent mastery on a year’s worth of state or Common Core Standards in another subject, usually math. In other words, a math class would meet the goal if students’ individual mastery scores, when averaged, were 70 percent or better. Middle-school teachers use the same yardstick, but only in their specialized subject.

Relay’s cofounder and president, Norman Atkins, talks movingly about the crisis in inner-city teaching and the need to “grow a pipeline of effective teachers who can make an immediate difference.” But the true value of Relay’s model may go beyond potentially improving the teaching in the classrooms where Relay’s graduates work. Robert Pianta, dean of the University of Virginia’s Curry School of Education, explained that Relay is creating a “feedback loop,” using child-level data to measure the outcomes of its teacher-training program, and using those measures to make decisions about program design. “This is how systems get better,” he told me.

Spreading accountability from the teacher back to the education school is an idea the Obama administration is also promoting in its efforts to remake teacher training. This spring, a federal panel looking at teacher-preparation programs debated, among other things, rating ed schools based on how much their teachers add to student learning. That possibility riles ed school deans, among others, but “individual accountability is coming down the pike,” says Kate Walsh, president of the National Council on Teacher Quality, a research and advocacy group.

Even Relay’s admirers concede that it’s too soon to tell whether the model works. It’s operating in just two cities: New York, where it’s offering a master’s degree to 206 students this year, and Newark, New Jersey, where so far it has state approval only to offer a one-year teaching certificate and has enrolled 64. Relay’s first class won’t graduate until 2013. Philanthropies are still footing much of the bill.

Relay has hired a research director, but Atkins says it may not open itself to independent researchers for another four years. Its students—with undergraduate degrees from the likes of the University of Virginia, Lafayette, and Georgetown—are atypical for an ed school, which could complicate comparisons with other teacher programs. Above all, trying to measure student achievement and a teacher’s role in improving it is hard to do.

Still, says Arthur Levine, former president of Columbia University’s Teachers College and a member of Relay’s board, Relay is helping to reinvent teacher education. “Relay is the model,” he told me. “It is the future.”

Nuts and Bolts

If there were ever a system in need of reinvention, it would be teacher education. Decades of studies, reports, and blue-ribbon commissions have criticized ed schools for low entrance requirements, mediocre standards, an emphasis on theory over practice, and outdated curricula. “It’s an accepted truth that the field is broken,” Walsh told me. The problem with fixing it, she added, is that “nobody has known what to do.”

What Relay is doing largely breaks the mold. Its students are full-time elementary- and middle-school teachers, almost all of them fresh out of college, almost none of them with a traditional teaching degree. The program is heavy on practice and nuts-and-bolts technique. It is competency-based: students can be waived out of Designing Assessments, for example, if they can show they are already adept at writing tests.

Relay’s method flips the classroom, with an online lesson at the start of every module or teaching unit (about 40 percent of instruction is online) and in-class discussions and exercises afterward. Twice-monthly night classes, once-monthly Saturday classes, and two summer terms are taught by master teachers and charter school heavyweights. Online instructors include Lee Canter, author of Assertive Discipline, charter school founders and principals, and Relay professors and deans.

Modules vary in duration and range from the nitty-gritty of classroom management—how to arrange furniture, how to grade papers, how to deal with families, how to open and close a lesson—to big-picture subjects, including literacy instruction, writing development, learning disabilities, unit planning, and character development. For a class called Benchmarking and Tracking Progress, scheduled to last 11¼ hours, the catalog says Relay students will create a spreadsheet to track their own students’ progress against year-end goals, and use the data to customize their teaching. In a class called Behavior Management Plans, Relay students will write a set of classroom rules and learn “how to engage in the very necessary practice of correcting students when they misbehave.”

Everybody Engaged

I logged onto an online lesson for a module titled Engaging Everybody, taught by Doug Lemov, managing director of Uncommon Schools. In the 3¾-hour lesson, Lemov lectured for three or four minutes on each of four techniques that he promotes to keep youngsters involved in class, techniques he labels “wait time,” “everybody writes,” “cold call,” and “call and response.” Each of Lemov’s minilectures was followed by a few pages of online reading from his book Teach Like a Champion, and an essay question or two that students answer online (see “Tools for Teachers,” book reviews, Spring 2011). Then came several short videos showing teachers using each technique in the classroom, with Lemov noting the teachers’ use of an apt pause or effective gesture.

Ed schools typically separate lectures from experience, Arthur Levine pointed out to me; by putting lessons online, Relay can blend them. Next came practice scenarios—what do you do if only three children raise their hands to a question about angles?—online group exercises, and instructions to prepare a lesson plan that incorporates the techniques.

Evening classes are on pedagogy—how to teach—and Saturday classes are on subject matter—what to teach. At the second Engaging Everybody evening class, Relay students are expected to present a 10-minute video of themselves using the techniques in their own classrooms. A complex “rubric” describes how students will be assessed on each: on the wait-time technique, students are evaluated on whether they wait at least three seconds between posing a question and calling on a child for an answer, and whether they “strategically narrate” the wait with encouraging comments.

The classroom lessons are heavily scripted. During the first three minutes of the Engaging Everybody class, for example, the Relay students are to report on how often they’re using the four techniques. The script then lists four paragraphs of narrative and questions for the Relay professor to pose over the next four minutes. For five minutes after that, there’s a review, with 10 questions for the professor to ask, and then a suggested transition: “All right, our minds are fresh on today’s content and we’re ready to move.” Then there’s a guided 7-minute “table discussion,” 5 minutes of class discussion, 11 minutes of partner feedback, and so on.

“It is the most self-consciously designed program I’ve ever seen at a university level. Everything was thought out,” said David Steiner, dean of the Hunter College School of Education, which hosted Relay’s predecessor, Teacher U, beginning in 2008. (Teacher U’s last class of about 147 students will graduate this summer.)

Relay students “model” the kind of behavior they hope to see in their own classrooms, so their hands fly up at questions, they rush to stack chairs and pass out papers, they snap their fingers or waggle their hands to show approval. Relay’s scripts do the same kind of modeling by showing students how to effectively use their limited class time, Steiner explained.

Feedback Loop

Relay’s class of first-year students had moved to the 6¾-hour Designing Assessments module when I visited in the spring. At the first of the module’s two evening classes, they had practiced writing “exit tickets,” quick quizzes to measure kids’ understanding of that day’s lesson. At the second evening class, students were to write an end-of-the-week test. Again, the script divided the evening into increments, with 45 minutes for students to practice writing test questions and the final 10 minutes for “team building.”

The Newark class, held at North Star Academy Charter School, was looser than the script suggested—and heavier on inspiration. James Verrilli, director of the Newark program and founding principal of North Star, opened with a clip from a Hollywood film about an innocent man’s decades-in-the-making escape from prison, and asked how it related to urban teaching.

The communal answer was that the escape seemed doomed—“like some people look at our kids,” one young woman said—but that perseverance and vision will yield success. “Shout-outs” followed that, with Verrilli singling out students, and students singling out each other, for exemplary work in their classrooms that week. The evening closed with an animated call-and-response reading of Relay’s creed, which ends with the lines, “We touch lives daily. We are teachers.”

In between, the discussion ranged from how to align test questions with the state standards to the layout of a test paper. There was agreement that some questions on the New Jersey tests included extraneous information that obscured the lesson. But “if the state is doing it, we don’t want our kids walking in blind,” said another young woman, who suggested that everyone write a few wordy questions for their own students to practice.

A few days earlier, I attended a Saturday class at Baruch College Campus High School. It gathered most of the Relay students working in New York City schools (mostly in charters, but a few in district schools where Teach For America has assigned them), divided them into subject specialties, and then again by elementary- and middle-school levels. Relay says 61 percent of the New York class is working toward a master’s degree in childhood education, 9 percent in middle-school math education, 5 percent in middle-school science, 3 percent in social studies, 12 percent in English, and 10 percent in general middle-school education.

In a class called Geometry, Fractions, and Measurements, Nicole Chalfoun—a former Bronx 5th-grade teacher—asked Relay students to design a “remediation strategy” for a child whose answer to the equation 2/5 + 3/8 is 5/13. “Where would you start?” she asked, as her students discussed which manipulatives would best convey to the child why uncommon denominators can’t be combined.

In Teaching Middle School Social Studies III, Ali Brown—director of history achievement for the Achievement First schools—asked her class to write an “essential question” that would frame a unit they were soon to teach on the American Revolution. “What is best going to make your kids think hard?” she asked. In an Elementary School Literacy class—the last of 10 sessions in the module—students were critiquing videos they had made of themselves teaching a reading lesson. In Teaching Middle School Math III, the morning began with a game called Buzz and a discussion of ways to modify it to include higher-level math, including calculus.

Every class meeting ends with a survey—was the lesson helpful, how could it be improved?—with comments fed back to course designers, says Hostetter.

The students I talked with—almost all of them first-year teachers—told me that Relay’s lessons were helping them plan their classes, practice their presentations, keep their kids engaged. “Everything I learn here I can use the next day,” said Milan Reed, who graduated from the University of Virginia with a major in political and social thought and now is teaching at Newark’s Spark Academy charter school.

“I get ideas, I get practice, I get feedback,” added Adam Feiler, a 2008 Georgetown graduate and Teach For America volunteer who’s teaching 4th grade at North Star’s Vailsburg Campus elementary.

Many also told me that Relay’s lessons have changed their classroom culture. “The culture went from being compliant to being invested,” said Max Silverstein, a Penn State business major now teaching in an early-childhood classroom at Newark Legacy Charter School. I heard the same thing from Alonte Johnson, a Morehouse College English major who is teaching middle-school English at Kings Collegiate Charter School in Brooklyn. A few days earlier, his students designed a seating chart that paired the better and slower readers. “The environment is more interdependent instead of everyone working for me,” he said.

On a Mission

Norman Atkins, who founded Uncommon Schools and its North Star Academy as well, said he, David Levin, cofounder of the KIPP charter schools, and Dacia Toll, founder of Achievement First schools, began talking about an education school when they found themselves competing for the same teachers. “Rather than fight over a shallow pool of talent, we were interested in what it would take to build a generation of teachers,” he told me in Relay’s spare Manhattan offices above a public library. (Levin and Toll are on the Relay board, but don’t hold executive positions in the nonprofit.)

They approached 10 college presidents looking for a partner institution, Atkins said, and Hunter’s David Steiner, himself a critic of teacher training, “was waiting for us with open arms.” Last year, seeking more autonomy, Atkins launched Relay and began phasing out Teacher U. The new school takes its name from research suggesting that a “relay” of three years of good teachers can erase the average educational disadvantage of low-income children.

The idea of holding Relay students—and before them, students at Teacher U—accountable for their students’ progress was “one of the very first things we talked about,” Atkins said. The school settled on the 70 percent average mastery floor after looking at the New York math and language tests, where proficiency generally is defined as a score of 70 percent correct answers. The 80 percent stretch goal (for the honors degree) was less data-based but is “at the nexus of ambition and feasibility,” said Brent Maddin, Relay’s provost.

Some 95 percent of Teacher U’s 2010 graduates and 98 percent of its 2011 graduates met the 70 percent targets, he said, although the graduation rate over the two-year master’s program is lower, between 70 percent and 80 percent because of attrition, Hostetter noted.

Reading progress can be assessed with any of six tests, including Fountas and Pinnell Benchmarks and Pearson’s DRA2. But subject tests don’t exist for all subjects and grades. Relay’s handbook says its students instead can use tests they acquire elsewhere or even write themselves, if the assessments show mastery of state or Common Core standards, or of standards set by charter networks or individual schools.

I asked Verrilli, head of the Newark program, how Relay could analyze achievement among youngsters taking so many different tests (a half dozen of his Newark students who are teachers in district schools are writing their own year-end assessments). It wasn’t an apples-and-oranges comparison, he said, but one between “McIntosh and Golden Delicious.” The comparison isn’t among tests, but about mastery levels, he said.

I put that to Scott Marion, associate director of the National Center for the Improvement of Educational Assessment, who told me that Relay was “dreaming” if it hoped to compare performance across schools, but he otherwise sounded supportive. “I care about the end determination: Is the teacher effective or not,” Marion said, “not ‘did these kids achieve a number’?”

Atkins is a serial social entrepreneur who also started the Robin Hood Foundation, which invests in schools and antipoverty programs in New York. His plans for Relay set a breakneck pace: Next year, he expects Relay to enroll 500 to 550 students in New York and New Jersey. It will add classes for high school teachers, including chemistry, biology, and physics. It has applied to New Jersey to begin a master’s program. And Relay expects to extend its reach further into district schools under an agreement to train up to 60 NYC Teaching Fellows in the Bronx.

Atkins said he expects Relay to be fully supported by tuition and client-school fees in three or four years. For now, philanthropies are footing about $13,000 of the $35,000 two-year tuition bill. Students pay about $4,500, with charter schools and federal grants and subsidies making up the rest. Arthur Levine, the board member, agrees with Atkins’s aim. “For innovation to survive, it has to be self-sustaining. If something’s not self-sustaining, it’s not serious,” he told me. And Relay is nothing if not serious.

“What calls us every day is the sad and tragic circumstance” of urban education, Atkins told me, and in one phrase or another, everyone at Relay says the same thing. The other thing they all told me turns on its head the notion of what makes a great teacher.

“We’re saying great teachers are made,” James Verrilli told me, “not born.”

June Kronholz is a contributing editor of Education Next and a former foreign correspondent, bureau chief, and education reporter for the Wall Street Journal.

Read the unabridged version of this report here.

---

“The United States’ failure to educate its students leaves them unprepared to compete and threatens the country’s ability to thrive in a global economy.” Such was the dire warning issued recently by an education task force sponsored by the Council on Foreign Relations. Chaired by former New York City schools chancellor Joel I. Klein and former U.S. secretary of state Condoleezza Rice, the task force said the country “will not be able to keep pace—much less lead—globally unless it moves to fix the problems it has allowed to fester for too long.” Along much the same lines, President Barack Obama, in his 2011 State of the Union address, declared, “We need to out-innovate, out-educate, and out-build the rest of the world.”

Although these proclamations are only the latest in a long series of exhortations to restore America’s school system to a leading position in the world, the U.S. position remains problematic. In a report issued in 2010, we found only 6 percent of U.S. students performing at the advanced level in mathematics, a percentage lower than those attained by 30 other countries. And the problem isn’t limited to top-performing students. In 2011, we showed that just 32 percent of 8th graders in the United States were proficient in mathematics, placing the U.S. 32nd when ranked among the participating international jurisdictions (see “Are U.S. Students Ready to Compete?” features, Fall 2011).

Admittedly, American governments at every level have taken actions that would seem to be highly promising. Federal, state, and local governments spent 35 percent more per pupil—in real-dollar terms—in 2009 than they had in 1990. States began holding schools accountable for student performance in the 1990s, and the federal government developed its own nationwide school-accountability program in 2002.

And, in fact, U.S. students in elementary school do seem to be performing considerably better than they were a couple of decades ago. Most notably, the performance of 4th-grade students on math tests rose steeply between the mid-1990s and 2011. Perhaps, then, after a half century of concern and efforts, the United States may finally be taking the steps needed to catch up.

To find out whether the United States is narrowing the international education gap, we provide in this report estimates of learning gains over the period between 1995 and 2009 for 49 countries from most of the developed and some of the newly developing parts of the world. We also examine changes in student performance in 41 states within the United States, allowing us to compare these states with each other as well as with the 48 other countries.

Data and Analytic Approach

Data availability varies from one international jurisdiction to another, but for many countries enough information is available to provide estimates of change for the 14-year period between 1995 and 2009. For 41 U.S. states, one can estimate the improvement trend for a 19-year period—from 1992 to 2011. Those time frames are extensive enough to provide a reasonable estimate of the pace at which student test-score performance is improving in countries across the globe and within the United States. To facilitate a comparison between the United States as a whole and other nations, the aggregate U.S. trend is estimated for that 14-year period and each U.S. test is weighted to take into account the specific years that international tests were administered. (Because of the difference in length and because international tests are not administered in exactly the same years as the NAEP tests, the results for each state are not perfectly calibrated to the international tests, and each state appears to be doing slightly better internationally than would be the case if the calibration were exact. The differences are marginal, however, and the comparative ranking of states is not affected by this discrepancy.)

Our findings come from assessments of performance in math, science, and reading of representative samples in particular political jurisdictions of students who at the time of testing were in 4th or 8th grade or were roughly ages 9‒10 or 14‒15. The political jurisdictions may be nations or states. The data come from one series of U.S. tests and three series of tests administered by international organizations. Using the equating method described in the methodology sidebar, it is possible to link states’ performance on the U.S. tests to countries’ performance on the international tests, because representative samples of U.S. students have taken all four series of tests.

Comparisons across Countries

In absolute terms, the performance of U.S. students in 4th and 8th grade on the NAEP in math, reading, and science improved noticeably between 1995 and 2009. Using information from all administrations of NAEP tests to students in all three subjects over this time period, we observe that student achievement in the United States is estimated to have increased by 1.6 percent of a standard deviation per year, on average. Over the 14 years, these gains equate to 22 percent of a standard deviation. When interpreted in years of schooling, these gains are notable. On most measures of student performance, student growth is typically about 1 full standard deviation on standardized tests between 4th and 8th grade, or about 25 percent of a standard deviation from one grade to the next. Taking that as the benchmark, we can say that the rate of gain over the 14 years has been just short of the equivalent of one additional year’s worth of learning among students in their middle years of schooling.

Yet when compared to gains made by students in other countries, progress within the United States is middling, not stellar (see Figure 1). While 24 countries trail the U.S. rate of improvement, another 24 countries appear to be improving at a faster rate. Nor is U.S. progress sufficiently rapid to allow it to catch up with the leaders of the industrialized world.

Students in three countries—Latvia, Chile, and Brazil—improved at an annual rate of 4 percent of a standard deviation, and students in another eight countries—Portugal, Hong Kong, Germany, Poland, Liechtenstein, Slovenia, Colombia, and Lithuania—were making gains at twice the rate of students in the United States. By the previous rule of thumb, gains made by students in these 11 countries are estimated to be at least two years’ worth of learning. Another 13 countries also appeared to be doing better than the U.S., although the differences between the average improvements of their students and those of U.S. students are marginal.

Student performance in nine countries declined over the same 14-year time period. Test-score declines were registered in Sweden, Bulgaria, Thailand, the Slovak and Czech Republics, Romania, Norway, Ireland, and France. The remaining 15 countries were showing rates of improvement that were somewhat slower than those of the United States.

In sum, the gains posted by the United States in recent years are hardly remarkable by world standards. Although the U.S. is not among the 9 countries that were losing ground over this period of time, 11 other countries were moving forward at better than twice the pace of the United States, and all the other participating countries were changing at a rate similar enough to the United States to be within a range too close to be identified as clearly different.

Which States Are the Big Gainers?

Progress was far from uniform across the United States. Indeed, the variation across states was about as large as the variation among the countries of the world. Maryland won the gold medal by having the steepest overall growth trend. Coming close behind, Florida won the silver medal and Delaware the bronze. The other seven states that rank among the top-10 improvers, all of which outpaced the United States as a whole, are Massachusetts, Louisiana, South Carolina, New Jersey, Kentucky, Arkansas, and Virginia. See Figure 2 for an ordering of the 41 states by rate of improvement.

Iowa shows the slowest rate of improvement. The other four states whose gains were clearly less than those of the United States as a whole are Maine, Oklahoma, Wisconsin, and Nebraska. Note, however, that because of nonparticipation in the early NAEP assessments, we cannot estimate an improvement trend for the 1992‒2011 time period for nine states—Alaska, Illinois, Kansas, Montana, Nevada, Oregon, South Dakota, Vermont, and Washington.

Cumulative growth rates vary widely. Average student gains over the 19-year period in Maryland, Florida, Delaware, and Massachusetts, with annual growth rates of 3.1 to 3.3 percent of a standard deviation, were some 59 percent to 63 percent of a standard deviation over the time period, or better than two years of learning. Meanwhile, annual gains in the states with the weakest growth rates—Iowa, Maine, Oklahoma, and Wisconsin—varied between 0.7 percent and 1.0 percent of a standard deviation, which translate over the 19-year period into learning gains of one-half to three-quarters of a year. In other words, the states making the largest gains are improving at a rate two to three times the rate in states with the smallest gains.

Had all students throughout the United States made the same average gains as did those in the four leading states, the U.S. would have been making progress roughly comparable to the rate of improvement in Germany and the United Kingdom, bringing the United States reasonably close to the top-performing countries in the world.

Is the South Rising Again?

Some regional concentration is evident within the United States. Five of the top-10 states were in the South, while no southern states were among the 18 with the slowest growth. The strong showing of the South may be related to energetic political efforts to enhance school quality in that region. During the 1990s, governors of several southern states—Tennessee, North Carolina, Florida, Texas, and Arkansas—provided much of the national leadership for the school accountability effort, as there was a widespread sentiment in the wake of the civil rights movement that steps had to be taken to equalize educational opportunity across racial groups. The results of our study suggest those efforts were at least partially successful.

Meanwhile, students in Wisconsin, Michigan, Minnesota, and Indiana were among those making the fewest average gains between 1992 and 2011. Once again, the larger political climate may have affected the progress on the ground. Unlike in the South, the reform movement has made little headway within midwestern states, at least until very recently. Many of the midwestern states had proud education histories symbolized by internationally acclaimed land-grant universities, which have become the pride of East Lansing, Michigan; Madison, Wisconsin; St. Paul, Minnesota; and Lafayette, Indiana. Satisfaction with past accomplishments may have dampened interest in the school reform agenda sweeping through southern, border, and some western states.

Are Gains Simply Catch-ups?

According to a perspective we shall label “catch-up theory,” growth in student performance is easier for those political jurisdictions originally performing at a low level than for those originally performing at higher levels. Lower-performing systems may be able to copy existing approaches at lower cost than higher-performing systems can innovate. This would lead to a convergence in performance over time. An opposing perspective—which we shall label “building-on-strength theory”—posits that high-performing school systems find it relatively easy to build on their past achievements, while low-performing systems may struggle to acquire the human capital needed to improve. If that is generally the case, then the education gap among nations and among states should steadily widen over time.

Neither theory seems able to predict the international test-score changes that we have observed, as nations with rapid gains can be identified among countries that had high initial scores and countries that had low ones. Latvia, Chile, and Brazil, for example—were relatively low-ranking countries in 1995 that made rapid gains, a pattern that supports catch-up theory. But consistent with building-on-strength theory, a number of countries that have advanced relatively rapidly were already high-performing in 1995—Hong Kong and the United Kingdom, for example. Overall, there is no significant pattern between original performance and changes in performance across countries.

But if neither theory accounts for differences across countries, catch-up theory may help to explain variation among the U.S. states. The correlation between initial performance and rate of growth is a negative 0.58, which indicates that states with lower initial scores had larger gains. For example, students in Mississippi and Louisiana, originally among the lowest scoring, showed some of the most striking improvement.  Meanwhile, Iowa and Maine, two of the highest-performing entities in 1992, were among the laggards in subsequent years (see Figure 3). In other words, catch-up theory partially explains the pattern of change within the United States, probably because the barriers to the adoption of existing technologies are much lower within a single country than across national boundaries.

Catch-up theory nonetheless explains only about one-quarter of the total state variation in achievement growth. Notice in Figure 3 that some states are well below the line (e.g., Iowa and Maine) while others are well above  (e.g., Maryland and Massachusetts). Note also that Iowa, Maine, Wisconsin, and Nebraska rank well below that line. Closing the interstate gap does not happen automatically.

What about Spending Increases?

According to another popular theory, additional spending on education will yield gains in test scores. To see whether expenditure theory can account for the interstate variation, we plotted test-score gains against increments in spending between 1990 and 2009. As can be seen from the scattering of states into all parts of Figure 4, the data offer precious little support for the theory. Just about as many high-spending states showed relatively small gains as showed large ones. Maryland, Massachusetts, and New Jersey enjoyed substantial gains in student performance after committing substantial new fiscal resources. But other states with large spending increments—New York, Wyoming, and West Virginia, for example—had only marginal test-score gains to show for all that additional expenditure. And many states defied the theory by showing gains even when they did not commit much in the way of additional resources. It is true that on average, an additional $1000 in per-pupil spending is associated with an annual gain in achievement of one-tenth of 1 percent of a standard deviation. But that trivial amount is of no statistical or substantive significance. Overall, the 0.12 correlation between new expenditure and test-score gain is just barely positive.

Who Spends Incremental Funds Wisely?

Some states received more educational bang for their additional expenditure buck than others. To ascertain which states were receiving the most from their incremental dollars, we ranked states on a “points per added dollar” basis. Michigan, Indiana, Idaho, North Carolina, Colorado, and Florida made the most achievement gains for every incremental dollar spent over the past two decades. At the other end of the spectrum are the states that received little back in terms of improved test-score performance from increments in per-pupil expenditure—Maine, Wyoming, Iowa, New York, and Nebraska.

We do not know, however, which kinds of expenditures prove to be the most productive or whether there are other factors that could explain variation in productivity among the states.

Causes of Change

There is some hint that those parts of the United States that took school reform the most seriously—Florida and North Carolina, for example—have shown stronger rates of improvement, while states that have steadfastly resisted many school reforms (Iowa and Wisconsin, for instance), are among the nation’s test-score laggards. But the connection between reforms and gains adduced thus far is only anecdotal, not definitive. Although changes among states within the United States appear to be explained in part by catch-up theory, we cannot pinpoint the specific factors that underlie this. We are also unable to find significant evidence that increased school expenditure, by itself, makes much of a difference. Changes in test-score performance could be due to broader patterns of economic growth or varying rates of in-migration among states and countries. Of course, none of these propositions has been tested rigorously, so any conclusions regarding the sources of educational gains must remain speculative.

Have We Painted Too Rosy a Portrait?

Even the extent of the gains that have been made are uncertain. We have estimated gains of 1.6 percent of a standard deviation each year for the United States as a whole, or a total gain of 22 percent of a standard deviation over 14 years, a forward movement that has lifted performance by nearly a full year’s worth of learning over the entire time period. A similar rate of gain is estimated for students in the industrialized world as a whole (as measured by students residing in the 49 participating countries). Such a rate of improvement is plausible, given the increased wealth in the industrialized world and the higher percentages of educated parents than in prior generations.

However, it is possible to construct a gloomier picture of the rate of the actual progress that both the United States and the industrialized world as a whole have made. All estimations are normed against student performances on the National Assessment of Educational Progress in 4th and 8th grades in 2000.  Had we estimated gains from student performance in 8th grade only on the grounds that 4th-grade gains are meaningless unless they are observed for the same cohort four years later, our results would have shown annual gains in the United States of only 1 percent of a standard deviation. The relative ranking of the United States remains essentially unchanged, however, as the estimated growth rates for 8th graders in other countries is also lower than for estimates that include students in 4th grade (see the unabridged report, Appendix B, Figure B1).

A much reduced rate of progress for the United States emerges when we norm the trends on the PISA 2003 test rather than the 2000 NAEP test. In this case, we would have estimated annual growth rate for the United States of only one-half of 1 percent of a standard deviation. A lower annual growth rate for other countries would also have been estimated, and again the relative ranking of the United States would remain unchanged (see the unabridged report, Appendix B, Figure B2).

An even darker picture emerges if one turns to the results for U.S. students at age 17, for whom only minimal gains can be detected over the past two decades. We have not reported the results for 17-year-old students, because the test administered to them does not provide information on the performance of students within individual states, and no international comparisons are possible for this age group.

Students themselves and the United States as a whole benefit from improved performance in the early grades only if that translates into measurably higher skills at the end of school. The fact that none of the gains observed in earlier years translate into improved high-school performance leaves one to wonder whether high schools are effectively building on the gains achieved in earlier years. And while some scholars dismiss the results for 17-year-old students on the grounds that high-school students do not take the test seriously, others believe that the data indicate that the American high school has become a highly problematic educational institution. Amidst any uncertainties one fact remains clear, however: the measurable gains in achievement accomplished by more recent cohorts of students within the United States are being outstripped by gains made by students in about half of the other 48 participating countries.

Politics and Results

The failure of the United States to close the international test-score gap, despite assiduous public assertions that every effort would be undertaken to produce that objective, raises questions about the nation’s overall reform strategy. Education goal setting in the United States has often been  utopian rather than realistic. In 1990, the president and the nation’s governors announced the goal that all American students should graduate from high school, but two decades later only 75 percent of 9th graders received their diploma within four years after entering high school. In 2002, Congress passed a law that declared that all students in all grades shall be proficient in math, reading, and science by 2014, but in 2012 most observers found that goal utterly beyond reach. Currently, the U.S. Department of Education has committed itself to ensuring that all students shall be college- or career-ready as they cross the stage on their high-school graduation day, another overly ambitious goal. Perhaps the least realistic goal was that of the governors in 1990 when they called for the U.S. to be first in the world in math and science by 2000. As this study shows, the United States is neither first nor catching up.

Consider a more realistic set of objectives for education policymakers, one that is based on experiences from within the United States itself. If all U.S. states could increase their performance at the same rate as the highest-growth states—Maryland, Florida, Delaware, and Massachusetts—the U.S. improvement rate would be lifted by 1.5 percentage points of a standard deviation annually above the current trend line. Since student performance can improve at that rate in some countries and in some states, then, in principle, such gains can be made more generally. Those gains might seem small but when viewed over two decades they accumulate to 30 percent of a standard deviation, enough to bring the United States within the range of, or to at least keep pace with, the world’s leaders.

Eric A. Hanushek is senior fellow at the Hoover Institution of Stanford University. Paul E. Peterson is director of the Harvard Program on Education Policy and Governance. Ludger Woessmann is head of the Department of Human Capital and Innovation at the Ifo Institute at the University of Munich. An unabridged version of this report is available at hks.harvard.edu/pepg/

Over the past few years, as cash-strapped states and school districts have faced tough budget decisions, spending on teacher compensation has come under the microscope. The underlying question is whether, when you take everything into account, today’s teachers are fairly paid, underpaid, or overpaid. In this forum, two pairs of respected economists offer very different answers. Andrew Biggs of American Enterprise Institute and Jason Richwine of the Heritage Foundation argue that, considering skills, workload, and benefits, today’s teachers are, on average, overpaid. Lawrence Mishel of the Economic Policy Institute and Joydeep Roy of Columbia University and New York City’s Independent Budget Office argue that Richwine and Biggs are off the mark, and that teachers deserve a raise.  Read on, and decide for yourself.

Jason Richwine & Andrew Biggs: Public school teachers are “desperately underpaid,” Secretary of Education Arne Duncan said recently. A documentary film called American Teacher, which premiered last fall, portrayed heroic teachers struggling to get by on paltry incomes. The book on which the film is based went so far as to claim that “a teaching career guarantees a life of subsistence earning—month to month and hand to mouth.”

Is public-school teacher compensation really inadequate? Like all public workers, teachers should be paid at a level commensurate with the market value of their skills, which represents the compensation needed to attract and retain a given set of workers. But comprehensive assessments of teacher compensation—covering salaries, fringe benefits, and job security—are uncommon.

As we will show, the analyses that do exist tend to be misleading and incomplete. While much variation in teacher pay exists across the nation and within the profession—younger teachers are paid considerably less than older ones, for example—the average public school teacher is compensated considerably better than comparably skilled private-sector workers.

Salaries

We begin with an oft-cited data point: after controlling for differences in education, experience, race, gender, marital status, and other earnings-related characteristics, public school teachers receive considerably lower total annual salaries than private workers. In fact, when treated as full-year employees, i.e., excluding the value of teachers’ longer summer vacation, by our own estimate public school teachers receive a 19 percent salary penalty.

Yet this type of analysis, which uses linear regression to adjust for skill differences, cannot support strong conclusions about the salaries of a single occupation. Unobserved ability differences, systematic errors in the observed variables, and varying work conditions could all be influencing the observed salary gap.

Imagine analyzing other occupations. If we added to the regression an indicator for architects, for example, we would find that architects receive a salary premium over seemingly comparable workers. Yet few people would immediately conclude that architects are “overpaid,” since architects could easily have skill characteristics not captured by the existing variables. Those who use the standard regression to argue that teachers are underpaid must also conclude that architects are overpaid, food-service workers are underpaid, computer programmers are overpaid, and so on.

The first argument for why the standard regression is misleading concerns the use of years of education (or highest degree obtained) as a measure of teacher skill. The implicit assumption is that education’s effect on future earnings is consistent across fields of study. Does a person who majored in education possess the same skills as the average college graduate, much less one who majored in engineering? Probably not. Students majoring in education score lower than other college students on tests like the SAT and GRE.

Research by Richard Arum, Josipa Roksa, and Esther Cho, based on the Collegiate Learning Assessment, which administers tests of critical-thinking and writing skills upon entrance to college and in follow-up years, concludes that education majors acquire considerably fewer general skills during college than students majoring in the social sciences, humanities, math, or science. If education programs enroll less-talented students and impart less knowledge along the way, one would expect education majors to earn less after graduation than majors in other fields, whether they worked as teachers or in other jobs.

For a more objective measure of skills, we turn to the National Longitudinal Survey of Youth (NLSY). Participants in the NLSY took the Armed Forces Qualification Test (AFQT), which measures word knowledge, paragraph comprehension, arithmetic reasoning, and mathematics knowledge. These skills are important in any job, and particularly in teaching, where the goal is to convey them to students. When controlling for AFQT scores rather than years of education, no wage gap between teachers and nonteachers is evident. In other words, teachers receive salaries around where we would expect based on the results of standardized tests.

Of course, teaching involves many skills that are not captured by standardized tests, including organizational and interpersonal skills. Are teachers adequately rewarded for these noncognitive abilities?

Using data from the Census Bureau’s Survey of Income and Program Participation, we are able to track changes in individuals’ salaries as they switch jobs. If teachers are not paid fairly for their noncognitive skills, we would expect teachers who shift to private-sector jobs to receive significant raises.

As Table 1 indicates, the control group that shifts from nonteaching jobs to other nonteaching jobs experiences an average real salary increase of 0.5 percent. Workers who switch from nonteaching to teaching receive an increase of 8.8 percent. Teachers who change to nonteaching jobs, in contrast, see their wages decrease by 3.1 percent. In other words, the effect on salaries of switching into or out of a teaching job is precisely the opposite of what one would expect if teachers were underpaid. State-level studies have found similar patterns. Unobserved skill differences between teachers and nonteachers, not a general salary bias against the teaching profession, make teacher salaries seem low.

What about work conditions? Do the daily demands of working in a classroom justify a salary premium? One of the more common claims about work conditions is that teachers work longer hours than nonteachers. But according to the best available data, teachers work around 40 hours per week, including work outside of the classroom, about the same as nonteachers.

A better way to assess whether teaching requires a compensating differential for work difficulty is by comparing public-school teacher salaries to private-school teacher salaries. When the standard regression is limited to teachers, those who work for public schools receive a 10 percent salary premium. University of Missouri economist Michael Podgursky has found that the premium is larger when the comparison includes only low-poverty suburban schools, a sector in which public and private schools are presumably most similar. This is inconsistent with the theory that the public-school salary premium reflects a compensating differential for working with more-difficult students.

In summary, based on 1) no evidence of a salary penalty when controlling for AFQT rather than years of education, 2) job switchers receiving higher salaries as teachers than as nonteachers, and 3) public school teachers receiving higher salaries than private school teachers, we conclude that the standard regression is, at best, highly misleading.

Fringe Benefits

We turn now to fringe benefits, where our starting point is the Employer Costs for Employee Compensation (ECEC) data set published by the Bureau of Labor Statistics (BLS). The ECEC data set indicates that public school teachers receive total fringe benefits equal to around 41.2 percent of their salaries. Our nonteaching control group—workers in establishments of 100 or more people, which tend to pay generous benefits—receive benefits equal to 41.3 percent of their salaries.

The initial data are shown in the first two columns of Table 2. Previous studies of teacher benefits have essentially stopped here, concluding that there is not much difference in benefits between teachers and nonteachers. As noted by the question marks, however, the ECEC data are incomplete.

The next two columns reflect important adjustments. First, paid leave for teachers in the ECEC data set is based on teachers’ shorter 185–day work year, versus roughly 260 days for most nonteachers. Paid leave for teachers is reported in the data set only if it takes place during the school year. Since we analyze total annual earnings rather than weekly or monthly pay, we must account for the fact that those earnings are generated by fewer weeks of work per year. The extra paid leaves amount to a 1 – 185/260 = 29 percent of salaries. The exact length of summer break is highly controversial, since some teachers likely work both before and after the official school year. However, reasonable adjustments to the work leave estimate do not change any of our conclusions.

Second, ECEC data for pensions are based on employer contributions to pension plans, not the benefits that employees will receive in retirement. Most public school teachers participate in defined benefit (DB) pension plans, which because of different accounting rules contribute significantly less today for each dollar of future retirement benefits than private-sector DB pensions or defined contribution (DC) pension plans. The adjusted data reflect the value of actual pension benefits accrued each year by teachers, not merely what the governments happen to contribute to their pension funds each year. A private-sector worker would need to save roughly 32 percent of his salary in a 401(k) to match the guaranteed benefits paid to the average teacher in a public-sector DB plan.

Third, retiree health benefits are not counted by the ECEC. These benefits are common in the public sector but increasingly rare and stingy in the private sector. They are generally funded on a “pay as you go” basis, meaning that employers make no explicit contributions for workers today to fund their future benefits.

Nevertheless, retiree health benefits can have significant value, and we can track that value through accounting disclosures. On average, future retiree health benefits for public school teachers are worth about 10 percent of their current salaries. These figures vary from district to district. Milwaukee public school teachers, for example, accrue retiree health benefits worth an extra 17 percent of their salaries each year.

After supplements and adjustments, total fringe benefits for teachers rise to about 101 percent of salaries, substantially higher than what is generally available to private-sector workers.

Job Security

Once hired, most public school teachers face a short probationary period when they may be fired relatively easily. After that, teachers rarely lose their jobs for poor performance. And when layoffs do occur, they are generally less severe than in the private sector.

During the recent recession and state and local budget crunch, some public school teachers were indeed laid off. Employment in education by local government declined by 2.9 percent between September 2008 and July 2011, according to BLS data (see “Public Schools and Money,” features, Fall 2012). Over this period, private-sector employment declined by 4.4 percent.

Public school teachers do have lower unemployment rates than other white-collar professionals, lower even than private school teachers, who lose their jobs almost twice as often. Using a simple utility model that makes some assumptions about risk aversion, we estimate that public-school teachers’ greater job security is worth around an extra 9 percent of pay, meaning that a typical teacher would be indifferent when faced with a choice between a 9 percent pay cut versus a reduction of job security to private-sector levels.

Discussion

Contrary to conventional wisdom, the combined wages, benefits, and job security enjoyed by public school teachers are above market levels. This fact has important implications. First, states and localities struggling to close budget deficits could reasonably consider restraining teacher compensation, particularly pension and retiree health benefits. The existing premium suggests that moderate pay reductions would not push the average teacher below his or her market-compensation level.

Second, across-the-board pay increases are unlikely to increase teacher quality. Put simply, if we already pay teachers above market levels, why has the quality of teachers not already improved? Why do school districts pay for skills that the average teacher does not possess?

One answer is that even when schools are offered better-qualified teachers, they often turn them down. Vanderbilt University economist Dale Ballou, writing in the Quarterly Journal of Economics, demonstrated that “important indicators of a strong academic background and cognitive ability do little to improve the prospects of an applicant for a public school teaching position.” Many applicants who graduate from more-competitive colleges, earn higher GPAs, or hold degrees in specialized areas such as math or science are turned down in favor of less-qualified candidates who took the traditional route of majoring in education.

Another answer is that traditional qualifications are poor predictors of teacher quality. The rigorous value-added models of student achievement tend to show considerable variation in teacher effectiveness, but very little of the variation can be attributed to the academic backgrounds of teachers. If traditional teacher qualifications are not correlated with student success, attempting to pay more for those same qualifications makes little sense.

Ideally, teacher compensation should be determined by capacity to help students advance toward their potential. A comprehensive system of merit pay could reward the best teachers and encourage the least-effective teachers to leave the profession. And the better our measures of teacher effectiveness become, the better a merit-pay system would work.

 

Lawrence Mishel & Joydeep Roy: Richwine and Biggs (hereafter R & B) provide an implausible and incorrect assessment that public school teachers are vastly overpaid. (In their technical report, they present results indicating that teachers receive a 52 percent compensation and a 21.5 percent wage advantage over comparable private-sector workers.) They reach this conclusion by putting a thumb on one side of the measurement scale. Their claim is implausible because they are simultaneously arguing that teaching is extremely well paid but has a low-cognition workforce. One wonders why, if this is the case, that the most elite college graduates have not flooded schools as they have the financial sector and how teaching remains the only female-dominated high-paying occupation.

Wage Comparison

R & B find that teachers make 21.5 percent more in wages (per work hour) than comparably skilled workers. This conclusion is primarily generated by two flawed estimates: one is their correction for “summers off” and the other their substitution of the AFQT test score for education in a standard wage regression.

R & B argue that teachers have lower cognitive abilities than other college graduates and therefore traditional comparisons using education controls do not adequately control for ability. They present three estimates of the teacher hourly-wage differential: the traditional one with education as the only “skill” measure; a second that adds a test-score measure; and a third that omits education but retains the test-score measure as the sole measure of education, cognitive ability, and/or skill. When they add the test-score measure to the traditional education controls, their estimate of the teacher wage disadvantage falls slightly (from –12.6 percent to –10.7 percent). Only when they omit any education controls do they find that teachers earn the same wages as other workers; this they adopt as their preferred estimate and conclude that teachers have the same annual wages as comparable workers. They use these results to dismiss their own initial estimate of a 19 percent annual wage disadvantage for teachers, equivalent to claiming that the wages teachers earn for the school year correspond to what comparable workers earn in a full year’s work.

This preferred estimate compares teachers to other workers with similar test scores, even though the comparison group has substantially less education (nearly all teachers have a bachelor’s degree, and about half also have a master’s, while about one-third of the general workforce has a bachelor’s or further education). This procedure is highly unusual, and R & B provide no empirical test to show that education controls are not good predictors of wages. Their wage results are entirely dependent on omitting education and using the AFQT test score as the sole “skill” variable. The fact that a widely cited previous study did so does not inspire unconditional acceptance, as that study did not obtain different results when it used AFQT scores alone. In R & B’s study, the results are dramatically different. In addition, noncognitive skills like interpersonal skills are probably at least equally relevant in a classroom setting, and such skills are unlikely to be captured in standardized tests.

To adjust for differing lengths of work years, R & B boost teacher wages by 29 percent to reflect a teacher work year of 185 days. Their calculation assumes teachers have 15 weeks off in the summer, or 3.5 months. Yet work years for teachers go from mid-August to mid-June, leaving roughly two months off. That disparity is because the 185–day work year does not include spring or winter breaks or any holidays. The implied comparison is to other workers who are working 52 weeks, and the data being used for the wage comparison, the March Current Population Survey (CPS), include weeks of paid vacation as part of the work year. It is common sense to interpret a teacher’s annual salary as being applied to the time between when the school year starts and ends. The R & B correction is quick, simple, and wrong. A 2008 paper by Sylvia Allegretto, Sean Corcoran, and Lawrence Mishel presents a 14.1 percent wage correction for “summers off” based on 188 work days, 9 paid holidays, and 15.6 paid vacation days in a school year; 15 percent of teachers working in summer school; and nonteacher college graduates averaging a paid work year of 51 weeks (based on March CPS tabulations). Also, R & B need not include paid leave in their calculations because paid leave is captured in March CPS annual wages.

Current Wages of Former Teachers

R & B argue that teachers are overpaid because former teachers earn less when they quit teaching to take a nonteaching job. This assertion rests on two erroneous assumptions. First, the authors assume that those who quit teaching to take up a nonteaching job have similar characteristics to the average teacher such that the experience of these “leavers” is representative of all teachers. The second assumption is that the salaries of these leavers in their current nonteaching jobs reflect their “true” worth as teachers, even though their current jobs may not have much in common with teaching.

Note first that the Survey of Income and Program Participation data R & B use yield very small sample sizes for looking at these job transitions, roughly 150 teachers leaving teaching for nonteaching jobs. The Teacher Follow-up Survey of the Schools and Staffing Survey (SASS-TFS) provides data designed to examine teacher turnover, and it has a much larger sample, 706 former teachers currently working in nonteaching jobs.

SASS-TFS data (see Table 3) show that teachers who quit teaching to work outside of education, particularly those going to the private sector of the economy, were generally among the lowest-paid teachers. Further, they represented just 1 percent of the teaching force, hardly a representative sample of the overall body of teachers.

The data do confirm that those teachers (who are not average in any way) suffer a decline in earnings when they move out of teaching, particularly when they work outside of the education sector. Instead of showing that teachers are overpaid, however, these data suggest that teacher salaries are inadequate, at least for this group of teachers. As critics of current teacher recruitment and retention policies argue, teachers are not generally fired, so why do these teachers voluntarily take up low-paying jobs in other sectors? Most probably, teaching did not compensate these people enough given the working conditions involved. In other research using data from the CPS, we find that the most popular destination occupations for former teachers outside of education are lower-paid positions such as librarians, cashiers, secretaries, and clergy.

Public versus Private School

R & B also argue that public school teachers are overpaid because their wages are significantly higher than those of private school teachers. However, this analogy is unlikely to be valid. There is significant movement of teachers across the two sectors. Presumably, most people considering a career in teaching are open to taking up a job in either sector. The data show that the background and educational qualifications of teachers working in private schools are quite similar to those of teachers working in public schools. With K–12 teaching being an integrated market, reducing public school pay would affect the ability of schools more generally to attract teachers, including private schools.

It is also instructive to note that teachers working in private schools quit teaching at a much higher rate than their counterparts in public schools, and almost two-thirds of these leavers rank an increase in salary to be very or extremely important in any possible decision to return to teaching. Richwine and Biggs are essentially attributing the wages paid to private school teachers as the market wage. The turnover from private schools to other sectors belies this point: private school salaries appear too low to maintain the workforce.

Compensating Differentials

Roughly 9 percentage points, or about one-fifth, of the 52 percent compensation advantage R & B claim is due to their estimated value of greater teacher job security. It is curious that R & B elect to monetize one aspect of teacher work but ignore all others. This is just one way they put their fingers on one side of the measurement scale. A more balanced assessment would consider other dimensions of teacher working conditions: the hierarchical nature of the job, the inflexible work hours, the relative inflexibility of vacation planning, the frequently unsafe working conditions, the lack of private office space, and the stress of being “on stage” nearly all day in front of students.

Moreover, it is far from clear that teacher job security is so special. Research by Alicia Munnell and colleagues at Boston College’s Center for Retirement Research, published in 2011, challenges whether public-sector job security has been greater in recent years, noting that “the peak-to-present drops in employment for state-local and private-sector workers can be projected almost perfectly based on the educational attainment of the respective sectors.”

Benefits

Much of the R & B claim of a large compensation advantage for teachers is due to their evaluation of pensions. The Bureau of Labor Statistics ECEC data indicate that teacher nonwage benefits (health, pensions, and payroll taxes) amount to 34.6 percent of wages, 4.6 percentage points higher than private-sector wages. In contrast to the BLS ECEC data, which indicate a 5.7-percentage-point pension benefit advantage (measured as a share of wages), R & B find a 25.8-percentage-point advantage (see retirement and savings in Table 2). This is because they triple the pension benefit for teachers compared to the BLS measure, estimating it to be 32 percent of wages rather than 11 percent. Fully 40 percent of R & B’s 52 percent teacher-compensation advantage estimate is thus based on their pension calculation.

Space limits an extended discussion here, but we note two conclusions from a 2012 article by Economic Policy Institute researcher Monique Morrissey, who explains that “the logical implication of Richwine and Biggs’s [pension] position is that public employers and taxpayers would be indifferent between current pension funding practices and investing in Treasury securities, even though this would triple the cost of pension benefits” and that R & B “selectively alternate between the cost of benefits to employers and the value to workers, and inappropriately equate the latter with the often much higher cost to individuals of obtaining equivalent benefits.” In a 2012 article, economist Dean Baker points out that, in effect, R & B’s critique of defined benefit plans is that even if they cost the same as a defined contribution plan, they should be reduced because they provide more security: one wonders if taxpayers feel the same way.
R & B also add the cost of retiree health care but exaggerate the cost differences between teachers and other workers, as noted by Morrissey and in a 2012 analysis by Rutgers University professor Jeffrey Keefe.

Plausibility

Teacher wages have certainly declined relative to comparable private-sector workers over several decades. Allegretto, Corcoran, and Mishel report that decennial census data, comparable to the March CPS data used by R & B, show that from 1960 to 2000 teachers’ annual wages declined relative to comparable workers roughly 20 percent overall: 28 percent among women and 11 percent among men. March CPS data show an erosion of teachers’ relative wages from 1979 to 2005 of 11 percent overall: 16 percent among women and 8 percent among men. Other studies by Peter Temin, by Eric Hanushek and Steven Rivkin, and by Paul Peterson have shown similar trends. Richwine and Biggs make an argument that some estimates of the current levels of teacher pay advantage are biased because of failures to control for cognitive ability and so on. These critiques of the levels of teacher relative pay, however, do not address the substantial erosion of teacher relative pay in recent decades. If teachers currently enjoy a compensation and wage advantage, then the advantage was substantially greater in 1980 or 1960. If so, then it is curious that teaching is not an employment magnet comparable to the financial sector. How do you explain why the brightest students from the most elite schools are not adopting teaching as their permanent career? How can R & B argue simultaneously that teaching has low-cognition workers but exceptional pay? Moreover, if teaching has been and remains so attractive, then why is it the only predominantly female high-paying occupation? In a 2012 analysis of gender segregation across occupations, Francine Blau, Peter Brummund, and Albert Yung-Hsu Liu conclude, “A large entry of men into predominantly female occupations is unlikely, in our view; as long as such jobs continue to pay less for workers with similar characteristics, men have little incentive to enter them in large numbers.” One can readily conclude that since men have not entered teaching it must not be or have been well paid.

Richwine & Biggs’s claim of a large compensation advantage enjoyed by teachers is both implausible and incorrect. We commend the 2011 analysis by Allegretto, Corcoran, and Mishel, which found a teacher wage penalty of 12 percent in 2010, up 10.5 percentage points from 1979 (most of the increase occurring between 1996 and 2001). Taking benefits into account, they find a 9 percent compensation penalty for public school teachers in 2010.

Richwine & Biggs: We urge readers to examine our full original report, which preempts many of Mishel & Roy’s technical criticisms and openly acknowledges the limitations they often use as rebuttals. We’ll pick two objections that we can dispense with quickly: pension valuation and job security.

By insisting on ignoring risk when valuing pensions, M & R are at odds with virtually the entire field of financial economics. Our approach to valuing pensions, which considers both the generosity and the risk of pension benefits, is entirely consistent with economic theory, the way in which liabilities of all types are valued in the private sector, public-sector accounting standards in Canada and Western Europe, academic writings, and the judgments of officials at nonpartisan government agencies such as the Congressional Budget Office, the Federal Reserve, and the Bureau of Economic Analysis.

M & R’s suggestion that public employees somehow do not have greater job security than comparable private workers is based on poor reasoning. The fact that public and private employment numbers have changed at similar rates says nothing about the relative levels of employment. Over the past decade, unemployment rates for public employees have been substantially lower than those for similarly skilled private-sector workers. The fact that both rates have risen and fallen in rough parallel does not imply that the levels are the same.

Regarding the inclusion of other compensating differentials in addition to job security, we would welcome that in a future pay comparison—provided the data are reliable, of course. We are not as confident as M & R, however, that other factors will be so unfavorable to teaching. For example, teaching offers a family-friendly work schedule. And individuals who return to teaching after a break, such as for the birth of a child, suffer no wage penalty, while in other professions wages upon return are significantly lower.

Mishel & Roy: The original claim by R & B was that teachers earn compensation and wages that are, respectively, 52 percent and 21.5 percent more than comparable workers. In other words, their findings suggest schools can cut compensation by as much as a third without harm, though in their current essay they only talk about how “moderate” pay reductions would not push the average teacher below his or her market-compensation level. There is scant evidence even behind this claim, and policymakers should be cautious in taking their results seriously.

R & B fail to address the fact that they substantially overvalue summers off, which they assert to be 15 weeks or 3.5 months long. And their decision not to account for differences in education as well as test scores relies on one paper in the literature, contradicting the overwhelming practice of labor market economists. Their estimates are fundamentally flawed. Nor do they respond to evidence that teachers moving to nonteaching jobs are not in any way average. The notion of extremely well-paid teachers is hard to square with reality, especially the failure of men to take over the teaching field.

On pensions, the issue is the cost to taxpayers (i.e., the employer’s costs) rather than a speculative value to an individual (R & B’s approach), because the policy context is state and local budgets. Using the expected rate of return on assets rather than the risk-free rate provides an unbiased projection according to accepted accounting standards (and to R & B) of actual employer outlays. Using a risk-free rate artificially inflates the value of the compensation of public employees. It only serves to obtain an inflated number to attract attention and is no guide to policy.

“Multiplication Is for White People”: Raising Expectations for Other People’s Children
by Lisa Delpit
The New Press, 2012, $26.95; 256 pages.

As reviewed by Mark Bauerlein

Lisa Delpit has won awards from the MacArthur Foundation, American Educational Studies Association, and Teacher Magazine; her book Other People’s Children has sold 250,000 copies; and she has a named chair at Southern University. Nonetheless, she is angry. In fact, the phrase “I am angry” appears 11 times in the introduction to this summary critique of the education of African American students. Delpit’s anger stems from two things: one, the persistently low achievement of those students, and two, school policies and attitudes that cause and maintain it. Straight off, even the most generous reformers come under indictment. Bill and Melinda Gates, for instance, devote much of their gigantic philanthropy to getting black and brown kids ready for college, yet they earn her scorn for “corporate foundations, which indeed have those funds because they can avoid paying taxes that the rest of us must foot.” It gets worse, as she adds, “I am left in my more cynical moments with the thought that poor black children have become the vehicle by which rich white people give money to their friends.”

The charge extends to the very problem the Gates Foundation, the Broad Foundation, Teach For America, and other groups criticized by Delpit propose to remedy: low African American performance. She dismisses the usual explanations—poverty, poor preparation, homes with no books—and identifies two other causes, one the result of the other.

Low performance begins with American racism. Our society, Delpit writes, has a “deeply ingrained bias of equating blackness with inferiority,” and it “seems always ready to identify African Americans with almost all negative behaviors.” At tender ages, black students undergo a series of “microaggressions…small psychic insults” that debilitate them. Black males perform poorly because “our young men have internalized all of the negative stereotypes.” Sometimes black students are invisible, unnoticed, and disrespected, and sometimes they are “hypervisible,” their normal youth behaviors magnified into pathologies. They end up estranged from school culture (“disidentification”), mistrusting their own capacities and fulfilling belittling expectations.

Teachers misinterpret them again and again, Delpit alleges, mainly by disregarding the culture black students inhabit. This is the second cause of low achievement. The classroom is a white, middle-class space often hostile to African American norms. It downplays collaboration, she notes, even though these students need it to “feel more secure and less vulnerable.” It ignores past contributions to learning and science by African Americans. It neglects spirituality, whereas “traditional African education” incorporates “education for the spirit” into everyday lessons.

Delpit assembles classroom anecdotes, including her daughter’s experiences, with research on “stereotype threat” to prove the point. Voices of black students bespeak the demoralizing results, as with the middle schooler who announces, “Black people don’t multiply; black people just add and subtract. White people multiply.” On the other hand, Delpit provides counterexamples of success, for instance, Afrocentric assignments, inspiring teachers who love and sympathize but maintain rigor, and a beloved white teacher whom the students consider “black” for this reason: when asked “how he felt as a white man teaching black history…tears came to his eyes as he answered that when he learned about Emmett Till and other terrible things white people had done to black people, it sometimes made him ashamed to be white.”

Of course, tales and profiles and selective research don’t amount to proof, nor do they serve as grounds for policy revision. Delpit identifies a significant problem—the clash of school culture with African American out-of-school culture—but her racial lens casts it simply as one of respect and morale, not of effective education. She believes that the former produces the latter, for “African American students are gifted and brilliant,” and they would prosper if schools and teachers became sensitive to their culture.

But this translation of teacher sensitivity into student achievement is precisely what remains to be demonstrated. Delpit praises Afrocentric curricula, but her support focuses entirely on inputs and premises, not on outcomes. A unit that instills math by taking racial profiling as the subject wins her admiration, but her only evidence for its effectiveness comes from a student who professes, “now I realize that you could use math to defend your rights and realize the injustices around you.” But what about the math scores those students attain in 12th grade? What grades do they get in first-year college calculus? Delpit claims that schools impart the message that “you must give up identifiably African American norms in order to succeed,” but she never shows that embracing those norms produces higher college enrollment or workplace readiness.

If that evidence doesn’t exist, then Delpit’s argument isn’t with schools. It’s with U.S. history, society, culture, economics. Many pages in “Multiplication Is for White People” suggest that this is, indeed, the case, such as the indignant section on racist actions after Hurricane Katrina. If society at large is racist, though, then schools should receive more credit than Delpit allows. She asserts that “Typical university curricula leave out contributions of people of color to American culture, except in special courses in African American studies,” a flatly false claim. Syllabi in U.S. history, literature, music, and other areas at nearly every campus amply represent African American creators. Her complaint really is that schools haven’t sufficiently countered popular attitudes.

Delpit’s prescription that schools show more respect for African American culture, then, may have the effect of cultivating an adversarial posture among students. If American society is anti–African American, then a “culturally relevant curriculum” necessarily conflicts with it. If high schools offer an Afrocentric curriculum, will students find university offerings uncongenial and drift toward African American studies and away from STEM fields, where job prospects are brighter? Will a high school teacher ashamed of his whiteness alienate students from white college teachers and employers not so ashamed? Delpit notes that yelling is often assumed in African American culture to be a sign of caring, but won’t failing to inform students of the inappropriateness of yelling in public and in workplaces set them up for future tensions?

These are open questions, and this book doesn’t begin to consider them. We might easily dismiss it as an expression of resentment—the shadow of Jim Crow looms on every page—but we do better to take the starting point seriously: we have a culture clash in the classroom. Rather than expounding the pains and injustices and prescribing a “sensitivity” reform, however, let’s examine various schools and curricula on the standard accountability measure. Do they produce graduates who proceed to college and workplace and thrive?

Mark Bauerlein is professor of English at Emory University.

Florida’s gains in reading and math achievement, as measured by the National Assessment of Educational Progress
Checked by Marcus A. Winters

Among the 50 states, Florida’s gains on the National Assessment of Educational Progress (NAEP) between 1992 and 2011 ranked second only to Maryland’s (see “Is the U.S. Catching Up?” features, Fall 2012). Florida’s progress has been particularly impressive in the early grades. In 1998, Florida scored about one grade level below the national average on the 4th-grade NAEP reading test, but it was scoring above that average by 2003, and made further gains in subsequent years (see Figure 1). Scores on Florida’s own state examinations revealed an equally dramatic upward trend.

Many have cited the series of accountability and choice reforms that Florida adopted between 1998 and 2006, under the leadership of Governor Jeb Bush, as the driving force behind the large and rapid improvement in student achievement (see “Advice for Education Reformers: Be Bold!” features, Fall 2012). Others have insisted that Florida’s NAEP scores do not represent true improvements in student reading achievement. Boston College professor Walter Haney, for example, argues that the scores are “dubious” and “highly misleading.” He contends that it is “abundantly clear” that Florida’s aggregate test-score improvements are a mirage caused by changes in the students enrolled in the 4th grade after the state began holding back a large number of 3rd-grade students in 2004 (all school years are reported by the year in which they ended). His argument has been touted by other researchers, most notably by some at the National Education Policy Center, and it has been cited in testimony presented before state legislatures considering the adoption of Florida-style reforms.

It is certainly true, as Haney has said, that one of the Florida reforms was to curtail social promotion of underachieving students from 3rd to 4th grade. In most school districts, students who do not warrant promotion on academic grounds move on to the next grade regardless, because many educators believe that keeping students with their peer group is desirable. But in Florida, those students who completed 3rd grade in the spring of 2003 and since have had to meet a minimum threshold on the Florida Comprehensive Assessment Test (FCAT) reading examination in order to be promoted to the 4th grade, unless they receive a special waiver. As a result, the percentage of students retained in 3rd grade increased substantially. In the two years prior to the policy change, only 2.9 percent of 3rd-grade students were retained, while in the two years following the policy’s implementation, 11.7 percent of Florida’s 3rd-grade students were told they had to remain in the same grade for the coming year.

Haney and others have concluded that this policy change artificially drove up 4th-grade test scores, because it removed from the cohort of students tested those who were retained in 3rd grade, the very students most likely to score the lowest on standardized tests. Although the point would seem to be well worth considering, it has not been subjected to serious empirical analysis. Does the holding back of the lowest-performing students in 3rd grade explain all the 4th-grade gains in Florida, as Haney contends? Does it explain some of the gains? Or none at all? The best way to answer the question is to look at changes in student test-score performance among those in 3rd grade for the first time, as their test scores are unaffected by the retention policy. If the gains observed for 4th graders were a function of differences in the type of students entering that grade due to the retention policy, then the performance of those entering 3rd grade should look essentially the same after 2002 as it did before the retention policy was put into place.

Drawing on information on student performance available from the Florida Department of Education, I was able to analyze test-score trends of students enrolled in the 3rd grade for the first time. I find that the gains among initial 3rd graders were not as dramatic as those shown on the 4th-grade NAEP, thereby suggesting that the 4th-grade scores did create the appearance of steeper achievement growth than actually took place. Nonetheless, the gains among initial 3rd graders were very substantial, about 0.36 standard deviations between 1998 and 2009, and more than enough to justify Florida’s claims that its gains have outpaced those in most other states.

Reading Test Scores for 3rd Graders

I first analyze changes over time in the FCAT test scores of students in their initial 3rd-grade year in order to discern the extent to which Florida’s elementary-school students made true achievement gains during the period in question. Because the state has not yet identified students for retention, the test scores of students the first time they are in the 3rd grade are not affected by any change in the student cohort resulting from the retention policy.

The administrative data set for the State of Florida contains individual test scores and demographic information for the universe of test-taking students in grades 3 through 10 in Florida from 2001 through 2009. The data set includes a unique student identifier, which allows me to follow the progress of each student over time and to determine which students have been retained.

Figure 2 shows the changes since 2001 in the performance of students at the 25th, 50th, and 75th percentiles in their initial 3rd-grade year. The figure documents clear positive movement across the test-score distribution for the first cohort of students that needed to reach a minimal score on the FCAT exam in order to be promoted from the 3rd to the 4th grade (2003). The achievement distribution makes another leap forward the following year (2004), which was the first year that began with a sizable number of retained students due to implementation of the policy. Student achievement continued to grow in subsequent years.

The test-score improvements shown on the figure are substantial. By 2009, the median reading test score of students in their initial 3rd-grade year had improved by more than one-third of a standard deviation since 2001, as had nearly all points on the distribution. A gain of this magnitude amounts to roughly a full year of academic progress for students in the early elementary grades. The test-score gains among the state’s lowest-performing students were even more impressive; for instance, students at the 10th percentile improved by more than half a standard deviation. The gains made by initial 3rd-grade students on the math exam are even larger than the gains in reading at all points on the distribution.

The results do suggest, however, that the aggregate test scores on the 4th-grade NAEP could well be inflated by the retention policy. The improvement in the median reading score for those students entering 3rd grade is smaller than the NAEP increase for 4th graders over the same time period. Even so, the 3rd-grade gains remain noteworthy enough to substantiate the basic claims of those who praise the Florida track record.

Rescaling NAEP Reading Scores

To assess how well Florida performed relative to the rest of the nation, one can use the results for initial 3rd-grade students on the FCAT to rescale the state’s 4th-grade scores on the NAEP reading exam. The rescaling assumes that test-score improvements on the FCAT for cohorts in their initial year as 3rd graders are a good proxy for gains in reading achievement made by Florida’s students in the next elementary grade. Though imperfect, this assumption is justified to the extent that most consider 4th-grade NAEP scores to be an assessment of overall elementary-school performance.

Because Florida did not participate in the NAEP in 2000, I use as the state’s baseline score its median score on the 4th-grade NAEP reading exam in 1998. Thus, I also assume that the state made no meaningful gains in 4th-grade reading between 1998 and 2000 that would have shown up on NAEP, which squares with the scores on the state’s own reading assessment. I then use the improvements of the median reading test score for initial 3rd-grade students on the FCAT since 2001 in order to rescale the state’s mean NAEP test score in the spring of the same year.

In addition to providing the originally reported NAEP score trend in median scores between 1998 and 2009 for Florida and the United States as a whole, Figure 1 shows the rescaled trend in Florida after making the adjustment described above. The first class affected by the retention policy entered the 4th grade during the 2004 school year, and thus the first NAEP score that could have been influenced by the exclusion of low-performing students from the 4th-grade NAEP sample was the spring 2005 administration.

The figure shows that Florida’s reading gains prior to the introduction of the policy were actually larger on the NAEP than on the FCAT. Such a difference cannot be explained by the retention policy, because students had not yet been retained. After introduction of the policy, Florida’s achievement on the state exam after accounting for sample selection increased between 2003 and 2005 in a way that did not show up on the NAEP scores. But the state’s NAEP scores quickly caught up to the FCAT performance. Adjusting the state’s NAEP scores for sample selection in 2007 and 2009 leads to a decrease in the state’s performance of about 0.07 and 0.08 standard deviations, respectively. However, Florida’s adjusted median score remains above the median score for all U.S. public-school students, and it continues to show substantial improvements relative to the prior decade.

Even after the adjustment, Florida’s students still made larger gains in reading than did the rest of the nation. The national gain, at 7 points (or about 0.19 standard deviations), was only slightly larger than half Florida’s rate. Prior to the adjustment, only Washington, D.C., made larger gains on the 4th-grade NAEP reading exam during this period. After the adjustment, only D.C. and Delaware made a larger test-score improvement.

What Reforms Might Have Produced the Reading Gains?

Putting a finger on exactly which policy changes produced the test-score improvements is remarkably difficult, because the state adopted a wide array of policies that may have had a beneficial effect. It is possible, however, to rule out some potential candidates.

For example, some have noted the state’s participation in the federal Reading First program, in which public schools received grant money to implement instructional and assessment tools. Florida also supplemented the Reading First grants with its own financing of reading coaches for schools across the state. The data clearly show, however, that any additional test-score gains made by schools that participated in Reading First or had reading coaches were far too small to explain the substantial improvements observed on both the NAEP and the FCAT.

Others have found it tempting to argue that the state’s constitutional amendments to reduce class size and provide universal pre-kindergarten services—both of which could have a sustained positive effect on young kids—are the most likely driver of the gains. Perhaps those reforms will prove effective. The 3rd-grade class of 2003, for which the large gains begin, however, was subject to neither policy.

Current research findings for the accountability and choice reforms adopted by Florida during this time period also appear insufficient to explain such large test-score improvements. Florida assigned letter grades—A, B, C, D, and F—to schools based on their performance on the FCAT. It put into place a school voucher program for students who were attending schools that received the grade of F twice in a row. A tax credit provided scholarships for low-income students. Studies of all these programs have shown that each had a positive effect. And studies have also shown that the retention policy has a positive impact on the performance of students who were retained. Though each of these policies has been tied to student test-score improvements, either the effect size was too small or the policy affected too few students to alone account for the substantial test-score improvements seen on the NAEP and FCAT.

Conclusion

The evidence presented here shows that Florida’s elementary-school students did in fact make large improvements in reading proficiency in the 2000s. As critics contend, the state’s aggregate test-score improvements on the 4th-grade FCAT reading exam—and likely on the NAEP exam as well—are inflated by the change in the number of students who were retained in 3rd grade in accordance with the state’s new test-based promotion policy. Large test-score improvements are also observed, however, among students whose scores were not influenced by changes in the sample selected.

Though somewhat smaller than what is apparent on the NAEP test, the portion of Florida’s reading test-score improvements during this time period that cannot be attributed to changes in the sample of students tested due to the retention policy is nonetheless substantial. Identifying the causes of these improvements remains an important task for future research.

Marcus A. Winters is senior fellow at the Manhattan Institute’s Center for State and Local Leadership and assistant professor at the University of Colorado–Colorado Springs.

They sauté them with garlic and serve them over a bed of guacamole—worms, that is, in Puebla, Mexico. You can order them with a side of ants’ eggs, which are soft and buttery. In Oaxaca, grasshoppers are more popular fare, appearing in tortillas as a main course or covered in chocolate as a dessert.

When I learned that I would be a participant in the Fulbright-Hays 2011 Summer Seminar in Mexico, a five-week program run by the U.S. Department of Education, I was eager to taste the cuisine in each of the eight states on the itinerary. It never occurred to me that I’d be eating bugs—at least not on purpose.

As a high school teacher, I’d always thought of cultural differences as opportunities to broaden my perspective. Yet there’s something about having to eat the culture that makes accepting cultural differences more personal and much more challenging.

When I teach literature, I talk about the importance of perspective in interpreting novels. Our way isn’t necessarily the right way; it’s just the way we know. With that credo in mind, I lathered my worm in guacamole, closed my eyes, and swallowed.

The trip awakened me to other cultural misconceptions as well. When visiting San Cristóbal de las Casas in Chiapas, I toured San Juan Chamula, a Mayan community. Our guide, archaeologist Chip Morris, began at the graveyard, which frankly resembled the outskirts of a garbage dump. Empty plastic soda bottles littered the areas around headstones. I saw this as a sign of disrespect. Morris set me straight: in the Chamula tradition, he explained, the dead must be remembered and honored. Having graveside parties and leaving bottles show that the family is meeting its obligations.

Travel regularly yields such epiphanies.

Travel is also a great way to discover and reflect on the sometimes surprising interactions between cultures. Morris next took us into the church in the town’s central square, where we saw a significant blending of ancient Mayan practices and Catholic influences. Chickens are sacrificed, as the statues of Catholic saints look on. Posh, a homemade rum drink, is offered by families who want to invite others to witness their audible prayers. And so is Coca-Cola.

Coke as a Mayan ceremonial beverage?

It’s true. In the 1960s, Coca-Cola made local Mayan leaders partners in the distribution of their beverages. By the 1970s, community leaders agreed that Coke and other soft drinks could be substituted for posh, deemphasizing the use of alcohol during religious ceremonies. The billboard on the road coming into Chamula shows a man in traditional festival dress celebrating with a Coke. Whatever one might think about Coke—and it has a checkered record in Latin America—it has played a significant role in reducing alcohol abuse in Mayan communities.

By the end of the trip, I had even come to have a better understanding of the Mayan practice of human sacrifice. After visiting Chichén Itzá and other Mayan sites, I came to see that these sacrifices involved not only enemies, but also what was most important to the Maya. They sacrificed their bravest soldiers during wartime. They sacrificed children and women, who shed the most water in tears during times of draught. In short, they sacrificed not because life was cheap, but because it was precious and their gods deserved the best of who they were.

I choose seemingly outrageous examples because they best illustrate why teachers must travel. We rightly insist that students share different points of view, but we often don’t demand the same of ourselves. And until we are out there “eating” another culture, we might not be scrutinizing our own misconceptions about place, people, and history.

Elaine Griffin is the English Department chair at the University School of Milwaukee.

Public school shepherds endlessly scream “wolf.” Yet, with one minor exception in the early 1980s, no fiscal predator has ever penetrated the perimeter constructed by public education stakeholders. Now, however, after four years of economic slowdown, the United States is facing an unusual alignment of unfavorable fiscal forces. It is increasingly doubtful that public education advocates can continue to protect their flocks. A cry of “wolf” may be justified.

Not all relevant financial figures are available yet, but reasoned extrapolations from private- and public-sector employment data suggest that U.S. schooling may be on a historic glide path toward lower per-pupil resources and significant labor-force reductions. If not thoughtfully considered, budget-balancing decisions could damage learning opportunities for schoolchildren.

Education managers are typically inexperienced in and often reluctant to initiate cost-savings actions. Budget cuts may be poorly targeted, and students, particularly economically disadvantaged students, are swept up in the process as collateral damage.

In California and Washington, bad budget cutting has already begun. Governors in these two states have acquiesced to employee demands and have protected educator jobs at the expense of students’ time to learn.

The greatest risk of all is to the past quarter century of efforts to render America’s schools more effective. Unless means are identified for making schools more productive, that is, doing better with less, reform momentum is in serious jeopardy.

Evolving Context

Many members of the general public and the policy community believe that school districts are going bankrupt, teachers are underpaid, and educator layoffs are rampant (see “The Compensation Question,” forum, Fall 2012, forthcoming). Inaccurate media reporting, naive celebrity comments, education-advocate laments, social-media babble, and talk-show dialogue reinforce this view.

What are the facts? Total K–12 public-school spending approaches $700 billion annually. Inflation-adjusted per-pupil school spending has increased over the last century by, on average, 2.3 percent per year. There have been a few plateau years during recessions, but never a significant decline (see Figure 1).

As a consequence, the United States now spends more money on K–12 schooling than any other nation in the world. More is spent by the United States, in the aggregate, than by hugely populous nations such as China and India. Spending per pupil is higher in the U.S. than in every country except Switzerland.

Achievement levels in the U.S. are not commensurate with spending, however. Many nations exceed the United States in science and math test scores, for example.

Spending increases have been directed overwhelmingly toward adding school employees. Professional-to-pupil ratios have become ever more favorable. Whereas 30 years ago there was one professional educator employed for every 18.6 public school students, the equivalent figure today is one for every 15.4 students. When other personnel are added to the mix—cafeteria workers, custodians, clerks, and so forth—the ratio falls to one employee for every 7 students.

School productivity, measured as educational outcomes divided by labor or financial inputs, has declined dramatically. Indeed, relative to sectors such as communication, finance, manufacturing, and agriculture, the public schools are highly labor-intensive. The productivity picture is made worse by the resistance of schools to augment teachers’ efforts with new instructional technologies.

Why School Productivity Matters

A new normal of public-sector fiscal austerity is emerging.

Forty-two states and the District of Columbia face budget shortfalls. (Only a few fossil fuel–rich or agricultural states are able to sidestep the issue.) Although federal tax revenues are far short of anticipated spending, the federal government is not about to step in with still another stimulus package. Congress and the president are deadlocked over a path to economic recovery. Eurozone economies are in disarray and have had their credit ratings lowered, which jeopardizes U.S. exports.

Through deep and painful experience with cyclical growth and recession, U.S. private-sector firms have learned to deal with contraction. There have been nine recessions in the United States since 1955. During each of these, employment in the private sector declined. Employment subsequently turned up, but conventional private-sector response to recession has been workforce contraction. Private-sector managers know how to hone their labor force to balance cost cutting with the retention of scarce skilled talent and how to invest in labor-saving technology. These dynamics render the private sector ever more efficient, sustaining the production of goods and services with lower labor costs.

Here is an example of just how productive the private sector has become during the most recent recession: By the final quarter of 2011, gross domestic product (GDP) had returned to its 2008 prerecession level. It did so, however, with 5 million fewer private-sector employees.

School districts demonstrate the flip side of this dynamic. Cost-saving actions in public education, such as layoffs, school closures, salary freezes, benefit reductions, and decreasing school days, are possible but unusual. Taking such uncomfortable steps is legally cumbersome and politically treacherous. Cutbacks frequently fail to generate anticipated savings and can trigger hard-to-heal labor-management wounds. In recent recessions, when the private-sector workforce was contracting, school-district hiring continued apace.

It is important to note that much of the employment decline in the private sector during recessions is the result of firms going out of business. In difficult economic times, private firms must either become more efficient or fail. The public-school sector faces no such threat, which may be why schools have historically added jobs, regardless of economic conditions.

Figure 2 depicts growth in private-sector and public-school employment. Here one can see that from 1955 to the start of the most recent recession, the private sector experienced nine labor-market contractions—on average, one downturn per decade. Conversely, until the current recession, employment in public schools had only one downturn, in 1982–83.

The downturn in public-school employment in the early 1980s came on the heels of two recessions, one that stretched from January to July 1980 and the other from July 1981 to November 1982. The fact that teaching jobs were shed after these recessions were officially over should not be surprising, given that school budgets are set, teacher contracts are made, and federal and state funding are allocated ahead of time, causing the public-school sector to respond to tough economic times more slowly than the private sector.

The same condition prevailed in the wake of the most recent recession. Following 2009, when the private sector began adding wage earners, the public schools began to shed teachers. Figure 3 shows this in greater detail.

From June 2008 to March 2012, public schools shed more than 250,000 jobs, 3 percent of their total workforce. It is of particular note that this shrinkage in the public-education workforce took place in spite of the added revenues from the American Recovery and Reinvestment Act (ARRA), which were intended to prevent such a decline.

Nonetheless, as Figure 3 also indicates, a larger share of school employees who were working in 2008 were still on the job in 2012 than the share of workers still employed in the private sector in 2012.

Impact of Revenue Decline

There is no overstating the painful consequences of organizational downsizing, be it private or public. Closing a manufacturing plant, shutting down a large distribution center, and curtailing hours at a backroom financial operation trigger layoffs, and, depending on the context of the contraction, can imperil an entire community. Individuals, parents, children, and even a geographic region can be hurt.

Serious and sustained school revenue declines are at least as bad and in some ways worse. Layoffs almost always involve the least experienced or most recently employed teachers and other staff. If the financial situation necessitates the closure of one or more schools, then the pain spreads wider and may threaten the survival of a community.

School cutbacks may also disproportionately affect low-income students. As mentioned previously, California and Washington have reacted to budget shortfalls in ways that harm students: reducing the length of the school year and the number of days that schools operate. While this saves money and jobs, as teacher salaries are reduced and layoffs avoided, time in school is most important for disadvantaged students. Middle-class families can compensate for the loss of school hours with enrichment activities such as trips to museums and libraries. Low-income students are seldom so insulated from schooling adversity.

If the entire public-education system could be rendered more productive, that is, if higher levels of achievement could be coaxed from existing resource levels, some of the pain could be avoided or at least mitigated.

Improving Productivity

Several integrated strategies offer the prospect of protecting, possibly promoting, education reform in the face of a new fiscal austerity. These strategies involve 1) accurately informing the general public and the policy community regarding the condition of schools, that is, their financing, their achievement, and the relationship between the two; 2) conducting empirical research aimed at understanding issues of productivity in education; 3) informing policymakers and school managers regarding means by which budget cuts can be made without eviscerating instructional effectiveness; and 4) solving challenges to wider adoption of instructional technologies.

Federal and state governments have expended hundreds of millions of dollars to ensure that local schools have Internet access and plentiful computing hardware. Grants have also been available for purchase of software and teacher training.

These efforts have seldom proved sufficient to transform America’s public schools. Instruction continues to rely almost exclusively on labor-intensive practices. Government policies have ignored the savings that private firms have shown can result from technological innovation. Put bluntly, why should a tenured classroom teacher go to the effort of altering her long-standing instructional protocols to adopt new technologies when her pay, professional status, and job security are only remotely related to improving her effectiveness or her clients’ satisfaction?

Strategies must be constructed that will attract classroom teachers to the use of technology to enhance their effectiveness. Whatever strategy emerges in this regard is likely to have to involve teacher and school performance evaluations linked to student achievement gains. If teachers, principals, and entire schools see that their professional status and remuneration are becoming more tightly linked to student achievement, then they will be more open to seeking technologies that will enhance instructional effectiveness.

There are those who contend that online learning will simply bypass schools, that conventional school classes will be disrupted by new digital models that operate outside the brick-and-mortar school. But there is only a modest chance of this happening. A state initiative in Florida, the Florida Virtual School, is promising in this regard. So is the spectrum of well-constructed subject-matter units that can be found at the Khan Academy web site. But the obstacles are almost too numerous to mention. Among them are the monopolistic nature of many public-school systems, the custodial function entrusted to schools by law, and the attractiveness to students of the social interactions that take place in school. If in fact conventional schools are to be disrupted by technology, it is unlikely to happen soon.

While waiting for technologies to augment the work of a teacher, what can be done by state and district officials to wring the maximum effect out of every dollar they have?

First, states and districts can discontinue costly practices that have not been shown to enhance student achievement, including paying educators for out-of-field master’s degrees and salary premiums for experience; following “last in, first out” personnel provisions; relying on regular classroom instructional aides; and adhering to mandated limits on class size. Regulations that mandate inefficiency, such as legislatively precluding outsourcing, requiring intergovernmental grants to “supplement not supplant” existing spending, and prohibiting end-of-budget year surplus carryover, can also be revised to encourage smarter spending.

In place of the practices above, states and districts can adopt strategies that foster efficiency at both the school and district level, such as adopting “activity-based cost” (ABC) accounting; empowering principals as school-level CEOs; adopting performance-based dollar distribution formulas and school-level financial budgeting; centralizing health insurance at the state level; and outsourcing operational services where proven to save money. By adopting these practices, districts and states may be able to ease the burden of the transition to the coming period of fiscal austerity and increase long-term efficiency in schooling.

James W. Guthrie, currently superintendent of public instruction in Nevada, is senior fellow and former director of education policy studies at the George W. Bush Institute, where Elizabeth Ettema is research associate in education policy.