In 2000, public elementary and secondary schools spent roughly $180 billion on teachers’ salaries and benefits, about half of their total expenditures; most of it was distributed according to fixed salary schedules that considered only a teacher’s education and years of experience. This system has its origins in the first half of the 20th century and was partly a response to the racial and gender discrimination that existed under more discretionary systems at that time.

However, over the past 20 years more educators have wondered whether such pay packages can attract, motivate, and retain high-quality teachers in a highly competitive professional world (see Forum). In response to such concerns, there was a flurry of merit pay activity in the early 1980s. Twenty-nine states had initiated some sort of merit pay program for teachers by 1986. Since then, however, almost all of them have been diluted or discontinued. A 1997 study by economists Dale Ballou and Michael Podgursky reported that 12 percent of school districts were using merit pay in some way, but the amount of incentive in these districts averaged only two percent of base pay.

Critics of merit pay argue that the falloff in such programs was due to the fundamental technical difficulties of accurately identifying effective teachers and rewarding good teaching practices. Proponents of performance-based pay insist that these experiments were too limited in scope and were destined to fail in the face of stiff opposition from teachers and unions.

Despite widespread pessimism among educators about whether merit pay systems can effectively reward good teachers, most of the limited empirical evidence has been surprisingly positive. For example, two studies (in 1992 and 1997) found that the math and reading test scores of students in South Carolina improved significantly when the students were taught by teachers receiving merit pay. Similarly, related and more recent literature suggests that mathematics students learn more when their teachers have certification in mathematics.

Policymakers should be cautious in interpreting this sort of evidence, however, as the apparent benefits of certification could merely reflect differences in the students placed in their classrooms. For example, teachers who receive merit pay may tend to select schools and classes whose students are high achievers for other reasons. Likewise, parents especially engaged in their children’s education may work to ensure their assignment to teachers with strong credentials. Such subtle differences may not be visible in the data typically available to researchers.

New Evidence

The effectiveness of these short-lived merit pay programs is exceptionally difficult to measure because of these selection effects. However, the fortuitous overlap of two Tennessee programs from the mid-1980s and 1990s provides an unusual opportunity to circumvent this problem. Project STAR (Student Teacher Achievement Ratio) was a large-scale class-size experiment that began with kindergarten students in the fall of 1985. At roughly the same time, the Volunteer State began directing pay increases to teachers deemed meritorious under a Career Ladder Evaluation System.

The fact that both teachers and students in schools participating in Project STAR were assigned randomly to classrooms allows for an especially rigorous test of whether a merit pay system can effectively reward good teachers.

Before describing the Tennessee programs in detail, however, we need to take a closer look at some of the objections to merit pay for teachers. One concerns the problem of designing valid evaluation procedures for measuring teacher performance. Under an efficient merit pay plan in any industry, employers should be able to explain clearly why an employee did not receive merit pay and what he or she would need to do to get it. Whether these conditions can be met in the teaching profession, where there is no single blueprint for effective practice, has been the most contentious issue surrounding merit pay. This evaluation problem is further complicated by the fact that schools have goals other than cognitive achievement (for instance, promoting citizenship, fostering individual development, and reducing drug use and violence) that are difficult to measure and are often achieved only with teachers’ cooperation.

These concerns raise the possibility that attempts to reward meritorious teachers could even have perverse consequences. For example, merit pay systems may discourage cooperation among teachers or otherwise foster a demoralizing and unproductive work environment.

While these problems may explain why merit pay plans have often been dismantled, some researchers suggest that they are excuses, not reasons. Dale Ballou, an economist at Vanderbilt University, has argued that merit pay is widely and successfully used in private schools, which suggests that there is nothing unique about education that makes merit pay infeasible or unattractive. Ballou notes that the amount of merit pay in private schools is quite large and that the teachers who report receiving it have earnings that are nearly 10 percent higher than their nonmerit counterparts. In contrast, the earnings of merit pay teachers in public schools are only 2 percent higher than their nonmerit colleagues. Ballou attributes the frequent dismantling of alternative compensation for public school teachers to union opposition.

The Career Ladder

Can we devise a merit pay system that overcomes the challenges of definitional clarity and valid measurement? Can we do so without directly incorporating measures of students’ progress on standardized tests?

The Tennessee programs initiated by Governor Lamar Alexander in 1984 offer some reason to believe that we can. At the same time, they underscore the considerable difficulty of doing so in a fair, equitable, and effective manner.

Part of Tennessee’s Comprehensive Educational Reform Act, the Career Ladder Evaluation System was both well funded and sophisticated in its approach to teacher evaluation. As Richard M. Brandt, a professor at the University of Virginia, wrote in 1995, the program was “perhaps the country’s most comprehensive experiment in summative evaluation.”

Governor Alexander, who would go on to become secretary of education under President George H. W. Bush and is now a U.S. senator, was more colloquial in his description, calling the program “an old-fashioned horse trade with teachers. Taxpayers said to teachers, ‘The state will pay you up to 70 percent more based on your performance if you’ll promise to be evaluated every five years.'”

Rung by Rung

While it lasted–for 13 years–the now-defunct Career Ladder had many of the elements that merit pay backers believed a good program should have, including multidimensional evaluations and a hierarchy of professional development (in other words, a career ladder) that was coordinated with significant financial and professional rewards.

The ladder had five distinct stages, ranging from probationary to master. Fast-track options allowed those who had been teaching before 1984 to advance immediately, subject to successful evaluations, to a career level matching their experience.

For new teachers, however, the first rung of the career ladder was a one-year probation supervised by two tenured teachers from their school. Subject to a favorable review by the school district, using state-approved criteria, these teachers were then placed on apprentice status for three years. At the end of those three years, the school district could recommend that the teacher be granted a five-year certification for professional, or Career Level I, status, which included a $1,000 salary supplement from the state.

Then, at the end of the five-year Level I stage, a teacher could either apply for another five-year Level I certification or seek a five-year certification as a Level II teacher. Advancement required evidence of superior performance, as defined by a state commission and the state board of education, but it also came with a $2,000 state supplement for those who chose a 10-month contract and $4,000 for those choosing an 11-month contract, a significant bonus to teachers’ salaries at the time.

At the end of the Level II certification period, the same kind of option was available: a teacher could seek recertification at Level II or pass more rigorous evaluations to receive a Level III certification and a salary supplement of as much as $7,000.

The evaluations that occurred at each stage of the career ladder assessed teachers on multiple “domains of competence” using several distinct data sources (such as student and principal questionnaires, peer evaluations, a teacher’s portfolio, and a written test). On the first three rungs of the ladder (probation, apprentice, Level I), the local school districts were responsible for evaluating and certifying performance. The key evaluator at these stages–typically the principal–received three to five days of state training on evaluation instruments and procedures. In contrast, the evaluations for certifications at Levels II and III were conducted largely by a three-member team of peers from outside the teacher’s district. These evaluators received three to four weeks of training and were often Level III teachers from other districts who had been borrowed for a year by the state certification commission. The extensive training provided to the Level II and Level III evaluators was considered appropriate since they fielded more complex evaluation instruments intended to discriminate among “good, superior, and outstanding” teachers.

Under the original formulation of the career ladder, participation was optional for veteran teachers and mandatory for new teachers. It was initially expected that new teachers who failed to advance to Level I status after their apprenticeship would be fired, since they would no longer be eligible for the state portion of their salary. However, in 1987, the career ladder was revised to make it optional for all teachers. The major consequence of failing to advance to Level I status was essentially the lost opportunity for the salary supplement.

Interestingly, it appears that relatively few teachers faced this cost. Nearly all of the state’s teachers (94 percent of them, according to one report) chose to enter the career-ladder program. A state audit in 1991 revealed that 95 percent of eligible teachers had achieved Level I certification, prompting criticism that the standards for this designation had been severely diluted. However, among teachers applying for certification at Levels II and III, the success rate was only 79 percent.

Though most teachers chose to participate, and the success rates for certification were quite high, some expressed criticisms that echoed issues often raised by merit pay critics: for example, that three classroom visits (some of them prearranged) were inadequate for evaluating teaching performance objectively and that separating the staff into levels strained relations among teachers and hurt morale. Even the application process was criticized for emphasizing, as the Christian Science Monitor reported, “cunning and endurance . . . rather than merit.” The criticisms suggest that, despite the relative sophistication of the career ladder, its efficacy in rewarding high-quality teachers remains an open question.

Project STAR

Coincidentally, a compelling way to evaluate the success of the career ladder system comes via data from Governor Alexander’s Student Teacher Achievement Ratio program. Project STAR was an experimental study of class-size reduction that also began in the fall of 1985. That year, it included 6,325 kindergarten students from 79 participating schools. The experiment lasted for three more years, following students through the 3rd grade. Overall, roughly 11,600 students participated, with additional students entering the participating schools in the 1st, 2nd, and 3rd grades. Participating schools were drawn from around the state and, by legislative mandate, included inner-city and suburban schools from larger metropolitan areas (Knoxville, Nashville, Memphis, and Chattanooga) as well as rural schools and those from smaller towns. All students in classrooms included in the experiment were given the Stanford Achievement Tests in math and reading in the spring of each year.

Pooling the information from the experiment’s four years yields a single data set with roughly 24,000 student observations for each subject. Roughly one-third of these observations are for black students, and nearly half were for students eligible for the free-lunch program. Fully 91 percent of the student observations in the dataset come from classrooms taught by teachers participating in the career ladder: 15 percent had teachers with probationary or apprentice status, 69 percent had teachers at Level I, while just seven percent had teachers who had reached Level II or III.

The key feature of the experimental design of Project STAR was that students and teachers within participating schools and grades were randomly assigned to one of three class types: small classes, regular-sized classes, or regular-sized classes with teacher aides. These random assignments allow us to use the STAR data to compare the performance of students assigned to career-ladder teachers with the performance of students in the same school and grade who were assigned to nonparticipating teachers.

Restricting the comparison to students attending the same school is essential because student-teacher pairings were random only within a given school. That is, the experiment did not move students and teachers to schools they would not otherwise have attended or staffed. This unfortunately means that some schools in the data set–those with classrooms taught by teachers with the same career-ladder status–do not offer useful information for looking at the effects of career-ladder status.

It should also be noted that student attrition from schools participating in the experiment was high, ranging from 20 to 30 percent each year, and that roughly 10 percent of students moved between small and regular classes. While most of the movement between classes was due to parental complaints or behavioral problems, the attrition figures could also reflect other factors unrelated to the study, such as students’ moving out of a school’s geographic zone or having to repeat a grade. However, if parents of students with unobserved propensities for high achievement sought out master teachers by class reassignment or by moving to another school altogether, our results would overstate the quality of career-ladder teachers.

Fortunately, we expect that these problems are less important for a study of the career ladder than for one about class size. Unlike a multiyear assignment to a particular class size, a one-year assignment to a particular teacher does not provide a strong incentive for attrition or reassignment. Students would be assigned a new teacher in the next academic year. By contrast, students placed in a large class were expected to remain in large classes through the 3rd grade.

Still, to evaluate whether the experiment successfully matched students and teachers randomly within schools, we examined the association between students’ traits and their assignment to a teacher of their own race. If the pairings of students and teachers were indeed random and remained so over time, we should find no within-school association between observed student traits and exposure to teachers in the career ladder. As expected, students’ race, gender, age, eligibility for the free-lunch program, and class-size assignment all exhibit small and statistically weak within-school relationships with assignment to a career-ladder participant.

Finally, because the student-teacher pairings were initially random, any statistically significant difference in performance between students with and without career-ladder teachers should be attributable to true differences in the quality of the teachers. The most conventional interpretation of such performance differences would be that the program provided effective incentives for teachers and that the evaluations carefully discriminated among teachers of high and low quality. However, the high pass rates on career-ladder evaluations suggest that these assessments were not particularly discriminating (at least through Level I). This raises the possibility that, if career-ladder teachers were more effective, it was simply because better teachers were more willing to negotiate the bureaucratic impediments to advancing on the career ladder. Nonetheless, even if the career ladder led only to self-sorting of teachers by quality, it would indicate that the program successfully directed its financial and professional rewards to meritorious teachers.

Results

To see what these Tennessee programs tell us about merit pay, let’s first look at the effects simply of having a teacher in the career-ladder program, ignoring for the moment the teacher’s specific level of accomplishment. To eliminate the effects of any chance differences in performance caused by other observable characteristics, our analysis takes into account students’ age, gender, race, and eligibility for the free lunch program; whether they had been assigned to a small class; and whether they were assigned to a teacher of the same race–which earlier research using these same data found to have a large positive effect on student performance (see “The Race Connection,” Spring 2004). We also include as control variables two conventional indicators of teacher quality: experience and possession of a graduate degree.

Our main results indicate that students with career-ladder teachers scored nearly 3 percentile points higher in mathematics than students with other teachers. They also suggest that reading scores were nearly 2 percentile points higher among these students, though the results for reading fall just short of conventional levels of statistical significance (see Figure 1).

The estimated effects on reading scores are statistically indistinguishable from zero primarily because they are less precise. If the effect on reading performance of having a career-ladder teacher were as precisely estimated as the effect of being in a smaller class, it would also be statistically significant. That it is not may reflect the fact that the experiment was designed to evaluate the effects of differences in class size, not the career-ladder program.

Regardless, our best guess is that having a career-ladder teacher in either subject had a quite large effect. The estimated gains associated with assignment to a career-ladder teacher equal 40 to 60 percent of the gains associated with assignment to a class with roughly 15 students rather than 22. Furthermore, the gains are approximately equivalent to a third of the black-white gap in test scores among students in the experiment.

When evaluating these results, it is important to keep in mind that 91 percent of the student observations in the data set came from classrooms with teachers certified by the career ladder. The benefits of having a career-ladder teacher are measured relative to a somewhat atypical base–namely, the small group of students whose teachers chose not to apply for the program or were unsuccessful in their application.

Our second analysis, therefore, considered not only the teacher’s participation in a career ladder, but also the teacher’s status within the program. That is, we looked separately at the effects of having a teacher at the probationary or apprentice level, at Level I, and at Level II or III.

In math, the career-ladder teachers at the probationary/ apprentice level and at Level I were the most successful at promoting achievement. In contrast, career-ladder teachers at the master level did not have a statistically significant effect on math scores (see Figure 2).

This surprising pattern could in theory reflect the success of the career ladder in attracting (and retaining) new, high-ability math teachers and in providing these new teachers with early mentoring and professional development. However, an alternative explanation is that novice teachers, many of whom quickly leave teaching, happen to be particularly adept at teaching math. The fact that we have already controlled for differences in teachers’ experience makes this explanation unlikely. Moreover, a similar pattern emerges when we look only at students with teachers having five or more years of experience, a good number of whom remained at the probationary/apprentice level (perhaps because fast-track options were not available in their area). In short, it appears that the career ladder simply was not very effective at distinguishing superior or outstanding math teachers from those who were merely competent.

In reading, by contrast, assignment to a Level II or Level III teacher was associated with a large and statistically significant increase in reading achievement, while estimates of the effects of having a teacher from both of the other two groups remained positive but statistically insignificant. This suggests the career ladder may have been modestly successful in identifying the most outstanding teachers in reading.

Conclusions

Overall, our results suggest that Tennessee’s Career Ladder Evaluation System was at least partially successful at rewarding teachers who were relatively effective at promoting student achievement. Though the program was voluntary for veteran teachers, the combination of large bonuses and relatively undemanding evaluations–at least at the lower levels–led the vast majority of teachers to enter. Nonetheless, assignment to a teacher who had been certified by the career-ladder evaluations led to large and statistically significant increases in mathematics scores and sizable, though statistically insignificant, increases in reading scores.

But our findings also suggest that the teachers who were on the highest rungs of the career ladder (and received the largest pay increases) were not consistently better at promoting student achievement. In reading, only students with a teacher at the highest levels of the career ladder made statistically significant gains. In contrast, the math-score gains associated with having a career-ladder teacher actually appear to have been concentrated among students with teachers on the lowest rungs of the career ladder. These mixed findings underscore the challenge of designing a system of teachers’ compensation that rewards quality in a fair and equitable manner–a political challenge as much as a technical one.

Despite some success in rewarding teachers for producing better student outcomes, the career ladder was a target of the same criticisms that challenge virtually all attempts to tinker with systems of teachers’ compensation. A few years of budgetary constraints helped kill the will to keep it all together. Thus, having made participation in the career ladder voluntary for teachers in 1987, it was perhaps inevitable that the Tennessee legislature in 1997 voted to prevent additional teachers from entering the program and becoming eligible for merit bonuses. Teachers already in the program, though no longer subject to regular evaluations, were allowed to keep their bonuses for the duration of their careers.

As Lamar Alexander lamented at the time, “Those who questioned the Model-T Ford didn’t try to kill it. They replaced it with something better.” Continuing debates over merit pay programs in districts in Tennessee and beyond indicate that efforts to find such a replacement are under way. But it may still be too early to tell whether the future for merit pay for teachers will resemble that of the Edsel or the Mustang.

-Thomas S. Dee is an assistant professor in the Department of Economics at Swarthmore College. Benjamin J. Keys is a graduate student in the Department of Economics at the University of Michigan.

The post Dollars and Sense appeared first on Education Next.

Surprisingly, neither the defenders nor the critics of education schools have produced research that answers that central question–which is why my colleague Susan Rozen and I embarked on a project several years ago to evaluate the course syllabi at selected schools of education. Our results were first presented as part of a Washington conference, sponsored by the American Enterprise Institute and the Progressive Policy Institute, then published as a chapter in the 2004 book, A Qualified Teacher in Every Classroom. We intended our work to be a first step in responding to the need for useful data about schools of education. What we hadn’t expected was the degree of interest–pro and con–and the passions that our conclusions would attract. In fact, I was invited back to the Progressive Policy Institute a year after our report came out to debate one of our critics. (See my discussion of Dan Butin below.) This piece addresses the responses to our work, and I hope it will continue to move the discussion about this important topic forward.

Some Background

Almost all the research to date about the quality of teacher preparation has been based on highly aggregated data that makes no distinctions between education schools or on interviews with teachers-college faculty members and their students (future teachers) about their personal experiences in college.

Stanford education professor Linda Darling-Hammond and several colleagues, for instance, once surveyed 3,000 beginning teachers in New York City to examine the relationship between their accounts of their preparation program and their confidence in their readiness for teaching. The survey results suggested that “Teachers prepared in a single formal program of preparation feel better prepared than those who take a series of courses from different institutions, who in turn feel better prepared than those who enter through alternative programs . . . and those who enter without prior experience or training.” But the study did not correlate teachers’ feelings of preparedness with measures of effectiveness, or, indeed, with any evaluation (other than self-evaluation) of their teaching ability. Much earlier (1989), Ken Howey and Nancy Zimpher, then at Ohio State, analyzed six exemplary–so they concluded–preservice teacher-education programs, all in the Midwest, but did not correlate their interview findings with program effectiveness. The key assumption of the study was that claims of ownership and identification with a program could be linked to program effectiveness–but no external validation of such assumptions was offered in the study. Nor did Howey and Zimpher offer any detailed analysis of course syllabi at the programs they reviewed.

A rare exception to this trend was research conducted by University of Georgia professor Peter Smagorinsky and Melissa Whiting of the University of Southern Mississippi nearly a decade ago. Smagorinsky and Whiting looked at syllabi for English methods courses from 81 universities for their book How English Teachers Get Taught: Methods of Teaching the Methods Class (1995) and, based on their findings, recommended learner- and activity-centered (“constructivist”) rather than traditional teacher-led programs. Of all these researchers, only Smagorinsky and Whiting actually reviewed a collection of syllabi, all in the field of English, to understand what prospective teachers were learning.

What We Intended

Beginning our research in 2002, my coauthor, Susan Rozen (Director of Reading/Literacy at the Bedford Public Schools in Bedford, Massachusetts), and I proceeded in the direction suggested by Smagorinsky and Whiting and decided to assess the nature of our future teachers’ schooling through an evaluation of the courses they were required to take. We expanded on that effort, however, and included a broader range of subjects and greater number of syllabi in our purview.

We reviewed a sample of syllabi in 16 schools of education, 14 of which were ranked in the top 30 in the nation by U.S. News and World Report (see Figure 1). We looked at schools like Harvard, Stanford, and UCLA (top ranked by U.S. News) as well as Eastern Michigan University and Sonoma State (not in the magazine’s top 30). We focused on initial certification programs and, within those programs, on the professional sequence required for certification. As a result, we looked at undergraduate programs, with the exception of those universities in states that required certification programs to be completed at the postbaccalaureate level (such as California). In certain universities, the traditional four-year undergraduate certification program has been expanded to five years, and it was therefore the five-year program we reviewed (e.g. Michigan State University). Where a university offered both a graduate and an undergraduate certification program, we chose the undergraduate program. We chose courses that were part of the required professional sequence because these courses are designed to prepare prospective teachers for teaching. We did not include in our review courses that were part of the general education requirements or courses that were specific to majors.

The syllabi we analyzed were in the domains of educational foundations, reading, and general methods courses (which typically include a student-teaching experience). In our original conference paper, we also presented findings in mathematics, but we were persuaded that in this field the academic courses were so integral to the professional preparation programs that we should not publish those findings. These domains form the heart of the education mission. If schools of education are essential, as their defenders argue, it is here that we would find evidence of the fundamental pedagogical skills that such schools provide.

Analyzing the course catalog in each university, we determined which courses in the professional sequence were required within the domains that we were reviewing. Then we gathered the syllabi for those courses from various sources, including professors, education school administrators, and the Internet. In a few cases we could not acquire a full set of the course syllabi in the domain under analysis, and so did not include that domain in our formal findings. In total, we reviewed 165 course syllabi that fell within our research criteria. Of these, 45 were in the area of the foundations of education, 61 in reading, and 59 in general methods and/or student teaching.

A Crucial Decision

As researchers, we then faced an important choice. We could simply record which texts were used in these courses and how often they were used, or we could list the books most often required (with numbers) and indicate which books were included rarely (again with numbers) or not at all. Although identifying worthy books that were not included on syllabi would open us up to criticism (as we soon discovered), the first option would have resulted in a long inventory of no clear significance, except perhaps to those within the profession. Naming books not included enabled us to draw out the significance of the books that were included. By juxtaposing counts of the most frequently required readings with the absence or near-absence of others, we were able to provide a first portrait of what future teachers are–and are not–learning at leading schools of education.

In the domain of foundations of education, the books most often required by the programs we reviewed were authored by Anita Woolfolk, Jonathan Kozol, Henry Giroux, Paulo Freire, Joel Spring, Howard Gardner, and John Dewey. Woolfolk’s work is a textbook in educational psychology, and one of Joel Spring’s volumes is a textbook in educational foundations. The rest are well-known works that embrace a constructivist and/or progressive standpoint. Conspicuously absent from almost all such syllabi were works that took a very different approach to teaching, such as those by E. D. Hirsch or Diane Ravitch. (We found Hirsch on two syllabi, Ravitch on just one.) Equality of education is a central theme of these courses, as evident from the included authors. Nonetheless, not one of the foundations courses, in the 15 schools of education for which we had complete data sets for that domain, asked students to read The Black-White Test Score Gap, at the time of our review arguably the leading collection of scholarly writings on that subject. We also noted that eight of the programs of teacher certification we reviewed did not cover either the philosophy or the history of education among the courses required for certification.

In our review of the courses in the teaching of reading, we followed the same approach. Here, however, we could draw on the findings of the National Reading Panel (NRP) and the National Research Council (NRC) to determine what was missing from the syllabi. Although recent research on reading is incorporated into some programs, work by Louisa Moats, Jeanne Chall, and Marilyn Adams–whose books and articles have been referenced frequently and used to support conclusions by the NRP and NRC–is rarely required. Analyzing the assignments and assessments listed in the syllabi, we also noted how rarely students were required to demonstrate competence in teaching reading skills and strategies. They were infrequently asked to demonstrate knowledge through presenting a lesson on critical reading skills in class or through taking a quiz or test.

Finally, we looked at the general methods courses and student-teaching experience. Here the norm is that the student first takes a methods course and then spends time in a public school classroom. While we again looked at required readings, for a gauge of the teaching experience part of the curriculum we focused on the handbooks that were provided to students as they entered their practicum and the descriptions of the practicum available in the course catalogs and/or on the web. Of the 11 schools from which we could get full data sets, just two indicated that they videotape their students’ teaching. Handbooks and catalog descriptions also suggested that professors at schools of education do not go to the schools to observe their students teach, but typically rely on adjunct instructors to report on the student teachers’ instructional performance. Finally, we found no evidence that in the practicum student teachers were being assessed on the basis of measured student performance. However, we have since learned that a very small number of schools of education are beginning to introduce this mode of assessment.

Strong Reactions

As a first effort to look at syllabi of required courses, our research represents a beginning. Fair concerns can be raised about technical matters. We were careful to name the 16 schools of education, and to define and list the number of courses and programs in each of the three domains we studied (foundations, reading, and methods), thus allowing other researchers to fully replicate our work. But we did not list the individual syllabi or name the particular schools (out of the 16) from which we were able to get full syllabi sets in each subdomain. It was clear when we sought syllabi from faculty members that, had we not assured their anonymity, we would not have obtained many of those syllabi: thus our choice to present aggregated data. Our data set was relatively small; certainly students take more than those courses that are required for certification, and a full survey would examine the many electives. We hope to address these issues in follow-up studies. When we conducted the research, more than half the schools of education listed many of their course syllabi on the web. Since that time, more are doing so. Ideally, this will allow future studies to include greater disaggregation of the data set.

Technical questions were not, however, the subject of most of the criticism we received at the Washington conference and in subsequent written and oral critiques. Kati Haycock, executive director of the Education Trust, said that we shouldn’t have focused on this “proxy stuff” and should have instead studied the links between individual teachers and their students’ achievement. We could agree with that–and have said that we encourage such research, although it is statistically very challenging. Meanwhile, our study focused elsewhere, on the quality of teacher preparation. Arthur Wise, president of the National Council for Accreditation of Teacher Education (the group that accredits almost half of the nation’s schools of education), argued that he didn’t think our survey sample was representative of “what’s going on at the institutions that prepare most of the nation’s teachers.” This is a fair objection, which we (and others) are now addressing, but a review of the top-ranked schools tells us what the profession looks toward as its best models. The faculty at elite schools are very influential in national policy debates, and they also train the future faculty at many less elite schools. Thus an analysis of elite schools, we believe, gives us important data from which we can draw some, albeit tentative, conclusions.

Other critics were harsher. Stanford education professor Darling-Hammond, author of The Right to Learn and well-known in our field for her emphasis on the importance of schools of education and professional certification for teachers, accused me of using “personal and political” standards and castigated our study as one of those “diatribes that come at the problems ideologically.” David Labaree, also from Stanford and author of The Trouble with Ed Schools, argued that we were wrong to take syllabi as serious evidence of anything: they are “more an ideological portrait of a course than actual substance.” As an observation about the nature and limits of syllabi, Labaree’s point is not to be denied: syllabi vary in their level of detail, ranging from a schematic account of texts, topics, and course requirements to a week-by-week specification of readings and assignments, along with supplementary readings, grading rubrics and formulas, and lengthy expositions of the perspective of the instructor. The thinner syllabi capture less information, and even the most complete will not capture what videotapes, for instance, would–the manner of presentation, the mode of teaching, the reactions of the students in class. But none of this undercuts our findings. The choice of readings is explicit in every syllabus and clearly indicates the formal content that the student should expect to encounter in the course. And by the same token, it is unlikely that much time in a class is devoted to texts that are not on the syllabus.

The critics were not done. The most recent critique of our efforts, from Dan Butin of Gettysburg College, calls me “disingenuous” for distinguishing empirical research about learning to read from opinions of professors who teach foundations courses. Professors speaking off the cuff at conferences or writing letters to others in the profession have fed into the controversy. Internet education blog Eduwonk.com eventually decried what it called “a nasty whispering campaign.” Frequently, we have had to point out that courses cited to us as evidence against our findings were not required courses but electives, just as our paper had made clear.

The strength of the discourse can, in all fairness, partly be blamed on us. Our complaint about the syllabi is a classic liberal one: students were not being exposed to a variety of important arguments and points of view. But it is true that we were deeply surprised by what our research found, and we expressed that surprise in our conference paper in strong language. We were perhaps too quick in using provocative phrases in our initial paper. Such phrases as “intellectually barren” and “too focused on indoctrination” unsurprisingly fueled incendiary comments. However, a problem of language cannot be taken to invalidate our conclusions.

Our Answers

Whispers–and personal attacks–aside, it is worth responding to the more substantive arguments of our critics. In a piece published electronically by Columbia University Teacher College’s TCRecord last August, Dan Butin raised an important issue of causality in an extensive critique. “There is an extremely loose coupling between what is written and what is taught,” he argued, “between what is taught and what is learned, and between what is learned and what is subsequently enacted in an efficacious way.” And though Butin rightly points out that we did not address those linkages, he says about his own paper, “This study will not address these assumptions either.” Butin’s point, of course, is equally applicable to all formal education: taken to its logical conclusion it suggests–counterintuitively–that we should not care about what is taught since education makes no difference to our future practice. At the very least, if future teachers are taught to believe in one set of principles and are subsequently forced to abandon them in practice, their professional preparation is problematic. It is worth adding that Butin criticized me for insisting on balance: “Steiner’s criteria of balance . . . focus on the wrong agenda.” Butin argues that I should have realized that reading books from only one perspective will enable “deep analysis.” Butin’s intellectual embrace of fundamentalism, the absolutism of his antiliberal stance, once again comes as something of a shock.

As for Linda Darling-Hammond’s claim that our standards of review were personal and political, we couldn’t disagree more. And we have explained and defended our standards in our published chapter: where there is empirical evidence for the positive impact of good teaching practices, we used it to infer what should be included in a teacher preparation course–thus the standards we suggested for the teaching of reading. Scholars disagree, naturally, about which empirical studies are most persuasive. However, our work is based on a broad set of major national research findings. In this domain, at least, the charge of playing personal politics looks a little shaky. In other cases, however, such as in our review of the foundations of education courses, no such empirical data are available. So, for example, we looked for readings that have stood the test of time and/or have proved influential in our current education debate. And we sought evidence that professors who were teaching their students about good teaching practices were also analyzing video of their students actually engaged in their student teaching. We were basing those desiderata not on research-based evidence, but on the assumption that education-school faculty would naturally wish to give direct feedback to this critical part of their student’s preparation for their profession.

It is therefore in these two areas–foundations courses and teaching methods–that one imagines that Darling-Hammond’s critique was intended to cut most sharply. Only a personal or political decision, apparently, could explain our hope for finding Plato as well as Dewey, or works by critics as well as defenders of multiculturalism. (I intend “multiculturalism” to be understood here not as describing our condition, but as embodying a particular progressive political and pedagogical program.) To ask that students be required to read E. D. Hirsch’s critique of Dewey while also asking that they read Dewey is thus to render a personal–since it is not an empirically or research-based–opinion. It is not, however, a political decision, unless, of course, one believes a call for balance is in itself political.

It is also important to note, in this context, that at the time of writing up our research, the coursework Stanford’s school of education required for teacher certification was on the web. Heading up the material was a list of suggested readings for new students in the program. At the time we reviewed the list, it comprised works by Gloria Ladson Billings, Elizabeth Cohen and Rachel Lotan (eds.), Linda Darling-Hammond, Howard Gardner, Jonathan Kozol, Deborah Meier, Jeannie Oakes, Laurie Olsen, Vivian Paley, Mike Rose, Frank Smith, David Tyack and Larry Cuban, and Guadalupe Valdes. There is as little empirical evidence that reading these books will improve one’s teaching as there is for any other list of foundational texts. To take one example, if reading Deborah Meier’s moving book (The Power of Their Ideas) about turning around a school enabled her readers to duplicate her performance as principal of Central Park East, urban education in the United States would be very different. What standards, therefore, did Darling-Hammond and her colleagues use in recommending their own books and those of the others they place on the list? Most likely, personal and political opinion, grounded in deeply held convictions.

But this response may occlude a deeper set of issues. The premise of Darling-Hammond’s charge is that there are standards for the choice of foundation course readings that are neither personal nor political. What could such standards be? On what “objective ground” does one choose Dewey over Plato, or ask for X rather than Y? One reasonable response is to suggest that such choices, at their thoughtful and considered best, represent not simply opinions but judgments, and to insist on the difference between the two. The judgment that future teachers should have read a Socratic dialogue is based on the experience of two millennia of readers, who have found in those dialogues an inexhaustible set of questions, problems, and insights. By contrast, a close review of that list of readings that Stanford University recommended to its new student-teachers is instructive. All of these authors are American, and all but one of their books was written in the past 30 years. In short, one of our top schools of education is convinced that nothing from China or Europe or Africa, and nothing written before 1974 merits inclusion. Rather than characterize the Stanford list (or our own call for balance) as the result of personal and political opinion, I would suggest it embodies an effort of judgment: a serious effort to shape the fundamental worldview of future teachers. It is this judgment that we should be discussing. My own is that as teachers, we join a community and a craft that stretch back for millennia. We owe it to the next generation of teachers to introduce them properly to the richness and depth that is the legacy of those who have come before.

David Steiner was until recently the chairman of the Department of Education Policy at the School of Education at Boston University. He is currently on leave to serve as Director of Arts Education at the National Endowment for the Arts. The opinions expressed here are his own.

The post Skewed Perspective appeared first on Education Next.

So, is it asking too much to expect those in the media charged with writing about education and NCLB to make some effort to describe them accurately? And shouldn’t we expect one of the nation’s most visible and influential education journalists to get it right?

I’m sympathetic to the myriad challenges that journalists face, but NCLB’s heft and the political battles around it are no excuse for someone like Michael Winerip, who writes the weekly “On Education” column for the New York Times (he is currently on a sabbatical to write a book), to distort the law into a vague semblance of reality. Take his September 24, 2003, column, for instance. Under the headline “On Front Lines, Casualties Tied to New U.S. Law,” Winerip reported that NCLB funding shortfalls were “devastating” for New York City. But he neglected to mention that the city had received more than $260 million in new dollars for poor students alone under NCLB in the previous two years (see “Who Got the Raw Deal in Gotham?” page 72).

It is especially problematic when the distortion is in the nation’s putative “newspaper of record.” As former Los Angeles Times education writer Richard Colvin, who now heads the Hechinger Institute on Education and the Media, says, Winerip’s “On Education” is “agenda setting” because of its influence on policymakers: “Editors across the country read what’s in that column and it informs their decisions.”

Winerip’s misleading writing about NCLB is particularly surprising because he has produced an impressive body of important journalism in a career of more than 20 years at the Times, reporting on everything from city politics in the 1980s to vital national issues in the 1990s. In 2000, Winerip was part of a Times team that produced a Pulitzer Prize-winning series on race in America. Many of his education columns are outstanding because of his clear eye for important subjects, including those that go unnoticed. For instance, he is one of the few reporters writing about the difficulties gay students face in school. And he can turn out delightfully quirky and interesting profiles, like his April 21, 2004, column about an 81-year-old woman returning to law school alongside hypercompetitive 20-somethings.

Obsessed with NCLB

But the very skills that produced this career of important work are missing in his writing about NCLB, which, needless to say, is an enormously important issue in education right now. In fact, since his first “On Education” column (January 8, 2003) to his last before going on sabbatical (May 26, 2004), Winerip devoted 15 columns-23 percent of his total-to NCLB, which he opined (October 1, 2003), “may go down in history as the most unpopular piece of education legislation ever created.”

It’s hard to know why Winerip became a knee-jerk NCLB-basher. But it’s not hard to see that he is one; nor need one be a partisan of the Bush administration (which I certainly am not) to grasp that his stance is costing the Times a chance to engage constructively in the debate about the law. There are problems with NCLB that scream for thoughtful explanation, rigorous attention, and public concern. This is not surprising with a federal law as complicated as NCLB. Any writer seeking to focus on it enjoys what the military would describe as a “target rich environment” and could make a genuine contribution to the debate on school improvement by engaging in a discussion of those liabilities. Winerip, however, has managed to avoid almost completely an accurate description of the act’s most important issues.

A good example is a February 19, 2003, column about Gonzales Elementary School in Tolleson, Arizona, a school that Winerip implies had been designated, because of NCLB, as “underperforming.” The Times columnist described a Kafkaesque series of hoops the school’s principal had to jump through because of the designation, including devising a detailed school improvement plan. But Winerip conflated the state’s accountability system and NCLB. The school would have received the same designation and been required to take the same steps in the absence of NCLB, a fact that Winerip omitted, while writing, “Unfortunately, last year the 5th grade did not make adequate yearly progress on the state competency exams. And that’s all it takes under the great new federal law.” Incidentally, whatever steps the school took seem to have worked. It did fine the following year under both Arizona’s accountability system and No Child Left Behind.

Are the Feds to Blame?

Routinely rushing to lay blame for local administrative upset at the feet of the feds, Winerip frequently fails to differentiate between state and federal requirements, much less offer readers information to understand why states differ from one another in their policies. For instance, in an April 28, 2004, column, Winerip described a school in Florida as unfairly penalized by NCLB, but he failed to mention that the school reported low overall test scores and had significant achievement gaps between white and minority students. He tartly noted that Texas (was the Lone Star state chosen to make a political point?) operates under different rules than Florida. But he failed to mention that Texas officials simply designed a different accountability plan, and that Florida could have done the same thing.

Likewise, in a September 3, 2003, column examining the differences between state and federal accountability systems, Winerip looked at North Carolina, where, he said, some schools that were doing just fine under the state’s previous accountability system were now being flagged as needing improvement under NCLB. He cited this as evidence of the folly of NCLB. What Times readers were not told, however, was that before NCLB, North Carolina, like almost every state, did not hold schools accountable for the performance of various subgroups, like minorities and special-needs students. Thus schools with good overall scores often had distinct groups of students that lagged far behind-a glaring inequity that the new federal initiative was explicitly designed to detect, but information Winerip conveniently left out of his story.

Winerip returned to North Carolina in his October 8, 2003, column, this time to bemoan the arbitrary demographic subgroup size of 40 (the minimum number of students that must be in the subgroup for the school to be held accountable for that group). But he failed to tell readers that it was the state, not the feds, that chose that number, that many states have even smaller subgroup sizes (and some larger, too), or that many advocates for poor, minority, and disabled students want small subgroup sizes in order to ensure that students are not lost in overall averages. Instead, without any context, Winerip readers are again left to conclude, on the columnist’s authority, that there is something arbitrary, if not downright crooked, about all these federal rules.

Missing the Point

Moreover, the inequity teased out by the subgroup rule is the core difference between NCLB and the 1994 reauthorization of the Elementary and Secondary Education Act. Understanding this is integral to making sense of the apparent conflict between state and federal systems. But Winerip referred only in passing to a “totally different federal formula.” Regrettably, through all the NCLB trashing, Winerip never described these issues so that lay readers, almost certainly unfamiliar with this context, could at least understand the current environment let alone learn about the complexity of the law and the challenges of designing policy for our decentralized system of schooling.

It’s not that Winerip doesn’t understand this context himself. Two of his most interesting columns focus on the challenges of the racial achievement gap. Yet inexplicably he never makes the leap from the problems he eloquently describes to NLCB’s intent, or potential, to help ferret out and rectify those same racial inequities. Similarly, No Child Left Behind’s left-leaning supporters like the Education Trust, Citizens’ Commission on Civil Rights, and Council of the Great City Schools, which are all concerned about the achievement gap, do not enter the Winerip conversation either.

Context is also lacking in his September 3 column, where he noted, “The federal system uses a single yearly proficiency goal-for North Carolina, 68 percent of students reading on grade level this year-and requires all schools to make that number.” In fact, the “safe harbor” provisions in NCLB mean that all schools do not have to meet fixed targets across the board each year, but only make some improvement in order to make adequate yearly progress. Certainly, provisions like this are somewhat arcane. But a reporter with Winerip’s skills could surely make them understandable for readers.

Besides, North Carolina’s relatively high bar is the result of a now somewhat problematic NCLB provision known as the “20 percent rule,” which was inserted into the law’s “adequate yearly progress” provisions. The concern was that some states would be starting with such low percentages of minority students at grade level that just requiring that as a starting point would subject the law to ridicule for having embarrassingly low standards. However, to ensure rigor, the alternative formula figured a state’s starting point as a function of 20 percent of the state’s grade-level population of students. In practice the provision has allowed some laggard states, at least initially, to get off easier than states that have been doing the right thing. Thus, paradoxically, many states that have been working to improve their school systems have more schools identified as failing to make adequate yearly progress under NCLB than trailing states. Though this is a complicated issue, it is a classic example of the trade-offs that federal policymakers routinely face and one that a thoughtful education writer with a national platform seemingly could explore for readers.

No Room for Subtlety

But, unfortunately, this sort of nuance finds no home in Winerip’s NCLB writing. He criticizes the federal law for basing school accountability on a single year’s test scores and holding schools accountable for the performance of transient students. In fact, NCLB does not require states to base school accountability on a single year’s test scores, but instead allows scores to be averaged over multiple years and permits states to use various statistical tools to help ensure the validity of those numbers. Nor does the law hold schools accountable for recent transfers. Similarly, Winerip wrote several times about testing disabled students, blaming requirements about assessments for special-needs students on the “Washington Brain Trust.” Yet, as mentioned earlier, he neglects to point out that many groups representing disabled students want these students included in state accountability systems. He also fails to share with Times readers any perspective on why it can be important to do so to help ensure that these students receive a quality education. Moreover, while criticizing NCLB for requiring new assessments for profoundly retarded students, Winerip makes little effort to explain what alternative assessments for such students would entail, to point out that there was ongoing regulatory debate about how best to design the federal policy in this area, or even to note that these students are a small fraction of the overall special-education population (most of whom benefit from access to mainstream standards).

Through all this, Winerip himself doesn’t do what policymakers must do every day-offer solutions to, or ideas about, vexing problems. Aren’t any NCLB problems being solved somewhere? What are the alternatives? What’s the most creative and serious thinking about addressing equity problems through means other than NCLB? Instead, it’s all griping and disparagement. Granted, he’s a columnist, not a policymaker, but many columnists regularly offer policy ideas, particularly in the midst of continuous slashing criticisms.

Whether bias matters in a news analysis column like this is debatable and, obviously, for the Times to decide. Hechinger’s Colvin argues that while news analysis does not require the same balance as a news article and can have a viewpoint, “that perspective needs to be backed up with reporting and context.” Thomas Toch, a longtime education journalist now working as writer-in-residence at the National Center on Education and the Economy, goes further. Toch notes the importance of the venue, but says Winerip should “be given a lot of leeway to interpret the education landscape as he sees it, to exert his voice in the debate.” Either way, there is a difference between viewpoint or voice and a relentlessly misleading presentation.

It’s important to note that the problem is not editorializing in the news coverage. In fact, the Times’s editorial position is largely supportive of NCLB. Instead, the problem with Winerip’s NCLB columns is that they often turn on incomplete, outlandishly selective, and even inaccurate presentations of the facts.

In the end it is beside the point to parse the motives of Winerip’s anti-NCLB mania. There are plenty of excellent education journalists out there who could offer readers less slash and burn and more nuance and context about a debate as important as this. In fact, Samuel Freedman, who has been writing “On Education” during Winerip’s book leave, is a terrific example. His writing is varied and thought-provoking, but also apparently without ideological blind spots and tendentious selective presentation. In other words, he brings to the page everything an important column like this should regularly deliver.

-Andrew J. Rotherham is director of education policy at the Progressive Policy Institute in Washington, D.C., and editor of www.eduwonk.com.

The post No Distortion Left Behind appeared first on Education Next.

Michael Winerip, “On Education”
columns for the New York Times,
September and October, 2003

When public schools opened in New York City in September 2003 amid reports of widespread classroom overcrowding, parents, educators, and policymakers demanded an explanation. And there, at the ready, was Michael Winerip, the education columnist for the New York Times.

The crowding, wrote Winerip in the first of a series of hard-hitting columns in September and October of that year, was caused by the “new students with challenging problems” whose parents took advantage of the No Child Left Behind law allowing them to transfer from a persistently failing school to one that was better. The 1,000-word reports were so convincing that even Mayor Michael Bloomberg began blaming NCLB for what Winerip called the “overloading” of Gotham’s middle and high schools. “What the federal law says is that if you are in a failing school, you have a right to demand that your child go to a good school,” the mayor said. “There aren’t any seats in the good schools. Those are full already anyway.”

Unfortunately, it wasn’t true. Months after the Winerip columns appeared, after the federal law had been trashed by editorial writers, teachers, and parents, city education officials released transfer tallies showing that kids looking for better schools under NCLB had gotten a bad rap. If there was new overcrowding-and even that was doubtful in many cases-the federal law had not caused it. (In many cases, in fact, overcrowded classrooms turned out to be in school buildings that had additional space, suggesting that the problem was one of local management.)

Of course, the Times columnist wasn’t alone in fanning the flames of the anti-NCLB hysteria, but his influential columns made it easier for city officials to avoid talking about the real reasons for the problems, namely, a demographic bubble that was moving its way through the city’s middle and high schools, combined with the city’s new initiative to create experimental “schools within schools” in certain buildings, a policy that required a cap on enrollment in some schools at the cost of crowding elsewhere.

For some reason Winerip was intent on finding fault with NCLB. Take Intermediate School 89, in Lower Manhattan, which he profiled in his September 10, 2003, column. He illustrated the problem of the 300-student school’s “newly overcrowded classes” by describing a classroom of 37 students where some children were sitting on benches and on the floor. Winerip quoted a teacher who worried that class size could swell to 40 before all was said and done. The principal, Winerip wrote, had been “informed by city officials that she would be receiving many additional student transfers under the federal No Child Left Behind law.”

Two weeks later, Winerip’s column focused on crowding caused by NCLB at Booker T. Washington Middle School on West 107th Street in Manhattan. “The two 6th-grade classes, taught by Erica Williams and Robyn Block, started with 32 each,” he wrote, “and thanks to federal transfers, ballooned to 43 and 41.” Concluded Winerip, “Like many other middle schools in New York City, this one is overloaded with children who have transferred in under the federal No Child Left Behind law.”

A week later, Winerip crusaded against NCLB overcrowding yet again, announcing in his first paragraph that NCLB “may go down in history as the most unpopular piece of education legislation ever created.” Describing crowded conditions at Public School 19 in the Bronx, the column noted that “Ms. O’Boyle’s 8th grade class had 40 students”-again suggesting that NCLB kids were to blame.

The same column lamented the crowding at Middle School 141 in the Bronx, which grew by 73 students that year. It also noted that enrollment at Intermediate School 192 in the Bronx grew to 1,370, an increase of more than 13 percent from the 1,211 the previous year. All caused by NCLB, Winerip suggested.

What the records show is that most of the worst increases had nothing to do with NCLB.

At P.S. 19, for example, where Ms. O’Boyle had 40 8th graders in her room, records showed that none of those students were there because of the federal law. At Intermediate School 89, where Winerip had described children sitting on the floor, the official data showed that while enrollment grew by 18 students, only 12 were NCLB transfers, and even those were spread out over three grades-seven in 6th grade, four in 7th grade, and one in 8th grade. At Middle School 141, which Winerip complained was overcrowded with 73 new students, it turned out that only 29 of them had transferred to the school under NCLB. At I.S. 192, only 51 students, not 159, as Winerip suggested, had transferred in under NCLB. And at Booker T., the records showed that only nine NCLB transfers ended up in the 6th-grade, even though Winerip had suggested that the 6th-grade classrooms of teachers Williams and Block increased by 20 new students because of the federal law.

The other interesting thing to note about Booker T. was that, NCLB aside, there wasn’t any overcrowding in the school: According to the New York City School Construction Authority, the school was built to hold 1,104 students and was still, officially, 13 children under capacity.

The I.S. 192 building, which Winerip complained had a double-digit increase in its student body, was built for 1,527 students, meaning it still had room to spare after welcoming all the new students in 2003 (only a third of whom, as we saw, actually arrived because of NCLB). And while it isn’t clear why Winerip chose I.S. 89 to profile for his first anti-NCLB installment, it is clear why the city’s education department was steering children in that direction: there was room for an additional 159 students. (After several weeks of school, another 6th-grade teacher was added, further minimizing the impact caused by the seven children who transferred to that grade under NCLB.)

In Winerip’s rush to judgment, he confirmed the fears of many who opposed the transfer provisions of NCLB, including some parents, teachers, and administrators who were eager to blame crowded classrooms on the outsiders who entered under the law.

No doubt there were numerous problems with the way New York City school officials implemented NCLB in the law’s first two school years, as indicated by the city’s recent decision to limit NCLB transfers in the 2004-05 school year. And, undoubtedly, local school administrators in New York City, as most anywhere, can be expected to say their school is overcrowded.

The facts in this case, however, make it difficult to show that transfers under the law caused overcrowding in 2003, when Winerip was complaining; they even suggest that school officials did an uncharacteristically competent job of linking transferring students with schools that were in the best position to accommodate them. An analysis I did for my newspaper, the New York Daily News, compared all 6,979 transfers in the city for 2003-04 with the official building capacity of the receiving schools the previous school year and found that nearly 60 percent of those transfers went to schools with more than enough room to spare.

“We did our best not to overcrowd schools,” said Jeremy Lack, of the city’s No Child Left Behind office. “And at the end of the day, no student who followed through the process was denied a transfer.”

Chancellor Joel Klein even addressed the bad rap that NCLB transfer students were getting in an e-mail to the city’s 1,200 principals that fall (2003). In the e-mail, Klein scolded some principals-though taking care not to name those who complained to Winerip-for trying to “protect their own school regardless of what it means for everyone else.”

Wrote Klein: “Whatever else you may say about No Child Left Behind, it is forcing us to come to grips with a moral imperative that many of us might prefer to ignore: that we have a collective responsibility for the education of over 1 million children.” Needless to say, the New York Times education columnist did not mention Klein’s memo.

But Winerip, by playing fast and loose with the facts, may have succeeded in killing school choice in New York City after all. In the summer of 2004, school officials announced that they were cutting the transfer rate from 7,000 to 1,000. In the midst of a political campaign, federal officials seemed ready to acquiesce in this gutting of one of NCLB’s most important provisions.

Joe Williams is a staff writer on education for the New York Daily News.

The post Who Got The Raw Deal in Gotham? appeared first on Education Next.

Diana Jean Schemo, “Nation’s Charter Schools Lagging Behind, U.S. Test Scores Reveal,” New York Times, August 17, 2004, page A1

It is not unusual for interest groups to issue reports that further their own political agendas–and to muddle the facts in the process. For this reason, newspapers generally ignore them, treat them with great skepticism, or make sure they properly vet the research with independent observers.

Not so in the case of the study of charter schools leaked by the American Federation of Teachers (AFT) to the New York Times, which then placed it in the right-hand column of the front page of its August 17 edition–a slot typically reserved for the day’s biggest story. Headlined “Nation’s Charter Schools Lagging Behind, U.S. Test Scores Reveal,” the story sent shock waves through the charter school movement and left more than a few education reformers scrambling for cover.

Using the data tool on the National Center for Education Statistics website, the authors of the AFT study called up some basic numbers on the performance of students from a nationally representative sample of charter schools. Their conclusion: “Charter schools are underperforming.” Their evidence: data from the National Assessment of Educational Progress (NAEP), often called the nation’s report card, showing students in charter schools doing less well than students in other public schools nationally, as well as in a small number of more focused comparisons.

The Times had a field day with the news. The AFT’s findings, the paper reported, “dealt a blow to supporters of the charter school movement, including the Bush administration”–a blow made all the more powerful (and credible) by the fact that the AFT had “historically supported charter schools.” Amy Stuart Wells, a sociology professor at Columbia University Teachers College, was quoted as saying the data were “really, really important” as they “confirm what a lot of people who study charter schools have been worried about.” Would that it were that simple.

Where do we begin to sort out the outlandish claims of the AFT study–and those made by others on its behalf? For starters, saying the AFT has historically supported charter schools is like saying that the Chicago Cubs are historically a World Series champion baseball team. While technically true (legendary AFT president Albert Shanker helped introduce the concept in a 1988 speech to the National Press Club), the union’s position on the issue has changed so markedly that it is now one of the staunchest opponents of charter schools around the nation. In recent years the AFT has criticized charter schools in a series of reports, of which August’s was only the latest and best publicized.

But hardly the most sophisticated. Indeed, on a methodological level, the AFT analyses are sufficiently pedestrian to be laughable. And most mainstream newspapers around the country–once the Times had made it the story of the hour–had the good sense to present a more critical view of the study’s import. In the title and lead paragraph of its coverage, USA Today noted that “achievement [is] not so simply measured” and that critics had already pointed out that “the report is hardly a fair look at whether charter schools help kids improve.” The Seattle Times quoted University of Washington researcher Mary Beth Celio’s dismissal of the study as “one of the most unsophisticated, low-level analyses I’ve ever seen.” The editorial board at the Chicago Tribune went further, deeming the AFT findings “about as new as a lava lamp, as revelatory as an old sock, and as significant as a belch.”

A Flawed Report

What’s wrong with the study? The basic problem is straightforward: raw comparisons showing charter school students scoring lower than public school students on standardized tests may simply reflect the fact that charter schools serve students in low-performing districts with high concentrations of poor and minority children. Many states allow charter schools to form only where students are having difficulties, and many charter schools are then asked to accept the most challenging of students. Any credible analysis of their effectiveness must account for these facts on the ground.

Indeed, if the AFT believes its own findings, it must also concede that private religious schools outperform public schools (see Figure 1). According to the same NAEP data that are the basis for the new AFT study, religious private schools outperformed the public schools nationwide by between 9 and 17 points, a gap at least as large as the public school-charter school difference that the AFT–with considerable help from the Times–is trumpeting. On past occasions, the AFT has objected vehemently to interpreting such findings as evidence that religious schools are superior on the grounds that they attract an especially able group of students. But for charter schools, it seems, the problem of selection effects need only be addressed in the most superficial of ways.

The authors’ sole strategy to “enhance the fairness of the analysis” was to look separately at students in 14 categories, including those from six different states, those who qualified for the federal free-lunch program (and those who didn’t), those from different ethnic backgrounds, and those living inside and outside a central city. As a strategy to control for the background characteristics that differentiate students in charter and traditional public schools, this approach is feeble. At best it can eliminate the effects of differences with respect to one background characteristic at a time. But it may not even be effective for that purpose if, for instance, the students eligible for a free lunch who attend charter schools come from even poorer families than eligible students in traditional public schools.

Even so, in most of the comparisons holding just one characteristic constant, the performance differences between charter and traditional public school students attenuate to the point of statistical insignificance. Twenty-one of the 28 comparisons the AFT conducted using 4th-grade average scale scores are statistically insignificant. As previous research has found that ethnic differences in achievement are large, it is especially noteworthy that all comparisons within ethnic groups in the NAEP charter school data cut against the AFT’s overall conclusions. The small differences that remain when looking separately at white, African-American, or Hispanic children are all statistically insignificant–a fact that is not apparent in either the Times story’s text or the tables that accompanied it.

But do any of these comparisons–within ethnic groups or otherwise–tell us anything meaningful about the quality of traditional public, charter, or religious private schools? Not a bit.

Plainly, to account adequately for the influences of a child’s family, home environment, and community on his or her learning capacity, one must do much more than look separately at students grouped by free-lunch status, ethnicity, or school location. At a minimum, it is essential to gather detailed data on students’ background characteristics and to put them to good use. Control variables now standard in education research include parents’ education and marital status, household income, and the quality of learning resources in the home, to name but a few. And rather than using aggregate comparisons within subgroups to eliminate the effects of differences in one background characteristic at a time, as the AFT has done, the influence of all of these factors must be addressed simultaneously.

But all this may just scratch the surface. As schools of choice, charters are likely to attract students who are not doing well in their traditional public schools. Moreover, many charter schools explicitly target “at-risk” students. Both of these facts would lead you to expect students in charter schools to perform at a low level even after taking into account their observable back.ground characteristics.

Ideally, one would therefore study charter schools in the context of a randomized field trial, assigning students randomly to attend either a charter or a traditional public school, gathering data on their performance at baseline, and tracking their progress over time. In the absence of that possibility, it is vital to use data from multiple years to track the learning trajectory of students in both charter and traditional public schools.

Yet another critical flaw in the AFT’s analysis is its failure to account for the length of time that a charter school has been in place–a factor known to affect any school’s performance. Having just hired new staff and teachers, implemented new curricula, and acquired building facilities, new schools often face considerable start-up problems. Almost one-third of the charter schools nationwide were less than two years old when the 2003 NAEP was administered, raising doubts about whether even meaningful findings about charter school performance would apply when more of them are well established.

Encouragingly, research on charter schools using more reliable methods to gauge school quality is under way. Nonetheless, it will be some time before definitive conclusions about the merits of one of the nation’s most prominent, and popular, reform strategies can be drawn. In the meantime, the AFT’s study does not even amount to a good interim report.

Why All the Fuss?

Given all of these problems, why would the Times see fit to bestow instant credibility on the AFT study by granting it glowing, page-one coverage? While we have no special insight into the motives of the newspaper’s editorial staff, the coverage itself suggests two factors that are important.

The first concerns alleged chicanery by the U.S. Department of Education, which, reported the Times, had buried the flawed charter school findings in “mountains of data . . . released without public announcement.” According to the authors of the AFT study, “a combination of intuition, prior knowledge, considerable digging, and luck” was required just to locate the data. Such sleuthing makes for dramatic storytelling–for the next best thing to doing it oneself, in the newspaper business, is reporting (exclusively, one hopes, so you can break the news) on someone else’s discovery of a cover-up.

As Bella Rosenberg, one of the report’s three authors, explained to the press, “Analyses are always welcome, but first things first. . . . Surely the interests of children are better served by timely and straightforward information about whether charter school performance measures up to the claims made for it.” In a letter to the Times, educational psychologist Howard Gardner praised the AFT for its act of public service in issuing the study and then asserted that the Department of Education’s decision not to highlight the findings was ideologically driven: “If the results had been positive, the Education Department would doubtless have heralded them. Across the policy spectrum, the pattern of the administration is all too clear: Call for evidence-based results, tout them when supportive, hide them when not, spin them when possible.”

Perhaps. But we draw a slightly different conclusion. Timeliness and transparency are important, but bad information is worse than none. And uncovering misleading information and presenting it out of context does a greater disservice to the “interests of children” than the Department of Education’s decision not to issue a report that does not control for student background characteristics. From this perspective, the AFT study and the Times‘s breathless coverage of it only made a bad situation worse.

The second probable reason for the prominent attention the Times gave the study stems from the fact that charter schools represent one of several remedies for schools deemed chronically failing under George W. Bush’s No Child Left Behind Act. (Other remedies include replacing much of the school’s staff or turning its operations over to the state or to a private company.) Thus the story’s import was magnified by the politics of education reform: it suggested a flaw in the Bush administration’s game plan. The very next day, the lead Times editorial heralded the report as “a devastating setback” to the Bush administration’s education program.

Ironically, however, it is not at all clear that political cleavages over charter schools follow strictly partisan lines. Indeed, federal financial support for the charter school movement has its origins in the Clinton era. Democratic presidential candidate John Kerry was an enthusiastic supporter of charter schools. And while Secretary of Education Rod Paige was a vocal proponent of charter schools, President Bush said hardly a word about charters on the campaign trail–nor, for that matter, did he say much about them from the White House.

What the NAEP Data Do Tell Us

While the statistics on the nation’s charter schools currently available from the NAEP are not at all useful for assessing these schools’ effectiveness, they do offer, for the first time, a glimpse of the makeup of a nationally representative sample of the students who attend them. As a result, one important fact about charter schools now appears incontrovertible: they are not bastions of wealth and privilege.

As Figure 2 shows, almost 62 percent of the roughly 3,000 4th graders in the NAEP charter school sample attend a school located in a central city, compared with just 32 percent of NAEP 4th graders in traditional public schools. Roughly 33 percent of the charter school students are African-American, compared with only 18 percent of the public school students. Fifty-four percent of elementary charter school students qualify for free or reduced-price lunch programs, compared with 46 percent of public school students. The analogous differences for the 8th graders tested by the NAEP are even more pronounced, perhaps reflecting the fact that a large number of middle and high school charters target at-risk students.

Given the conditions under which states and districts accept charter schools, the language of their mandates, and the characteristics of families most eager for alternatives to traditional public schools, these differences can hardly come as a surprise. For the foreseeable future, charter schools are likely to serve high concentrations of poor and underprivileged students. What remains unclear is how much they can do for this population. Sadly–and despite the impression given by the gray lady of American journalism–the AFT study tells us nothing about that.

William G. Howell is an assistant professor of government at Harvard University. Martin R. West is a research fellow at the Program on Education Policy and Governance at Harvard University and the research editor of Education Next.

The post Gray Lady Wheezing appeared first on Education Next.

Just about everyone with experience in public schooling knows what a teachers’ salary schedule looks like. (You can see a specimen in our Forum section, accompanying Brad Jupp’s story, “The Uniform Salary Schedule,” page 10.) Along one axis are 15 or so “steps,” usually based on years of experience in that school system. Across the other are categories of teacher “qualifications,” with higher pay depending on how many university courses and degrees the teacher has acquired. In the specimen from Denver, for example, a “step 1” teacher (in this case, that’s a step above “new hire”) with a bachelor’s degree is paid $32,971, while a “step 10” teacher with a master’s degree earns $46,860.

Such schedules have three profound failings. First, they pay no attention to the labor market for people possessing a specialty. Thus the gym teacher earns exactly the same as the AP physics teacher, who earns the same as a middle-school social studies teacher, though all know that their job options outside of teaching are entirely different.

Second, they pay no attention to the relative challenge of teaching in particular schools within a district. Thus an instructor in the toughest downtown high school gets paid the same as one teaching in the most serene of suburban-like elementary schools. Especially for poor districts, such schedules-usually devised and agreed to via collective bargaining-make it brutally hard to manage the educationally productive distribution of limited dollars and human resources.

Third, they ignore teacher effectiveness. Despite all we have learned about huge differences in teaching skills and the profound impact such differences have on children’s learning, the mediocre (or worse) teacher earns exactly the same as the classroom superstar.

Insane, no? It’s impossible to picture how the United States will radically upgrade the quality of its three-million-member K-12 teacher cadre, attract more able people to enter and stick with this field, ensure needy kids’ access to the best instructors, or span the curriculum with people who really know their subjects as long as this archaic, civil-service pay system persists.

This issue’s Forum section illustrates the emerging consensus, bringing together three people who might otherwise have very different perspectives on public education. On this they agree: It’s time to replace the uniform salary schedule with something that takes account of (at minimum) subject specialty, school environment, and classroom effectiveness. The Forum also illustrates some of the tantalizing innovations now visible on this front, from Denver’s pioneering teachers’ pay system to the Milken Family Foundation’s Teacher Advancement Plan.

The innovations, however, remain tiny and tentative alongside the massive challenges.

Timorousness in the face of obdurate teacher unions (Denver being a happy exception) is perhaps the greatest cause of our halting progress in this domain. But reformers must also face the reality that getting an alternative system properly calibrated, making it work, and holding onto it are exceptionally challenging.

This we can see in Thomas Dee’s account of Tennessee’s 13-year attempt to install a workable “career ladder” for the Volunteer State’s public-school teachers (see “Dollars and Sense,” page 60). Despite the fact that it seems to have enhanced teacher effectiveness, issues arose from day one. Whether to “ration” the upper rungs of the ladder or let all teachers ascend it. Whether to judge teachers’ readiness to climb according to their students’ achievement, supervisors’ (or peers’) appraisal of their classroom prowess. What’s fair to teachers? What’s administratively workable? What can the state afford? What will the union tolerate?

In the end, just about every Tennessee teacher clambered onto the career ladder. Salary differentials shrank; grounds for ascending became ever more diverse and less exacting; and never did the program deal with such variables as subject specialty or working conditions.

A failed experiment? No, a cautionary tale: Everyone knows we must erase the uniform salary schedule. But close attention must be paid to what will replace it.

-CHESTER E. FINN JR.

The post Paying Teachers Properly appeared first on Education Next.

The post Supplemental services; keeping good teachers appeared first on Education Next.

The post The Softening of American Education appeared first on Education Next.

The post Reading, Writing, and Willpower appeared first on Education Next.

The post Book Alert appeared first on Education Next.