“Well, I said we’d go from 10:30 to noon,” Bob Ward reminded the crowd, trying to end a sold-out public policy forum on “Charter Schools in New York and the Nation.”

The session in the second-floor seminar room at the Nelson A. Rockefeller Institute of Government in Albany had already featured a detailed presentation of “charter facts” from the new executive director of New York’s Charter Schools Institute and a dozen friendly questions from the mostly pro-charter audience. Ward, the dignified and cerebral deputy director of the institute, seemed anxious to wrap things up. “So, thank you—”

He stopped in midsentence. A hand had shot up at the back of the crowded room. “Eva, did you want to ask a question?” he asked.

All eyes turned to the dark-haired woman sitting on a folding chair along the back wall of the room. Some eyes rolled, as most of the group knew Eva Joseph, the embattled superintendent of Albany Public Schools (APS). They had seen her at countless education forums, on the local nightly news, and in the daily paper at every turn of the school budget clock, determinedly defending her district and, increasingly, railing against charter schools. “I’ll make it quick,” said Dr. Joseph. “I do want to thank you for acknowledging the situation in Albany, but going to the heart of what’s real, we have 10 charter schools in Albany with a total public school population of 10,500 students. Compare that to 23 charter schools in the Big 5, with the exception of New York City. Buffalo, Syracuse, Rochester, Yonkers. Twenty-three total charter schools and you total up their enrollment. The proliferation here. The oversaturation, per pupil and per capita, is glaring. And it has serious implications for the district. It destabilizes it on many fronts….”

Standing a few feet away, as Joseph plunged on, a man leaned against the wall, smiling. It was not a smug or obvious smile, nor the smirk of a man who was mocking or scornful. Tom Carroll was smiling because he had heard the speech before and because he knew, as founder of the charter school foundation that had siphoned off nearly a quarter of Dr. Joseph’s 10,500 students, that he was at least an immediate cause of the vitriol. It was the smile of victory.

The Holy Grail of Charter Schooling

During much of the previous nine years, Carroll had overseen the launch of eight charter schools (see Table 1) in Albany, a small city (pop. 94,172), as Joseph suggested, for so many charters (to see additional images of the schools and their students please click here). Joseph, who began her Albany tenure as an assistant superintendent in 1997 and took the top job in September 2004, had been engaged in the charter battle for most of that time. “Fifth Albany Charter School Approved” was the headline just two months after she became superintendent. “SUNY adviser suggests city district cut staff, rent out extra space as students depart.”

What had been especially maddening for Dr. Joseph and her school board, which issued routine condemnations of the charters, was not just the presence of so many of the new schools—“the proliferation, the oversaturation”—but that they were so good. The destabilization was real and deep, creating not just viable, but quantifiably better, educational alternatives for children. And this is the singular accomplishment of Carroll’s charter organization: they had found the Holy Grail of charter schooling, quality and scale.

It is still an elusive goal for the charter school movement, which has grown to include more than 4,500 schools and 1.3 million students in 40 states and the District of Columbia. This remains a drop in the public school bucket (nationally there were more than 94,000 public K—12 schools and more than 49 million students in 2007), which is why “market share” is considered a crucial milestone, one of the few ways to pinch traditional schools in their pocketbooks. According to the National Alliance for Public Charter Schools, in 2007 New Orleans had reached 55 percent market share, Washington, D.C., 31 percent, and Southfield, Michigan, and Dayton, Ohio, 28 percent each.

Albany at that time came in at 17 percent, tied with Buffalo for 12th place nationally. But it had already distinguished itself as the only member of the market-share club with consistently high academic outcomes.

In fact, the failure of charters to offer a meaningful choice, i.e., a better education, has become a sore point among charter promoters. Education Next editor Chester E. Finn Jr., in a 2007 confessional in his Thomas B. Fordham Institute newsletter (the Education Gadfly), wrote, “Why are so many charter schools inadequate, even mediocre? What went wrong?” Finn noted that in his own “charter-saturated” Dayton, where Fordham was born, things had gone terribly awry. And though charters have taken to putting a good face on things by comparing themselves to their local district schools, which is fair, the truth about quality is uncomfortable. In the fall of 2008, for instance, the Dayton Daily News published a story headlined, “Most charter schools made gains; most Dayton district schools saw losses.” But an accompanying chart revealed that 12 of Dayton’s 19 K—8 charters did not make Adequate Yearly Progress (AYP); 8 of the 10 charter high schools fell short of their AYP benchmark.

By contrast, all of Tom Carroll’s Albany charter schools made AYP. Not only that, they had become the best schools in the city. “Last year our students were number one in math in every single grade,” says Carroll, who now runs the Foundation for Education Reform & Accountability (FERA), which furnishes research help to charters, and serves as chairman of the Brighter Choice Foundation (BCF), which provides start-up financing aid. “In English, we were number one in 4th and 7th grades. We’re expecting to do even better this year.” And they have. On the 2009 state test in English language arts, in four of the six grades tested, the top school in Albany was one of Carroll’s charters.

Albany public school parents, mostly black and mostly poor, not only have a choice; they have one that will make a significant difference in their children’s futures. The Brighter Choice network has turned largely forgotten students into serious achievers. These schools have not only closed the achievement gap; they have reversed it.

An Unlikely Road to School Reform

“You might say that our success is the revenge of the amateurs,” jokes Carroll, over a recent lunch at a downtown Albany bistro. “We didn’t really know anything about education when we started—and perhaps that’s why we have succeeded.”

A veteran of the sharp-elbowed politics of New York’s infamously dysfunctional state legislature (called the worst in the nation by the Brennan Center for Justice in 2004), Carroll assembled a team of equally determined and savvy colleagues, most of whom had honed their political skills in those same tough trenches. They sounded like a law firm: Carroll, Backstrom, Murphy, Bender, and Brooks. Eva Joseph called them, disparagingly, “the white guys.” In fact, they knew money and they knew politics, and when they stumbled on to the disastrous state of public education, they became determined to know schooling.

“We were all focused on budget and tax issues that would make New York a more job-friendly place,” recalls Peter Murphy, who had worked with Carroll in the state’s budget office. “But we did lots of work examining various parts of the government, including education.” Increasingly, more budget and tax roads led to education, which consumed more than a quarter of the state’s revenues.

While Murphy continued work with a think tank called the Empire Foundation for Policy Research, helping to produce a report on education choice in 1993, Carroll and Brian Backstrom, another veteran of the budget office, and Jason Brooks, a fresh-faced triple major (history, religion, and political science) from Syracuse University, started FERA in 1998. FERA would seal their fates as education reformers when Virginia Gilder, then the wife of one of their major political reform benefactors, Wall Street financier Richard Gilder, asked Carroll what she could do to help fix the schools. Since there was no charter law in New York at the time, the group launched a voucher program. And in an early test of their market-share strategy, Carroll and company decided to spend all of Gilder’s money at just one school, offering $2,000 to 153 students, a third of the student body, at Giffen Elementary, “one of the worst public schools in New York State,” according to Forbes magazine, which featured the program on its cover.

The focus on one school, the national attention, and Giffen being “within spitting distance of the State House,” as Forbes put it, ensured that FERA would be an education reform player and an immediate thorn in the local school district’s side. Even Fred LeBrun—an influential columnist for the Times Union who once called Carroll and his political friends “a blustery gathering of overstuffed three-piece suits with watch fobs”—sympathized, praising the group for “walking the walk, not just talking the talk.” (The message was lost on Albany Public Schools: to this day, Giffen remains hopelessly behind the academic eight ball, with just 46 percent of its 168 remaining students in grades 6—8 able to pass proficiency tests in English and 57 percent passing in math. Carroll still oversees the voucher program, which continues to provide options for 38 Albany public school students each year.)

The next opportunity to walk the walk came in 1998 when the group helped write the state’s charter school law, passed in December, making New York the 34th state to have one.

The Road to a Brighter Choice

Carroll and his colleagues, now fully engaged in their roles as school reformers, immediately established the Charter School Resource Center to offer technical assistance to anyone willing to set up a charter school.

They knew their market. Almost overnight 90 applications for charter schools (the law set a cap of 100) were submitted to the state education department’s Board of Regents and the newly formed Charter Schools Institute at the State University of New York (SUNY). “You know you’ve hit on something when there is that level of interest across the state for doing something different in public education,” says Backstrom.

Carroll, Backstrom, and Brooks traveled the country and spent months on the phone quizzing successful school leaders: what works in your school, what doesn’t work? “And time and time again we were struck with how similar the answers were,” says Backstrom. Longer school day. Longer school year. Content-rich curriculum. School uniforms. Even the single-sex school, they learned, which had been all but driven off the education landscape by Title IX, was being tried, if quietly, and was working.

In November 2000 the group submitted a 300-page application to the Board of Regents to open the Brighter Choice Charter School for Boys (BCCS-Boys) and Brighter Choice Charter School for Girls (BCCS-Girls), which, initially, would be housed in the same building.

“The City of Albany is an educational tale of two cities,” the applicants wrote. Some 30 percent of Albany students already attended private schools—almost double the state average—and only 2 of the city’s 15 public schools managed even “respectable test scores.” Failure rates in the rest of the schools ranged from 50 percent to 80 percent in 4th-grade reading tests; citywide, the failure rate was 64 percent.

This, said the charter applicants, was “not acceptable.” The Regents agreed and granted charters to the two schools. And the rest, as Eva Joseph and a city of doubters would soon learn, is history.

Political Savvy Meets Commitment to Excellence

Even though they would have fewer than 100 places when the two schools opened in September 2002 (with just two grades, K—1, and plans to expand through grade 5), the schools received a thousand applications. This for a place that promised longer days (an hour more than the regular public schools), an extra 25 days of school per academic year, tough discipline, uniforms, and rigorous academic standards.

Success, of course, was anything but guaranteed. “They fought us every step of the way,” recalls Carroll of the APS and the teachers union. The New York State United Teachers (NYSUT), which boasted 600,000 members and had a retirement fund of $70 billion, was considered the most powerful lobbying group in the state. “They bring the cash to the legislators in wheelbarrows,” says Peter Murphy, only half-joking, “and not a day goes by that they aren’t trying to kill charters.” On one occasion, NYSUT slipped an amendment on to an obscure law that would have limited the market share of charters in Albany to 5 percent. Thanks to their many legislative connections, Carroll and team heard about the move and sent a busload of parents and students to the legislative hearing room. “If we had not had a legislative political background,” says Carroll, “they would have taken us to the cleaners.”

And then came Chris Bender, who was 10 minutes into pitching his school insurance product to Carroll when Carroll said, “You should be on my board.”

It was not an impetuous offer. Carroll knew that Bender was a fifth-generation Albany native and heir to a local publishing fortune. “We had people who knew a lot about charter schools,” explained Carroll. “But Chris Bender’s family had been here for 400 years, since the Dutch arrived. Between the two of us there are very few people of any significance in Albany that one of us doesn’t know. If we needed advice on construction, for instance, we could get to the best people. Who knows historic preservation? Who knows environmental regulation? We could figure all that out. If you were just starting out, if you helicoptered in from another country and tried to do it, you would trip over yourself a million times. With a master’s degree in education from Teachers College at Columbia University, Bender was also the only member of the team with real education credentials.

Three months after joining the board Bender became executive director of the group’s new nonprofit, the Brighter Choice Foundation, a technical and financial resource organization for Albany charter schools that would become the key to scale.

Through BCF, Carroll expanded the “what works” operating principle to include not just re-creating specific proven policies and practices, but replicating whole schools. The Achievement Academy middle school, which opened in the fall of 2005, was modeled after the Amistad Academy in New Haven, Connecticut, a pioneer of charter success. Albany Prep, which also opened in 2005, was modeled on the International Baccalaureate program and offered extended instructional periods for core academic subjects. The BCF would not only scout out potential charter school operators; it would build them a building; arrange for financing; and provide operational start-up money and free technical assistance, including community relations, politics, media relations, vendor advice, and legal advice if needed. The BCF lobbied KIPP (Knowledge Is Power Program) founder David Levin to open a school in Albany, offering KIPP both financial and logistical support and the freedom to focus on academic excellence. KIPP Tech Valley, a middle school, opened in 2005.

The financing scheme is both ingenious and, Carroll argues, replicable (see sidebar, page 36). For instance, BCF turned a $15 million loan from the Walton Foundation into a “revolving loan fund,” explains Carroll, that “allows us to build a facility and then take out a mortgage on it.” Once the building is built and the kids are in the chairs, BCF secures a short-term mortgage, which it pays while the new school is starting. When the school reaches full enrollment, usually within four or five years, it can then issue tax-exempt bonds and buy the building from BCF.

Getting It Right

The financing strategy, which includes start-up grants to new schools of up to $500,000, “helps schools start smaller, so they can start good,” says Chris Bender, who also believes that the smart leveraging of small amounts of money is what makes the BCF model replicable.

With such a model in hand, “a paradigm shift for us,” says Carroll, Brighter Choice was able to go to scale with quality and begin to make deep changes in the city’s educational system. The strategy is working so far.

Can Anyone Do This?

The results of the Albany effort are promising. An obvious question is whether what Brighter Choice has created can be replicated elsewhere.

Tom Carroll suggests that successful full-scale replication would require six preconditions: a strong charter-school law (which would allow the issuance of as many charters as are needed); a core leadership team (to provide strategic direction, execute on-time and on-budget decisions, provide oversight, and wield enough political skill to keep opponents at bay); a market with good economics (Albany is a reasonable-cost market with per-pupil charter aid of more than $12,000); access to facilities (a cooperative district leader, available land or buildings, and a reasonable zoning and planning process); a strong commitment to, and mechanism to ensure, quality (Albany was a good location because there was only one bad local charter school); and seven-figure philanthropic support.

How much philanthropic money would be needed depends on the size of the market, the cost of the market relative to the per-pupil charter aid, the number of schools contemplated, and whether a replicator would, as in Albany, adhere to school models that start small and remain small.

To achieve scale with quality in Albany has required spending of about $500,000 per school for start-up grants, with an annual central office expense (for the Foundation for Education Reform & Accountability and the Brighter Choice Foundation) of around $1 million. The Brighter Choice Foundation spends another $1 million annually on parent outreach, community organizing, and direct mail and advertising. Though Carroll has been able to tap into a network of Wall Street contacts, he believes that raising such funds, over $2 million a year, is possible in markets like Albany, which has a metropolitan area population of 1.1 million.

Importantly, once the Albany charter schools reach full enrollment, they no longer receive any philanthropic subsidy at all, reflecting Carroll’s distaste for school models that require ongoing philanthropic life support.

At BCF’s flagship schools, Brighter Choice for Girls and Brighter Choice for Boys, 3rd and 4th graders have been outperforming their district counterparts almost from the beginning on the statewide English language arts and mathematics exams (see Figure 1 for the 2009 test results).

“The Brighter Choice and KIPP schools are even outscoring the white suburban districts surrounding Albany,” says Carroll. “KIPP beat Bethlehem Middle School and Shaker Junior High (both in Albany’s affluent white suburbs), which is North Colonie’s middle school, in 7th grade English and math last year. The point is that we are not just beating crappy mediocre Albany schools; we’re beating the top public school districts in the area.”

Things have not always gone perfectly for Carroll and company. There was early backsliding in performance indicators at the Brighter Choice charter schools, for instance. The Brighter Choice board, which Carroll chairs, immediately asked the state department of education for permission to postpone adding a 5th grade as had been planned. They then did what traditional public schools seem so reluctant to do: they immediately changed leadership personnel. Brighter Choice went through three principals in three years, “until we got it right,” says Carroll.

Carroll’s missteps highlight the secrets to Brighter Choice’s success: constant vigilance, constant adaptability. When the Achievement Academy scores tanked, coming in below the district average in 2006, its second year, the school’s board immediately fired the principal and several teachers (who work with one-year contracts), changed textbooks as well as some systems and routines, and saw, the following year, a marked increase in test scores.

Scale Counts

The second part of the Brighter Choice story has been taking that quality to scale, to provide the kind of pressure to change that will improve the educational landscape.

“In a country this size, creating 50 or 60 really good schools barely creates a ripple,” explains Carroll. “It has a profound impact, of course, on the children educated in them, but it doesn’t challenge any of the institutional structures.”

Scale also provides a certain element of political protection, as Carroll has learned. “If you’re a single school and they close it down, most political people are willing to take that hit. But they’re not willing to close down schools serving thousands of kids.”

Though no one on the Brighter Choice team ever imagined shepherding eight schools into existence (the ninth, a girls’ high school, Albany Leadership, will open in 2010, and applications for two new middle schools are in the pipeline), they were even less sanguine about their chances of moving Albany’s public school system to change.

“We learned early on,” says Brian Backstrom, “that you couldn’t get the schools you needed by changing the schools you had. So that’s why we decided to build new schools, from the bottom up.”

But there have been signs of change at Albany Public Schools, which suggests that scale can count. “They have done some things,” says Backstrom. “Uniforms in one school. They’ve renovated all their facilities, gotten small class sizes—of course, they’ve done that in part because we’ve taken 2,200 kids away from them, but they’ve done it.”

The district has also lengthened its school day, albeit by only 30 minutes, but it is the first change in the union-contracted school day since the district was created in the mid-1970s. And they’ve even closed a perennially underperforming school (the Livingston Middle School). “Admittedly,” says Carroll, “our decision to locate [KIPP Tech Valley] charter school across the street from their weakest school was not subtle.”

Jason Brooks went through the minutes of Albany’s board of education meetings and discovered that, in fact, they were indeed listening, watching—and talking about Brighter Choice. At a March 17, 2005, meeting, for instance, board member Bill Barnett exclaimed, “I think that it’s high time for this district and the Board to have an in-depth discussion around the implications and the associated cost of increasing the number of instruction days.”

Added board member Scott Wexler, “We cannot compete with charter schools with 200 days while our…calendar committee [has] not [had an] instructional discussion [but] an employee benefit discussion. Our employees apparently need to understand our desire to have more time on task so we can be more successful.”
It’s a hopeful start and Brooks created a memo called “The Positive ‘Ripple Effect’ of Charter Schools” to note it.

And, though the district won’t admit the connection, Carroll believes that the recent overall rise in district elementary- and middle-school test scores is the result of “competition from charter schools [which] has forced an increased district focus on measurable outcomes.”

Finally, on March 25, 2009, just two weeks after her Rockefeller Institute charter critique, Eva Joseph announced her resignation. “She said the job has consumed so much of her time, sometimes seemingly 24 hours a day, that she looks forward to relaxing mentally,” wrote Scott Waldman in the Albany Times Union. “Under Joseph, Albany also became ground zero for the charter school movement, with the city’s 11th school expected to open soon. Joseph and the city’s charter school leaders often were at odds.”

“The question remains,” says Carroll, “as charter schools continue to grow in the city—within a year of this September roughly a third of public school children in Albany will be in charter schools—will the district put its head in the sand or finally be forced to reform its schools in order to compete?”

Carroll and company are not waiting around to see what happens. “Our opponents would love to freeze us in our tracks through a moratorium or a market-share cap,” notes Carroll. “But, as long as Albany has a shortage of good schools, the demand from parents and students for more charter schools will not diminish. Why would we want to stop creating charter schools to meet this demand when we know the alternative is for these children to attend bad schools?” A good question.

Peter Meyer, former news editor of Life magazine, is a freelance writer and contributing editor of Education Next.

An unabridged version of this article is available here.

An interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future is available here.

---

School choice supporters, including hundreds of private school students in crisp uniforms, filled Washington, D.C.’s Freedom Plaza last May to protest a congressional decision to eliminate the city’s federally funded school voucher program after the next school year (to see additional images of this event please click here). That afternoon, President Obama announced a compromise proposal to grandfather the more than 1,700 students currently in the District of Columbia Opportunity Scholarship Program, funding their vouchers through high school graduation, but denying entry to additional children. Both program supporters and opponents cite evidence from an ongoing congressionally mandated Institute of Education Sciences (IES) evaluation of the program, for which I am principal investigator, to buttress their positions, rendering the evaluation a Rorschach test for one’s ideological position on this fiercely debated issue.

School vouchers provide funds to parents to enable them to enroll their children in private schools and, as a result, are one of the most controversial education reforms in the United States (to see an interview with Patrick Wolf about his evaluation of the D.C. Opportunity Scholarship Program and about its likely future please click here). Among the many points of contention is whether voucher programs in fact improve student achievement. Most evaluations of such programs have found at least some positive achievement effects, but not always for all types of participants and not always in both reading and math. This pattern of results has so far failed to generate a scholarly consensus regarding the beneficial effects of school vouchers on student achievement. The policy and academic communities seek more definitive guidance.

The IES released the third-year impact evaluation of the Opportunity Scholarship Program (OSP) in April 2009. The results showed that students who participated in the program performed at significantly higher levels in reading than the students in an experimental control group. Here are the study findings and my own interpretation of what they mean.

Opportunity Scholarships

Currently, 13 directly funded voucher programs operate in four U.S. cities and six states, serving approximately 65,000 students. Another seven programs indirectly fund private K—12 scholarship organizations through government tax credits to individuals or corporations. About 100,000 students receive school vouchers funded through tax credits. All of the directly funded voucher programs are targeted to students with some educational disadvantage, such as low family income, disability, or status as a foster child.

Nineteen of the 20 school voucher programs in the U.S. are funded by state and local governments. The OSP is the only federal voucher initiative. Established in 2004 as part of compromise legislation that also included new spending on charter and traditional public schools in the District of Columbia, the OSP is a means-tested program. Initial eligibility is limited to K—12 students in D.C. with family incomes at or below 185 percent of the poverty line. Congress has appropriated $14 million annually to the program, enough to support about 1,700 students at the maximum voucher amount of $7,500. The voucher covers most or all of the costs of tuition, transportation, and educational fees at any of the 66 D.C. private schools that have participated in the program. By the spring of 2008, a total of 5,331 eligible students had applied for the limited number of Opportunity Scholarships. Recipients are selected by lottery, with priority given to students applying to the program from public schools deemed in need of improvement (SINI) under No Child Left Behind. Scholars and policymakers have since questioned the extent to which SINI designations accurately signal school quality because they are based on levels of achievement instead of the more informative measure of achievement gains over time.

The third-year impact evaluation tracked the experiences of two cohorts of students. All of the students were attending public schools or were rising kindergartners at the time of application to the program. Cohort 1 consisted of 492 students entering grades 6—12 in 2004. Cohort 2 consisted of 1,816 students entering grades K—12 in 2005. The 2,308 students in the study make it the largest school voucher evaluation in the U.S. to employ the “gold standard” method of random assignment.

Voucher Effects
Researchers over the past decade have focused on evaluating voucher programs using experimental research designs called randomized control trials (RCTs). Such experimental designs are widely used to evaluate the efficacy of medical drugs prior to making such treatments available to the public. With an RCT design, a group of students who all qualify for a voucher program and whose parents are equally motivated to exercise private school choice, participate in a lottery. The students who win the lottery become the “treatment” group. The students who lose the lottery become the “control” group. Since only a voucher offer and mere chance distinguish the treatment students from their control group counterparts, any significant difference in student outcomes for the treatment students can be attributed to the program. Although not all students offered a voucher will use it to enroll in a private school, the data from an RCT can also be used to generate a separate estimate of the effect of voucher use (see sidebar, page 50).

Using an RCT research design, the ongoing IES evaluation found no impacts on student math performance but a statistically significant positive impact of the scholarship program on student reading performance, as measured by the Stanford Achievement Test (SAT 9). The estimated impact of using a scholarship to attend a private school for any length of time during the three-year evaluation period was a gain of 5.3 scale points in reading. That estimate provides the impact on all those who ever attended a private school, whether for one month, three years, or any length of time in between (see Figure 1). Consequently, the estimate should be interpreted as a lower-bound estimate of the three-year impact of attending a private school, because many students who used a scholarship during the three-year period did not remain in private school throughout the entire period. The data indicate that members of the treatment group who were attending private schools in the third year of the evaluation gained an average of 7.1 scale score points in reading from the program.

What do these gains mean for students? They mean that the students in the control group would need to remain in school an extra 3.7 months on average to catch up to the level of reading achievement attained by those who used the scholarship opportunity to attend a private school for any period of time. The catch-up time would have been around 5 months for those in the control group as compared to those who were attending a private school in the third year of the evaluation.

Over time, in my opinion, the effects of the program show a trend toward larger reading gains cumulating for students. Especially when one considers that students who used their scholarship in year 1 needed to adjust to a new and different school environment, the reading impacts of using a scholarship of 1.4 scale score points (not significant) in year 1, 4.0 scale score points (not significant) in year 2, and 5.3 scale score points (significant) in year 3 suggest that students are steadily gaining in reading performance relative to their peers in the control group the longer they make use of the scholarship. No trend in program impacts is evident in math.

What explains the fact that positive impacts have been observed as a result of the OSP in reading but not in math? Paul Peterson and Elena Llaudet of Harvard University, in a nonexperimental evaluation of the effects of school sector on student achievement, suggest that private schools may boost reading scores more than math scores for a number of reasons, including a greater content emphasis on reading, the use of phonics instead of whole-language instruction, and the greater availability of well-trained education content specialists in reading than in math. Any or all of these explanations for a voucher advantage in reading but not in math are plausible and could be behind the pattern of results observed for the D.C. Opportunity Scholarships. The experimental design of the D.C. evaluation, while a methodological strength in many ways, makes it difficult to connect the context of students’ educational experiences with specific outcomes in any reliable way. As a result, one can only speculate as to why voucher gains are clear in reading but not observed in math.

Student Characteristics
The OSP serves a highly disadvantaged group of D.C. students. Descriptive information from the first two annual reports indicates that more than 90 percent of students are African American and 9 percent are Hispanic. Their family incomes averaged less than $20,000 in the year in which they applied for the scholarship.

Overall, participating students were performing well below national norms in reading and math when they applied to the program. For example, the Cohort 1 students had initial reading scores on the SAT-9 that averaged below the 24th National Percentile Rank, meaning that 75 percent of students in their respective grades nationally were performing higher than Chart 1 in reading. In my view, these descriptive data show how means tests and other provisions to target school voucher programs to disadvantaged students serve to minimize the threat of cream-skimming. The OSP reached a population of highly disadvantaged students because it was designed by policymakers to do so.

Did Only Some Students Benefit?

Several commentators have sought to minimize the positive findings of the OSP evaluation by suggesting that only certain subgroups of participants benefited from the program. Martin Carnoy states that “the treated students in Cohort 1 were concentrated in middle schools and the effect on their reading score was significantly higher than for treated students in Cohort 2.” Henry Levin likewise asserts that “the evaluators found that receiving a voucher resulted in no advantage in math or reading test scores for either [low achievers or students from SINI schools].”

The actual results of the evaluation provide no scientific basis for claims that some subgroups of students benefited more in reading from the voucher program than other subgroups. The impact of the program on the reading achievement of Cohort 1 students did not differ by a statistically significant amount from the impact of the program on the reading achievement of Cohort 2 students, Carnoy’s claim notwithstanding. Nor did students with low initial levels of achievement and applicants from SINI schools experience significantly different reading gains from the program than high achievers and non-SINI applicants. The mere fact that statistically significant impacts were observed for a particular subgroup does not mean that impacts for that group are significantly different from those not in the subgroup. For example, Group A and Group B may have experienced roughly similar impacts, but the impact for Group A might have been just large enough for it to be significantly different from zero (or no impact at all), while Group B’s quite similar scores fell just below that threshold.

From a scientific standpoint, three conclusions are valid about the achievement results in reading from the year 3 impact evaluation of the OSP:

• The program improved the reading achievement of the treatment group students overall.
• Overall reading gains from the program were not significantly different across the various subgroups examined.
• Three distinct subgroups of students—those who were not from SINI schools, students scheduled to enter grades K-8 in the fall after application to the program, and students in the higher two-thirds of the performance distribution (whose average reading test scores at baseline were at the 37th percentile nationally)—experienced statistically significant reading impacts from the program when their performance was examined separately. Female students and students in Cohort 1 saw reading gains that were statistically significant with reservations due to the possibility of obtaining false positive results when making comparisons across numerous subgroups.
  Why examine and report achievement impacts at the subgroup level, if the evidence indicates only an overall reading gain for the entire sample? The reasons are that Congress mandated an analysis of subgroup impacts, at least for SINI and non-SINI students, and because analyses at the subgroup level might have yielded more conclusive information about disproportionate impacts for certain types of students.

Expanding Choice

The OSP facilitates the enrollment of low-income D.C. students in private schools of their parents’ choosing. It does not guarantee enrollment in a private school, but the $7,500 voucher should make such enrollments relatively common among the students who won the scholarship lottery. The eligible students who lost the scholarship lottery and were assigned to the control group still might attend a private school but they would have to do so by drawing on resources outside of the OSP. At the same time, students in both groups have access to a large number of public charter schools.

The implication is that, for this evaluation of the OSP, winning the lottery does not necessarily mean private schooling, and losing the lottery does not necessarily mean education in a traditional public school. Members of both groups attended all three types of schools—private, public charter, and traditional public—in year 3 of the voucher experiment, although the proportions that attended each type differed markedly based on whether or not they won the scholarship lottery (see Figure 2). In total, about 81 percent of parents placed their child in a private or public school of choice three years after winning the scholarship lottery, as did 46 percent of those who lost the lottery. The desire for an alternative to a neighborhood public school was strong for the families who applied to the OSP in 2004 and 2005.

These enrollment patterns highlight the fact that the effects of voucher use reported above do not amount to a comparison between “school choice” and “no school choice.” Rather, voucher users are exercising private school choice, while control group members are exercising a small amount of private school choice and a substantial amount of public school choice. The positive impacts on reading achievement observed for voucher users therefore reflect the incremental effect of adding private school choice through the OSP to the existing schooling options for low-income D.C. families.

Parent Satisfaction
Another key measure of school reform initiatives is the perception among parents, who see firsthand the effects of changes in their child’s educational environment. Whenever school choice researchers have asked parents about their satisfaction with schools, those who have been given the chance to select their child’s school have reported much higher levels of satisfaction. The OSP study findings fit this pattern. The proportion of parents who assigned a high grade of A or B to their child’s school was 11 percentile points higher if they were offered a voucher, 12 percentile points higher if their child actually used a scholarship, and 21 points higher if their child was attending a private school in year 3, regardless of whether they were in the treatment group. Parents whose children used an Opportunity Scholarship also expressed greater confidence in their children’s safety in school than parents in the control group.

Additional evidence of parental satisfaction with the OSP comes from the series of focus groups conducted independently of the congressionally mandated evaluation. One parent emphasized the expanded freedom inherent in school choice:

“[The OSP] gives me the choice to, freedom to attend other schools than D.C. public schools….I just didn’t feel that I wanted to put him in D.C. public school and I had the opportunity to take one of the scholarships, so, therefore, I can afford it and I’m glad that I did do that.” (Cohort 1 Elementary School Parent, Spring 2008)

Another parent with two children in the OSP may have hinted at a reason achievement impacts were observed specifically in reading:

“They really excel at this program, `cause I know for a fact they would never have received this kind of education at a public school….I listen to them when they talk, and what they are saying, and they articulate better than I do, and I know it’s because of the school, and I like that about them, and I’m proud of them.” (Cohort 1 Elementary School Parent, Spring 2008)

These parents of OSP students clearly see their families as having benefited from this program.

Previous Voucher Research

The IES evaluation of the DC OSP adds to a growing body of research on means-tested school voucher programs in urban districts across the nation. Experimental evaluations of the achievement impacts of publicly funded voucher and privately funded K—12 scholarship programs have been conducted in Milwaukee, New York City, the District of Columbia, Charlotte, North Carolina, and Dayton, Ohio. Different research teams analyzed the data from New York City (three different teams), Milwaukee (two teams), and Charlotte (two teams). The four studies of Milwaukee’s and Charlotte’s programs reported statistically significant achievement gains overall for the members of the treatment group. The individual studies of the privately funded K—12 scholarship programs in the District of Columbia and Dayton reported overall achievement gains only for the large subgroup of African American students in the program. The three different evaluators of the New York City privately funded scholarship program were split in their assessment of achievement impacts, as two research teams reported no overall test-score effects, but did report achievement gains for African Americans; the third team claimed there were no statistically significant test-score impacts overall or for any subgroup of participants.

The specific patterns of achievement impacts vary across these studies, with some gains emerging quickly, but others, like those in the OSP evaluation, taking at least three years to reach a standard level of statistical significance. Earlier experimental evaluations of voucher programs were somewhat more likely to report achievement gains from the programs in math than in reading—the opposite of what was observed for the OSP. Despite these differences, the bulk of the available, high-quality evidence on school voucher programs suggests that they do yield positive achievement effects for participating students.

Conclusions
School voucher initiatives such as the District of Columbia Opportunity Scholarship Program will remain politically controversial in spite of rigorous evaluations such as this one, showing that parents and students benefited in some ways from the program. Critics will continue to point to the fact that no impacts of the program have been observed in math, or that applicants from SINI schools, who were a service priority, have not demonstrated statistically significant achievement gains at the subgroup level, as reasons to characterize these findings as disappointing. Certainly the results would have been even more encouraging if the high-priority SINI students had shown significant reading gains as a distinct subgroup. Still, in my opinion, the bottom line is that the OSP lottery paid off for those students who won it. On average, participating low-income students are performing better in reading because the federal government decided to launch an experimental school choice program in our nation’s capital.

The achievement results from the D.C. voucher evaluation are also striking when compared to the results from other experimental evaluations of education policies. The National Center for Education Evaluation and Regional Assistance (NCEE) at the IES has sponsored and overseen 11 studies that are RCTs, including the OSP evaluation. Only 3 of the 11 education interventions tested, when subjected to such a rigorous evaluation, have demonstrated statistically significant achievement impacts overall in either reading or math. The reading impact of the D.C. voucher program is the largest achievement impact yet reported in an RCT evaluation overseen by the NCEE. A second program was found to increase reading outcomes by about 40 percent less than the reading gain from the DC OSP. The third intervention was reported to have boosted math achievement by less than half the amount of the reading gain from the D.C. voucher program. Of the remaining eight NCEE-sponsored RCTs, six of them found no statistically significant achievement impacts overall and the other two showed a mix of no impacts and actual achievement losses from their programs. Many of these studies are in their early stages and might report more impressive achievement results in the future. Still, the D.C. voucher program has proven to be the most effective education policy evaluated by the federal government’s official education research arm so far.

The experimental evaluation of the District of Columbia Opportunity Scholarship Program is continuing into its fourth and final year of studying the impacts on students and parents. The final evidence collected from the participants may confirm the accumulation of achievement gains in reading and higher levels of parental satisfaction from the program that were evident after three years, or show that those gains have faded. Uncertainty also surrounds the program itself, as the students who gathered on Freedom Plaza in May currently are only guaranteed one final year in their chosen private schools. What will policymakers see as they continue to consider the results of this evaluation? The educational futures of a group of low-income D.C. schoolchildren hinge on the answer.

Patrick J. Wolf is professor of education reform at the University of Arkansas and principal investigator of the D.C. Opportunity Scholarship Program Impact Evaluation. The opinions expressed in this article are his own.

An unabridged version of this article is available here.

---

Methodology Notes
If one’s purpose is to evaluate the effects of a specific public policy, such as the District of Columbia Opportunity Scholarship Program (OSP), then the comparison of the average outcomes of the treatment and control groups, regardless of what proportion attended which types of school, is most appropriate. A school voucher program cannot force scholarship recipients to use a voucher, nor can it prevent control-group students from attending private schools at their own expense. A voucher program can only offer students scholarships that they subsequently may or may not use. Nevertheless, the mere offer of a scholarship, in and of itself, clearly has no impact on the educational outcomes of students. A scholarship could only change the future of a student if it were actually used.

Fortunately, statistical techniques are available that produce reliable estimates of the average effect of using a voucher compared to not being offered one and the average effect of attending private school in year 3 of the study with or without a voucher compared to not attending private school. All three effect estimates—treatment vs. control, effect of voucher use, and impact of private schooling—are provided in the longer version of this article (see “Summary of the OSP Evaluation” at www.educationnext.org), so that individual readers can view those outcomes that are most relevant to their considerations.

I have presented mainly the impacts of scholarship use in this essay. Those impacts are computed by taking the average difference between the out comes of the entire treatment and control groups—the pure experimental impact—and adjusting for the fact that some treatment students never used an Opportunity Scholarship. Since nonusers could not have been affected by the voucher, the impact of scholarship use can be computed easily by dividing the pure experimental impact by the proportion of treatment students who used their scholarships, effectively rescaling the impact across scholarship users instead of all treatment students including nonusers. I focus here on scholarship usage because that specific measure of program impact is easily understood, is relevant to policymakers, and preserves the control group as the natural representation of what would have happened to the treatment group absent the program, including the fact that some of them would have attended private school on their own.

Questions of educational adequacy and school spending have long been a point of contention in school reform. Amid the recent economic turmoil and gaping state budget shortfalls, questions of whether court-ordered funding remedies have delivered—and why they have or have not—have taken on particular import. This forum offers two sharply different takes on our experiences to date, and what lessons they offer going forward. Eric Hanushek and Alfred Lindseth are the authors of Schoolhouses, Courthouses, and Statehouses: Solving the Funding-Achievement Puzzle in America’s Public Schools (Princeton University Press, 2009), in which they propose a system of performance-based funding focused on improving student achievement. Michael Rebell is executive director of the Campaign for Educational Equity at Teachers College, Columbia University, and is the author of Courts and Kids: Pursuing Educational Equity through the State Courts (University of Chicago Press, forthcoming), in which he proposes a new functional separation of powers among the executive, legislative, and judicial branches to promote education reform and student achievement.

Education Next: Over the past four decades, many states have revised their funding of schools, through either judicial or legislative initiatives, in an effort to improve schools serving disadvantaged children. Too often, however, these actions have not yielded improved student achievement. Looking to the future, what kinds of judicial or legislative remedies are most likely to fulfill the promise of improved student outcomes?

Eric Hanushek and Al Lindseth: This question is particularly timely, as national policies on education embodied in the federal No Child Left Behind (NCLB) law are in a state of flux and likely to change under President Obama. At the same time, an economic crisis has engulfed not only our country, but most of the world, suggesting that significant increases in funding for education budgets are unlikely in the foreseeable future. The challenge is to find ways to develop a well-educated workforce that are not only more effective than those relied on in the past, but also do not depend on significant annual increases in education appropriations.

Since about 1970, the achievement levels of U.S. students on the reading and math tests of the National Assessment of Educational Progress (NAEP) have remained largely flat despite massive financial and other efforts to improve them. The problem is particularly acute for poor and minority students, with average black and Hispanic students lagging three or four grade levels behind the average white student. While lack of sufficient funding is often cited as the principal reason for low student performance, the United States already spends more on K—12 education than all but a few countries. Moreover, spending has increased dramatically over the past several decades, with today’s per-pupil expenditures almost four times, in inflation-adjusted dollars, what they were in 1960.

The underlying system, which governs how money is spent, has remained largely unchanged over that period. It is characterized by, among other things, a compensation scheme that pays teachers and administrators without regard to the results they get in the classroom; rules that make it extremely difficult to terminate unqualified teachers or assign the good ones where they are most needed; an assessment and rating system that discriminates against good teachers who are assigned to schools with significant numbers of at-risk students; a monopolistic structure that insulates public schools from competition; and numerous union and other work rules that prevent principals from effectively running their schools. It is a system more concerned with the adults and their rights than it is with ensuring the success of its students. Although some reforms have taken place in the last decade or so—the adoption of statewide standards, limited choice options, and increased accountability—they have not been sufficient to overcome the obstacles posed by the underlying system.

Given this sobering assessment, what can be done in the future to improve student achievement? The solution, we believe, lies in performance-based funding: a system of integrated education policies and funding mechanisms designed to drive and reward better performance by teachers, administrators, students, and others involved in the education process. Such a system will ensure more effective use of education dollars through better decisionmaking, eliminate perverse incentives that reward mediocrity or failure, and most important, energize and motivate those involved in the education of our young people. The essential components of a performance-based funding system cannot be ordered à la carte. These components interlock and depend on each other for their success. While various states have adopted some of these components—state-level academic standards, for example—none have implemented the integrated system we recommend, and the results have been clearly unsatisfactory.

A performance-based system of funding would contain the following nine features:

1) A focus on improving outcomes rather than on increasing inputs. States must set high and uniform achievement goals for every child to strive to meet. While every child may not reach the highest goals, high expectations will encourage children to do their best.

2) Local school administrators and teachers with the flexibility to determine how their schools can best meet high standards. Often even the most dedicated teachers and principals are hampered by severe limitations on spending and programmatic decisions by ineffective state regulations, constraints such as those that come with categorical funding, and a variety of state and local laws and contractual arrangements. The idea is to let those who are most familiar with the problems faced in the schools take the lead in deciding how to solve them.

3) Rewards for both teachers and administrators based on their success in improving student achievement. In almost every school district in the country, teachers are currently paid based solely on their years of experience and degree level, despite a consensus in the scientific community that these two factors bear little relationship to their success in improving student performance. The single-salary pay schedule—which makes it virtually impossible to pay good teachers more, to offer bonuses for teaching in hard-to-staff schools, and to pay higher salaries to teachers in shortage areas, such as math, science and special education—must go, and a pay system implemented based upon the just named considerations.

4) Greater accountability commensurate with increased authority and discretion. Teachers, schools, and principals must be held accountable for results. Just as they should be rewarded if they are successful, they must experience the consequences if they are not. Each state should adopt an accountability plan that sets clear goals as well as significant and enforceable consequences if goals are not achieved within a reasonable period.

5) Rewards and accountability based on factors within the control of the local district. Schools, teachers, and administrators should be judged and, if appropriate, rewarded based on the “value” they add during the school year, not on absolute test scores. The latter may be influenced by students’ homes and neighborhoods and may give teachers in middle-class suburban communities an advantage over those teaching in less advantaged communities. Under current practice, schools with disadvantaged students are almost always labeled “failing,” no matter how good the teachers are. Once value-added assessments are put in place, it will be possible to isolate the contributions made by the schools, teachers, and programs in raising achievement from external factors also affecting achievement and to act accordingly by following a model of continuous improvement.

6) Schooling options for parents and children who judge their school less than satisfactory. Schools must know that, if they are not successful, parents have alternatives for their children. Therefore, the finance system should also support charter and other choice schools.

7) Reasonable funding levels based on the needs of particular student enrollments and other factors outside of district control, but also discretion by local district taxpayers to augment the funding of their schools. Base funding would adjust for district poverty and external labor-market factors. Supplementation should incorporate “equalization” funds by the state to recognize differences in the ability of districts to raise funds locally when levying the same tax rate, but would permit parents and taxpayers to express directly their satisfaction with educational plans and policies.

8) Transparency incorporating value-added measures. Parents, taxpayers, and other stakeholders can then readily gauge how good a job the schools are doing.

9) A commitment to evaluating school and programmatic effectiveness. Expensive new strategies, such as large-scale class-size reduction programs, should be implemented only if they also provide for regular, independent evaluations to determine their effectiveness. Unsuccessful programs should not be allowed to continue and proliferate year after year just because they have strong sponsors.

The path to such reform will not be an easy one. While elements such as state standards, accountability measures, and value-added measures either are not controversial or are gaining acceptance, other important components, especially performance-based pay and increased choice options, are opposed by powerful forces with vested interests in the current system. Most powerful are the politically connected teachers unions. They, for example, clashed with Washington, D.C., schools chancellor Michelle Rhee over a proposal to couple higher pay with greater risk of termination because of ineffectiveness. The unions vigorously opposed those efforts, leading Rhee to move instead to improve the teaching force by terminating unqualified teachers. Unless this system is changed, it seems unlikely that outcomes will measurably improve in the district, already one of the highest funded and worst performing in the country.

The responsibility to enact and implement performance-based funding systems will fall primarily on the political branches of government, the state legislatures and governors. Although judicial remedies have played a significant role in school finance in the past, that era is drawing to a close. Beginning in the early 1970s, advocacy groups, frustrated with legislative efforts, began turning to the courts, initially to seek more equity in the allocation of education funds and later to seek vastly increased appropriations from state legislatures through “educational adequacy” lawsuits based on vaguely worded state constitutional provisions. A significant number of state courts responded positively to plaintiffs’ pleas and ordered unprecedented increases in K—12 funding in their states. Unfortunately, basic problems in the underlying systems of delivering education services were often ignored. In this sense, the courts mirrored what had been going on in the state legislatures, and the results were, not surprisingly, much the same: large amounts of money expended, but little or no improvement in student outcomes. An analysis in our recently published book examines the NAEP test-score trends in the four states that have implemented court remedies the longest, and demonstrates that, despite spending increases amounting to billions of dollars, the achievement patterns in three of them—Wyoming, New Jersey, and Kentucky—are largely unchanged from what they were in the early 1990s, before the court-ordered remedies commenced. Only in Massachusetts, where much deeper and broader reforms were instituted, has there been some improvement, although even there the state’s black students have not benefited from the remedy.

Perhaps due in part to this track record, the courts have begun to step back, opting instead to leave decisions regarding education policy and appropriations in the hands of the political branches of government, where they have traditionally resided. In the last five years, court decisions in approximately 15 states have disposed of educational adequacy cases, and, with one or two minor exceptions, the courts have either dismissed the cases or granted minimal relief. While this could change in a number of cases still pending, we believe the likelihood of significant court-ordered remedies in the foreseeable future is small.

Performance-based funding is not by itself a panacea that will solve all problems of substandard achievement or eliminate the achievement gap. Many of the problems that plague American education are beyond the control of the schools and will have to be addressed by other means. Performance-based funding will, however, put the nation’s schools back on the right track, help to raise the achievement of all students significantly, and once again make our students competitive on the world stage.

President Obama has called for increased funding to support NCLB, and Congress has provided substantial stimulus money for schools. A wise use of that money would be to underwrite transition costs in states moving to implement a performance-based funding system. For example, support for the improvement of student testing, for the development of improved databases and value-added measures, and for initial payments of expanded salaries under performance-based pay could provide important incentives for the states to move toward more logical and more effective funding systems.

Michael Rebell: The basic premise of the book and essay by Eric Hanushek and Al Lindseth—and of the question posed by Education Next—is that although “massive” amounts of money have been spent on education over the past 40 years, the results have been meager. Hanushek and Lindseth claim that states in which courts have ordered “extraordinary spending increases,” or at least the select few they have studied, have shown no improvement in student test scores. They then argue that certain “performance-based” accountability mechanisms that they recommend, rather than increased funding, should be the focus of future efforts.

I strongly dispute these premises, and I doubt that the reforms that Hanushek and Lindseth recommend are feasible, or that if enacted, they would constitute the panacea for the nation’s education ills that they imply. Extensive inequities in education funding, by which students with the greatest needs receive the fewest funds, still prevail in many parts of the United States; for that reason, state courts continue to have a critical role in ensuring meaningful educational opportunities for all children. The evidence strongly indicates that money well spent does make a significant difference in student achievement, and as Education Sector’s Kevin Carey has noted in reviewing one of Mr. Hanushek’s books: “There is little evidence that starving schools of needed funds is a catalyst for innovation, or that well-funded schools are more likely than others to be inefficient.” Moreover, although I agree that additional accountability measures are needed, continued involvement of the state courts, working in concert with the executive and legislative branches in a new, functional separation-of-powers mode, is essential for holding all parties accountable and for attaining the nation’s education goals.

Let me first put the spending issues into perspective. Hanushek and Lindseth claim that per-pupil spending in the U.S. has quadrupled since 1960. This is a gross exaggeration. According to recent analyses by Economic Policy Institute research associate Richard Rothstein, the cost of school services, when adjusted by the consumer price index, increased by 157 percent from 1967 to 2005, but when adjusted by the more relevant net services index (which omits shelter rent and medical care. the increase was only 92 percent. Moreover, these general statistics mask the fact that much of this increase has gone to special education, a sector that has dramatically expanded and substantially improved the lives of millions of students with disabilities over this time period. According to Rothstein, from 1967 to 2005 the share of educational expenditures going to regular education dropped from 80 to 55 percent and the share going to special education increased from 4 to 21 percent.

Second, for the past two decades, the United States has been committed to the historically unprecedented mission of simultaneously promoting excellence and equity in education. The standards-based reform movement seeks both to equip all of our high-school graduates to compete in the global marketplace and to narrow the achievement gap between our advantaged and disadvantaged student populations. Obviously, attaining these critical goals will require substantial resource infusions, especially for the high-need schools that historically have been treated inequitably by state education finance systems. Thus far, neither Congress, which has not even come close to fully funding the No Child Left Behind Act, nor most states, which have raised their academic standards but not their funding levels to a commensurate degree, have stepped up to the plate.

Third, Hanushek and Lindseth assert that “the United States already spends more on K—12 education than all but a few countries.” Although the U.S. is fourth among the 30 industrialized democracies that comprise the OECD (Organisation for Economic Co-operation and Development) in per-pupil spending on K—12, it is in the middle of the pack (13) in education spending as a percentage of GDP. Moreover, since, comparatively speaking, the U.S. starves health care, economic security, housing, and other areas of social welfare provision, the schools must bear an enormous burden in overcoming the impact of concentrated poverty for the poor and minority children they are committed to educating to high levels. In 2005, the childhood poverty rate in the U.S. was 21.9 percent, the highest, with the exception of Mexico, of the 24 OECD countries listed, and far higher than the 3 percent childhood poverty rate of countries like Denmark and Finland.

Given the extent of poverty in our society and the heavy burden that has been placed on the schools to alleviate its impact, it is astounding how much educational progress has been made. For example, from 1990 to 2007, black students’ scale scores increased 34 points on the NAEP 4th-grade mathematics tests (compared with a 28-point increase for whites), and the black-white achievement gap declined from 32 to 26 points during this period. Nevertheless, even greater progress can and should be made. I doubt, however, that “performance-based funding,” the solution Hanushek and Lindseth offer, will prove to be the silver bullet that will “help to raise the achievement of all students significantly, and once again make the nation’s students competitive on the world stage.”

Hanushek and Lindseth announce their performance-based funding prescriptions as if they will instantly solve the nation’s educational ills. But test-based outcomes, merit pay for teachers, rewards and sanctions, and voucher and charter alternatives have been part of the reform agenda of most states for years. Studies of each of these approaches have generally shown mixed results, and there is no strong empirical basis for dramatically expanding their use. Hanushek and Lindseth have an answer to this criticism: This is not a menu of options that can be ordered à la carte, they say. These components interlock, and they must be implemented as an “integrated system.”

Leaving aside the objections I have to many aspects of their program, full implementation of their “integrated systems approach” is clearly a pipe dream. In a democratic polity, no single reform approach can ever be fully put into effect, much less maintained, in its pure form. Policymaking for public education in a democracy inevitably is shaped by politics, and any reform proposal will inexorably be subject to compromise and modification. Although there was an unprecedented degree of bipartisan support for passage of the No Child Left Behind law in 2001, for example, that support came at a high price. As Education Next editors Rick Hess and Chester Finn recently observed, NCLB is a “Christmas tree of programs, incentives, and interventions that are more an assemblage of reform ideas than a coherent scheme. NCLB’s remedy provisions bear all the marks of concessions to various ideologies, advocates, and interest groups, with scant attention paid to how they fit together, the resources or authority they require, or whether they could be sensibly deployed through the available machinery.”

How to forge a better package of education reforms out of the positive aspects of NCLB is the main education policy challenge for the Obama administration, and how to make standards-based reform really work is the parallel problem that state education policymakers need to face. Hanushek and Lindseth’s performance-based funding proposal adds little of real value to this equation. However, the state courts’ wide experience in recent decades with fiscal equity and education adequacy litigations, which these authors roundly criticize, does provide significant possibilities for developing productive policy compromises and significantly advancing prospects for meaningful education reform.

Since 1973, when the U.S. Supreme Court held that education was not a “fundamental interest” under the federal constitution, education advocates, frustrated by continuing inequities in the funding of public education, have turned to the state courts. As Hanushek and Lindseth acknowledge, a “significant number of state courts responded positively to plaintiffs’ pleas.” In fact, during this era, cases have been filed in 45 of the 50 states, and plaintiffs have won more than 60 percent of them; since 1989, when the legal emphasis shifted from “equity” cases that seek equal funding levels for all students to “adequacy” cases that look to provide all students a basic quality education consistent with state standards, plaintiffs have prevailed in two-thirds of the final high-state-court decisions.

Hanushek and Lindseth claim that the courts have begun to step back from their support of constitutional rights in this area. But, in fact, there has been no diminution in the willingness of state supreme courts to issue strong rulings on students’ basic constitutional right to an adequate education. What has changed in recent years is that more cases have reached the remedy stage and more courts are experiencing difficulty in seeing constitutional compliance through to a successful conclusion. Some courts have cut short their remedial oversight out of frustration with the political complications and complexity of effecting meaningful change.

In other words, the adequacy movement has matured, and the courts are now grappling with many of the same implementation and compliance issues that have stymied governors and legislatures for years. The problems raised by judges in these remedial proceedings call for thoughtful responses and nuanced solutions, rather than the cavalier rejection of “judicial activism” that Hanushek and Lindseth and other opponents of adequacy articulate. (The title of a recent book that Hanushek edited and to which Lindseth contributed accuses judges of “harming our children.” This kind of hyperbole is clearly unwarranted.) As University of Wisconsin law professor Neil Komesar has insightfully pointed out, “All societal decision makers are highly imperfect.” Governors, state education departments, legislatures, and the federal Congress have been unable to solve the nation’s educational problems over the past half century, so why should anyone expect judicial interventions to achieve immediate, decisive results?

Where courts have persevered in their efforts, there have often been substantial improvements in student achievement. Hanushek and Lindseth allude to NAEP test-score trends in a few states with long-standing court orders that they claim have resulted in no improvement in student achievement in three out of four cases. The NAEP scores they focus on do not correspond in most of the cases to the relevant years in which the court orders were actually implemented; they ignore the fact that, as in Kentucky, initial increases in funding are sometimes followed by substantial decreases in later years; and their use of NAEP scores makes no sense in a state like New Jersey, where the court orders covered only a subset of the state’s students ( i.e., students in 31 poor urban school districts) and not the full statewide populations represented by NAEP scores. Recent, more finely tuned data for New Jersey, provided by Peg Goertz, a University of Pennsylvania researcher who has closely followed developments in the Garden State, indicate that from 1999 to 2007 substantial gains were made in the Abbott districts, which were the focus of the judicial remedies. For example, in 4th-grade mathematics, the achievement gaps between the Abbott districts and the rest of the state were cut by more than one-third. Similarly, Kentucky, which was near the bottom of the national rankings in virtually all performance indexes before its 1989 court decision, now ranks above the national averages in reading and science and almost at the national average in math.

Despite these gains, to fully meet our nation’s challenging goals for excellence and equity in our public school systems, clearly more needs to be done. What is required is a concerted effort by all three branches of government to bring their relative functional strengths to bear on ensuring constitutional compliance and solving the nation’s educational ills. In a forthcoming book, I propose a “successful remedies model” that is based on the extensive empirical experience that dozens of courts have had in dealing with legislatures, governors, and state education departments in crafting remedies. It is a process approach that is compatible with Hanushek and Lindseth’s performance-funding focus or any other policy perspective, or as is more likely, whatever mix of policies a state’s elected representatives choose to endorse. This process seeks to ensure that, whatever reform path state policymakers pursue, the compromise package they assemble is cohesive, adequately funded, and consistently implemented; moreover, the state should be committed to seeing the reforms through over time so that lasting results can be achieved. “Success” in implementing standards-based reforms under this model is defined not in terms of test scores in a limited number of subject areas, but broadly, in terms of providing all students a sound basic education on a sustained basis.

To achieve such success requires effective, programs and ongoing “colloquy” among the three branches of government. The courts’ role in this process is to outline in general, principled terms the expectation that the legislative and executive branches will develop challenging standards, fair and adequate funding systems, and effective programs and accountability measures, but to leave to the programs and the political branches the full responsibility for actually formulating these policies. Legislatures should make basic educational policy decisions; state education departments and local school districts should determine how best to implement educational reforms. Once the state has decided on its policy position, however, a judicial presence should be maintained to ensure that the chosen policy is fully funded, is implemented in a coherent manner, and results in substantially improved student performance, as measured by validated assessments of academic achievement and of students’ ability to function as capable citizens and workers.

Since significant compliance cannot be achieved overnight, in most cases courts will need to maintain nominal jurisdiction for a multiyear period, probably 10 to 15 years. The mere fact that judicial oversight remains in place can ensure continued adherence to implementation of stated policy goals, and actual interventions should be rare, especially if it is clearly understood that all the courts would be enforcing are the state’s own policy goals. A judicial presence is especially important to ensure that the reform process—and reasonable funding levels—are maintained in times of economic stress or recession like the present, where children’s needs and constitutional values are often given short shrift.

In short, then, my answer to the question posed by the editors of Education Next is that what is most likely to fulfill the promise of improved student outcomes in the future is not any silver bullet remedy, but rather a pragmatic process that allows courts, legislatures, state education departments, and school districts to work collaboratively to implement meaningful reforms on a sustained basis.

Hanushek and Lindseth: Notwithstanding his obfuscation, Michael Rebell’s solution is essentially more of the same. Beginning by misstating spending increases (based on incorrect data and flawed adjustments) and ignoring pertinent performance data, he rewrites the constitution of every state to give judges the major policy-setting role in a “new, functional separation of powers mode.” He further recommends that judges and legislators be guided in their efforts by a “successful remedies model” to be drawn from previous adequacy litigation—perhaps tempting if such “successful” models actually existed. Quite surprisingly, he cites New Jersey’s tortured 35-year-old Abbott litigation as an example of “success,” but neglects to mention that the state’s black students, the principal beneficiaries of the remedy, are still scoring at about the same relative levels on the NAEP tests as in 1992. In Kentucky, he relies on data for all students, which mask the fact that black students, the state’s principal minority group, have regressed compared to their peers nationally during the remedial period.

Our solution may not be a “silver bullet” for everything that ails American education, but it surely presents a better chance for our children than continuing the demonstrably failed practices of the past. In the end, Rebell basically concludes that political forces are too strong to bring about the fundamental changes we recommend, so we should just continue plowing more money into the current system. If we do, no one should be surprised in 2040 when our students are still performing, as they are now, at 1970 levels.

Rebell: If I didn’t know that Rick Hanushek was an outstanding economist and that Al Lindseth was a master litigator, I would think from some of the provocative phrases they use in their writings that they were sensationalist journalists, looking to attract readers with shocking but misleading headlines and catchphrases. They claim that I am proposing to “rewrit[e] the constitution of every state to give judges the major policy-setting role.” A detailed examination of the positions they actually take in their writings, and especially in their recent book upon which this Forum is based, indicates, however, that we agree that money—if well spent—does matter, that education finance cases have had a significant equalizing effect on state education funding formulas, and that court orders can “support legislators who want to address serious problems in education.”

The fact is that the unproven, business-model, and privatization practices they propose as education reforms have no chance of being adopted as an “integrated system,” especially in the present political climate. I would, therefore, ask Hanushek and Lindseth to stop tilting at windmills and to join with me in instituting a dialogue in major areas in which we do agree, like the fact that courts can and should hold states and school districts accountable for better performance, and that “school funding policies must recognize the underlying heterogeneity of students and their educational challenges and ensure that all schools have the means to succeed” (Hanushek and Lindseth, Schoolhouses, Courthouses, and Statehouses, page 218). That kind of conversation might help to promote real changes that might provide truly meaningful educational opportunities to all of our children.

Recent months have brought an ever-louder drumbeat in support of state-level participation in the Programme for International Student Assessment (PISA), with a weaker chorus calling for states to participate in the Trends in International Mathematics and Science Study (TIMSS). What would be gained if, in addition to the nation as a whole, individual states were to participate directly in these assessments by testing a much larger and more representative sample of students?

Not as much as many advocates would have us believe, and probably not enough to justify the considerable cost. Despite the growing infatuation with international comparisons of student performance and the illuminating feedback they can provide on how young Americans are doing relative to students in other countries, current international assessments cannot generate a great deal of reliable policy advice. In other words, they’re better at showing how our children’s academic performance (in certain subjects) compares with that of their overseas agemates than at guiding us toward stronger U.S. schools.

If states choose to spend a fortune on such tests, it’s because their leaders believe, in the words of the National Governors Association (NGA), that doing so will lead to “policy solutions to U.S. education system shortcomings.” This is the first of two justifications given for these efforts, which are widely referred to as “international benchmarking.” The second purpose is the straightforward comparison of U.S. students and their peers in other countries. Here, the resulting “league tables” are the center of attention, and these results are often described, and almost as often decried, as a “horse race” or an “Olympics contest” with an attendant fixation on who’s winning and who’s losing. Absent is an acknowledgment that these assessments embody particular assumptions and beliefs about what should be measured.

As states consider funding and administering international assessments for themselves, such facts must be kept in mind and key questions addressed. Among them are, what will states get as a result of participation? How much will it cost? Are there easier and cheaper ways of obtaining the desired information? We begin with a comparison of PISA and TIMSS, the two most prominent international assessments (see sidebar for additional information).

PISA v. TIMSS

PISA is a self-proclaimed “yield study” assessing the “literacy” of 15-year-olds and is not tied to any specific curricula. PISA claims to be assessing the skills young adults will need in the emerging global economy. Never modest, the report releasing the 2003 PISA results was titled “Learning for Tomorrow’s World” and the 2006 PISA results “Science Competencies for Tomorrow’s World.”

---

A Quick Guide to International Student Assessments

The three main programs are known widely by their acronyms:

• PIRLS (Progress in International Reading Literacy Study) is an assessment of 4th-grade reading administered every five years under the auspices of the International Association for the Evaluation of Educational Achievement (IEA). Fifty-five countries are expected to participate in 2011.http://nces.ed.gov/surveys/pirls/
• TIMSS (Trends in International Mathematics and Science Study) is IEA’s assessment of student achievement in 4th- and 8th-grade science and math and is conducted every four years. Some 67 countries administer TIMSS.http://nces.ed.gov/timss/
• PISA (Programme for International Student Assessment) is an evaluation of reading, math, and science “literacy” among 15-year-olds. The Organisation for Economic Co-operation and Development (OECD) conducts PISA every three years, emphasizing one of the subjects on a revolving basis. The emphasis in 2006 was science; in 2009 it will be reading. Participation in PISA has grown from 43 countries in 2000 to an expected 65 countries in 2009.http://nces.ed.gov/Surveys/PISA/

---

TIMSS is more grade- and curriculum-centered and far more modest in its claims. As described in the release of the 2007 data, “TIMSS is designed to align broadly with mathematics and science curricula in the participating countries. The results, therefore, suggest the degree to which students have learned mathematics and science concepts and skills likely to have been taught in school.”

PISA has not hesitated in making the most of these differences, and the high visibility of the Organisation for Economic Co-operation and Development (OECD) has ensured that PISA is the most prominent of the international assessments. Indeed, while in public the NGA is careful to say that it is in favor of international benchmarking in general and not in favor of any particular test, in practice the organization has tended to favor PISA. For example, a background paper the NGA released in February 2008 concludes by promising that “the National Governors Association will work with states interested in state-level PISA administration to identify available resources and cost-savings options to support state participation.” (In this three-page memo, TIMSS rated just one sentence.)

More recently, the NGA, the Council of Chief State School Officers, and Achieve released a report titled “Benchmarking for Success: Ensuring U.S. Students Receive a World-Class Education,” calling for state scores on international assessments. PISA was mentioned far more frequently than TIMSS, and the head of the PISA project at OECD, Andreas Schleicher, was repeatedly quoted.

The Alliance for Excellent Education has made an even stronger appeal for state participation in PISA, calling our failure to participate at that level “myopic.”

Most state interest is in the science and math assessments of PISA and TIMSS. In 1995, TIMSS participants included five states and one consortium of school districts. In 1999, just before PISA was launched, 13 states plus 14 consortia of school districts participated in TIMSS. Only two states, Massachusetts and Minnesota, funded their own participation in the 2007 TIMSS. No U.S. states have independently participated in PISA or PIRLS thus far.

The results of the three assessments are often compared to those of the National Assessment of Educational Progress (NAEP), which in the United States currently dwarfs the international assessments in scope. In 2007, the NAEP 8th-grade math exams involved more than 150,000 students in approximately 7,000 schools. The 8th-grade 2007 TIMSS assessed some 7,400 U.S. students in fewer than 250 schools. PISA in 2006 involved only 5,600 15-year-olds in around 170 schools.

The difference in size of these assessments is important: in reading and math in 4th and 8th grade, NAEP can report state-by-state performance every two years and, through the Trial Urban District Assessment program, NAEP can present data on a growing number of large school districts. In contrast, the small size of the PISA and TIMSS samples in most states makes state-level reporting difficult, though not impossible.

Statistical Benchmarking
One relatively inexpensive way in which state performance can be gauged against international measures is through statistical linking. Figure 1 shows TIMSS scores for 8th-grade math for seven states plus the District of Columbia and an equal number of countries. This figure is based on work by Gary Phillips of the American Institutes for Research, who has estimated 2007 TIMSS scores for all states by placing TIMSS and NAEP on the same scale. Such statistical linking can be more easily done with TIMSS than with PISA because TIMSS is administered at the same grade levels as NAEP, and their purposes and frameworks are similar.
Figure 1 presents patterns more precisely than we might infer from NAEP or TIMSS alone. For example, we know from the 2007 NAEP math report that Massachusetts and Minnesota are high-performing states, that South Carolina and Missouri perform around the national average, that Mississippi and Alabama are low-performing states, and that the District of Columbia, is the lowest-performing “state” in the nation. The 2007 TIMSS ranks countries, ranging from the highest-performing, including Chinese Taipei (Taiwan) and the Republic of Korea, through the middle ranks, including the United States, and going down to the lowest-performing countries, including Turkey (the lowest shown in Figure 1), Syria, Egypt, Algeria, and Colombia.

By combining these two sets of results, we know that even the best-performing American states do not score nearly as high as Chinese Taipei or Korea, that the average-performing American states are about on par with England, the Russian Federation, and Lithuania, and that the District of Columbia’s performance is more comparable to those of Thailand and Turkey.

While the Phillips method creates this interwoven list at minimal cost (because he is estimating state scores from already extant data), obtaining actual state-level scores is very expensive. How much a more precise list would be worth is an issue that states must consider.

Generating Policy Advice

Figure 1 reflects the “how are we doing” aspect of benchmarking. But what policy advice can be garnered from these international studies? The quick answer is not as much as many would have you believe. This is particularly true of PISA, which is very aggressive in drawing policy implications out of the data. Reflecting the mission of the OECD, which combines both statistical and policy functions, the very organizational structure of PISA combines the collecting and release of statistical data with policy advice in a single unit. In contrast, in the United States, strict Office of Management and Budget guidelines separate the release of federal statistical reports by time and place from any policy statements.

The pressure from policymakers for advice based on PISA interacts with this unhealthy mix of policy and technical people. The technical experts make sure that the appropriate caveats are noted, but the warnings are all too often ignored by the needs of the policy arm of PISA. As a result, PISA reports often list the known problems with the data, but then the policy advice flows as though those problems didn’t exist. Consequently, some have argued that PISA has become a vehicle for policy advocacy in which advice is built on flimsy data and flawed analysis. Much of the critical work has come from Germany, where PISA has had a profound effect. There has also been debate in Finland, in part because students do well in PISA but did relatively poorly on TIMSS on 1999.

Among the limitations are the fact that both PISA and TIMSS produce cross-sectional (single point in time) data and therefore do not allow longitudinal analysis (comparisons over time) at the student level, which is how researchers prefer to measure growth in student achievement and to identify the factors associated with such changes. But too often, countries and education ministers demand information and don’t want to know how cross-sectional data using flawed measures can lead to bad advice or advice that seems “right” but has little or no basis in fact.

To be sure, international data can in some cases be useful for addressing some policy questions, and often these data are the only way to examine the consequences of differences in policy that vary across countries. But the obstacles to drawing strong causal inferences based on such data are substantial. While both the International Association for the Evaluation of Educational Achievement (IEA) and the OECD work hard to make sure that measures are comparable, there are large variations in how people in different countries understand similar questions, and how statistical agencies measure and report indicators. This is especially true for the United States, where, as a federal system of continental size in which education is still predominantly a state and local function, variance is often more important than the mean national value calculated by many international studies. Unfortunately, both TIMSS and PISA data are used, too often with little regard for their limitations, to formulate policy recommendations.

It should be noted that NAEP has taken a different approach to releasing its results, one that PISA has rejected. While early NAEP reports resembled the current voluminous PISA reports, over time, the Nation’s Report Cards have become shorter and more tightly focused on data that show cross-sectional results by state and by reporting categories called for in U.S. legislation (for example, race or ethnicity and income). NAEP reports also compare the performance of students at the same grade level over time; these comparisons have become a critical tool for measuring the nation’s progress. In short, these are highly focused and valuable benchmarking reports. Further, a typical NAEP release is far more accessible to policymakers, reporters, and the general public than PISA reports are. And NAEP reports do not contain policy implications or recommendations: the dividing line between statistical reporting and policy prescription is honored.

Within guidelines designed to protect confidentiality, data from all three assessments are made widely available both online and through other avenues of dissemination, allowing researchers to explore statistical patterns and the policy implications of the data. However, with PISA, the OECD megaphone attracts the attention of policymakers and drowns out alternate interpretations.

Accountability Problem
OECD is a high-level intergovernmental organization with wide sweep across many domains, including finance, trade, environment, agriculture, technology, and taxation as well as education. Its pronouncements have a gravitas that cannot be matched by the IEA. Just as OECD does not have a modest presence in describing the scope of PISA, its voice in pushing the policy lessons that can be “learned” from PISA has been equally loud. But that loud voice has a few cracks in it.

For example, consider Chapter 5 of the 2006 PISA report, “Science Competencies for Tomorrow’s World,” which is devoted to analyzing the effects of school resources on student performance. Clearly, the stakes are high in identifying the best ways to allocate money, teachers, school leadership, and the like. Box 5.1 on page 215 notes all the appropriate caveats: PISA is cross-sectional; some things are not measured well; and other important factors are unmeasured. Moreover, it notes that the school characteristics measured are from the student’s current school, and in many countries a 15-year-old might have been in that school for only a year or two. The report warns that “these restrictions limit the ability of PISA to provide direct statistical estimates of the effects of school resources on educational outcomes.”

Nonetheless, the chapter proceeds with dozens of charts and tables relating different school resources to student outcomes. Finally, the chapter ends with “policy implications,” even though the foundations for these implications are weak. For example, the report reads, “what is noticeable about the strongest effects measured in this chapter is that they are not the ones most closely associated with finite material resources, such as the distribution of good teachers. Rather, such effects are related to how schools and the school system are run—for example, the amount of time that students spend in class and the extent to which schools are accountable for results.”

This is powerful stuff: hold schools accountable, and other issues (such as the allocation of good teachers) don’t matter. What is the evidence for PISA’s confident endorsement of accountability?

On close examination, “accountability” turns out to mean merely the public posting of school-level results. No other measures of accountability were tied to PISA scores at a level of statistical significance once students’ socioeconomic status was introduced. But the report uses the highly evocative word “accountable” rather than “posting.” The reader can easily have the mistaken impression that a broad set of conditions has been found to be important.

Standards of evidence and the research practices that are found in this analysis would not pass muster in the equivalent statistical agencies and among most researchers in the United States. Indeed, the National Center for Education Statistics (NCES) frequently objects to many of the findings in PISA reports, but is almost as often politely (and sometimes not politely) ignored.

Questions Remain
Despite these limits, the chorus for state-level assessments will continue to grow louder. But how would state-level international assessments fit into the already complex world of large-scale assessments? In the 2006 PISA assessment, the U.S. barely made the minimal school participation rate to be included in the analysis. If governors and chief state school officers are behind a state administration of PISA or TIMSS, getting school participation may be easier, but student engagement in low-stakes tests declines as students get older—likely a situation that even gubernatorial exhortations cannot resolve.

If it is true that what gets tested gets taught, are governors and chief state school officers really willing to allow OECD or IEA to drive their curricula? In the rush to employ rigorous international standards, we need to keep in mind that TIMSS and PISA each embody particular views about what those standards should be.

Can PISA really inform a state’s policymakers about how to improve their school system? PISA assesses mathematical and science “literacy,” a broad domain encompassing skills and knowledge learned both inside and outside of school. PISA 2006 cautions, “If a country’s scale scores in reading, scientific or mathematical literacy are significantly higher than those in another country, it cannot automatically be inferred that the schools or particular parts of the education system in the first country are more effective than those in the second. However, one can legitimately conclude that the cumulative impact of learning experiences in the first country, starting in early childhood and up to the age of 15 and embracing experiences both in school and at home, have resulted in higher outcomes in the literacy domains that PISA measures.” In other words, PISA itself admits that it cannot reliably identify which parts of the education pipeline are working well and which need improvement.

How expensive are these tests, and are there less costly ways to get these or similar data? To get a state-level score, about 1,500 students in the state need to be tested. The cost to test that number of students is around $700,000 for PISA and, based on the experience of Massachusetts, approximately $500,000 per grade for TIMSS. Nationwide, that’s around $25 million per PISA assessment and $15 million per grade in TIMSS if all states participated. There are less expensive alternatives, such as the TIMSS estimates for the states generated by Gary Phillips and noted earlier. While the alternatives would not produce all the details that might come from the full assessment (and mercifully avoid the temptation to use these data for unwarranted policy analysis), they could produce reliable estimates of state performance relative to international performance. Phillips has recently worked out the technical details for obtaining international benchmarks with PISA, by embedding PISA items within the state assessment. The PISA items on the state assessment are then used to link the state assessment to the PISA scale. Once this is done, the state can benchmark its performance against PISA without having students take the entire PISA assessment.

An alternative to statistical linking for PISA would be “small area estimation.” These estimates are model-based and “borrow” information from other data available for the state together with any state-level PISA data collected. The results are known as “indirect” projections to distinguish them from standard or “direct” estimates. NCES produced such state and country estimates of adults with low literacy based on the National Assessment of Adult Literacy (NAAL), for example. Further research would be needed to determine the feasibility of conducting small area estimates to generate state-level PISA scores. And it is likely that the PISA sample would have to be increased somewhat. This would add costs, but far less than full state-level assessments.

Double-Edged Sword

Compared to students in most other OECD countries, American students do not perform well on PISA. In the aftermath of the dismal results from the 2006 PISA, the clamor for producing state-level PISA scores was thunderous. State interest in administering PISA has since receded, with the current financial crisis and the fading of the fanfare surrounding PISA 2006, but the economy will eventually turn around, PISA 2009 is in the offing, and the Obama administration and many governors are sympathetic to the idea of international benchmarking.

While TIMSS has its partisans and states have chosen to participate in past years, momentum is behind PISA. If more states do choose to participate in PISA, what should they expect? First, they would get a PISA score that would allow them to compare themselves to other PISA participants. In some cases, this would provide bragging rights (“Our students are better than those in Turkey”). In most states, disappointing results would provide reform-minded governors with ammunition to push for their own legislative agenda. But if states choose a full state assessment, along with the PISA scale score would come all of the OECD’s policy advice and its approach to standards, which might make it harder for reform-minded governors to choose the options they prefer. Caveat emptor.

Mark Schneider is a vice president at American Institutes for Research and former commissioner of the National Center for Education Statistics.

Complete survey results available here.

---

What do Americans think about their schools? More important, perhaps, what would it take to change their minds? Can a president at the peak of his popularity convince people to rethink their positions on specific education reforms? Might research findings do so? And when do new facts have the potential to alter public thinking? Answers to these questions can be gleaned from surveys conducted over the past three years under the auspices of Education Next and Harvard’s Program on Education Policy and Governance (PEPG). (For full results from the 2009 survey, download the PDF; for the 2007 and 2008 surveys see “What Americans Think about Their Schools,” features, Fall 2007, and “The 2008 Education Next—PEPG Survey of Public Opinion,” features, Fall 2008).

In a series of survey experiments, we find a substantial share of the public willing to reconsider its policy prescriptions for public schools. But this responsiveness is not uniform: presidential appeals are more persuasive to fellow partisans than to those who identify with the opposition party, research findings have the greatest impact when an issue remains unsettled, and learning basic facts has the biggest impact when those facts are not well known. None of this comes as a surprise, until one considers how stable aggregate public opinion has been over time.

Individual Volatility but Collective Stability
The opinions expressed by individuals, when surveyed on political issues to which they have not given much thought, can appear so fragile as to be meaningless. More than one psephologist has shown that it is not uncommon for people, when repeatedly asked the same question, to give a positive response the first time, offer a negative one on the second occasion, and then return to a positive position the third time around. In such situations, opinions seem to be so lightly held they lack any content whatsoever.

Our own data likewise reveal a fair amount of volatility in the views expressed in the three Education Next—PEPG surveys by individual respondents, many of whom participated in multiple years. Of those asked to grade the nation’s public schools in both 2008 and 2009, for example, only 59 percent assigned the same grade both years. Among those who gave a grade of “A” or “B” in 2008, 46 percent awarded a grade of “C” or lower in 2009.

Numerous respondents also expressed different views on controversial policy issues across survey years. Among those who either completely or somewhat supported merit pay in 2008, 34 percent did not give that support one year later. Conversely, 29 percent of respondents who either completely or somewhat opposed the policy in 2008 did not express that opposition the next year. Similar churning is evident in the responses to questions concerning single-sex public schools, charter schools, and national standards.

The flip-flop that characterizes as much as one-third of individual responses does not produce equally large fluctuations in aggregate public opinion, however. On the contrary, the percentage of Americans holding to a particular point of view typically remains stable from one year to the next. On two-thirds of the domestic issues studied by political scientists Benjamin Page and Robert Shapiro, opinion did not change by more than 5 percentage points, despite the fact that years separated the fielding of different surveys. In the aftermath of major events—wars, economic recessions, or a terrorist attack—the views of the public as a whole may change abruptly and dramatically. More commonly, though, public opinion either holds firm or eases slowly in one direction or another.

Thinking on education policy follows the general pattern. In the three years of Education Next—PEPG surveys, we found little change in the responses to many of the questions posed in identical or similar ways across successive years (see Figure 1). Public opinion held steady on such issues as the introduction of merit pay for teachers, setting of uniform educational standards across the country, and the desirability of single-sex education.

Nor did the public’s evaluation of American schools change much between 2007 and 2009, despite the media drumbeat of negative information about dropout rates and test scores. Indeed, the percentage of those surveyed willing to give the nation’s schools an “A” or a “B” slipped by just four points, from 22 percent in 2007 to 18 percent in 2009. Meanwhile, the share of adults giving schools a “D” or an “F” hovered around 25 percent throughout the three-year period (see Figure 2).

What accounts for the differences between individual and aggregate public opinion? Undoubtedly, part of the explanation is measurement error. Some of those answering our survey questions may have simply misread or misunderstood the questions in one year or the other, so their opinion seems to have changed when in fact it did not. Ordinarily, that kind of error balances itself out, as mistakes by one individual offset opposite errors by another.

But it seems unlikely that a third of our respondents would make such mistakes, and a substantial body of research on political behavior suggests that something else is going on as well. One prominent theory emphasizes the influence of public discourse. When people answer a survey item, they often draw upon a recent media report they have heard or conversation they have had with friends, relatives, or co-workers. Individual responses, then, vary from week to week as people are exposed to different claims. Collective opinion, however, remains constant so long as the general discourse does. If that theory is correct, then opinion in the aggregate changes only when public discourse shifts—either by a major event or with the introduction of a new fact or a new political force.

On some education issues, public discourse has changed since 2007. For instance, support for the federal No Child Left Behind Act has eroded, as evidence accumulated that the federal law was not living up to the promise of its grossly overstated name and politicians in both major parties found it to be an easy target (see Figure 3). Between 2007 and 2008, the share of adults who thought the law should be renewed (with no more than minor changes) fell by 7 percentage points. Support for the law stabilized after 2008, however, and roughly half the population still supports its reenactment with no more than modest revisions. And as we saw in previous years, a randomly selected group of respondents who were asked about “federal accountability policy” rather than “No Child Left Behind” expressed even higher levels of support.

Similarly, as the current recession deepens, we see hints of growing taxpayer resistance to the rising cost of education. Support for increased spending on public education fell from 51 to 46 percent between 2007 and 2009. Confidence that spending more on schools would enhance school quality fell by a similar amount, from 59 to 53 percent. Still, these changes remain modest. Facing the most significant economic downturn since the Great Depression, most Americans continue to support increased spending on their local public schools.

What would it take, then, to move aggregate public thinking decisively in one direction or another? Might influential public figures, research findings, or factual knowledge lead at least some portions of the American public to update its thinking? To find out, we divided the more than 3,000 respondents to our 2009 survey into randomly chosen groups. The first group was simply asked its opinion about a policy question, while the second (and often a third or fourth) group was given some additional piece of information, such as the president’s position on the issue, a research finding, or a key fact. By comparing answers given by the different groups, which should be similar in composition, it is possible to gauge the impact of these additional sources of information on the public’s views. (For more methodological details, see sidebar.)

Professors or Politicians: Who Is More Influential?
We fielded our survey in March of 2009, when newly elected president Barack Obama enjoyed public approval ratings above 60 percent. The timing of the survey provided an ideal opportunity to estimate the impact an endorsement by a popular president can have on policy views.

To ascertain the president’s influence, we conducted some simple experiments. On three topics—merit pay, charter schools, and school vouchers—one group of survey respondents was asked its opinion without any special prompt. Another group was first told the president’s position on the issue before being asked for its own. A third group was instead told about evidence from research on the policy’s effects on student learning. We did not specify a specific study, as the point was not to estimate the influence of any particular piece of research but rather the potential impact such evidence might have.

Merit Pay: When asked for an opinion straight out, a slight plurality of Americans sampled—43 percent—supported the idea of “basing a teacher’s salary, in part, on his or her students’ academic progress on state tests.” Twenty-seven percent opposed the idea, with the remaining 30 percent undecided. As noted above, that pattern of opinion has hardly budged since 2007.

Such stability over time, however, masks a propensity of some Americans to alter their views in light of an appeal by a popular political leader. Those informed of President Obama’s support for merit pay favored the idea by 13 percentage points more than those not so informed (see Figure 4). Obama’s backing had a particularly dramatic impact on African Americans, whose support jumped by 23 percentage points. Even many teachers were persuaded. Initially, only 12 percent of those not informed of Obama’s opinion thought merit pay a good idea, but that number jumped to 31 percent among those told of the president’s position. Obama’s endorsement caused support among Democrats to rise from 41 to 56 percent. Among Republicans, too, backing for the idea rose, albeit by a lesser amount (from 48 to 59 percent).

By comparison, policy research on the topic had a modest impact on public thinking. Among those told that “a recent study presents evidence that students learn more when their teachers are paid, in part, according to their students’ academic progress on tests,” support for merit pay climbed by just 6 percentage points above the support given when that information was withheld. The one subgroup to register especially large changes was African Americans, among whom support skyrocketed by 28 percentage points. Democrats were somewhat more responsive to research evidence than other segments of the public, with their support for merit pay increasing by 10 percentage points.

School Vouchers: Public opinion on school vouchers varied somewhat, depending on the way in which the question was worded. To one group of respondents we presented the issue as follows: “A proposal has been made that would give low-income families with children in public schools a wider choice, by allowing them to enroll their children in private schools instead, with government helping to pay the tuition. Would you favor or oppose this proposal?” In this instance, 40 percent of the respondents gave a favorable reply and 34 percent a negative one, with 27 percent taking a middling position. But when we posed the question slightly differently—asking about a “proposal that would use government funds to help pay the tuition of low-income students whose families would like them to attend private schools”—just 35 percent supported the idea. In this instance, a small alteration in wording shifted public opinion by 5 percentage points.

We also find that public support for vouchers declined by 5 percentage points between 2008 and 2009, perhaps as a result of the opposition to vouchers expressed by most Democratic presidential candidates during that party’s extended primary-election campaign, which conceivably could have altered the balance of public discourse. That interpretation is reinforced by the impact that President Obama’s position can have on public opinion. Overall, the percentage favoring vouchers was 11 points lower among those informed of the president’s opposition than among those not so informed (35 percent to 24 percent, see Figure 4). We also observed large partisan differences in the president’s influence on this issue. Whereas just 30 percent of Democrats expressed opposition to vouchers when asked outright, 52 percent did so after hearing of Obama’s opposition. By comparison, opposition among Republicans increased only slightly, from 50 to 54 percent. African Americans expressed higher levels of support for vouchers than did the population as a whole (57 percent), but support also was 12 percentage points lower among those African Americans told of presidential opposition.

A study that “presents evidence that students learn no more in private school than in public schools” depressed support for vouchers by 10 percentage points overall, an impact almost as large as presidential position taking. The same research evidence reduced support among Democrats by 15 percentage points, as compared to 6 percentage points for Republicans.

Charter Schools: Most Americans have yet to make up their minds about charter schools. Though 39 percent expressed support and only 17 percent signaled opposition in 2009, 44 percent remained undecided. These responses look much as they did in both 2007 and 2008, an indication that public discourse on charters has not changed significantly in recent years.

Despite that stability of public opinion about charters, aggregate support increased by 11 percentage points when respondents were told that Obama backed them (see Figure 4). We again found evidence that Obama’s impact has a partisan tinge. Among his fellow Democrats, Obama’s support is an unmitigated asset for charter school advocates, lifting support from 35 to 47 percent. But among Republicans, the percentage favoring charters increased by only 5 points (from 47 to 52 percent) upon learning of Obama’s endorsement. That endorsement actually decreased the proportion of Republicans who “completely” supported charter schools, from 22 to 15 percent.

When it comes to charter schools, research findings appear every bit as influential as a popular president. Told that recent research showed “students learn more in charter schools than in public schools,” support for charter schools rose by 14 percentage points. Among African Americans, the percentage who “completely” supported charter schools climbed by fully 23 percentage points, from 14 to 37 percent. Hispanics, meanwhile, were least persuaded by the evidence; only 5 percent altered their opinions. As they did on the previous items, Democrats appear to be more impressed by research than Republicans. Among those given evidence that charter schools enhance student learning, Democratic support for charter schools shot upward by 18 percentage points to 53 percent (compared to 35 percent among those not so informed), while the percentage of Republicans favoring such schools shifted by just 12 percentage points.

When all three issues—merit pay, vouchers, and charters—are considered together, a case can be made that new policy research, if communicated widely, can have an impact rivaling that of an influential president at the peak of his popularity. Admittedly, evidence from the research community does not have the same consistent impact on opinion as Obama’s position taking, which at the time of our survey could move overall public opinion by anywhere from 11 percentage points (in the case of charters) to 13 percentage points (in the case of merit pay). But the impact of a study is of comparable magnitude, ranging from 6 percentage points (in the case of merit pay) to 10 percentage points (in the case of vouchers) to 14 percentage points (in the case of charters). Research appears particularly influential among Democrats and when the general public’s own views have yet to take shape. That half the public has yet to make up its mind about charter schools may provide researchers with an opportunity to shape the public conversation going forward.

Stubborn Facts

How about raw facts concerning the state of American education? What does the public actually know about the performance of the nation’s public schools and the resources devoted to them? And is the public willing to update its views when told the truth?

We conducted additional experiments to investigate these issues. In 2007, we asked respondents to estimate average per-pupil expenditures within their local school district and the average teacher salaries in their states. When we discovered that those surveyed, on average, underestimated per-pupil expenditures by more than half and teacher salaries by roughly 30 percent, we wondered whether people had equally poor information about the performance of American high schools (see “Educating the Public,” features, Summer 2009). So in 2009 we asked a random third of our sample to estimate high school graduation rates and another third to estimate the international standing of U.S. 15-year-olds in math. The remaining two-thirds of the sample was told the truth about one or the other of these matters, allowing us to see whether people’s assessments of their schools differed when given accurate information.

To our surprise, the public had a far more accurate understanding of student performance than they had of teacher salaries and per-pupil spending. When it comes to high school graduation rates nationwide, the best available estimates from the U.S. Department of Education suggest that roughly 75 percent of those who enter 9th grade graduate within four years, a far cry from the goal of universal high school completion to which the president of the United States and all 50 governors in 1989 committed themselves to reaching by the year 2000. When asked to give their own estimate, without any hint or help as to what the right answer might be, those surveyed came up with an even more pessimistic estimate of 66 percent, 9 percentage points below actual levels. Excluding those respondents who gave answers of less than 25 percent (on the grounds that they may have misunderstood the question or not taken it seriously) increases the average estimate only slightly to 69 percent. Either estimate is nonetheless a good deal closer to, and a good deal less optimistic about, the truth than the wildly inaccurate estimates that the public offered about teacher salaries and school expenditures.

The public was only slightly less accurate when it came to estimating how well 15-year-olds in the United States do in math, as compared to students in 29 of the leading industrialized countries. Here the correct answer, according to the latest tests administered by the Organisation for Economic Co-operation and Development’s Program on International Student Assessment (PISA), is 24th out of 29th. Both the average and median guess was 18th, a bit more optimistic than actual PISA results but not too far off the mark. Clearly, Americans have not been deceived into believing that our students are outperforming their counterparts abroad.

So what happens when the public is told the truth? Not much, it turns out, if people already have a pretty solid grasp of the relevant facts. When informed that 75 percent of students graduated from high school, the public took that as neutral to mildly good news, as the percentage giving schools an “A” or “B” increased by a trivial 2 points and the percentage getting a “D” or “F” dropped by 1 point (both statistically insignificant changes). Learning the truth about the international standing of American students had a bigger impact, reducing the share of respondents giving a grade of “A” or “B” from 18 to 13 percent and increasing the share of respondents giving a “D” or “F” by 10 percentage points (see Figure 5a).

In the case of spending, however, learning the truth shifted opinion by a larger margin (see Figure 5b). For the nation as a whole, overall support for higher spending levels dropped by 8 percentage points (from 46 to 38 percent) when respondents were informed of actual per-pupil expenditures in their own district. The impacts of this information varied widely across subgroups. Told the truth about per-pupil expenditures, the share of African Americans willing to support additional spending plummeted from 82 to 48 percent. Perhaps not surprisingly, teachers held firm in their commitment to higher spending.

Even larger impacts are observed on support for increased teacher salaries. When informed about actual average teacher salaries in their state, respondents’ support for higher salaries dropped by 16 percentage points (from 56 to 40 percent). In this instance, roughly comparable impacts are observed for all three ethnic groups. But as one might again expect, teachers’ support for high salaries was relatively undiminished, dropping just 6 percentage points (from 77 to 71 percent).

Why does the public have a generally accurate understanding of school performance but a gross misunderstanding of the amount that is spent on education? The answer may have to do with the availability of information on these issues. It is true that the U.S. Department of Education regularly releases information on all four topics in the same document, the Digest of Education Statistics. But student dropout rates and student performance on international tests receive much more extensive attention in the news media than information about per-pupil spending in individual school districts or teacher salaries in specific states. The cost of education is divided among federal, state, and local governments, and the total sums are difficult to assemble until that is done by the federal government several years after the fact.

It is unlikely that organizations outside of the media are likely to pick up the slack. With a large share of the population convinced that schools and teachers should be given more money, or at least be held harmless, few if any interest groups or politicians have an incentive to dramatize the fact that spending levels and teacher salaries are much higher than most people believe. So school reformers instead focus on low test scores and high dropout rates as justification for merit pay, school accountability initiatives, and other school choice reforms. The public may only learn about the true cost of education when a popular political figure stakes a political career on telling them. That, we suspect, is as likely as the Cubs winning the Super Bowl.

Surveys and Realities
Our experiments only hint at what could happen in the real world of school politics. It is one thing to inform a captive audience of survey respondents about the president’s position, the results from research, or a key fact about American education. Reaching the entire American public is a completely different matter. To change opinions, one must get the public’s attention. And that is no easy task, when jobs, family life, entertainment, and sports command a higher priority in most households. Only 38 percent of the respondents to our survey report paying “a great deal” or “quite a bit” of attention to education issues. And even the power of presidents is limited by the large number of issues to which they must attend. President Obama’s genuine thoughts on such matters as merit pay, charters, and vouchers, however deeply held, necessarily command far less of his time and energy than the multitude of foreign policy, economic, and other domestic problems to which he must devote his attention.

Still, our findings suggest that a well-publicized stance taken by a popular president on an education issue might shift the opinions of large segments of the American public. Similarly, scholarship appears to be a potent weapon for groups with policy agendas they wish to pursue, as the committed can broadcast research findings with great repetition. Indeed, any group that seeks to change public opinion without gathering research to back its positions is leaving a flank unprotected. Finally, advocates are well advised to search for facts the public does not understand, and then to communicate those facts as widely as they can. Just as nothing affects opinion about an ongoing war as quickly as communiqués from the front, so too a better understanding of the facts about the public schools could in the long run shape American education.

William G. Howell is Sydney Stein Professor of American Politics at the Harris School of Public Policy at the University of Chicago. Paul E. Peterson is professor of government at Harvard University, senior fellow at the Hoover Institution, and editor-in-chief of Education Next. Martin R. West is assistant professor of education at the Harvard Graduate School of Education and executive editor of Education Next.

---

Survey Methods

This survey, sponsored by Education Next and the Program on Education Policy and Governance (PEPG) at Harvard University, was conducted by the polling firm Knowledge Networks (KN) between February 25 and March 13 of 2009. KN maintains a nationally representative panel of adults, obtained via list-assisted random digit—dialing sampling techniques, who agree to participate in a limited number of online surveys. Because KN offers members of its panel free Internet access and a WebTV device that connects to a telephone and television, the sample is not limited to current computer owners or users with Internet access. When recruiting for the panel, KN sends out an advance mailing and follows up with at least 15 dial attempts. The panel, then, is updated quarterly. Detailed information about the maintenance of the KN panel, the protocols used to administer surveys, and the comparability of online and telephone surveys is available online (www.knowledgenetworks.com/quality/).

The main findings from the Education Next—PEPG survey reported in this essay are based on a nationally representative stratified sample of U.S. adults (age 18 years and older) and oversamples of Hispanics and non-Hispanic blacks, public school teachers, and residents of Florida (the last group for supplemental analyses not reported here). The combined sample of 3,251 respondents consists of 2,153 non-Hispanic whites, 434 non-Hispanic blacks, 481 Hispanics, and 183 members of other ethnic groups; 709 public school teachers and 948 residents of Florida; and 1,694 self-identified Democrats and 1,265 self-identified Republicans. We use post-stratification population weights to adjust for survey nonresponse as well as for the oversampling of teachers and Floridians. These weights ensure that the observed demographic characteristics of the analytic sample match the known characteristics of the national adult population.

On many items we conducted experiments to examine the effect of variations in the way questions are posed. The figures and tables present separately the results for the different experimental conditions. In these instances, respondents were randomly assigned to exactly one of at least two possible conditions. Reported effects in the figures and tables reflect differences observed across the baseline and experimental conditions.

In general, survey results based on larger numbers of observations are more precise, that is, less prone to sampling variance than those made across groups with fewer numbers of observations. As a consequence, answers attributed to the national population are more precisely estimated than those attributed to subgroups. With 3,251 total respondents, the margin of error for responses given by the full sample in the Education Next—PEPG survey is 1.7 percentage points (for items on which opinion is evenly split). The results presented for subgroups within the sample have larger margins of error, depending on their actual size. However, any differences in opinions or changes in opinions over time reported in the text are statistically significant unless otherwise noted.

Of the 3,251 respondents surveyed in 2009, approximately 300 had also been interviewed in 2008. For this group, it was possible to identify the consistency of responses to identical questions asked in both years.

Percentage totals do not always add to 100 as a result of rounding to the nearest percentage point.

---

If you haven’t heard the news that the newspaper industry is dying, you must not be reading the newspaper anymore. Which is entirely possible. According to the Pew Research Center, newspaper readership fell 5 percent in just the past year, and advertising revenues are down 23 percent over the past two years. The third quarter of 2008 saw the worst decline in print ad revenue in nearly 40 years, reports the Newspaper Association of America. Several major chains are in bankruptcy, and a few big papers have disappeared entirely. With the economy in deep recession, the situation only looks to grow worse.

This is a topic that excites editorialists, who are, of course, members of the journalism profession themselves. Many bemoan the demise of the daily newspaper, arguing that it signals the end to the educated citizen. Others worry that Americans will retreat to ideological safe havens—cable TV channels, Internet sites, and blogs that conform to their strongly held views—which will lead to even greater divisiveness in our politics and culture (see Figure 1).

Are these concerns valid when it comes to coverage of education? To find out, I interviewed some of the smartest minds in education journalism, including Richard Lee Colvin of the Hechinger Institute; Richard Whitmire, president of the National Education Writers Association (EWA); Jim Bencivenga, formerly the education editor of the Christian Science Monitor; Virginia Edwards, the publisher of Education Week; and Elizabeth Green, an editor of the online upstart GothamSchools.

They all agree that the demise of the daily newspaper is bad for the local “conversation” around education. And even if papers survive, already the education beat is being squeezed. Those reporters who remain on the job (see Figure 2) are asked to cover higher education as well as K—12 schools, meaning they have less time to develop expertise in specific areas. They are pushed to write shorter articles, leaving little space for in-depth reporting. And editors want stories that are hyperlocal, at the school level, not missives about the latest school board policies, or dry accounts of state regulatory actions.

As a result, says Whitmire, decisions go unchallenged. “What is lost is that the superintendent will bring in a new program, and nobody will be there to explain to the community whether similar programs have worked or failed in other places.” Colvin goes even further: “We hear from superintendents that the coverage is worse than ever.” All the reporters seem to want is a “couple of quotes” for a “sensationalist” story. So when it’s time for leaders to make the case for, say, budget increases, they have no credible vehicle through which to explain their rationale to the broader public, beyond their own communication outlets, and no independent third party to present opposing views in a fair manner.

“The people who will be excluded from the conversation will be people without kids in the schools,” explains Bencivenga. Those with a vested interest—the teachers unions, realtors—will continue to get their message out. But there will be no one to counter these self-serving opinions. As Edwards argues, “An ill-informed public will benefit people who can push an agenda without accountability and public scrutiny.”

The national scene is unlikely to look so bleak, argue the experts. In part that’s because everyone expects a few of the great national papers to survive. But it’s also because of the well-developed community of think tanks, blogs, and trade publications that follow the education issue, and aren’t disappearing anytime soon. “National will always be the easiest to do,” says Whitmire.

But even at that level, there are signs of trouble. Right now a CEO or university president might skim an education editorial or op-ed while flipping through the Wall Street Journal or New York Times. But imagine if such elites stop getting newspapers and only read articles online that are of immediate interest. The larger public that engages in the K—12 education debate could shrink dramatically, to just partisans engaged in the war of ideas around schooling.

Still, nobody expects high-quality education journalism to disappear without a fight. Already new models are starting to emerge. One of the most talked about is Green’s GothamSchools, an online site that follows New York City’s education scene, and also covers national education happenings. An initiative of the nonprofit Open Planning Project, GothamSchools aims to be a “one stop shop in New York City for education news,” according to Green. It covers education the way newspapers used to, with its two reporter/editors showing up at every meeting of the city council and the state assembly that touches on the public schools. But it also dives in to do more in-depth analyses and investigative pieces.

There are similar experiments up and running in Philadelphia, Cleveland, and Chicago. Plus a growing number of cities, such as Minneapolis and San Diego, are home to new online nonprofit newspapers that feature strong education reporting.

Meanwhile, the national players are tweaking their approach. Hechinger is going into the content-production business, with a growing staff that will write original, long-form articles about key education trends. The EWA has a new “public editor,” former Washington Post reporter Linda Perlstein, who helps cub education writers around the country hone their craft and offers high-level feedback to more-seasoned pros. And Education Week is contemplating how it might distribute its content to daily newspapers.

All of these “business models” have promise, but also big question marks. First, they depend on philanthropic support for start-up revenue; with the stock market, and thus endowments, down some 30 percent, it’s not known whether there will be enough grant dollars to go around. Even Education Week, which has attracted foundation support for years, only receives 25 percent of its revenues from such sources and must bring in the rest from subscriptions, advertising, and elsewhere. Second, it’s unclear whether any of these approaches are sustainable once the grant money inevitably runs out.

So should education reformers try hard to help solve these problems? Would the further diminishment of the newspaper be bad for the cause of change? That’s not quite clear, either. On the one hand, newspaper editorial pages have generally been friendly to supporters of greater accountability, transparency, and parental choice. They have been particularly bullish on charter schools (“Opinion Leaders or Laggards?” what next, Summer 2008). And to be sure, the organized interests of the status quo—particularly the powerful teachers unions—will always find a way to get their message out to core audiences. They would only profit from a world without the fourth estate playing referee.

On the other hand, reporters have never been particularly astute at covering “change,” particularly the variety that causes pain for adults. In this way, the media have not been very friendly to reformers. Colvin puts it best: “Journalists never get out front of reform. They are always the trailing entity. Anyone whose ox is gored by reform is going to be outspoken and resist it. Journalists must understand that there’s always pain and disruption in great change. But they rarely frame it that way.”

Now that it’s the journalists who face pain and disruption, the question is whether they will get out in front of the changes happening in their own industry. No doubt you can follow this developing story in a local newspaper—if you still subscribe to one.

When the status quo is protected by vested interests, then school reform must be driven by ideas backed by clear evidence. Results from our 2009 national poll tell us that a solid research finding has the capacity to shift public support for charter schools from 39 to 53 percent, a substantial increase (see “The Persuadable Public,” features). A study’s power to persuade turns out to be as potent as Barack Obama’s persuasive capacity two months after he assumed the presidency.

To get a better sense of how research can influence the real world of policymaking, consider recent events in Massachusetts, where Boston’s longest-serving mayor (1993—present), Thomas Menino, is seeking reelection for an unprecedented sixth term. “Mumbles,” as the mayor is affectionately called, is best known for his commitment to snow removal, neighborhood parks, and symbiotic relationships with political insiders. On education matters, he appoints the school board and lets the members run the Boston schools as they please—so long as they avoid upsetting the local teachers union. But on the eve of his current campaign, Menino asked the legislature to expand charter school operations in Boston.

Why did Mayor Menino suddenly get charter school religion? Only recently, teachers unions seemed to be riding high in the saddle, enjoying for the first time in more than a decade a government unified under the union-friendly leadership of a Democratic governor and a legislature controlled by the same party. The mayor has generally distanced himself from education issues, and Boston’s best-known school reform consists of “pilot” schools, which have more than usual autonomy but are still subject to the district’s education-crushing collective bargaining agreement. Governor Deval Patrick, in a nod to the mayor, backed legislation that would expand pilot schooling throughout the state while curtailing charter school operations (see “Accountability Overboard,” features, Spring 2009).

The nail in the charter school coffin was expected to come with the release of a charter and pilot school evaluation initiated by the Boston Foundation, a reliable public school supporter. The foundation had nonetheless arranged for its evaluation to be conducted under the leadership of economist Thomas Kane of the Harvard Graduate School of Education, who has considerable credibility on all sides of the charter school debate. The study was designed as a gold-standard randomized field trial, in which students were (by means of a lottery) randomly given the opportunity to go to charter schools or not. The achievement of students who won the lottery and attended charter schools was compared with the achievement of students who entered but failed to win the lottery. Ditto for pilot schools.

To the surprise of the Massachusetts education establishment, the charters won—and the pilots lost—the research contest. No matter how the data were analyzed, charter schools routinely outperformed both Boston’s pilot schools and its traditional public schools. Pilots turned out to be no improvement on the status quo whatsoever.

The research findings reinforced the pro-charter campaign led by a local think tank, the Pioneer Institute. Statewide, newspapers editorialized in favor of charters and against the governor’s so-called reforms. Even the liberal Boston Globe climbed on board the school reform train. It didn’t hurt that the state legislature was riddled by scandal and Governor Patrick’s tax, fiscal, and transportation policies were going nowhere.

Politically, it was time for Mayor Menino to separate himself from the nonsense emanating from the state capitol. The best way for a popular mayor to remain that way is to catch a changing wind before it acquires gale force, in this case a wind set in motion by the Kane evaluation. When circumstances are right, professors can be as powerful as politicians.

Well…let’s not exaggerate. Mayor Menino may have climbed out of the teachers union bed but only into a twin bed in the same room. The mayor’s call for action will need to be accompanied by well-timed use of mayoral muscle inside the state legislature if more charter schools are to come to Boston. Still, research has nudged the thinking of one of Massachusetts’s most savvy politicians—no small feat.

Students will test the limits of acceptable behavior in myriad ways better known to school teachers than to judges; school officials need a degree of flexible authority to respond to disciplinary challenges; and the law has always considered the relationship between teachers and students special.

— Supreme Court Justice Stephen Breyer