The post What <em>Cabrini</em> Can Teach Us about the School Choice Movement appeared first on Education Next.

The post Hoosiers Score Benefits from Private School Choice appeared first on Education Next.

The May-June issue of  Harvard Magazine carries an article, “The Risks of Homeschooling,” promoting the argument of Harvard Law School Professor Elizabeth Bartholet that the U.S. should enact “a presumptive ban” on homeschooling. Homeschooling is essentially unregulated, Bartholet argues, and many parents adopt this method of educating their children for nefarious reasons including indoctrinating the parents’ values into their children, isolating the children from society, and abusing them. Parents should be assumed to be incompetent and dangerous educators of their children. Therefore, specific parents may homeschool their children only if government officials determine that allowing them to educate their children at home is worth the risk.

The article prompted a tsunami of critical responses, in Education Next (see “Harvard Professor’s ‘Absurd’ Claim that Homeschooling is Child Abuse”) as well as here, here, here, here, here, here, here, and here. We seek here to move the discussion beyond the 1,000-word Harvard Magazine article that sparked such opprobrium by carefully considering Bartholet’s 80-page Arizona Law Review article that inspired the story. We expected it to be rigorous and fact-based but were sadly disappointed.

We are empirical social scientists who have studied and written about homeschooling along with other education policy topics. One of us, Wolf, homeschooled one of his now-adult children years before the Covid-19 pandemic turned tens of millions of Americans into some version of home-schoolers. Another, Watson, is homeschooling by necessity now. Upon reviewing Professor Bartholet’s article, we conclude that it suffers from contradictions, factual errors, statements of stereotyping, and a failure seriously to consider that the alternative to homeschooling–public schooling–shares the problems that she attributes to home education. One need not have personal experience with homeschooling to identify the article’s many flaws.

Professor Bartholet often seems to be arguing with herself. Early in the article, she states: “We have no way of identifying, based on existing information, the total group of homeschoolers.” For the remainder of the article, however, she confidently describes what is true of “many” (used 90 times), “most” (53 times), and “a majority of” (6 times) homeschoolers. If no one knows the denominator, how can anyone say what is true for many, most, or, especially, a majority of the homeschool population? Similarly, in the section where Bartholet argues that “many” homeschooling parents are incapable of educating their children, she asserts that homeschoolers are disadvantaged relative to non-homeschoolers. In the later section where she dismisses the empirical research on homeschooling outcomes, she argues that the superior results for homeschooled students are simply because their families are advantaged relative to non-homeschoolers. Throughout the article, homeschoolers are characterized as almost monolithically (“up to 90%”) conservative Christians seeking to indoctrinate their children. On page 10, however, they are described more accurately as highly diverse in their religious affiliations and motivations. On page 5, Bartholet states, accusingly, “Most all miss out on extracurricular activities.” Five pages later she admits that many parents who homeschool “make efforts to enable their children to participate in certain school programs such as sports.” If families homeschool primarily to indoctrinate their children, shield them from any outside influence, and abuse them, as she argues throughout the essay, why do most families homeschool a child for only part of their K-12 education, as is implied by statistics on page 9 of the law review article?

One contradiction stands out. Professor Bartholet says “homeschooling in its current unregulated form poses serious risks of abuse and neglect.” That is the central claim of her paper. Later in the article, she lists a dizzying array of government regulations of homeschooling judged by courts to be “legal.” These include her preferred regulations that parents be preapproved by the state before they can educate their children, be certified instructors, use a specified curriculum, provide annual reports, administer the state test to their child, and allow government officials to inspect their home. So homeschooling is unregulated but her prescribed regulations of homeschooling are reasonable because they operate legally today?

The article contains a number of assertions about homeschooling that are clearly undermined by the facts and unsupported by her sources. Bartholet’s central claim is: “Many families choose homeschooling precisely because it enables them to escape the attention” of Child Protective Services. Her source that supposedly proves this explosive charge merely states that there “is anecdotal evidence” of such ill intent. Similarly, Bartholet asserts that the “majority” of families that homeschool are “descendants of the original conservative Christian wing.”  She writes that estimates range “from a majority up to 90%.” The sources for this claim include a federal government survey that indicates only 16% of parents homeschool primarily to provide “religious instruction,” an article from The Atlantic on homeschoolers suggesting that “[R]oughly two-thirds are Christian,” and speculation by an author that 90% of families that homeschool “are religious.” If only 67% of homeschoolers are even Christian, how could “up to 90%” be specifically conservative Christians? The numbers don’t add up.

Bartholet states that many families that homeschool “are at the low end of the socioeconomic ladder, with 19% below the poverty line and 36% between poverty and 200% of poverty, significantly more than that of those in public and private schools.” She is claiming that 55% is a comparatively high proportion of low-income students in a population. It is not. Poverty and near-poverty rates are either missing or unreliable for most private school students, but over 52% of the public school student population has incomes at or below 185% of poverty, meaning that the proportions of students at or below 200% of poverty are similar for homeschool and public school students, in spite of the fact that only 28% of homeschool families contain two adults working outside the home. She asserts, without evidence, that children who are homeschooled lose “out on opportunities to learn things that are essential for employment and for exercising meaningful choices in their future lives.” A recent study by the research center CARDUS, which Bartholet cites at other points in her article, reports, to the contrary, that employment rates and earnings are similar for homeschooled and non-homeschooled adults, and the two groups are “not much different in their pursuit of new experiences in life.” Bartholet claims, “Homeschooling presents…democratic concerns.” The only empirical study of that question that we know of concludes, to the contrary, that students who were homeschooled display significantly higher levels of political tolerance than otherwise similar students who attended public schools.

Speaking of tolerance, some of the statements in the Bartholet article concern us. She states: “Members of a variety of religious groups are included today in this conservative Christian wing, including many Mormons, Jehovah’s Witnesses, and Seventh-day Adventists. These homeschooling groups hold similar ideas about the importance of keeping their children isolated from conflicting cultural values.” Professor Bartholet presumes to define which religious groups are and are not “conservative Christians,” then engages in stereotyping by ascribing the same “isolationist” values to individuals in those groups. She criticizes parents who homeschool for having values that run counter to those of the larger society, thus undermining the values of pluralism and toleration of people with views of the good life that differ from one’s own. She demonstrates prejudice against immigrants by asserting that non-English-speaking parents are incapable of homeschooling their children. She argues that some parents who homeschool “are mentally ill or disabled, or caught up in substance abuse,” as if having a disability necessarily disqualifies a parent from educating their child. These are some of the statements that leapt out as offensive to us.

Child abuse is a serious problem. It concerns us greatly. It occurs in the homes of some parents who homeschool their children but also in public schools around the country. Bartholet’s article pays scant attention to the usual alternative to homeschooling–public schools–and the problems of child abuse and disturbingly low levels of civic knowledge reported there. A rich debate could be had about when unconventional approaches to parenting cross the line into abuse and neglect. We did not find such a balanced or nuanced discussion in the 80 pages of this article.

Professor Elizabeth Bartholet’s claims that homeschooling contributes significantly to the scourge of child abuse fail to survive scrutiny. Ironically, Bartholet pleads with us to trust government officials to decide which select parents have the capability and correct values to be safely granted the privilege of educating their children at home. We now know what that looks like. Last month, government officials ordered almost all K-12 children to be educated at home in order to promote the safety and wellness of themselves and our society. We are all homeschoolers now, by government decree. That development appears to have left Professor Bartholet, well, schooled.

Patrick J. Wolf is a distinguished professor of education in the Department of Education Reform at the University of Arkansas and co-editor of Educating Citizens: International Perspectives on School Choice and Civic Values.

Matthew H. Lee is a distinguished doctoral fellow in the Department of Education Reform at the University of Arkansas.

Angela R. Watson is a senior research fellow at the Johns Hopkins Institute for Education Policy.

The post Harvard Law Professor’s Attack on Homeschooling Is a Flawed Failure. And Terribly Timed, Too. appeared first on Education Next.

“Everything works somewhere; nothing works everywhere,” writes Dylan Wiliam in his book Creating the Schools Our Children Need. To that I would add, everything works at something; nothing works at everything.

Together, the two maxims describe what my research team found when we evaluated the Louisiana Scholarship Program, a statewide school-voucher initiative: the program satisfied some of its goals but fell well short of others, including that of raising student scores on state tests. What follows is a cautionary tale about good intentions and seemingly reasonable decisions resulting in unintended consequences.

The Louisiana Education Scene

Student performance on standardized tests in Louisiana has trailed national averages for decades. In the 2017 8th-grade reading results on the National Assessment of Educational Progress, Louisiana public schools tied for 42nd in the nation and rated significantly higher than only one jurisdiction, the District of Columbia. Only 25 percent of Louisiana 8th graders scored as proficient or above in reading, similar to the 23 percent rate in 2015 but higher than the abysmal 17 percent rate in 1998. NAEP reading scores for Louisiana 4th and 12th graders have been similarly disappointing, as have their math and science scores.

The private-school sector in the Pelican State is relatively large, for three likely reasons: the traditionally low academic performance of the state’s public schools; Louisiana’s French-Catholic heritage, which has given rise to many parochial schools; and the state’s troubled history of racial segregation. In 2011–12, when this story begins, Louisiana had 394 private schools enrolling 112,645 K–12 students, or nearly 16 percent of Louisiana K–12 students, well above the national private-school average of 11 percent. The private-school sector in Louisiana is a diverse blend of religious and secular schools, with Catholic and evangelical Christian schools dominating the scene. Annual tuition rates in 2013 ranged from $2,000 to $19,660, with a school-level average of about $6,000.

In 2005, Hurricane Katrina raged in, devastating the city of New Orleans and environs. The flood damage to more than 300 public schools was so extreme they had to be condemned. Since the storm left many area private schools intact, the Catholic Archdiocese of New Orleans, waiving tuition, initially took in thousands of students whose public schools had been ruined. Several months later, after Hurricane Rita ravaged parts of the Houston area, the federal government established hurricane vouchers for the two storm-damaged regions, temporarily covering the private-school tuitions of educationally displaced children.

In the wake of Katrina, Louisiana lawmakers established two major private-school choice programs. The first was the Elementary and Secondary School Tuition Deduction policy, enacted in 2008. This initiative allows families to deduct on their state income-tax return up to $5,000 per child in private-school educational expenses. The families of more than 106,000 of the 112,000 students attending Louisiana private schools in 2012 claimed the deduction. The second program was the Student Scholarships for Educational Excellence Program, which in 2009 began providing private-school vouchers to 624 low-income students in the parishes of Orleans and nearby Jefferson. This program served as a pilot for the larger, statewide Louisiana Scholarship Program, launched in 2012.

State policymakers also dramatically refashioned the public-school system in New Orleans. Gone were residential attendance zones, the teacher collective-bargaining agreement, and almost all of the public schools the Orleans Parish School Board directly operated. In their place arose a new kind of urban public school system, dubbed the Recovery School District. Overseen by state education officials, the new district was composed almost entirely of public charter schools that would be held accountable for student achievement on state tests. Douglas Harris at Tulane University concluded that this package of market-based reforms—expanded school choice coupled with results-based accountability—substantially improved the test scores of students attending the city’s public schools (see “Good News for New Orleans,” features, Fall 2015).

 

The Louisiana Scholarship Program

The pilot voucher program had 1,950 students enrolled in 2012 when it expanded statewide and became the Louisiana Scholarship Program. Demand for the program was strong from the start: a total of 9,736 students applied that first year, with 5,296 receiving vouchers and 4,944 using them to attend a private school. It is this 2012–13 applicant cohort, a majority of whom lived outside New Orleans, that we evaluated over the course of four years. The program enrolled 6,695 students in 2016–17, a drop of 9 percent from its enrollment peak of 7,362 in 2014–15 (see Figure 1).

Participation in the voucher program is restricted to low-income students in low-performing public schools. To qualify, the income of a child’s family must be at or below 250 percent of the federal poverty line, which in 2016–17 was $60,750 for a family of four. Applicants also must either be entering kindergarten or be attending a public school graded C, D, or F by the state’s test score-driven school accountability system. About one third of the K–12 students in Louisiana, or nearly 250,000, were eligible for the scholarship program in 2012, when 4 percent of the eligible student population applied to the program. Nearly 90 percent of the eligible applicants were African American and over 80 percent were entering grades 1–6 that year.

Applicants win spots in the program by means of a government-run lottery, and the students who did not win the lottery provided the control group for our study. Students in D or F schools receive priority in this lottery, so very few students from C schools have received scholarships. Students with disabilities also receive priority, as they are placed automatically in their private school of choice if a seat is available. The lottery simultaneously awards students with scholarships and placement in a specific private school, drawing from the school preferences listed by parents. Unlike most private-school choice programs, the scholarship award takes place after the school-shopping process, not before it.

The voucher value is limited to 90 percent of the state and local per-pupil funding in a student’s school district or the tuition rate the student’s chosen private school charges, whichever is less. Annual tuition at participating private schools ranges from $2,966 to $8,999, with an average of $5,437, compared to average state and local per-pupil funding of $8,500 in Louisiana’s public schools in 2010–11.

Why might a private school decide not to participate in the Louisiana Scholarship Program? One reason could be that participating schools must submit to regulations regarding financial practices, curriculum, student enrollment, mobility, safety, and achievement. They must provide financial audits to state officials every year and maintain a curriculum that the state Department of Education judges to be of equal or higher quality to the curriculum in public schools. Participating schools cannot select their voucher students and must instead admit them solely via the placement lottery administered by the state.

All participating private schools must administer the state’s accountability tests annually to voucher students in grades 3–8 and again in grade 10. Participating schools with at least 10 voucher students per grade are assigned school-performance ratings based on voucher-student scores on the state test.

As in the test-based accountability model applied to public charter schools in Louisiana, the state can sanction private schools participating in the voucher program. Any of three conditions leads to state sanctions. First, a school is sanctioned if the average rate of gain of voucher students on the state test is so low that the school qualifies for an F grade. Second, a school is sanctioned if less than 25 percent of its voucher students score at or above the state benchmark for proficiency. Finally, participating schools are sanctioned if their annual financial audit reveals improprieties or raises concerns about the school’s future viability.

A private school that receives a sanction is prohibited from enrolling new voucher students until it remedies the condition that led to the penalty. By 2015–16, the final year of data collection for our study, 35 of the 122 private schools in the program had been sanctioned for at least one year.

These regulations on admissions, testing, and curriculum render the Louisiana Scholarship Program one of the most highly regulated among the 58 private-school choice programs in the country. Why put so many restrictions on the private-school choices available to parents?

The history and context of education in Louisiana likely led policymakers to opt for such a highly regulated model. The Pelican State still has many racially stratified schools. Most of the D and F public schools in the state serve a majority low-income student population. Thus, restricting the voucher program to students from low-income families attending failing schools, and requiring the schools to forgo any admissions standards, was expected to prevent the program from worsening school-level racial and income segregation.

The private schools in Louisiana ranged dramatically in tuition rates, which likely serve as a rough proxy for school quality. Officials anticipated that, as is common in voucher initiatives, few of the higher-tuition private schools would elect to participate in the program, since they would have to foot a large part of the cost of educating voucher students. Moreover, the state Elementary and Secondary School Tuition Deduction policy provided the fee-paying customers of private schools with a significant cost rebate that did not require the school to take on any additional state regulations or paperwork, reducing the incentive for schools to join the voucher program and accept its regulatory requirements.

Policymakers expected the lower-quality, under-enrolled private schools in Louisiana to sign up to participate in the voucher program, so they designed a results-based accountability system that would remove schools from the program if they produced low student test scores. A similar regulatory system appeared to be working well for the public charter-school sector in New Orleans. Mandating that private schools administer the state accountability test would allow an apples-to-apples comparison of school performance. It all seemed so sensible at the time.

Private School Participation

Fewer than a third of the private schools operating in Louisiana in 2012 agreed to participate in the Louisiana Scholarship Program. That was the smallest share of schools to participate in a statewide, means-tested private-school choice program among all such programs in the country, and we sought to understand why. Brian Kisida, Evan Rhinesmith, and I implemented a survey of Louisiana private-school leaders that revealed the top factors deterring private schools from joining the voucher program. We found that these leaders feared that more regulations might arise in the future and hamper their independence or threaten their religious identity. They also worried about the integrity of their admissions policies, and they didn’t like the pressure to adhere to the state’s curriculum standards.

Evidence from a survey experiment conducted in other states suggests that the kinds of regulations in place in Louisiana may discourage private schools from participating in choice programs. Corey DeAngelis, Lindsey Burke, and I sent brief email surveys to every private-school leader in Florida, California, and New York, asking if they would be likely to participate in a private-school choice program offering a $6,000 voucher. Survey takers were randomly divided into five groups. Those in the first group were asked about their willingness to participate in the hypothetical program with “no strings attached.” Those in the other four groups were asked about participating under a specific regulatory requirement.

One third of private-school leaders who were told they would face no additional regulations said they were “certain” they would participate in such a voucher program. Among school leaders who were told the hypothetical program would mandate an open-admissions policy, just 14 percent said they would be certain to participate. Of those who were told that they would be required to administer the state accountability test to all voucher students (with the results reported publicly), 24 percent said they would be certain to participate. The remaining two conditions had no significant effect on leaders’ expressed willingness to participate: a requirement that the school administer a norm-referenced test of its choosing; and a mandate that the school accept the voucher as the full cost of educating the child.

What kinds of private schools did participate in the Louisiana Scholarship Program? Compared to non-participants, participating schools were more likely to be Catholic, have lower enrollments, and have student populations that were disproportionately made up of racial or ethnic minorities. The typical participating private school was accustomed to serving socially disadvantaged students and eager to increase its enrollment. Surprisingly, average tuition levels were not a major factor in separating participating schools from non-participating ones in Louisiana.

Students Served

The scholarship program was designed to serve a highly disadvantaged population: kids from low-income families who were attending a public school that produced consistently low test scores.

It is therefore no surprise that Yujie Sude and I found almost no evidence that the voucher program “cream-skimmed” easy-to-educate students. At the application stage, the program attracted a highly disadvantaged population of students. Louisiana public-school students were more likely to apply to the voucher program if they were low income, African American or Hispanic, and lower test performers. Students in the earlier K–12 grades were much more likely to apply than older students, suggesting that parents are more willing to switch their child from a public school to a private one when they are younger and perhaps better able to adapt to their new school environment.

Which students stayed in the program? Several factors were associated with persistent voucher use over three consecutive years. Students with lower initial test scores were more likely to persist than were students with higher scores. Girls and students who enrolled initially in earlier grades were more likely to continue than boys or students who started in later grades. Students in private schools that had lower minority enrollments and that were located a shorter distance from home were also more likely to persist, as were students residing in public-school districts that had lower per-pupil spending.

After all of these factors played out, how select was the voucher-student population three years after application? Characteristics associated with student disadvantage tended to distinguish three-year voucher users from students who never applied to the program. Compared to those who didn’t apply, students who participated and continued for three years were more likely to be female, low income, and African American. The Louisiana Scholarship Program did succeed in expanding school choice to a relatively disadvantaged population of students.

Effects on Racial Segregation

Many commentators claim that parental choice programs worsen segregation. To test that claim, Jonathan Mills and Anna Egalite analyzed the effect that initial voucher users had on the level of racial integration in both the public schools they left and the private schools they joined (see “The Louisiana Scholarship Program,” features, Winter 2014). First, the researchers established a benchmark for racial balance in each school, pegged to the racial composition of the local community. Then they determined whether each student who transferred to a private school in 2012 under the voucher program had an integrating or segregating effect. For the public school the student was leaving, the student was deemed to have had an integrating effect if the race of the student was overrepresented at that school, given the community benchmark. But a transferring-out student had a segregating effect on that school if the student’s race was under-represented there. Conversely, a transfer into a private school better integrated it if the student’s race was under-represented and further segregated it if the student’s race was overrepresented.

The researchers found that 83 percent of initial student transfers under the program had an integrating effect on the racial composition of the public schools they left. In the private schools they joined, about half of the students generated better integration and the other half lessened integration. Even in the 34 Louisiana public-school districts still under a court order to integrate by race, 74 percent of the transfers under the voucher program brought the public schools closer to the goal of racial integration. Thus, the program has operated as a voluntary school desegregation program, even though the racial integration of public schools was not its primary purpose.

Academic Outcomes

The main purpose of the scholarship program was to improve academic outcomes. On that goal, it clearly fell short. Using gold standard experimental methods, Jonathan Mills and I determined that the effects of the program on student scores on the state accountability test tended to be negative, especially in math, as long as four years after initial scholarship use.

For the students who participated in the voucher lotteries and for whom we had both baseline (2011–12) and subsequent test scores, the effects of the scholarship program on achievement varied between negative and neutral (see Figure 2). At baseline, the math test performance of the lottery winners was statistically equivalent to that of the control-group students (that is, those who did not win a voucher placement). But effects on the math performance of the scholarship students were negative and large in the first year of the program, when students were adjusting to their new schools and, likewise, the schools were adjusting to them. The voucher students recovered some of their lost ground on the state math test the second year and delivered math scores that were statistically similar to the control group in year three, when the state switched to a new accountability test and neither public nor private schools were held accountable for the results. In year four, when test-based accountability was reestablished statewide, the voucher program again produced negative impacts on math scores; these were moderately large.

The reading test-score impacts of the program followed a trend similar to the math scores but with consistently smaller effects, most of which were not significantly different from zero. For those students for whom we had baseline test scores, the reading impacts of the program were negative and moderately large the first year, and much smaller and not statistically significant in the second, third, and fourth years.

The achievement impacts of the scholarship program were not uniform for all groups of students. In the fourth year, the negative effects of the program on test scores were less than half as large for African American students as for non-African American students. Also, voucher students in grades 4 and 8 took the main state accountability test, while those in grades 3, 5, 6, and 7 took a version of the test that was scored on a different scale. For the latter group, the negative test-score effects of the program were less than half as large as for those who took the main state test.

The effects of the voucher program on student test scores also varied based on key features of the school the student most preferred. Matthew Lee, Jonathan Mills, and I determined that the achievement effects of the program were much less negative, and in some rare cases even positive, for subgroups of students whose preferred private schools ranked in the top one third of participating schools in regard to higher tuition, higher total K–12 enrollment, or a greater number of instructional hours for students.

Educational Attainment

Educational attainment—how much schooling an individual ultimately completes—greatly influences his or her later life outcomes, and researchers have increasingly turned to studying the impacts of education programs on student rates of high-school graduation and college enrollment, persistence, and completion. Too few students in our experimental sample were old enough to have graduated from college for us to examine that important outcome. However, more than 1,000 students who participated in the voucher program’s lotteries completed high school and could have enrolled in college. Did the negative test-score effects of the program decrease their rates of college going?

Heidi Holmes Erickson, Jonathan Mills, and I found that the Louisiana Scholarship Program had no effect on college entrance for students. After high school, voucher users enrolled in a two-year or four-year college at a rate of 60.0 percent, which was equivalent to the 59.5 percent enrollment rate of the control group.

Mixed Outcomes

Opponents of private-school choice may use the results of our study to argue that “free market” approaches to education have failed. However, the initiative that we evaluated was clearly not a pure free-market education reform. Free markets generally require: 1) that new suppliers can easily enter the market; 2) that prices can vary across providers and different versions of a service; and 3) that consumers are the main judge of the quality of the service. None of those conditions held for the Louisiana Scholarship Program. New suppliers were virtually prohibited from emerging, as private schools had to operate for two years with only fee-paying customers before they were allowed to participate. The price of the service was largely fixed by the voucher formula, and parents were prohibited from paying more for a higher-quality education, at least within the program. A government-run lottery placed each voucher student in a school, and government overseers determined which schools could and could not continue to enroll new voucher students. And while parents were able to communicate their preferences about school placement, it was the state, not the parents, who made the final selection.

School choice programs often seek to reduce racial segregation in schools. The Louisiana Scholarship Program succeeded in doing so. Everything is good at something. The statewide launch of the program in 2012 better integrated Louisiana’s public schools, as most voucher-program participants left public schools in which their own race was overrepresented. Persistent scholarship users were more likely to be African American and to have had lower initial test scores compared to those who didn’t apply to the program. Furthermore, the program saves the state money, because the voucher maximum of less than $6,000 is only about two thirds of the combined state and local per-pupil funding in Louisiana’s public schools.

The regulatory framework of the program rested on several ideas: that private schools accepting government vouchers are comparable to public charter schools; that low-performing schools will improve with government incentives to do so or be kicked out of the program; and that student test scores are the best single measure of school performance. All of these notions are subject to question.

Private schools differ from public charter schools in two critical ways that affect how they might be effectively regulated. First, private schools have a choice in whether to participate in voucher programs, accepting government funding and the regulations that come with it. In Louisiana, where tax policies indirectly subsidize families who pay tuition out of pocket, private schools have lots of paying customers and thus greater latitude to turn down the chance to serve disadvantaged students on state scholarships if the offer is not attractive to them. Public charter schools have no such choice, as the government is their only reliable funding source.

There is little evidence that education overseers can help schools improve by using regulatory carrots and sticks. Student test-score performance is a product of many complex factors, including family background (see “How Family Background Influences Student Achievement,” features, Spring 2016), school culture, teacher quality, child nutrition, test preparation, and curricular alignment with the test. Actual levels of student learning are in there somewhere, but it is a crowded room of factors that most private schools with established educational programs and cultures cannot change over the course of a few years, even if a regulatory system incentivizes them to do so. The evaluation literature on aggressive efforts at systematic school turnaround reinforces this point: school improvement is a slow, difficult, evolutionary process.

Removing private schools from the program if they fail to boost student test scores might seem like an automatic method for improving outcomes by “chopping off the lower tail of the performance distribution,” but where do the affected students go instead? If the existing private schools already are full and new ones are prohibited from serving voucher students during a long probationary period, the state sanctioning system merely serves to limit the educational options of parents and students.

Finally, student scores and annual gains on the state-mandated accountability test are only one measure of school performance. They capture only a portion of what we would consider actual student learning, and advancing student learning is only one of many goals of publicly financed education. We also charge schools with instilling civic values in students and developing positive character traits of persistence, conscientiousness, and respect for others and the norms of society. When Thomas Stewart and I polled a room of 40 parents participating in the D.C. Opportunity Scholarship Program on how they assess the academic progress of their child in school, none of them selected “standardized test scores” as the metric. Instead, they said they used “student motivation to learn,” “student grades,” “positive student attitudes toward school,” and “positive student behaviors toward schools” as measures of whether their child was succeeding. The private schools participating in the Louisiana Scholarship Program may be delivering these more nuanced but vital outcomes to their voucher students, but determining that would require a more comprehensive assessment of school performance.

In order to accomplish anything, schools must keep their students safe. Recent surveys suggest that school safety now rivals academic concerns as the main reason parents seek private-school choice. Public regulators would be wise to work indicators of school safety and an orderly school environment into their measures of whether a school is performing at a satisfactory level.

The Louisiana Scholarship Program did not succeed in raising student scores on the state accountability test. In fact, it had a clear negative effect on math scores and a possible, though less severe, negative effect on reading scores. Nothing is good at everything. Our evidence suggests that about half of the negative test-score effect is likely due to public schools more carefully aligning their curriculum to the tested material and systematically implementing techniques for preparing students for the test. Still, it is clear that the program did not positively affect scores on the state test. Insofar as state accountability test scores are used as the litmus test for effectiveness, the Louisiana voucher program did not let the good times roll down in the Bayou.

Patrick J. Wolf is a distinguished professor and 21st Century Endowed Chair in School Choice in the Department of Education Reform at the University of Arkansas College of Education and Health Professions.

The post What Happened in the Bayou? appeared first on Education Next.

The Miami Herald recently re-published an op-ed originally appearing in The Conversation. In the essay, education professors Christopher Lubienski and Joel Malin wonder why states keep adopting and expanding private school choice programs when such initiatives, in their view, have failed. Their answer essentially is that school choice researchers and advocates have duped policymakers by “moving the goalposts,” emphasizing the positive effects of choice on outcomes besides the test scores that advocates initially stressed to sell the programs. Upon close examination, the Lubienski and Malin argument argument collapses like a house of cards.

(Corey DeAngelis has already published an evidence-rich response to Lubienski and Malin at The Washington Examiner that covers some of the flaws I discuss here.)

The Lubienski and Malin argument has four main components. First, programs must be evaluated only based on their effects on the outcomes promised by their advocates. Second, private school choice programs were sold to policymakers solely based on the promise that they would boost the test scores of participants. Third, recent evaluations of private school choice programs have demonstrated that all of the programs have permanent negative effects on the test scores of participants. And finally, fourth, “teams from the University of Arkansas” (meaning me, Jay Greene, and our fantastic doctoral students) only started focusing the attention of the academic and policymaking community on the non-test-score effects of school choice, including its effects on educational attainment and civic values, after the test-score outcomes from choice evaluations turned negative. All four of those claims are false.

Lubienski and Malin argue that programs only can be judged successful if they are effective at producing the specific outcomes promised by their supporters. Evaluators are unfairly “moving the goalposts” if they instead discover that a program has positive effects that weren’t originally predicted. On its face, their claim is ridiculous. Why would it be preferable to know less about a program’s effects than to know more? Why would anyone think that a narrowly-focused program evaluation is superior to a comprehensive one?

Certainly program evaluators themselves do not ascribe to the Lubienski and Malin position. For example, the authors of a prominent textbook on program evaluation state “Although input from stakeholders is critical, the evaluator should not depend solely on their perspective to identify the issues the evaluation will address.” They, instead, argue that program evaluations should be expansive, including considerations of all relevant outcomes that might be influenced by the initiatives. “[W]e must caution against an overly narrow interpretation of what information is useful,” the experts write. Research questions should come from multiple sources including evaluator expertise, program theory, and statutory mandates, in addition to “stakeholder claims.” As someone who teaches a graduate course in program evaluation and actually conducts comprehensive evaluations of private school choice programs regularly, I’m on the side of the evaluation experts: more knowledge is better than less.

But let’s assume, for the sake of argument, that Lubienski and Malin are correct and choice programs should be evaluated solely based on the outcomes promised by their supporters. Their next claim is that voucher advocates sold choice to the public solely based on predictions that such programs would boost participant test scores. A wealth of evidence contradicts that claim.

Milton Friedman, the originator of the modern idea of private school choice, justified choice mainly in terms of parent empowerment, the positive effects of competition on the performance of affected public schools, and civic values. Throughout the seminal essay that launched school vouchers, Friedman discusses the civic outcomes of government-run and privately run schools. After pointing out that government-run schools of the 1950s are highly stratified by race because they draw students based on segregated neighborhoods, and private schools of the 1950s are highly stratified by income because low-income families can’t afford the tuition, he writes that “The widening of the range of choice under a private system would operate to reduce both kinds of stratification.”

John Chubb and Terry Moe did focus on the positive effects of private schooling on participant test scores in their book Politics, Markets, and America’s Schools. Still, much of the debate over the desirability of private school choice always has centered on the questions of what effects it has on school-level racial segregation, the students “left behind” in public schools, and student non-cognitive outcomes such as civic values. Lubienski himself, in a book co-authored with Sarah Theule Lubienski, states “our analyses and the analyses of others indicate that [school choice] efforts can create a…more socially segregated system of schooling.” In that same book, the Lubienskis focus attention on the question of the competitive effects of school choice, denigrating the more than 20 rigorous studies of the question that show positive effects and applauding various critiques of this deep and vital research base. Since they study more than just the test-score effects of school choice, why can’t other people?

Choice skeptic Henry Levin of Teachers College has long argued that private school choice programs should be evaluated based on their effects on four outcomes: expanding options for parents, productive efficiency (including effects on educational attainment as well as test scores), racial integration, and civic values (see here & here). All six of my longitudinal evaluations of private school choice programs have tried to cover all four of the outcomes recommended by Levin, apparently to the chagrin of Lubienski and Malin. I didn’t move the goalposts in 2019. If anyone did, it was Henry Levin, way back in 1998.

So, we have determined that it flies in the face of professional standards to limit school choice evaluations only to their effect on participant test scores, and that questions of school integration, the competitive effects of choice on nonparticipants, and the non-achievement effects of choice on participants long have been central to debates over school choice. What about the Lubienski and Malin claim that all of the recent choice evaluations have found that choosers suffer initial achievement losses that they never make up? That statement is a gross exaggeration. Lubienski is notorious for cherry-picking only the private school choice results that confirm his ideological bias. That bias is on full display in this latest essay, where he and his co-author refer to the early set of 10 experimental studies reporting largely positive test-score effects of school choice as “a small set of studies” while somehow the recent set of only four studies, three of which show some enduring negative achievement effects, is a larger and more convincing evidence base.

A more accurate characterization of findings from recent studies is that some private school choice initiatives have persistent negative effects on student achievement, mainly in math. Only David Figlio and Krzysztof Karbownik’s quasi-experimental evaluation of the Ohio EdChoice program found statistically significant negative achievement effects in reading in the final year of the evaluation that were robust to different analytic techniques. The negative reading impacts of the Indiana Scholarship Program were sensitive to changes in samples and student matching techniques. The negative reading effects of the Louisiana Scholarship Program similarly were inconsistent across samples and statistical models, a vital detail that Lubienski and Malin omit from their essay. The Ohio, Indiana, and Louisiana evaluations all reported negative effects on math achievement in the final year of the evaluations that were robust. Importantly, the recently concluded evaluation of the D.C. Opportunity Scholarship Program found that initial negative effects of that program on student math scores that were trumpeted in the Lubienski and Malin essay disappeared completely by the third and final year of the evaluation. In direct contradiction to a Lubienski and Malin claim, students in one of the recent private school choice evaluations completely made up the initial achievement ground they lost by switching to a private school.

The final claim by Lubienski and Malin is the easiest to debunk. They assert that the University of Arkansas research team only focused attention on the civic values and attainment effects of private school choice after the test-score effects started coming up negative (with the Figlio & Karbownik study in 2016). Jay Greene and I are political scientists by training. From the very beginning of our long careers evaluating school choice programs, we have focused on the impacts of choice on civic outcomes. Jay started publishing evaluations of the effect of private schooling on civic values way back in 1998, 18 years before the first school choice evaluation reporting negative test score effects (see here, here, and here). My first school choice study focused on its effects on student civic values, including political tolerance, voluntarism, and patriotism. I followed that up that 2001 publication with a co-edited book on the topic, way back in 2004, and a systematic review of the many studies of school choice and civic values in 2007.

My research team was the first to report the effects of a private school choice program on high school graduation rates, not because I wanted to “move the goalposts” but because the U.S. Congress directed in law that educational attainment be an outcome evaluated in our study. That’s right, policymakers themselves, in their wisdom, have demanded that non-test-score outcomes be the subject of private school choice evaluations. As an evaluator, I didn’t move the goalposts. I just kicked the ball through them.

Patrick J. Wolf is a distinguished professor and 21st Century Chair in School Choice in the Department of Education Reform at the University of Arkansas College of Education and Health Professions.

A version of this article orginally was published at RedefineEd.

The post On Goalpost Moving appeared first on Education Next.

We used surveys to randomly assign different regulations commonly found in school choice programs to 4,825 private school leaders in the states of California and New York and asked them whether or not they would participate in a new private school choice program during the following school year. Here’s what we found.

Relative to no additional regulations, open-enrollment mandates – preventing private schools from having specific admissions policies – reduced the likelihood that private school leaders were certain to participate in a hypothetical choice program by 60 percent. State standardized testing requirements reduced the likelihood that private school leaders were certain to participate by 29 percent. However, we found no evidence to suggest that mandating private schools to accept the voucher as full payment or requiring them to administer a nationally norm-referenced test of their own choosing affected the willingness of private school leaders to participate.

These overall results largely mirror what we found in our previous experiment in Florida. Statistically significant overall effects can be found in the figures below.

Our overall results suggest additional government regulations, beyond those that all private schools face, largely reduce the number of options available to families. But it is possible that regulations were more likely to deter lower-quality private schools from participating in the programs. If so, regulations could have increased the quality level of the private schools participating in the hypothetical voucher programs, on average.

However, using four different measures of school quality – Google review scores, GreatSchools review scores, tuition levels, and enrollment trends – we did not find any statistically significant evidence to suggest that any of the regulations improved the average quality of participating private schools by disproportionately deterring lower-quality schools from participating.

In fact, the only marginally significant result detected indicated that state standardized testing requirements were more likely to deter private schools with higher Google review scores. Specifically, one model found that a one-point (on a five-point rating scale) increase in Google review scores was associated with a 14.5 percentage point larger negative effect of the state testing mandate on anticipated program participation for higher quality schools compared to lower-quality ones. As we’ve hypothesized before, regulations could actually reduce the average quality levels of participating private schools. Lower-quality private schools may be more likely to participate in voucher programs, regardless of the additional regulations, because they are more likely to be in great need of financial resources and enrollment.

Our study is not without limitations. The survey response rate was only 8.24 percent, so we cannot be sure that the results are representative of all private schools in California and New York. In addition, because random assignment of regulations to private schools participating in actual voucher programs is not feasible, we had to rely on private school leaders’ reported expectations to participate in hypothetical voucher programs.

Of course, public officials pushing for more voucher program regulations may only wish to help children get better educations. It might be possible for regulations to help achieve that admirable goal. But our study finds that voucher program regulations could have substantial negative effects on the amount of quality options available to the children that need them the most.

Corey A. DeAngelis is an education policy analyst at the Cato Institute. He received his Ph.D. in education policy from the University of Arkansas. Lindsey M. Burke researches and writes on federal and state education issues as the Will Skillman fellow in education policy at The Heritage Foundation. Patrick J. Wolf is a Distinguished Professor of Education Policy and 21st Century Endowed Chair in School Choice in the Department of Education Reform at the University of Arkansas.

The post Some Regulations Deter Private Schools from Participating in Voucher Programs appeared first on Education Next.

Schools are expected to help shape the character skills needed to live a good life and contribute to society. In theory, programs that help families send their children to a private school could improve character skills through exposure to peers and school cultures that discourage risky behaviors. Do private school choice programs actually help achieve this goal? Our just-released evaluation of the longest-running modern school voucher program in the United States—the Milwaukee Parental Choice Program (MPCP)—suggests that they do.

The study updates and extends our previous work on the same program with new data, including information on students’ criminal activity and involvement in paternity disputes as young adults. We gathered these data in fall 2018, when the 2,178 students we have been following in a longitudinal evaluation of the MPCP were roughly 25 to 28 years old. Half of these students were using vouchers to attend a private school in 8th or 9th grade in 2006; the rest are a carefully matched sample of students who attended Milwaukee Public Schools. Comparing the outcomes of these two groups reveals how exposure to the MPCP at that time affected students’ later outcomes, regardless of how long students initially using vouchers remained enrolled in private schools.

Our results confirm that the program reduces the incidence of both criminal activity and paternity disputes for young adults. Specifically, we find that participating in the program in 8^th or 9^th grade is associated with a statistically significant reduction of around 53 percent in drug-related convictions, 86 percent in property damage convictions, and 38 percent in paternity disputes.

Not surprisingly, the program’s effects on some outcomes differ for males and females. Males participating in the MPCP experienced a 53 percent reduction in drug-related offenses and an 87 percent reduction in property damage offenses when compared to their public-school peers. Females exposed to the program, on the other hand, experienced little or no reduction in convictions for those crimes. Because young males are at greater risk than young females to commit such crimes, the MPCP had greater scope to affect their behavior. The program’s effects on paternity suits were more similar by gender, with a 42 percent reduction for males and a 34 percent reduction for females.

The character-building effects of the school voucher program also varied with students’ initial math ability. For two outcomes—thefts and traffic offenses—exposure to the MPCP did more to reduce negative outcomes for students with high initial math scores than for students with lower scores. In contrast, students with lower initial math ability experienced a larger reduction in paternity suits due to their participation in the MPCP than students with higher initial math ability.

Of course, much more research on this topic is needed. While two random assignment studies find that winning a charter school lottery reduces crime for male students, no random assignment studies link private school choice to adult crime or paternity disputes. Even so, our new results suggest that the Milwaukee voucher program improves low-income urban students’ long-term prospects for life success by helping them avoid both prison and unplanned pregnancies.

Corey A. DeAngelis is an education policy analyst at the Cato Institute. He received his Ph.D. in education policy from the University of Arkansas. Patrick J. Wolf is a Distinguished Professor of Education Policy and 21st Century Endowed Chair in School Choice in the Department of Education Reform at the University of Arkansas.

The post Private School Choice Helps Students Avoid Prison and Unplanned Pregnancies appeared first on Education Next.

Yong Zhao, “Side Effects in Education: Winners and Losers in School Voucher Programs.” Phi Delta Kappan, 100(5), pp. 63-66.

Checked by Patrick J. Wolf

University of Kansas Professor Yong Zhao has jumped into the school choice debate with his recent essay in the Phi Delta Kappan. Welcome, Dr. Zhao.

Zhao makes several important claims about private school choice. He says that researchers who are proponents of school choice exaggerate the positive tilt of the findings. He claims that the debate over the achievement effects of school choice has largely ignored substantial variation in those effects across student subgroups. Finally, he concludes that we know almost nothing about the effects that school choice has on civic outcomes but should expect those effects to be negative.

School choice is new academic territory for Zhao, a Distinguished Professor who specializes in education technology and virtual learning. Permit me to offer him some friendly advice from someone who has studied the topic for over 20 years.

First, don’t call people names. Zhao labels school choice researchers as either “proponents” (a.k.a., “advocates”) or “independent researchers.” He doesn’t tell us how he arrived at those determinations, but two possibilities immediately come to mind. A scholar might be a school choice “proponent” if their intention is to promote choice and an “independent researcher” if their intention is to arrive at the truth about choice. To judge accurately who is which, Zhao would need to possess the capacity to look inside of the hearts of his fellow human beings and therefore observe their intentions. He would have to possess the God-like ability of comedian Woody Allen’s character in the movie Annie Hall, who said:

I was thrown out of NYU my freshman year for cheating on my metaphysics final, you know. I looked within the soul of the boy sitting next to me.

Human beings in general, and social scientists in particular, are terrible at judging other people’s intentions. We are better off classifying school choice researchers based on the rigor of their methods, not a feigned ability to know the deepest desires of their hearts.

Zhao might, instead, be classifying school choice scholars as “proponents” or “independent researchers” based on the nature of the findings that we report. It appears that, in Zhao’s view, a school choice “proponent” is any scholar who reports that a school choice program has positive effects on student achievement. In contrast, an “independent researcher” is any analyst who agrees with Zhao that school choice is bad. Zhao’s claim that school choice “proponents” report “more significant positive effects than independent researchers” (p. 64) thus reduces to a tautology. Proponents report more positive effects because reporting more positive effects makes them proponents. A better approach for Zhao and others would be to forgo the arbitrary and unscientific labels he foists on the researchers and focus, instead, on the quality and findings of the research itself.

Second, get your facts right. Zhao criticizes the body of research on the achievement effects of school choice for focusing “only on the average effect of school choice on the students who participate in it.” Really? Few education interventions have been evaluated with greater attention to their possibly varied effects on different subgroups than has private school choice. The first study published about a school voucher program in the U.S. examined differences in the achievement effects of the program across student subgroups. Almost every one of more than two dozen major private school choice evaluations in the subsequent 21 years has examined the extent to which voucher achievement effects vary by student characteristics. The focus of the three-city evaluation that informed William Howell and Paul Peterson’s seminal book The Education Gap: Vouchers and Urban Schools (see “Vouchers in New York, Dayton, and D.C.,” research, Summer 2001) was the finding that African American students experienced larger and more consistent achievement effects from school choice than students of other ethnicities. One can easily find additional peer-reviewed publications of the possible heterogeneous effects of vouchers in New York City here, here, here, and here; in Washington, DC here, and here; in Milwaukee, here and here; in Indiana here and another one under review; and in Louisiana here and here. Contrary to Zhao’s claim, examining possible variation in voucher achievement effects has been an obsession of school choice researchers, not a blind spot. Who does and does not benefit academically from school vouchers has been central to the debate.

There is a scientific method for determining if the effects of an intervention vary across subgroups that Zhao seems not to use. Zhao claims that voucher programs have heterogeneous achievement effects if the impacts are positive and statistically significant for some subgroups but not statistically different from 0 (i.e. null) for other subgroups. He makes the same claim regarding voucher effects that are null for some subgroups but statistically significantly negative for other subgroups. Zhao is making much of nominal differences in the pattern of voucher achievement effects that may be due to statistical noise endemic to the small subgroup samples in most of these studies. A program has a single, general effect on participants unless its effects on different subgroups are, themselves, significantly different from each other based on statistical tests. The only statistically valid heterogeneous pattern so far uncovered in a study of the achievement effects of school vouchers is the positive and statistically significant effect of the private school choice program in New York City on African American students, which itself was significantly different from the program’s null effect on non-African American students. School choice researchers have uncovered few scientifically valid differences in the impacts of school choice on achievement for different subgroups of students, though not for lack of trying.

Third, know the subject of which you speak. Zhao concludes his essay with the claim that, “Little, if any, empirical evidence has been collected concerning other equally important outcomes of schooling, such as preparing students for civic engagement and betterment of a shared society.” Actually, there is a deep and broad research literature on the mostly positive effects of school choice in general and private schooling in particular on civic values such as political tolerance, volunteering in one’s community, political knowledge, political engagement, social capital, and patriotism. I co-edited a book, Educating Citizens, on the topic in 2005. Rigorous studies demonstrating the positive private school effects on enhancing civic values can be found here, here, here, here, here, here, here, here, here, okay you get the picture. So many studies of this relationship exist—when Zhao claims there are “little, if any”—that comprehensive peer-reviewed summaries of the evidence have been published in these pages (see “Civics Exam,” research, Summer 2007) and elsewhere.

The evidence supporting the private school advantage in promoting civic values is so compelling that one would think it would be a settled matter by now. So long as doubters like Professor Zhao continue to ignore the wealth of published evidence to the contrary, empirical research on school choice and civic values will continue. Many commentators, however, insist on trusting their ideological preferences on the matter of school choice and civic values instead of their lying eyes. I would say that “proponents” of a public school advantage in promoting civic values present a different picture “than independent researchers”, but I don’t like to call people names.

Dr. Patrick J. Wolf is Professor and 21st Century Chair in School Choice in the Department of Education Reform at the University of Arkansas College of Education and Health Professions.

The post Article On School Choice Ignores Key Evidence appeared first on Education Next.

What happened in Louisiana? The first experimental evaluations in the world to find negative effects of a private school choice program on student achievement were of the Louisiana Scholarship Program. Education policy scholars have been debating what went wrong in the bayou for over three years now. Some researchers have argued that Louisiana’s onerous regulations deterred the best private schools from participating in the program. Others instead have claimed that the program would have done even worse without the quality-enhancing regulations in place. Of course, both theories are plausible. But which theory is supported by evidence?

Our just-released study  is the first experimental evaluation of the effects of various regulations on the willingness of private school leaders to participate in voucher programs.

We randomly assigned a hypothetical voucher program participation offer to 3,080 private school leaders in Florida in 2018. Specifically, we randomly assigned one of three different regulations – or no additional government regulation at all – to each of these private school leaders and asked them whether they would participate in the voucher program the next year. Because regulations increase the costs of program participation, we expected each regulation to reduce the likelihood that private school leaders would be certain to participate the following year.

Relative to no regulations, our most conservative statistical models find that open-enrollment mandates reduce the likelihood that private schools are certain to participate in a choice program by about 17 percentage points, or 70 percent. State standardized testing requirements reduce the likelihood that private schools are certain to participate by 11 percentage points, or 44 percent. We find no evidence to suggest that the prohibition of copayment affects anticipated program participation overall. Our main results suggest that private school leaders are most concerned about maintaining their specialized missions.

It appears that regulations tend to reduce private school participation in voucher programs overall. But that finding alone doesn’t explain why the Louisiana experiment found negative effects on student test scores. Although the Louisiana program includes all three regulations in our study, those requirements could have prevented a lot of low-quality private schools from participating in the program. But do regulations increase average quality levels of participating private schools? Or are they more likely to deter higher-quality private schools from participating in school choice programs?

Because we had access to two proxies for private school quality – tuition levels and enrollment trends – our analysis is able to examine this important question. The clearest result is that higher-quality schools – as measured by tuition – are more likely to be deterred by the regulation that mandates that all schools take the voucher amount as full payment. This result is intuitive, as it is much more costly for a school with tuition of $20,000 to take a $6,000 voucher as full-payment than for a school with tuition of $10,000 to do so.

In addition, our model with all controls finds that a $1,000 increase in tuition is associated with a 1.4 percentage point larger negative effect of a state standardized testing mandate on intended program participation, and a 10-percentage point increase in enrollment growth from 2014 to 2016 is associated with a 2-percentage point larger negative effect of the open-enrollment regulation on intended program participation.

Importantly, because random assignment leads to equivalence in expectation across treatment and control groups on both measurable and unmeasurable factors, each of our estimates of the impact of regulatory requirements on the expressed willingness of private school principals to participate in a private school choice program is causal.

Of course, one study certainly does not settle the voucher regulation debate. But our results suggest that costly regulations tend to reduce the quantity and quality of private schools that elect to participate in school choice programs. In other words, estimates from existing private school choice program evaluations could be lower-bounds of the true effects of private schooling in general. And decreasing voucher regulations could improve the efficacy of these types of programs.

— Corey A. DeAngelis, Lindsey Burke and Patrick J. Wolf

Corey A. DeAngelis is an education policy analyst at the Cato Institute’s Center for Educational Freedom. Lindsey Burke is the director of the Heritage Foundation’s Center for Education Policy and Will Skillman Fellow in Education. Patrick J. Wolf is a Distinguished Professor of Education Policy and 21st Century Endowed Chair in School Choice in the Department of Education Reform at the University of Arkansas.

The post Do Voucher Regulations Reduce Anticipated Voucher Program Participation and School Quality? appeared first on Education Next.

CHECKED: Robert C. Pianta and Arya Ansari, “Does Attendance in Private Schools Predict Student Outcomes at Age 15? Evidence from a Longitudinal Study,” Educational Researcher, Online First.

Checked by Patrick J. Wolf

“New Study Finds Low-Income Students Do Not Benefit from Private Schooling: Findings refute administration’s call for $1 billion for private school vouchers.” Such was the headline of a press release issued by the University of Virginia’s Curry School of Education. Not to be outdone, perennial school choice critic Valerie Strauss of the Washington Post announced, “No, Private Schools Aren’t Better at Educating Kids than Public Schools: Why this study matters.”

Of course, headlines about studies sometimes exaggerate the findings to draw attention to researchers and their research. In the text of the release, however, lead author Robert C. Pianta, Dean of the Curry School, states that the “assumption that private schools are more effective…is demonstrably ineffective and potentially harmful.”

So, has this new study, forthcoming in the prestigious journal Educational Researcher, proven that private-school vouchers harm children? Hardly. It isn’t even a study of school vouchers. It isn’t designed to determine what caused the student outcomes it examines. Its findings are inconclusive, not negative. And, finally, it doesn’t have a large enough sample to prove much of anything.

The study grew out of a larger project designed to measure an extensive set of school factors and find out which ones correlate with student outcomes. A comparatively small sample (more on that later) of 1,067 children who were newborns in 1991 were followed until age 15 in the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development. Participants were recruited from 10 cities across the country, none of which was home to a private-school voucher program during any of the 15 years of data collection. The research team measured students’ achievement, psychological well-being, and expectations for reaching various educational attainment goals, as well as their school type and family background. The measures of psychological well-being, including externalizing or internalizing behavior, social skills, risky behaviors, victimization, and future outlook, are especially important and distinctive features of this study. Data collection took place in four waves.

The authors, Pianta and his colleague Arya Ansari, analyzed the data observationally, meaning they looked for correlations between their variables of interest—whether a student had attended a private school between kindergarten and 9th grade and, if so, for how many years—and student outcomes. They report that the simple correlations between private schooling and desirable student outcomes are positive and statistically significant for every outcome measure. When variables are included in statistical models to control for student and family background characteristics, however, they find that all of the correlations between private schooling and student outcomes become statistically insignificant. It is from this “no effects of private schooling” finding that Pianta concludes that private schooling through vouchers harms children.

It bears repeating that the Pianta-Ansari study is not of a school-voucher program. Their unrepresentative sample of the private schools in 10 cities does not include a single private school serving a single student with a school voucher. The authors seem to admit to limitations in their private-school sample when they state that a “second consideration in evaluating private-school effects is the exceptionally wide variation in private schooling across the United States.” They can’t generalize their findings to all private schools in the country, and if their findings don’t necessarily apply to all private schools, then they almost certainly don’t apply to the distinctive set of private schools that educate students in voucher programs. The authors disregard their own concerns, however, and irresponsibly generalize the findings from private schools not participating in voucher programs in the 1990s and early 2000s to private schools participating in voucher programs now and in the future.

A second limitation of this study is its non-experimental research design. I have also performed a few non-experimental studies of school vouchers but only when an experiment proved to be infeasible. The authors acknowledge that experimental studies using random assignment of students to treatment and control groups “to support causal inferences regarding the impacts of private schooling are among the strongest scientific examinations of such effects.” They justify their use of a weak non-experimental design by claiming that prior school-voucher studies draw similar conclusions, whether the studies are experimental or non-experimental. That claim is false.

In a chapter in the forthcoming book School Choice at the Crossroads, I present the results of a systematic review of the existing research on school vouchers, which demonstrates that the academic effects of vouchers from experimental studies are more positive than those from non-experimental studies such as this one. In a reanalysis of the data from the first evaluation of the D.C. school voucher program, Kaitlin Anderson and I show that the results produced by non-experimental analyses are less positive than the true causal impacts revealed experimentally. If rigorous experimental findings point in one direction while less-rigorous observational findings point in the other direction, we should believe the experiments. The authors of the Curry School study instead privilege their non-experimental results regarding “non-vouchers” over the stronger and different experimental findings.

Third, there is no logic to the claim that non-significant findings from a study suggest that an education intervention harms children. No effect means no effect, not a negative effect.

Finally, the main findings of the study were essentially preordained given important features of its sample and analysis. Not only is the study sample unrepresentative of the private-school population nationally, but it is also different in ways that bias the findings against private schools. Only 45 percent of the study targets actually participated, for example. Children of non-English-speaking mothers, a large subpopulation in many school-voucher programs, were excluded from the sample. Also absent were cities famous for their large and reputable private-school sectors, such as New York, Indianapolis, and Minneapolis-St. Paul. What the authors found is that, more than a decade ago, private schools not known for being particularly effective may not have been particularly effective.

The researchers analyze outcomes from only 1,097 students, which they describe as a relatively large sample but is small in comparison to other school-voucher evaluations currently underway in D.C., Louisiana, and Indiana. The authors report a lot of missing data for these students on the “type of school attended” variable, which is central to their study. Whether or not a student attended a private school was unknown for between 7 and 16 percent of the students annually, depending on grade, leaving only 90 percent of the students with even six years of certain data regarding private schooling, out of a possible 10 years. The authors then use the least-scientific method available to account for these missing data, essentially assuming that a student attended a public school for any year in which their school type was unknown, and thus introduce extensive measurement error into their key variable of interest. This measurement error increases the likelihood of non-significant results, especially when analysts introduce control variables with less measurement error into their statistical models, as the authors do here.

The authors claim their study had the statistical power to detect private-school effects as small as 10 percent of a standard deviation overall and 20 percent for subgroups. Taken at face value, this implies that they would fail to detect school-voucher effects of the size typically found in actual voucher studies, which average 7 to 12 percent of a standard deviation. Moreover, their own results disprove their claim about the statistical power of their study, as a handful of private-school effects larger than 10 percent of a standard deviation are not statistically significant in their article’s crucial Table 4. An extensive set of study limitations converged to make the non-significant findings a virtual certitude, regardless of how effective the private schools actually were.

Pianta and Ansari conclude: “In sum, we find no evidence for policies that would support widespread enrollment in private schools, as a group, as a solution for achievement gaps associated with income or race.” Their lack of significant findings was obtained from an outdated, non-experimental, underpowered, sample-of-convenience analysis of places and people that were not participating in actual private-school voucher programs. They add the requisite, “findings should be interpreted with caution.” Readers would be wise to follow the sage advice that the authors themselves ignored.

Patrick J. Wolf is professor in the department of education reform at the University of Arkansas.

The post No, One Limited Study Does Not Prove School Vouchers Don’t Work appeared first on Education Next.