“Everything works somewhere; nothing works everywhere,” writes Dylan Wiliam in his book Creating the Schools Our Children Need. To that I would add, everything works at something; nothing works at everything.
Together, the two maxims describe what my research team found when we evaluated the Louisiana Scholarship Program, a statewide school-voucher initiative: the program satisfied some of its goals but fell well short of others, including that of raising student scores on state tests. What follows is a cautionary tale about good intentions and seemingly reasonable decisions resulting in unintended consequences.
The Louisiana Education Scene
Student performance on standardized tests in Louisiana has trailed national averages for decades. In the 2017 8th-grade reading results on the National Assessment of Educational Progress, Louisiana public schools tied for 42nd in the nation and rated significantly higher than only one jurisdiction, the District of Columbia. Only 25 percent of Louisiana 8th graders scored as proficient or above in reading, similar to the 23 percent rate in 2015 but higher than the abysmal 17 percent rate in 1998. NAEP reading scores for Louisiana 4th and 12th graders have been similarly disappointing, as have their math and science scores.
The private-school sector in the Pelican State is relatively large, for three likely reasons: the traditionally low academic performance of the state’s public schools; Louisiana’s French-Catholic heritage, which has given rise to many parochial schools; and the state’s troubled history of racial segregation. In 2011–12, when this story begins, Louisiana had 394 private schools enrolling 112,645 K–12 students, or nearly 16 percent of Louisiana K–12 students, well above the national private-school average of 11 percent. The private-school sector in Louisiana is a diverse blend of religious and secular schools, with Catholic and evangelical Christian schools dominating the scene. Annual tuition rates in 2013 ranged from $2,000 to $19,660, with a school-level average of about $6,000.
In 2005, Hurricane Katrina raged in, devastating the city of New Orleans and environs. The flood damage to more than 300 public schools was so extreme they had to be condemned. Since the storm left many area private schools intact, the Catholic Archdiocese of New Orleans, waiving tuition, initially took in thousands of students whose public schools had been ruined. Several months later, after Hurricane Rita ravaged parts of the Houston area, the federal government established hurricane vouchers for the two storm-damaged regions, temporarily covering the private-school tuitions of educationally displaced children.
In the wake of Katrina, Louisiana lawmakers established two major private-school choice programs. The first was the Elementary and Secondary School Tuition Deduction policy, enacted in 2008. This initiative allows families to deduct on their state income-tax return up to $5,000 per child in private-school educational expenses. The families of more than 106,000 of the 112,000 students attending Louisiana private schools in 2012 claimed the deduction. The second program was the Student Scholarships for Educational Excellence Program, which in 2009 began providing private-school vouchers to 624 low-income students in the parishes of Orleans and nearby Jefferson. This program served as a pilot for the larger, statewide Louisiana Scholarship Program, launched in 2012.
State policymakers also dramatically refashioned the public-school system in New Orleans. Gone were residential attendance zones, the teacher collective-bargaining agreement, and almost all of the public schools the Orleans Parish School Board directly operated. In their place arose a new kind of urban public school system, dubbed the Recovery School District. Overseen by state education officials, the new district was composed almost entirely of public charter schools that would be held accountable for student achievement on state tests. Douglas Harris at Tulane University concluded that this package of market-based reforms—expanded school choice coupled with results-based accountability—substantially improved the test scores of students attending the city’s public schools (see “Good News for New Orleans,” features, Fall 2015).
The Louisiana Scholarship Program
The pilot voucher program had 1,950 students enrolled in 2012 when it expanded statewide and became the Louisiana Scholarship Program. Demand for the program was strong from the start: a total of 9,736 students applied that first year, with 5,296 receiving vouchers and 4,944 using them to attend a private school. It is this 2012–13 applicant cohort, a majority of whom lived outside New Orleans, that we evaluated over the course of four years. The program enrolled 6,695 students in 2016–17, a drop of 9 percent from its enrollment peak of 7,362 in 2014–15 (see Figure 1).
Participation in the voucher program is restricted to low-income students in low-performing public schools. To qualify, the income of a child’s family must be at or below 250 percent of the federal poverty line, which in 2016–17 was $60,750 for a family of four. Applicants also must either be entering kindergarten or be attending a public school graded C, D, or F by the state’s test score-driven school accountability system. About one third of the K–12 students in Louisiana, or nearly 250,000, were eligible for the scholarship program in 2012, when 4 percent of the eligible student population applied to the program. Nearly 90 percent of the eligible applicants were African American and over 80 percent were entering grades 1–6 that year.
Applicants win spots in the program by means of a government-run lottery, and the students who did not win the lottery provided the control group for our study. Students in D or F schools receive priority in this lottery, so very few students from C schools have received scholarships. Students with disabilities also receive priority, as they are placed automatically in their private school of choice if a seat is available. The lottery simultaneously awards students with scholarships and placement in a specific private school, drawing from the school preferences listed by parents. Unlike most private-school choice programs, the scholarship award takes place after the school-shopping process, not before it.
The voucher value is limited to 90 percent of the state and local per-pupil funding in a student’s school district or the tuition rate the student’s chosen private school charges, whichever is less. Annual tuition at participating private schools ranges from $2,966 to $8,999, with an average of $5,437, compared to average state and local per-pupil funding of $8,500 in Louisiana’s public schools in 2010–11.
Why might a private school decide not to participate in the Louisiana Scholarship Program? One reason could be that participating schools must submit to regulations regarding financial practices, curriculum, student enrollment, mobility, safety, and achievement. They must provide financial audits to state officials every year and maintain a curriculum that the state Department of Education judges to be of equal or higher quality to the curriculum in public schools. Participating schools cannot select their voucher students and must instead admit them solely via the placement lottery administered by the state.
All participating private schools must administer the state’s accountability tests annually to voucher students in grades 3–8 and again in grade 10. Participating schools with at least 10 voucher students per grade are assigned school-performance ratings based on voucher-student scores on the state test.
As in the test-based accountability model applied to public charter schools in Louisiana, the state can sanction private schools participating in the voucher program. Any of three conditions leads to state sanctions. First, a school is sanctioned if the average rate of gain of voucher students on the state test is so low that the school qualifies for an F grade. Second, a school is sanctioned if less than 25 percent of its voucher students score at or above the state benchmark for proficiency. Finally, participating schools are sanctioned if their annual financial audit reveals improprieties or raises concerns about the school’s future viability.
A private school that receives a sanction is prohibited from enrolling new voucher students until it remedies the condition that led to the penalty. By 2015–16, the final year of data collection for our study, 35 of the 122 private schools in the program had been sanctioned for at least one year.
These regulations on admissions, testing, and curriculum render the Louisiana Scholarship Program one of the most highly regulated among the 58 private-school choice programs in the country. Why put so many restrictions on the private-school choices available to parents?
The history and context of education in Louisiana likely led policymakers to opt for such a highly regulated model. The Pelican State still has many racially stratified schools. Most of the D and F public schools in the state serve a majority low-income student population. Thus, restricting the voucher program to students from low-income families attending failing schools, and requiring the schools to forgo any admissions standards, was expected to prevent the program from worsening school-level racial and income segregation.
The private schools in Louisiana ranged dramatically in tuition rates, which likely serve as a rough proxy for school quality. Officials anticipated that, as is common in voucher initiatives, few of the higher-tuition private schools would elect to participate in the program, since they would have to foot a large part of the cost of educating voucher students. Moreover, the state Elementary and Secondary School Tuition Deduction policy provided the fee-paying customers of private schools with a significant cost rebate that did not require the school to take on any additional state regulations or paperwork, reducing the incentive for schools to join the voucher program and accept its regulatory requirements.
Policymakers expected the lower-quality, under-enrolled private schools in Louisiana to sign up to participate in the voucher program, so they designed a results-based accountability system that would remove schools from the program if they produced low student test scores. A similar regulatory system appeared to be working well for the public charter-school sector in New Orleans. Mandating that private schools administer the state accountability test would allow an apples-to-apples comparison of school performance. It all seemed so sensible at the time.
Private School Participation
Fewer than a third of the private schools operating in Louisiana in 2012 agreed to participate in the Louisiana Scholarship Program. That was the smallest share of schools to participate in a statewide, means-tested private-school choice program among all such programs in the country, and we sought to understand why. Brian Kisida, Evan Rhinesmith, and I implemented a survey of Louisiana private-school leaders that revealed the top factors deterring private schools from joining the voucher program. We found that these leaders feared that more regulations might arise in the future and hamper their independence or threaten their religious identity. They also worried about the integrity of their admissions policies, and they didn’t like the pressure to adhere to the state’s curriculum standards.
Evidence from a survey experiment conducted in other states suggests that the kinds of regulations in place in Louisiana may discourage private schools from participating in choice programs. Corey DeAngelis, Lindsey Burke, and I sent brief email surveys to every private-school leader in Florida, California, and New York, asking if they would be likely to participate in a private-school choice program offering a $6,000 voucher. Survey takers were randomly divided into five groups. Those in the first group were asked about their willingness to participate in the hypothetical program with “no strings attached.” Those in the other four groups were asked about participating under a specific regulatory requirement.
One third of private-school leaders who were told they would face no additional regulations said they were “certain” they would participate in such a voucher program. Among school leaders who were told the hypothetical program would mandate an open-admissions policy, just 14 percent said they would be certain to participate. Of those who were told that they would be required to administer the state accountability test to all voucher students (with the results reported publicly), 24 percent said they would be certain to participate. The remaining two conditions had no significant effect on leaders’ expressed willingness to participate: a requirement that the school administer a norm-referenced test of its choosing; and a mandate that the school accept the voucher as the full cost of educating the child.
What kinds of private schools did participate in the Louisiana Scholarship Program? Compared to non-participants, participating schools were more likely to be Catholic, have lower enrollments, and have student populations that were disproportionately made up of racial or ethnic minorities. The typical participating private school was accustomed to serving socially disadvantaged students and eager to increase its enrollment. Surprisingly, average tuition levels were not a major factor in separating participating schools from non-participating ones in Louisiana.
Students Served
The scholarship program was designed to serve a highly disadvantaged population: kids from low-income families who were attending a public school that produced consistently low test scores.
It is therefore no surprise that Yujie Sude and I found almost no evidence that the voucher program “cream-skimmed” easy-to-educate students. At the application stage, the program attracted a highly disadvantaged population of students. Louisiana public-school students were more likely to apply to the voucher program if they were low income, African American or Hispanic, and lower test performers. Students in the earlier K–12 grades were much more likely to apply than older students, suggesting that parents are more willing to switch their child from a public school to a private one when they are younger and perhaps better able to adapt to their new school environment.
Which students stayed in the program? Several factors were associated with persistent voucher use over three consecutive years. Students with lower initial test scores were more likely to persist than were students with higher scores. Girls and students who enrolled initially in earlier grades were more likely to continue than boys or students who started in later grades. Students in private schools that had lower minority enrollments and that were located a shorter distance from home were also more likely to persist, as were students residing in public-school districts that had lower per-pupil spending.
After all of these factors played out, how select was the voucher-student population three years after application? Characteristics associated with student disadvantage tended to distinguish three-year voucher users from students who never applied to the program. Compared to those who didn’t apply, students who participated and continued for three years were more likely to be female, low income, and African American. The Louisiana Scholarship Program did succeed in expanding school choice to a relatively disadvantaged population of students.
Effects on Racial Segregation
Many commentators claim that parental choice programs worsen segregation. To test that claim, Jonathan Mills and Anna Egalite analyzed the effect that initial voucher users had on the level of racial integration in both the public schools they left and the private schools they joined (see “The Louisiana Scholarship Program,” features, Winter 2014). First, the researchers established a benchmark for racial balance in each school, pegged to the racial composition of the local community. Then they determined whether each student who transferred to a private school in 2012 under the voucher program had an integrating or segregating effect. For the public school the student was leaving, the student was deemed to have had an integrating effect if the race of the student was overrepresented at that school, given the community benchmark. But a transferring-out student had a segregating effect on that school if the student’s race was under-represented there. Conversely, a transfer into a private school better integrated it if the student’s race was under-represented and further segregated it if the student’s race was overrepresented.
The researchers found that 83 percent of initial student transfers under the program had an integrating effect on the racial composition of the public schools they left. In the private schools they joined, about half of the students generated better integration and the other half lessened integration. Even in the 34 Louisiana public-school districts still under a court order to integrate by race, 74 percent of the transfers under the voucher program brought the public schools closer to the goal of racial integration. Thus, the program has operated as a voluntary school desegregation program, even though the racial integration of public schools was not its primary purpose.
Academic Outcomes
The main purpose of the scholarship program was to improve academic outcomes. On that goal, it clearly fell short. Using gold standard experimental methods, Jonathan Mills and I determined that the effects of the program on student scores on the state accountability test tended to be negative, especially in math, as long as four years after initial scholarship use.
For the students who participated in the voucher lotteries and for whom we had both baseline (2011–12) and subsequent test scores, the effects of the scholarship program on achievement varied between negative and neutral (see Figure 2). At baseline, the math test performance of the lottery winners was statistically equivalent to that of the control-group students (that is, those who did not win a voucher placement). But effects on the math performance of the scholarship students were negative and large in the first year of the program, when students were adjusting to their new schools and, likewise, the schools were adjusting to them. The voucher students recovered some of their lost ground on the state math test the second year and delivered math scores that were statistically similar to the control group in year three, when the state switched to a new accountability test and neither public nor private schools were held accountable for the results. In year four, when test-based accountability was reestablished statewide, the voucher program again produced negative impacts on math scores; these were moderately large.
The reading test-score impacts of the program followed a trend similar to the math scores but with consistently smaller effects, most of which were not significantly different from zero. For those students for whom we had baseline test scores, the reading impacts of the program were negative and moderately large the first year, and much smaller and not statistically significant in the second, third, and fourth years.
The achievement impacts of the scholarship program were not uniform for all groups of students. In the fourth year, the negative effects of the program on test scores were less than half as large for African American students as for non-African American students. Also, voucher students in grades 4 and 8 took the main state accountability test, while those in grades 3, 5, 6, and 7 took a version of the test that was scored on a different scale. For the latter group, the negative test-score effects of the program were less than half as large as for those who took the main state test.
The effects of the voucher program on student test scores also varied based on key features of the school the student most preferred. Matthew Lee, Jonathan Mills, and I determined that the achievement effects of the program were much less negative, and in some rare cases even positive, for subgroups of students whose preferred private schools ranked in the top one third of participating schools in regard to higher tuition, higher total K–12 enrollment, or a greater number of instructional hours for students.
Educational Attainment
Educational attainment—how much schooling an individual ultimately completes—greatly influences his or her later life outcomes, and researchers have increasingly turned to studying the impacts of education programs on student rates of high-school graduation and college enrollment, persistence, and completion. Too few students in our experimental sample were old enough to have graduated from college for us to examine that important outcome. However, more than 1,000 students who participated in the voucher program’s lotteries completed high school and could have enrolled in college. Did the negative test-score effects of the program decrease their rates of college going?
Heidi Holmes Erickson, Jonathan Mills, and I found that the Louisiana Scholarship Program had no effect on college entrance for students. After high school, voucher users enrolled in a two-year or four-year college at a rate of 60.0 percent, which was equivalent to the 59.5 percent enrollment rate of the control group.
Mixed Outcomes
Opponents of private-school choice may use the results of our study to argue that “free market” approaches to education have failed. However, the initiative that we evaluated was clearly not a pure free-market education reform. Free markets generally require: 1) that new suppliers can easily enter the market; 2) that prices can vary across providers and different versions of a service; and 3) that consumers are the main judge of the quality of the service. None of those conditions held for the Louisiana Scholarship Program. New suppliers were virtually prohibited from emerging, as private schools had to operate for two years with only fee-paying customers before they were allowed to participate. The price of the service was largely fixed by the voucher formula, and parents were prohibited from paying more for a higher-quality education, at least within the program. A government-run lottery placed each voucher student in a school, and government overseers determined which schools could and could not continue to enroll new voucher students. And while parents were able to communicate their preferences about school placement, it was the state, not the parents, who made the final selection.
School choice programs often seek to reduce racial segregation in schools. The Louisiana Scholarship Program succeeded in doing so. Everything is good at something. The statewide launch of the program in 2012 better integrated Louisiana’s public schools, as most voucher-program participants left public schools in which their own race was overrepresented. Persistent scholarship users were more likely to be African American and to have had lower initial test scores compared to those who didn’t apply to the program. Furthermore, the program saves the state money, because the voucher maximum of less than $6,000 is only about two thirds of the combined state and local per-pupil funding in Louisiana’s public schools.
The regulatory framework of the program rested on several ideas: that private schools accepting government vouchers are comparable to public charter schools; that low-performing schools will improve with government incentives to do so or be kicked out of the program; and that student test scores are the best single measure of school performance. All of these notions are subject to question.
Private schools differ from public charter schools in two critical ways that affect how they might be effectively regulated. First, private schools have a choice in whether to participate in voucher programs, accepting government funding and the regulations that come with it. In Louisiana, where tax policies indirectly subsidize families who pay tuition out of pocket, private schools have lots of paying customers and thus greater latitude to turn down the chance to serve disadvantaged students on state scholarships if the offer is not attractive to them. Public charter schools have no such choice, as the government is their only reliable funding source.
There is little evidence that education overseers can help schools improve by using regulatory carrots and sticks. Student test-score performance is a product of many complex factors, including family background (see “How Family Background Influences Student Achievement,” features, Spring 2016), school culture, teacher quality, child nutrition, test preparation, and curricular alignment with the test. Actual levels of student learning are in there somewhere, but it is a crowded room of factors that most private schools with established educational programs and cultures cannot change over the course of a few years, even if a regulatory system incentivizes them to do so. The evaluation literature on aggressive efforts at systematic school turnaround reinforces this point: school improvement is a slow, difficult, evolutionary process.
Removing private schools from the program if they fail to boost student test scores might seem like an automatic method for improving outcomes by “chopping off the lower tail of the performance distribution,” but where do the affected students go instead? If the existing private schools already are full and new ones are prohibited from serving voucher students during a long probationary period, the state sanctioning system merely serves to limit the educational options of parents and students.
Finally, student scores and annual gains on the state-mandated accountability test are only one measure of school performance. They capture only a portion of what we would consider actual student learning, and advancing student learning is only one of many goals of publicly financed education. We also charge schools with instilling civic values in students and developing positive character traits of persistence, conscientiousness, and respect for others and the norms of society. When Thomas Stewart and I polled a room of 40 parents participating in the D.C. Opportunity Scholarship Program on how they assess the academic progress of their child in school, none of them selected “standardized test scores” as the metric. Instead, they said they used “student motivation to learn,” “student grades,” “positive student attitudes toward school,” and “positive student behaviors toward schools” as measures of whether their child was succeeding. The private schools participating in the Louisiana Scholarship Program may be delivering these more nuanced but vital outcomes to their voucher students, but determining that would require a more comprehensive assessment of school performance.
In order to accomplish anything, schools must keep their students safe. Recent surveys suggest that school safety now rivals academic concerns as the main reason parents seek private-school choice. Public regulators would be wise to work indicators of school safety and an orderly school environment into their measures of whether a school is performing at a satisfactory level.
The Louisiana Scholarship Program did not succeed in raising student scores on the state accountability test. In fact, it had a clear negative effect on math scores and a possible, though less severe, negative effect on reading scores. Nothing is good at everything. Our evidence suggests that about half of the negative test-score effect is likely due to public schools more carefully aligning their curriculum to the tested material and systematically implementing techniques for preparing students for the test. Still, it is clear that the program did not positively affect scores on the state test. Insofar as state accountability test scores are used as the litmus test for effectiveness, the Louisiana voucher program did not let the good times roll down in the Bayou.
Patrick J. Wolf is a distinguished professor and 21st Century Endowed Chair in School Choice in the Department of Education Reform at the University of Arkansas College of Education and Health Professions.
This article appeared in the Fall 2019 issue of Education Next. Suggested citation format:
Wolf, P.J. (2019). What Happened in the Bayou? Examining the Effects of the Louisiana Scholarship Program. Education Next, 19(4), 48-56.
