Scripting the Moves: Culture and Control in a “No-Excuses” Charter School
by Joanne W. Golann
Princeton University Press, 2021, $27.95; 248 pages.

As reviewed by Robert Pondiscio

In the early days of KIPP, or the Knowledge Is Power Program, and other networks of urban charter schools that drafted in its considerable wake, the highly prescriptive form of classroom management and teaching these schools pioneered was a subject of intense fascination and considerable optimism. A 2006 New York Times Magazine article by Paul Tough titled “What It Takes to Make a Student” described the belief of KIPP founders David Levin and Mike Feinberg that middle-class kids learn certain methods for taking in information early on and employ them instinctively. KIPP students, by contrast, needed to be taught those methods explicitly. The network’s model included the technique known as “Slant,” an acronym that reminds students to sit up, listen, ask questions, nod, and track the speaker.

“To anyone raised in the principles of progressive education, the uniformity and discipline in KIPP classrooms can be off-putting,” Tough reported. “But the kids I spoke to said they use the Slant method not because they fear they will be punished otherwise but because it works: it helps them to learn.”

Fifteen years on, Vanderbilt University professor Joanne W. Golann’s new book, Scripting the Moves, revisits this highly prescriptive brand of teaching and finds it mostly wanting. The book takes its title and frame from Suzette Dyer, a school principal who observed that success needs to be “scripted” for students and even for teachers. “You’ve got to script the moves for students. You have to narrate the experience so students understand exactly what the outcomes are,” Dyer said in an interview for Restoring Opportunity by Greg Duncan and Richard Murnane.

The author writes that she was initially “not taken aback” by lessons at “Dream Academy” (her pseudonym for the high-performing middle school in a medium-sized northeastern city where she made her observations) that “literally spelled out what students needed to do to conform to school expectations for showing attention.” But the more time she spent at the school, the more she questioned the efficacy of these rigid behavioral scripts. The prescriptiveness, she writes, “left little room for them to develop what I call tools of interaction, or the attitudes, skills, and style that allow certain groups to effectively navigate complex institutions and shifting expectations.” Golann’s object is cultural capital. Middle-class students use it in schools and workplace as “a flexible tool, not a straightjacket,” she notes. “Scripting,” then, is a self-limiting factor, a kind of paint-by-numbers version of “what it takes” to succeed in school and beyond.

Working-class parents “already emphasize to their children ‘no excuses’ problem solving—to work hard and not bother others with requests for accommodations.” Middle-class parents, by contrast, encourage their children “to negotiate with their teachers and bend rules to their benefit.” If the school wanted to teach middle-class expectations to its students, Golann writes, “it should have taught them how to effectively make excuses.” Likewise, the rigid scripting Golann witnessed at Dream Academy allowed for little flexibility, leading teachers to “gloss over legitimate excuses, hiding the structural issues that shape students’ behaviors and actions.” This is a valid observation, if an ungenerous interpretation of a school model whose purpose, right or wrong, was never intended to cultivate unthinking compliance among students, but resiliency and determination.

In general, Golann’s observations are thoughtful, scholarly, and, in contrast to many who have sought merely to discredit the no-excuses model, mostly empathetic. There is a problem, however, and it’s a significant one: Her analysis rests largely on 18 months of fieldwork dating back to September 2012. That’s a long time ago, and an eternity in urban charter schools. She notes that no-excuses charter networks “have begun to reflect on the implications of their rigid behavioral scripts,” but this understates the considerable degree to which charter schools have dialed back their discipline practices and the prescriptiveness of their pedagogies, an iterative process that began a decade ago with KIPP’s disappointment over its graduates’ college-completion rates. This process accelerated more recently with concerns in American education at large about the disproportionate rates at which nonwhite children have been subject to school discipline and suspensions. The long lag time between Golann’s fieldwork and the arrival of the book (some portions were previously published in academic journals in 2015) means Scripting the Moves can read at times like a time-capsule glimpse into a category of schools that long ago recognized and responded to many of the author’s most important critiques.

Indeed, some of the data she presents remind us why “no excuses” came under such intense scrutiny after years of replication and fawning media coverage. Over the course of a single school year, “Dream Academy” teachers meted out an eye-popping 15,423 infractions to the school’s 250 students, an average of more than 60 per student. Only six students managed not to incur a single infraction; one 5th-grade boy drew 295. Numbers such as these caused critics, not unreasonably, to decry the inflexible behavioral demands of “no excuses” schools. On the other hand, Golann notes the school had very few major infractions, such as fighting, graffiti, and bullying, illustrating precisely the “sweating the small stuff” mindset that early no-excuses schools fetishized, taking their lead from the era’s “broken windows” policing model.

Many of the practices Golann describes are best left to molder on the classroom-management compost pile. At the start of the school year, students sit on the floor until they “earn” their seats. Minor behavioral infractions lead to students being “benched,” a tactic borrowed from KIPP’s practice of “porching” (“If you can’t keep up with the big dogs, stay on the porch.”) Benched students must wear their shirts inside out like a middle-school scarlet letter and are forbidden from interacting with peers. These kinds of wince-worthy punishments go a long way toward explaining how the model went from halo effect to heel turn in the minds of so many observers, including Golann.

“If ‘no excuses’ is supposed to be about the school making no excuses for student failure, it ends up being about the school accepting no excuses for deviating from the school’s rigid behavioral script,” Golann writes.

She’s not wrong about the excesses of rigid school cultures and behavioristic teaching, but her critique is at times overly broad. My own book based on a year of observations at New York City’s Success Academy noted that a lesson could be rich and invigorating in the hands of a talented teacher but excruciating under another teacher who seemed not to grasp the “why” behind behavior management, viewing it as an end in itself rather that the starting line for deep learning and inquiry. Similarly, while intellectual fashions have largely turned against no excuses, there is a danger in memory-holing the conditions that made the model’s practices appealing and effective, particularly to parents who prized the physical safety offered by tightly run schools that stood in stark contrast to chaotic neighborhood schools with low graduation rates and few opportunities for college acceptance or success.

Golann clearly means for us to see “scripting” as a problem and even a failure. But strong and successful institutions—from families and churches to the U.S. Marines—have long played an essential role in shaping character by “scripting the moves.” The challenging question implied in Golann’s book is whether the problem is the act of scripting or the particular script. “In many ways, no-excuses schools create an alternative universe for students . . . one that promises upward mobility if students will only follow the school’s scripts for success. Students are asked to ‘overcome’ their backgrounds and assimilate into dominant culture,” she writes. “But it is not easy—and perhaps not prudent—to insulate students from their home worlds. It risks not recognizing the ways in which students are affected by out of school factors and can potentially be detrimental to students’ sense of identity and feelings of connectedness.”

Golann’s critique is on point and resonant with the present moment. Still, one wonders where parents’ desires for their children fit into the calculus. For some, an “alternative universe” is a problem. For others, it’s the point.

Robert Pondiscio is a senior fellow at the American Enterprise Institute and the author of How the Other Half Learns.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Pondiscio, R. (2021). No Excuses, Revisited: A thoughtful but dated criticism of “no excuses” schools. Education Next, 21(4), 78-79.

Beyond Standards: The Fragmentation of Education Governance and the Promise of Curriculum Reform
by Morgan Polikoff
Harvard Education Press, 2021, $60; 192 pages.

As reviewed by Natalie Wexler

The many education reformers who have relied on academic standards to boost student achievement might outline their theory as follows: States broadly define what students should know and be able to do at specific grade levels. Publishers use these standards to create detailed curricula, which districts adopt. Teachers receive training in the standards’ requirements. Students’ progress is tracked by standards-based assessments. And educators are held accountable for the results. The expected outcome: markedly higher student achievement and a narrowing of racial and income-based gaps.

In Beyond Standards, Morgan Polikoff demonstrates that this theory hasn’t matched reality. He argues that standards are inherently too vague to enable teachers to arrive at a common or accurate understanding of what they need to teach or to identify the right materials for teaching it. Polikoff also points to decentralized governance as a problem. With over 13,000 school districts in the United States, he argues, it’s impossible to provide all students with a standardized educational experience.

Polikoff pins his hopes for improvement in K–12 education partly on high-quality curriculum rather than standards. As he notes, studies have shown that the effect of a high-quality curriculum on student outcomes can be as strong as the effect of having a veteran rather than a novice teacher. He also supports a more active role for states. Citing Louisiana as an example, he argues that states should identify or create high-quality curricula and exercise greater control over decisions typically left to districts, schools, or individual teachers (see “Louisiana Threads the Needle on Ed Reform,” features, Fall 2017). Polikoff even urges states to mandate that districts adopt curricula from among a limited set of state-approved options.

That aspect of his reform prescription is more problematic. As Polikoff is aware, the idea of state-mandated curriculum flies in the face of a strong American tradition of local control. Some states—20 or 25 by his estimate—issue lists of approved curricula or textbooks, but none require districts to use those on the list. In the other states, curriculum decisions are left entirely to districts. When Polikoff advised one state’s education leaders simply to collect data on which curricula districts were using, they protested that “district folks would freak out and assume the state was trying to usurp their authority over teaching and learning.”

Beyond the logistical or political obstacles, though, it’s not clear states can be relied on to make good curriculum choices—especially in the area of literacy. Polikoff argues that decentralization has led to a plethora of curricular approaches, but there is a standard approach to literacy, referred to as “balanced literacy.” While curricula vary in some respects, most of the commonly used ones fail to guide teachers effectively in teaching phonics, an area where their training is often deficient. And almost all emphasize reading comprehension skills and strategies, such as “finding the main idea” or “making inferences.” Students practice the skills on “leveled texts”—books on various topics that they can read easily and that may be well below their grade level. Polikoff doesn’t specifically address either component of this widespread approach.

When he refers to high-quality literacy curriculum, he seems to have in mind a handful of newer curricula grounded in evidence that many children need systematic instruction in phonics to read fluently and that comprehension depends far more on academic knowledge than “skill.” These curricula put content in the foreground and go deeply into topics in social studies and science. Although they haven’t been studied as much as math curricula, the evidence on their effectiveness that does exist is promising.

But will state decisionmakers gravitate to these newer curricula or stick with what’s familiar? Polikoff seems to assume they’ll opt for the good stuff if they involve teachers in the adoption process and rely on guidance from organizations like EdReports, which rates curricula according to their alignment to Common Core standards. Teachers, though, may prefer to work with what they’re used to. And although EdReports ostensibly includes knowledge building as one of its criteria—and although the organization has rated several knowledge-building curricula highly—it has made some puzzling decisions of late.

For example, EdReports gave its highest rating to McGraw-Hill’s Wonders, one of the 10 most popular reading programs. But a recent critique by Student Achievement Partners, which evaluates curricula for how well they align with research evidence, found that Wonders is overstuffed, fails to spend enough time on some foundational skills, lacks coherence, and doesn’t build content knowledge systematically.

It’s not hard to imagine a state being misled by guidance from EdReports. In fact, that may have already happened in Florida, where the state’s new standards profess to value knowledge over reading-comprehension skills. Strangely, the recently announced state adoption list failed to include any of the curricula that focus on building knowledge but did include Wonders, along with some other lower-quality options. At the same time, several Florida districts had already conducted their own reviews and chosen actual high-quality curricula, only to find that the state later failed to recommend them.

Fortunately, those districts can still adopt high-quality curricula, although it might be more difficult or cost them more money. Polikoff calls such state incentives to use approved curricula “modest,” but they may be more powerful than he thinks. It’s essentially the method Louisiana used to induce districts to adopt high-quality curricula, and, as he reports, over 80 percent of schools in the state are now using such materials. But if Polikoff had his way, Florida would be able to prevent the districts that want high-quality curriculum from purchasing it and require them to use inferior curriculum instead.

In the abstract, Polikoff’s prescription makes sense: why not have 51 decisionmakers rather than 13,000? We might even wish we could have just one, like the many developed countries that have national curricula. But centralizing the decisionmaking process only makes sense if the decisionmakers understand what they’re doing. In Louisiana, curriculum adoption at the state level worked because of a visionary state superintendent of education, John White. There may be others like him, but at this point we can’t count on one being at the helm of every state department of education—or even most of them.

I agree with Polikoff that standards-based reform hasn’t worked—and in the case of literacy standards, which reinforce the mistaken notion that reading comprehension is primarily a set of skills, I think we would be better off without them. But weaning schools away from what’s familiar and toward what’s aligned with science will unfortunately take a lot more than jettisoning standards and giving authority over curriculum to the states.

Natalie Wexler is an education writer and author of The Knowledge Gap: The Hidden Cause of America’s Broken Education System—And How to Fix It.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Wexler, N. (2021). A Sharp Critique of Standards-Based Reform: Polikoff pins his hopes on high-quality curricula selected by the states. Education Next, 21(4), 76-77.

The IRS recently sent $15 billion in Child Tax Credit payments to the families of 60 million children. Parents woke up to find the first monthly distributions (averaging $483 per family) had been directly deposited into their bank accounts. These dollars are part of the $110 billion in expanded child tax credits legislated by this spring’s $1.9 trillion American Rescue Plan Act. While these funds are not targeted at education, the program has opened a temporary window that could help vastly expand school choice.

The American Rescue Plan Act created an unrestricted, refundable child tax credit of $3,600 for children under 6 and $3,000 for children aged 6 to 17. The full credit applies to all families making less than $150,000, and the balance is paid as a cash transfer if the total exceeds their federal tax burden. After a year, unless Congress extends the increase, the credit will revert to $2,000 per child and become only partially refundable (see this issue’s forum, “Should Congress Make the Expanded Tax Credit Permanent?”).

To understand how the child tax credit might be made into something more educationally transformative, consider three things.

First, $3,000 may seem modest compared to the cost of schooling—the sum is less than a quarter of what public schools spend per child each year—yet it’s more than 60 percent of tuition at a typical Catholic elementary school. In fact, $3,000 isn’t much less than the average voucher awards (of $4,000 to $5,000) in states like Indiana, North Carolina, and Ohio.

Second, at this moment, governors and state legislators have exceptional financial flexibility to augment or supersize the child tax credit funds. State budget collections are coming in much higher than was anticipated last year, while the American Rescue Plan Act delivered state and local governments $350 billion in federal Covid aid—atop the $130 billion in K–12 aid.

Third, the child tax credit offers an opportunity to extend school choice to a broader swath of families. President Joe Biden and a Democratic Congress just opted to send child tax credit funds to the families of 80 percent of the nation’s children. If we were to stipulate that choice programs should serve the kids whom these officials deemed in need of these funds, that would vastly expand the ranks of the eligible. It would be awkward for Democrats to argue that middle-class parents need federal help paying their bills but are too well-off to merit state help in defraying the cost of schooling.

In short, state leaders have the opportunity to offer families expanded educational options at a time when support for school choice has exploded. The most promising tack is to augment the federal tax credit for any family that chooses to spend its funds on school tuition or another documented educational cost. A temporary 50-percent state match for families who use the funds to pay tuition could make the credit for many worth about as much as the voucher offered in leading school-choice states; a 100-percent match, funded with general Covid aid, could make it exceptionally large.

Such a move allows states to radically expand the ranks of the eligible while hewing to the eligibility criteria endorsed by Democratic leaders. (In our school life column for this issue, Robert Behning, the chair of the Indiana house education committee, makes precisely this argument.) The resulting program would only run as long as the expanded child tax credit, but it could spark a taste for choice among many.

State leaders should also explore ways to augment the credit with education savings accounts, and then help parents understand how to marry the two. Earlier this year, for instance, West Virginia enacted its first education savings account program, providing eligible families with $4,600 per child (See “School Choice Advances in the States,” features). Combined with the maximum child tax credit payment, that yields $7,600 per child—an amount that exceeds the cost of tuition in 9 out of 10 West Virginia private schools. Energetic use of the bully pulpit could encourage some families to think of child tax credit payments as education-choice funds.

The expanded child tax credit is a one-year program. If it goes away, children will have benefited and the constituency for choice will grow. And if it becomes permanent? Then state leaders will have the opportunity to do even more.

Frederick M. Hess
Executive Editor

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Hess, F. (2021). Tax Credit Could Boost Choice. Education Next, 21(4), 5.

Remember the Red for Ed movement? Three years ago, teachers in West Virginia massed on the state capitol by the thousands, demanding salary increases and seeking to stop legislation that would have brought private-school choice to the state. For the most part, the teachers succeeded. They got their raise and prevented all but the creation of a few public charter schools. Their tactics soon spread to other states, with large teacher protests leading to salary hikes in Arizona, Indiana, Kentucky, and Oklahoma.

Teachers in West Virginia not only reveled in their success but also vowed to punish their enemies. In last year’s primaries, they helped unseat a pair of state senators who were prominent supporters of school choice. Overall, however, teachers suffered political losses in the state. Republicans won supermajorities in both legislative chambers, and they wasted little time before passing one of the most expansive school-choice laws in the country.

West Virginia now offers universal education savings accounts, or ESAs. These differ from traditional vouchers in that they can be used not only for private-school tuition but also for tutoring, therapies, technology, and all manner of education-related services. The state’s new Hope Scholarship program, which launches in the 2022–23 school year, provides students with up to $4,600 each in state funding for a wide range of approved services (including those from out-of-state providers). The program is open to all K–12 students, with funds available to families who withdraw students from public schools or who don’t enroll new school-aged children there in the first place. What’s more, there’s a trigger provision that allows children who are already enrolled in private schools to receive vouchers, if participation in the ESA program does not exceed 5 percent of public-school enrollment within two years.

Other states have experimented with ESAs but, as with other choice initiatives, these scholarship programs have generally been limited by geographical area or to particular populations of students or families (such as children with disabilities or those from a military background). In West Virginia, ESAs are not capped and will be widely available. “They had no private-school choice a year ago, and now they have the most ambitious program in the country,” says Patrick Wolf, an education-policy professor at the University of Arkansas. “They went from zero to 100 in one session.”

Victory Laps

West Virginia stands out in its dramatic move toward school choice, but it was far from the only place where proponents were able to take a victory lap this year. Other states that had long resisted choice options suddenly changed their positions. In states that have consistently experimented with alternatives to traditional public schools, such as Arizona and Florida, the size and scope of their programs continued to grow. “It has been a breakthrough year,” says Robert Enlow, president and CEO of the advocacy group EdChoice. “This is, without a doubt—based on all our tracking over the years—the biggest year for educational school choice.”

Florida remains a leader in providing options for families. About one in four U.S. children taking advantage of private-school-choice programs live in Florida. The state is continually expanding its offerings. A law enacted in May 2021 merged various programs while increasing private-school scholarships to 100 percent of per-pupil funding. The bill prioritizes low-income families making under 185 percent of the federal poverty level ($49,000 for a family of four) but expands eligibility to encompass those making up to 375 percent of the federal poverty level, or just under $100,000 for a family of four. The law also eliminates the old requirement that students enroll in public schools before applying for a scholarship, and it exempts military families from program limits and waiting lists.

In Florida, choice supporters are well organized, enjoying strong champions both in the legislature and in Governor Ron DeSantis, a Republican who has appointed choice-receptive justices to the state supreme court. Daniel Aqua, the executive director of Teach Florida, suggests that the state’s population growth has also helped promote its school-choice experiment, with both private and public schools seeing increasing enrollment. “With tremendous population growth comes decreased competition for children,” he says.

All told, seven states created new programs this year, while 11 more expanded existing options. The Wall Street Journal declared 2011 “the year of school choice.” In 2021, advocates have enjoyed even greater success.

Several factors account for this achievement. The pandemic led to widespread school closures, which in turn caused real headaches for parents. It would be too much to say that all parents became choice parents, but clearly more of them—including affluent suburban parents who had generally taken pride in their local schools—suddenly found themselves scrambling for workable alternatives. By definition, this meant they were open to nontraditional instruction programs and methods, including private schools and learning pods. This dynamic helped expand the choice coalition beyond its core supporters among conservatives and libertarians paired, sometimes uneasily, with low-income minority parents from urban cores. Those groups, in turn, set aside their differences to work together in 2021. “Just as the country was splitting and becoming polarized, the school-choice movement was too,” Wolf says. “There didn’t seem to be as much cooperation between those two groups. But Covid changed that. There’s nothing like a crisis to bring people together and get them to focus on what unifies them.”

The pandemic not only shut classrooms but state capitols as well. With Covid-19 closing statehouses to the public for long stretches, teachers and parents opposed to choice programs were unable to gain easy access to legislators, let alone stage large demonstrations of the Red for Ed variety. This lack of access was a problem for advocates working in many policy areas, but it posed particular challenges for teachers and parents, who were already stretched thin owing to childcare difficulties and the demands of remote or hybrid instruction. There was less advocacy and legislative activity related to charter schools in 2021 simply because, as parents searched for options, they could already choose a charter school at no cost to themselves (in states that allow charters), while private-school tuition can be prohibitive for many families.

Finally, partisan politics played a role as well this year. The choice proposals that met success were launched almost exclusively in Republican-controlled states in the South and elsewhere. Not much happened in the Democratic blue states where schools were closed the longest. Although President Trump lost his reelection bid last year, he campaigned on choice and during his last month in office signed an executive order allowing states to use federal funds for vouchers for disadvantaged children. His education secretary, Betsy DeVos, was not able to move the needle much while in office, but she remains a formidable defender of choice. She and her family have helped to fund legislative candidates all over the country. In 2020, they devoted more than $10 million to political contributions in their home state of Michigan alone. Enlow, of EdChoice, says that the focus that DeVos and the Trump administration devoted to choice issues in the District of Columbia and nationally had a “unique, positive effect” on state action. Becky Pringle, president of the National Education Association, casts the former education secretary’s influence in a different light: “Now, after her voucher plans were rejected by Democrats and Republicans in Congress over the last four years, Betsy DeVos is funding efforts to push states with Republican-controlled legislatures to take scarce funding away from public schools and give it to private schools.”

Republicans gained or expanded their power at the state level last year, with Democrats failing to take control of a single legislative chamber. There have always been Republicans who oppose at least some kinds of school choice—Roger Hanshaw, the house speaker in West Virginia, voted against the ESA bill—but it became a lot harder for teachers unions and other choice opponents to kill a bill if they had to pick off 20 Republican votes rather than a handful. “This is a bipartisan issue, and it should be,” Enlow says, “but in places like West Virginia, the outcome of the election emboldened legislators.”

Building on Success

School-choice advocates held a much stronger hand this year than they’ve enjoyed in the recent past. The question now is whether they’ll be able to build on their success, or whether the pandemic created a singular set of political circumstances. History suggests that the momentum will continue. In many cases, only modest programs could muster enough support for initial passage, and that was so in some states last year. Once in place, however, programs have typically received additional funding, with eligibility requirements expanding over time. That’s been true in Florida, where bills to create or expand programs are passed nearly every year and where about 200,000 children are now enrolled in some sort of choice program.

Opponents of choice are worried about their recent losing streak continuing. Jack Schneider, an education historian at the University of Massachusetts Lowell, says that schools could reach an inflection point where such a significant number of tax dollars are diverted to homeschooling and private schools as to have a measurably damaging effect on traditional district schools, though it hasn’t yet happened in any state. “There’s a real danger of a negative feedback loop here,” he says. “You defund schools, the quality declines, and you’re able to make further arguments for, quote, choices for people, which further undermines the quality of schools.”

It’s been a full decade since Arizona launched its education savings account program. At the time, it looked like the hot new idea destined to spread rapidly to other states. That didn’t happen. In 2015, Nevada passed a law that promised nearly universal ESAs (96 percent of school-aged children in the state were eligible), but the program’s funding formula was blocked by the state supreme court, and the initiative was eventually killed when Democrats took control of the legislature.

This year, Indiana, Kentucky, Missouri, and New Hampshire all created ESA programs. Georgia expanded its special-needs scholarship program to make it act as a de facto ESA, allowing expenditures beyond tuition including tutoring, meals, transportation, speech and occupational therapy, and even uniforms. In 2020, North Carolina expanded its Opportunity Scholarship voucher program by boosting the family-income limit for eligibility and through other provisions.

“ESAs are sort of the sexy new model for private-school choice,” Wolf says.

By not limiting eligibility in any way, the West Virginia law lends credence to the time-honored complaint that such programs hand money to parents who aren’t sending their kids to public schools anyway. According to initial state estimates, the program will cost roughly $24 million to support students who leave public schools, but more than $100 million to subsidize those who have not been attending public schools.

Patricia Rucker, who chairs the West Virginia Senate’s education committee, says she “never understood why states would limit ESAs for certain types of kids and issues and geographical areas.” She says the public schools work well for most children—perhaps 95 percent of them—but there will always be students who are not well served by traditional schooling. As a parent of special-needs children herself, she has encountered moments when district schools weren’t able to be flexible enough to accommodate them properly. She often hears similar stories from other parents. “How can you ask a teacher with 30 kids to stop everything when one child is crying?” Rucker asks. “It’s really not fair to expect that.”

Whatever their benefits in terms of helping kids, ESAs turned out to have a secret ingredient that helped win political support for choice in some red states this year. Republicans dominate rural America politically, but winning rural voters over to the school-choice cause has been a struggle. Underpopulated areas typically offer families few options in terms of private schools, so there’s not exactly a groundswell of support for private-school choice among rural residents. Traditional public schools may be the only game in town. “In some of these small towns, not only is the school the major employer, but it’s the center of the community,” says Tim Benson, a policy analyst with the Heartland Institute, a conservative think tank in Illinois. “All the kids go there. Three or four generations have gone to that school. They don’t want anything to disturb it in any way.”

ESAs can introduce choice to rural areas, Rucker says, not only because parents can use the money to pay for tutoring and the like, but also because the funding that ESAs provide can potentially help build up the market for such services—a market that could include the public schools themselves acting as providers of discrete services. “There were legislators who had not supported choice in the past whose fears were greatly helped by pointing this out,” she says, “that you could use the money to pay for public-school services.”

Tough Fight in Arkansas

Ken Bragg has been trying to initiate school choice in Arkansas throughout his three terms as a state legislator, taking up a baton that other lawmakers had carried for years prior to his arrival. In 2020, he couldn’t move a bill to create a tax-credit scholarship program (in which the state essentially underwrites scholarships by giving a tax break to donors) out of committee. He was more optimistic this year.

Bragg combined $4 million for the scholarship program with $6 million in grant money that public schools could apply for. He had hoped that the 60–40 split in favor of public schools would win over skeptics, but public-school administrators still worried about the per-pupil state funding they would lose with every child who accepted a scholarship to attend private school. Support in the house looked favorable, but enough legislators were “intimidated,” Bragg says, by telephone calls from school superintendents the night before the vote that they balked, and the bill failed—by eight votes.

“They all tend to stick together,” Bragg says, “even the rural schools that wouldn’t be affected by it.”

Bragg and his colleagues persisted, coming up with a much more modest proposal that offered $2 million for tax-credit scholarships. His initial bill included an escalation clause to increase funding by 25 percent once the $4 million or $6 million limits were reached. That provision was eliminated, as were a number of accountability measures and audit requirements.

Lack of accountability is one reason choice opponents say these programs are misguided. Any time there’s a lot of public money flowing out in multiple directions, they say, there’s potential for fraud. The West Virginia ESA program, for example, allows anyone who is not a family member to receive funds for tutoring. “The way you continue to get money is using the money,” says Carol Burris, executive director of the Network for Public Education, an advocacy group that supports public schools. “It really opens the door for misuse of funds and lack of education for children.”

The stripped-down bill in Arkansas provides enough funding for scholarships covering 250 children out of a statewide student population of 480,000. Over the past couple of decades, legislators in many states have adopted this kind of incremental approach. Unable to win support for more ambitious ideas, they take baby steps, launching initiatives that amount to pilot programs. Once programs are created, they can be expanded—and generally they are. Montana provides one striking example along these lines. Last year, Republicans won the governor’s office for the first time in 16 years, which eased the way for a major expansion of that state’s tax-credit scholarship program. Previously, credits per donor were capped at $150. Now, they can go as high as $200,000.

Lawmakers also expanded tax-credit scholarship programs this year in Florida, Indiana, Iowa, Kansas, Nevada, Oklahoma, and South Dakota. Because of state policymakers’ tendency to expand existing programs, even small initiatives such as Bragg’s encounter fervent opposition, with superintendents and unions viewing them as the camel’s nose under the tent. It’s also why choice advocates take what they can get from their state legislature, hoping to lay down even a tiny foundation they can build on later.

“If it’s a good program and it’s helping kids, wouldn’t you want to expand it?” Bragg asks.

Testimonials from parents whose children took advantage of a privately funded scholarship program in Arkansas provided powerful lobbying tools for Bragg, he says. School-choice advocates frequently say that word of mouth from satisfied parents is one reason choice programs have grown over the years. Schneider, the education historian, suggests this parental support also points to something about how choice proponents are now approaching their advocacy activity. For years, data comparing private and charter schools to traditional district schools have been muddy, at best, in terms of showing anything like superior performance among alternative institutions. Rather than making comparisons related to student achievement, Schneider says, choice advocates are now mostly trying to make a different kind of sale. “There’s some evidence to believe that some charter schools do some things pretty well,” he says, “but that’s not the rhetorical basis for a reform movement. Instead, we are hearing that we need to empower families with options and we need to stop letting big government and the teachers union dictate what’s best for kids.”

Pursuing the Vision

School choice has enjoyed some measure of bipartisan support, with many prominent Democrats singing the praises of charter schools. That was about as far as they were willing to go, however. For most Republicans, charters were only the starting point. They are pushing hard to realize the vision laid out by economist Milton Friedman more than a half century ago, in which public education dollars become fully portable, following students to any type of school they might attend. Advocates haven’t come anywhere close to achieving that vision, but it is still being promoted by DeVos as well as influential conservative groups at the state level, including the American Legislative Exchange Council and think tanks that are part of the State Policy Network.

In 2021, everything broke the right way for school-choice advocates. The pandemic roused dissatisfaction among parents while limiting the lobbying activity of teachers unions and their allies. This state of affairs put many Democrats in an uncomfortable position, trapped between frustrated constituents and unions who insisted in many jurisdictions on keeping schools closed, even after the science suggested they were not high-risk locations for contracting Covid-19. Last summer, United Teachers Los Angeles put out a statement saying that as part of a “safe and equitable start of school,” the union wanted passage of Medicare for All, new taxes on wealth and millionaires, housing for homeless and low-income populations, defunding of police, and a moratorium on charter schools. This may be an extreme example of a teachers union stretching its demands past the schoolyard, but conservatives are delighted to suggest that unions have overplayed their hand. “The pandemic and closures gave legislators the backbone necessary to go ahead and cast a vote for choice programs, where before they’d be on the fence,” Benson says.

Will that momentum carry over into a post-pandemic world? Conditions will be different. With schools open everywhere, many parents won’t feel the same urgency to find alternatives. But as with so many other issues, school-choice advocates say that Covid-19 served to reveal systemic weaknesses that have been present all along. It may have been the straw that broke the back of opposition, at least within the GOP. The few red states that had long been holdouts when it came to creating choice programs, such as Arkansas and Kentucky, have at last hopped on the bandwagon.

In West Virginia’s case, they’re now leading the band. “I really do think there is a sense of inevitability about the expansion of school choice,” says Garrett Ballengee, executive director of the Cardinal Institute, a conservative West Virginia think tank that helped promote ESAs. “I don’t think we’ll see a groundswell for limiting these programs.”

Alan Greenblatt is a reporter who covers politics and government. He is coauthor of a textbook on state government, Governing States and Localities.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Greenblatt, A. (2021). School Choice Advances in the States Advocates describe “breakthrough year.” Education Next, 21(4), 18-25.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Education Next. (2021). “When Choice Really Works, It Lifts Up Everyone” State Rep. Robert Behning on Indiana’s voucher program. Education Next, 21(4), 83-84.

Hybrid Homeschooling: A Guide to the Future of Education
by Michael Q. McShane
Rowman & Littlefield, 2021, $60; 142 pages.

As reviewed by Michael B. Horn

Hybrid learning and homeschooling have become prominent models over the past school year as millions more students learned from home, whether part or full time, during the coronavirus pandemic.

Against that backdrop, Mike McShane’s new book, Hybrid Homeschooling, would seem both topical and timely.

It is both of those things, but not for reasons directly related to the pandemic or the various phenomena of blended and remote learning that became so widespread in much of the country beginning in March 2020.

McShane’s book is instead a treatment of a strand of homeschooling that has received relatively little attention: “hybrid homeschooling,” which he defines as “a school that for some part of the week educates children in a traditional brick-and-mortar building and for some other part of the week has children educated at home.”

At first glance, this concept might not seem to differ much from the enriched virtual-school models that have emerged over the past 15 years—schools in which students learn in person for a portion of the week and remotely online for another part of the week—or even schools in which students learn in person five days a week and learn at home during off hours. The big difference, McShane writes, lies in the definition of homeschooling, Hybrid homeschoolers have an education that is at least “partially controlled by parents, is partially provided by their parents, and takes place in the home for part of the school week. . . . The arrangement must meet three criteria: physical, regular, and substantial.”

The book serves ultimately as a survey-level primer on this phenomenon, which is an important one to understand because hybrid homeschooling may make homeschooling and school choice more accessible to millions of families in the years ahead. As McShane documents, prior to the pandemic, 10 percent of parents indicated a desire to home-school their children “if money or logistics” were no object. According to a February 2021 survey by EdChoice, where McShane is director of national research, 44 percent of parents would prefer a mix of home- and school-based education in the future—and, assuming hybrid homeschooling is available, parents in the original 10 percent are more likely to find a way to continue to home-school in the years ahead.

McShane leads into his primer with a brief but comprehensive summary of the research and the state of homeschooling more generally. As he documents, homeschooling has been on the rise since 1970, when “there were fewer than fifteen thousand homeschool students throughout the United States.”

Since then, he argues, it’s come “roaring back,” which is hard to dispute given that in 2016, according to the National Center for Education Statistics, 1.69 million students—or 3.3 percent of the schooling population—were home-schooled, up from 850,000 in 1999.

What McShane doesn’t mention is that the NCES estimate peaked at 1.773 million in 2012. Granted, the data are weak on the true numbers of students who are home-schooled, because of the wide variability in state policy relating to the practice—which McShane does a good job of summarizing—yet it seems clear that prior to the pandemic, the growth of homeschooling had plateaued. Although McShane shows evidence based on state-level data that the numbers may have started to rise again into 2019, homeschooling hadn’t been growing nearly as fast as its advocates like to assert.

Then again, what makes hybrid homeschooling so intriguing is its potential to make homeschooling more accessible to families by, for example, reducing costs or eliminating parents’ logistical challenges around childcare.

After reviewing studies on the effects of homeschooling and considering the views of its detractors, McShane concludes that it’s not possible to assert that homeschooling has a positive effect on academic achievement or social development, but it’s also clear that students who are home-schooled “run little risk of academic or social harm.”

The book provides a series of compelling case studies of families and educators who have made the leap into hybrid homeschooling. Each chapter begins with a story that illustrates a particular aspect of homeschooling and chronicles the experiences of parents, families, educators, and regulators. These stories serve to humanize the sometimes wonky details that McShane explores throughout.

There’s the story, for example, of a family whose children are enrolled in the Classical Christian Conservatory of Alexandria, Virginia, where the mother, Kristin Forner, is on the front lines of fighting Covid-19 as an anesthesiologist and palliative-care physician.

Forner told McShane that “we are not a typical homeschooling family,” as both she and her husband were educated in public schools and were not particularly excited about homeschooling at first. But they were drawn to the model because they wanted a classical, Christian education for their children, and there weren’t many schooling options around that fit the bill. When they realized they could afford the conservatory and that their children would learn at home two days a week, the benefits became clearer: quality time with their children, more time for creative play, greater transparency into what their children were learning, and the opportunity to teach controversial subjects on their own terms.

What emerges from the stories is an empathetic portrait of the individuals who choose to engage in hybrid homeschooling—and a realization of how diverse those individuals are.

McShane argues that families choose hybrid homeschooling for four primary reasons: the gift of time, personalization, being involved together in education, and mental health.

As for educators, they choose to participate in hybrid models for many of the same reasons, but also to create a stronger community than they could in a public school. That said, McShane describes the drawbacks to teaching in a hybrid homeschool environment—compensation chief among them—that for now will likely limit the numbers of educators who can commit to such schools.

One of the most interesting chapters provides a summary of policy on homeschooling. The chapter covers the various ways in which states treat homeschoolers and the challenges, inherent in models that aren’t built around seat time, of circumventing time-based Carnegie Unit requirements. It also highlights the opportunities to innovate that hybrid homeschooling affords public-school leaders when they choose to participate rather than fight those who opt for homeschooling. The public-school educators McShane chronicles come across as cage busters redefining the educational experience in positive ways. Kentucky’s superintendent of the year, Brian Creasman, from Fleming County Schools, for example, seized the opportunity to enroll hybrid homeschoolers in mastery-based programs and at last take advantage of the state regulations that waive the Carnegie Unit—regulations that were “staring at us in the face.”

Where the book most misses the mark is in the innovation chapter, which feels forced and a bit too academic. The discussion of design thinking in hybrid homeschooling isn’t so much wrong as it is stilted and too brief to resonate. And the use of Everett Rogers’s diffusion-of-innovation curve—a model that attempts to show the rate at which new ideas and technologies spread—feels premature at best. As a whole, the chapter reads like a needless add-on to an otherwise robust discussion of the growing hybrid-homeschooling phenomenon.

I would have preferred to see McShane explore how the funders that are looking for ways to reinvent schooling through entrepreneurship and innovation might exploit—or perhaps already are exploiting—hybrid homeschooling to help produce larger-scale changes in the aftermath of the pandemic. For funders looking for ideas, there are plenty of inspiring innovators and entrepreneurs in this book who may hold the keys to a bigger rethinking of how education has to work in this country. McShane’s volume is a great place to start.

Michael Horn is an executive editor of Education Next, co-founder of and a distinguished fellow at the Clayton Christensen Institute for Disruptive Innovation, and a senior strategist at Guild Education.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Horn, M. (2021). A Robust and Timely Discussion of a New Kind of Homeschooling: Hybrid approach combines at-home learning with school attendance. Education Next, 21(4), 74-75.

The Quick Fix: Why Fad Psychology Can’t Cure Our Social Ills
by Jesse Singal
Farrar, Straus and Giroux, 2021, $28; 352 pages.

As reviewed by Jay P. Greene

Jesse Singal’s new book, The Quick Fix, is an impressive display of social-science journalism. Singal manages to describe complicated and technical issues accurately and with nuance, a feat rarely achieved by researchers, let alone journalists. The book focuses on six niches of social-science study that over the past few decades have had widespread influence on policies and practices beyond the narrow confines of academia. He takes on the self-esteem movement, the “superpredator” theory in criminology, the use of “power posing,” positive psychology, grit, and the implicit association test for unconscious racial bias.

It would be too strong to say that Singal “debunks” the findings that drew attention to these six topics, but he does critique them and is particularly skeptical of claims that interventions or policies generated from these areas of study have the potential to significantly alter outcomes in real-world settings. He acknowledges the extent to which research supports such claims but points out the limited quality of that research, asserting that it is often so contingent on specific contexts that it does not apply more broadly.

For example, in the chapter on self-esteem, Singal discusses Carol Dweck’s ideas about growth mindset, the belief that academic performance can be altered through personal effort. He acknowledges that a large randomized experiment published in Nature by Dweck and two dozen co-authors found that a “mindset intervention . . . does appear to have some effects. . . . If this research holds, it could be argued that mindset interventions do offer a minor but legitimate boost to a subset of otherwise academically vulnerable students—a boost that is at least somewhat related to self-esteem.” His critique is not that self-esteem ideas are fundamentally mistaken, but that they have been grossly oversold and misapplied in contexts well beyond what can be supported by rigorous research.

Singal similarly concedes that a positive-psychology intervention, the Penn Resilience Program, or PRP, has had positive results: a study “found that while the PRP did appear to reduce depressive symptoms among students exposed to it, those reductions were small, statistically speaking.” In the chapter on grit, Singal notes that “both conscientiousness and grit do appear to be correlated with school performance—somewhat.” And in the chapter on the implicit association test, or IAT, to measure unconscious racial bias, Singal writes “there does appear to be a statistically significant correlation between IAT scores and behavior observed in studies; it’s just so small as to likely be meaningless in the real world.” Singal expresses plenty of reservations about how robust all of these research findings are, but he does not accuse their proponents of manufacturing false results. His real concern is about the use of these findings to attempt to shape and improve individual behavior in any meaningful way, especially on a mass scale.

If the main problem that Singal is identifying is one of overhyping and misapplying social-science research, it is unclear how much of the responsibility lies with researchers or others. Singal is inclined to place a fair amount of the blame on the researchers, who are drawn by the attention and resources that overhyped research can generate. This view does not seem entirely fair, given the extent to which politicians, foundations, reporters, and the general public are willing to lavish attention and resources on whichever researchers will confidently claim that they have consulted with the oracle of social science and divined guidance for how we should structure policy and live our lives. Education reform has especially suffered from this cultlike devotion to claims generated by social science, ignoring the glaring weakness of most social-science research while dismissing the useful insights of wisdom and experience.

The corruptibility of researchers is a problem, but that’s only part of the story—especially because in several chapters we learn that the researchers recanted their findings or otherwise attempted to temper misuse of their work. For example, in the chapter critiquing the 1990s-era claim that the country was facing an alarming rise in superpredator criminals, Singal notes that the main proponents of that theory later abandoned their claims, even authoring a U.S. Supreme Court amicus brief to rebut them. In the chapter on “power posing” as a strategy for advancing women’s careers, Singal reveals that one of the authors of the original study later posted a statement on her faculty website, underlined and in bold, saying, “I do not believe that ‘power pose’ effects are real.” In the chapter on enhancing grit to improve student success, Singal concedes that Angela Duckworth, who developed the concept, tried but failed to contain the misuse of her findings: “To her credit, Duckworth has been significantly more candid and transparent than other researchers who have found their ideas under scrutiny, and she has been generally open about the limitations of the research. . . . Duckworth has expressed frustration at the fact that she had, to a certain extent, lost control of the grit narrative.”

There is a larger story here, which Singal does not fully develop, about why we as a society invest an unreasonable amount of authority in social science. He hints at this in his concluding chapters about the implausibility that priming, nudges, and other subtle interventions have large and predictable effects on human behavior, given how complicated and deeply rooted our motivations likely are. But he doesn’t seem to see the problem as inherent in our overreliance on social science as a guide for life. He seems to think that if only researchers preregister their studies and exercise greater care, we can avoid these abuses. He favorably quotes “the champion of replication and transparency in psychological science,” Brian Nosek, who writes that reformers have “irrevocably altered the norms and accelerated adoption of behaviors like preregistration and data sharing. Thanks to them, psychological science is in a different place today than it was in 2011. Psychology in 2031 is going to be amazing.” Singal’s cautious agreement with this optimism strikes me as naïve, especially given all of the abuses he so carefully documents in his book.

Singal accurately captures the nuance and detailed shortcomings of research but seems to struggle in discussing the bigger picture with similar skepticism. The heart of the book lies in the chapters, some of which Singal published previously as standalone articles, about the weakness and misuse of particular research claims. In cobbling this material together into a book, Singal may not have given priority to identifying the unifying themes of his chapters. A plausible conclusion he could have drawn is that while social science can shed light on human behavior and even help guide it, it is not the only or necessarily the most reliable source of wisdom on how to live our lives. That’s also what the great religious traditions and their deference to experience and past practice are about. The Enlightenment values that gave rise to the social sciences can supplement the ancient teachings but need not replace them. Given how careful Singal is, perhaps he did not want to make an overly strong argument about unifying themes for fear of extending beyond his evidence, which is reasonable but makes the volume as a whole a little less compelling than it might have been.

Jay P. Greene is a senior research fellow at the Heritage Foundation.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Greene, J. (2021). The Fix Is In: A skeptical look at “the oracle of social science.” Education Next, 21(4), 72-73.

This spring, the Biden administration announced it would seek public comment on student race and school climate, which was roundly viewed as a precursor to restoring an Obama-era directive to reduce racial disparities in discipline practices. Those guidelines, which were rescinded by former Secretary Betsy DeVos, have been variously described as a critical means of protecting students’ civil rights and a dangerous overreach by the federal government that prevented schools from keeping students safe.

At issue is the school-to-prison pipeline—a term often used to describe the connection between exclusionary punishments like suspensions and expulsions and involvement in the criminal justice system. Black and Hispanic students are far more likely than white students to be suspended or expelled, and Black and Hispanic Americans are disproportionately represented in the nation’s prisons.

Is there a causal link between experiencing strict school discipline as a student and being arrested or incarcerated as an adult? Research shows that completing more years of school reduces subsequent criminal activity, as does enrolling in a higher-quality school and graduating from high school. Yet there is little evidence on the mechanisms by which a school can have a long-run influence on criminal activity.

To address this, we examine middle-school suspension rates in Charlotte-Mecklenburg Schools, where a large and sudden change in school-enrollment boundary lines resulted in half of all students changing schools in a single year. We estimate a school’s disciplinary strictness based on its suspension rates before the change and use this natural experiment to identify how attending a stricter school influences criminal activity in adulthood.

Our analysis shows that young adolescents who attend schools with high suspension rates are substantially more likely to be arrested and jailed as adults. These long-term, negative impacts in adulthood apply across a school’s population, not just to students who are suspended during their school years.

Students assigned to middle schools that are one standard deviation stricter—equivalent to being at the 84th percentile of strictness versus the mean—are 3.2 percentage points more likely to have ever been arrested and 2.5 percentage points more likely to have ever been incarcerated as adults. They also are 1.7 percentage points more likely to drop out of high school and 2.4 percentage points less likely to attend a 4-year college. These impacts are much larger for Black and Hispanic male students.

We also find that principals, who have considerable discretion in meting out school discipline, are the major driver of differences in the number of suspensions from one school to the next. In tracking the movements of principals across schools, we see that principals’ effects on suspensions in one school predicts their effects on suspensions at another.

Our findings show that early censure of school misbehavior causes increases in adult crime—that there is, in fact, a school-to-prison pipeline. Further, we find that the negative impacts from strict disciplinary environments are largest for minorities and males, suggesting that suspension policies expand preexisting gaps in educational attainment and incarceration. We do see some limited evidence of positive effects on the academic achievement of white male students, which highlights the potential to increase the achievement of some subgroups by removing disruptive peers. However, any effort to maintain safe and orderly school climates must take into account the clear and negative consequences of exclusionary discipline practices for young students, and especially young students of color, which last well into adulthood.

Desegregation in Charlotte-Mecklenburg

For decades, school enrollment and bus routes in the Charlotte-Mecklenburg school district were designed to achieve racial integration. The busing plan was ordered by a state judge and upheld by a unanimous U.S. Supreme Court decision in 1971, after the Swann family, who were Black, sued to reassign their 6-year-old son from an all-Black school to an integrated school closer to their home. The landmark Swann v. Charlotte-Mecklenburg Board of Education decision required the district to reassign students to new schools to balance their racial composition and influenced similar busing programs nationwide.

It was another parent lawsuit that ultimately ended mandatory busing and redrew school-zone boundaries in Charlotte-Mecklenburg again. In 1997, a white parent named William Capacchione sued the district because he believed his child was denied entrance to a magnet program based on race. This case led to a series of court battles that ended with a 2001 ruling by the Fourth Circuit Court of Appeals, which upheld an earlier state court order to stop using race in school assignments. The district had “eliminated, to the extent practicable, the vestiges of past discrimination in the traditional areas of school operations,” the court ruled.

As a result, over the summer of 2002, Charlotte-Mecklenburg Schools redrew school-attendance boundaries based only on classroom capacity and the geographical concentration of students around a building. This mechanical redistricting process rarely took advantage of environmental features such as streams and major roads, and was controversial because it often bisected existing neighborhoods. About half of all students changed schools between 2001–02 and 2002–03.

For some students, that meant going from a school where suspensions were relatively rare to a school with a different approach to discipline (see Figure 1 for an example). While all schools are held to the district’s code of conduct and guidance by the North Carolina Department of Education, different schools have higher or lower rates of suspensions and expulsions.

Many discussions about the school-to-prison pipeline center on the possibility that students experiencing suspension differ from other students in ways that could explain their higher levels of involvement in the criminal justice system later in life. The sudden reassignment of half of all Charlotte-Mecklenburg Schools students in the summer of 2002 meant that students who live in the same neighborhoods and previously attended the same school could be assigned to attend very different schools in the fall. This creates a natural experiment to identify the impact of a school’s approach to discipline, which we use to identify a school’s influence on a range of outcomes in adulthood, including educational attainment and criminal activity.

A Natural Experiment

Our analysis focuses on 26,246 middle-school students who experienced the boundary change because they were enrolled in a Charlotte-Mecklenburg school in both the 2001–02 and 2002–03 school years. We use district administrative records that track students from 1998–99 through 2010–11. The data include information on student demographics, test scores for grades 3 through 8 in math and reading, and annual counts of days suspended. Overall, 48 percent of students are Black, 39 percent are white, and 8 percent are Hispanic. On average, 23 percent of students are suspended at least once per school year, and the average suspension duration is 2.3 days.

District records also include each student’s home address in every year, which we use to determine individual school assignments under the busing and post-busing regimes. To define residential neighborhood, we use the 371 block groups from the 2000 Census that include at least one Charlotte-Mecklenburg student. We use address records to assign students to these neighborhoods and to middle-school zones for both the pre- and post-2002 boundaries.

To look at long-term outcomes, we first match district records to Mecklenburg County administrative data for all adult arrests and incarcerations from 1998 through 2013. Sixth graders in 2002–03 who progress through school as expected would enter 12th grade in the 2008–09 school year. Because our data on crime extends through 2013, we use two main measures of criminal activity: whether the individual was arrested between the ages of 16 and 21 and whether the individual was incarcerated between the ages of 16 and 21. This allows us to observe crime outcomes for all students who were in grades 6 through 8 in 2002–03.

We also track college-going data from the National Student Clearinghouse. That includes records for every student of college age who had ever attended a Charlotte-Mecklenburg school, including students who transfer to other districts or private schools or who drop out of school altogether. Because our data end in the summer of 2009, we cannot examine longer-run measures of educational attainment such as degree completion. Thus we focus on 7th- and 8th-grade students and measure whether they attended college within 12 months of the fall after their expected high-school graduation date.

Approximately 12 percent of our sample eventually drops out of high school, while 23 percent attend a 4-year college within 12 months of their expected graduation date. Between the ages of 16 and 21 years old, 19 percent are arrested at least once and 13 percent are incarcerated at least once. While well above the national averages in terms of suspensions and crime, Charlotte-Mecklenburg Schools is fairly representative of large, urban school districts in the Southern United States.

The Impacts of a Strict School

To quantify each school’s strictness, we use the same basic method commonly used to estimate individual teachers’ value-added to student test scores. We examine the number of days students are suspended both in and out of school to calculate strictness, while controlling for student characteristics such as test scores, race, gender, special-education status, and limited-English proficiency status, among others. This produces an estimate of each school’s predicted impact on suspensions based on how frequently it had suspended students in previous years.

We find that an increase of one standard deviation in school strictness expands the likelihood of being suspended in a given school year by 1.7 percentage points, or 7 percent. The average annual number of days suspended per year grows by 0.38, a 16 percent increase.

How does this affect student outcomes later in life? We look at criminal activity throughout Mecklenburg County and find that students who attend a stricter school are more likely to be arrested and incarcerated between the ages of 16 and 21.

Students assigned a school that is one standard deviation more strict are 17 percent more likely to be arrested and 20 percent more likely to go to jail, based on our estimated increases of about 3.2 percentage points for arrests and 2.5 percentage points for incarcerations. In looking at what types of crimes are involved, we find that school strictness increases later involvement in crimes related to illegal drugs, fraud, arson, and burglary, but not in serious violent crimes like murder, manslaughter, rape, robbery, and aggravated assault.

We also look at the impact on student academic performance and educational attainment and find no evidence that school strictness affects overall achievement. Because we measure the net effect across all students in a school, this may be due to a balancing of two opposing forces: negative effects of lost instructional time for those students who were suspended and positive effects of reduced number of disruptive peers in the classroom for students who were not.

However, we do find evidence that suspensions negatively affect educational attainment. A one standard deviation stricter school increases the likelihood that a student drops out of high school by 1.7 percentage points, or 15 percent, and decreases the likelihood of attending a 4-year college by 2.4 percentage points, or 11 percent.

We then compare effects by race and find outsized impacts for Black and Hispanic students. Being assigned to a school that is one standard deviation more strict increases the average number of days suspended each school year by 0.43 for Black and Hispanic students compared to 0.21 days for non-minority students. That number is even larger for Black and Hispanic males, who are suspended 0.82 more days each year, on average—more than three times the effect for non-minority males.

As adults, Black and Hispanic students assigned to stricter schools are more likely to be arrested and incarcerated than their non-minority classmates. A one standard deviation stricter school increases the likelihood of being arrested by 3.9 percentage points for Black and Hispanic students compared to 2.7 percentage points for non-minority students (see Figure 2). The effect on incarceration in adulthood is 3.1 percentage points for Black and Hispanic students compared to 1.9 percentage points for non-minority students. Negative effects are especially pronounced among Black and Hispanic male students, who are 5.4 percentage points more likely to be arrested and 4.4 percentage points more likely to be incarcerated as adults.

While the average impact of a strict school across all students is negative, we do find small positive impacts on academic achievement for white male students. White male students who are assigned a school that is one standard deviation stricter score about 0.06 standard deviations higher on middle-school math and reading tests. This is consistent with prior studies that show positive short-run academic benefits to some students from removing disruptive peers from the classroom. However, we find no long-run impact on educational attainment for white males, who also experience substantial increases in adult arrests and incarcerations of 4.9 and 3.7 percentage points, respectively.

What Drives School Strictness?

We investigate three potential factors driving differences in school strictness. First, we look at the potential role of school peers. Prior research has found that peers are important contributors to students’ educational experiences, but we find little relationship between school strictness and peer characteristics, suggesting that our results are not driven by changes in peer composition.

Second, we test our main school strictness results alongside two other measures of school effects, based on student-achievement gains and teacher turnover. We find that disciplinary strictness is the only predictor of students’ later involvement in the criminal-justice system. This serves as further evidence that our results are driven by school effects on suspensions rather than other aspects of school quality or simply the disruption caused by sudden changes in enrollment patterns.

Finally, we turn to the role of school leaders, who have considerable discretion in how they handle disciplinary action. Principals have the authority to set parental meetings, after-school interventions, and in-school suspensions. Even the process for short-term out-of-school suspension is almost completely up to school leaders in Charlotte-Mecklenburg; the superintendent’s approval is only required for long-term suspensions of 11 days or more. We look at the movements of principals across schools and find that when a principal who has been strict in prior years switches into a new school, suspensions in the new school increase. This suggests that school effects on suspensions are driven by leadership decisions.

These findings echo the public’s anecdotal understanding of the strong role that principals play in establishing school climate and discipline. Consider Charlotte-Mecklenburg’s recent approach to limiting suspensions among young elementary-school students. Suspending very young students has come under public criticism across the country, with policymakers in New York City, Colorado, and New Jersey weighing moratoriums on the practice. The Charlotte-Mecklenburg school board considered a moratorium but opted to limit principal discretion instead and now requires the superintendent’s approval. In 2017–18, the first year of the new policy, the number of suspensions for K–2 students fell by 90 percent.

Implications

Misbehaving peers can have strong negative impacts on other students in the classroom, and all students need a safe, predictable, and peaceful environment to thrive. But our findings show that the school-to-prison pipeline is real and poses substantial risks for students in strict school environments. On average, students who attend middle schools that rely heavily on suspensions are at greater risk of being arrested and incarcerated as young adults and less likely to graduate from high school and go to college. Further, these effects are most pronounced for Black and Hispanic males, who are dramatically underrepresented among college graduates and overrepresented in the nation’s prison system.

This raises a critical question for policymakers and educators who enforce strict school discipline: for whom are our schools safe? And it establishes an opportunity for principals and organizations that support school leadership to weigh the tradeoffs between strict discipline practices and longer-term outcomes for students. As the nation continues to grapple with questions about racial equity and police reform, the contributing causal role that school-discipline practices play in raising the risk of criminality in adulthood cannot be ignored.

Andrew Bacher-Hicks is assistant professor of education at Boston University. Stephen B. Billings is associate professor at the University of Colorado Boulder. David J. Deming is professor at the Harvard Kennedy School and Harvard Graduate School of Education.

For more, please see “The Top 20 Education Next Articles of 2023.”

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Bacher-Hicks, A., Billings, S., Deming, D. (2021). Proving the School-to-Prison Pipeline: Stricter middle schools raise the risk of adult arrest. Education Next, 21(4), 52-57.

Anyone who uses a smartphone or shops online has had their habits tracked, click by telling click. Big companies comb through that data to find patterns in human behavior and to understand, anticipate, and offer up goods and services we are most likely to purchase. Through predictive analytics, they identify trends and forecast our future choices.

This high-tech data crunch has become increasingly common in higher education, too. Colleges and universities are facing mounting pressure to raise completion rates and have embraced predictive analytics to identify which students are at risk of failing courses or dropping out. An estimated 1,400 institutions nationwide have invested in predictive analytics technology, with spending estimated in the hundreds of millions of dollars. Colleges and universities use these analyses to identify at-risk students who may benefit from additional support.

How accurate and stable are those predictions? In most cases, college researchers and administrators don’t know. Most machine-learning models used in higher education are proprietary and operated by private companies that provide little, if any, transparency about the underlying data structure or modeling they use. Different models could vary substantially in their accuracy, and the use of predictive analytics could lead institutions to intervene disproportionately with students from underrepresented backgrounds. It’s also not clear whether these expensive services and complex models do a better job of identifying at-risk students than simpler statistical techniques that take significantly less time and expertise to implement and that institutions therefore may be able to implement on their own.

We put six predictive models to the test to gain a fuller understanding of how they work and the tradeoffs between simpler versus more complex approaches. We also investigated different approaches to sample and variable construction to see how data selection and model selection work together. Our study uses detailed student data from the Virginia Community College System to investigate whether models accurately predict whether a student does or does not graduate with a college-level credential within six years of entering school. Using these same models, we also examine, for a given student, whether their predicted risk of dropping out is the same from one model to the next.

We find that complex machine-learning models aren’t necessarily better at predicting students’ future outcomes than simpler statistical techniques. The decisions analysts make about how they structure a data sample and which predictors they include are more critical to model performance. For instance, models perform better when we include predictors that measure students’ academic performance during a specific semester or term than when we include only cumulative measures of performance.

Perhaps most importantly, we find that the dropout risk predictions assigned to a given student are not stable across models. Where students fall in the distribution of predicted risk varies meaningfully from one model to the next. This volatility is particularly pronounced when we use more complex machine-learning models to generate predictions, as those approaches are more sensitive to which predictors are included in the models and which students and institutions are included in the sample. For example, among the students considered at high risk of dropping out based on predictions generated from a linear regression model, just 60 percent were also deemed high risk according to a popular machine-learning prediction algorithm called XGBoost.

Finally, we show that students from underrepresented groups, such as Black students, have a lower predicted probability of graduating than students from other groups. While this could potentially lead underrepresented students to receive additional support, the experience of being labeled “at risk” could exacerbate concerns these students may already have about their potential for success in college. Addressing this potential hazard is not as straightforward as just removing demographic predictors from predictive models, which we find has no effect on model performance. The most influential predictors of college completion, such as semester-level GPA and credits earned, are correlated with group membership, owing to longstanding inequities in the educational system.

Our findings raise important questions for institutions and policymakers about the value of investments in predictive analytics. Are institutions getting sufficient value from private analytics firms that market the sophisticated models? Even more fundamentally, since a primary goal of predictive analytics is to target individual students with interventions to keep them on track to completion, how reliable are these methods if a student’s predicted risk is sensitive to the particular model used? Colleges and universities should critically evaluate what they are getting for their investment in predictive analytics, which one estimate puts at $300,000 per institution per year, as well as the equity implications of labeling large proportions of underrepresented students as being “at risk.”

Who Goes on to Graduate?

The predictive analytics boom has coincided with growing pressure on colleges and universities to raise completion rates. About two thirds of U.S. states now use performance-based funding, which bases a school’s annual state aid amount on the outcomes of its students, not the size of its enrollment. Meanwhile, students are borrowing record amounts of money to fund their postsecondary education, and loan default rates are highest among students who drop out before finishing their degree.

Institutions have turned to predictive analytics to determine which students are most at risk of dropping out and to more efficiently steer advising and other interventions toward students identified as needing help. Such resources are relatively scarce after a decade-long decline in higher education funding—particularly at the non-elite, broad-access colleges and universities where most lower-income and underrepresented students enroll. If predictive analytics perform as intended, institutions can more effectively and efficiently target resources for the students who need them most.

For that to work, predictions must be accurate. We tested six models to see which do a better job of assessing student risk and which sorts of decisions we could make along the way to make models more or less accurate. These include three models that are commonly used by researchers due to their ease of implementation and interpretation: Ordinary Least Squares, Logistic Regression, and Cox Proportional Hazard Survival Analysis. We also tested three more complex and computationally demanding models: Random Forest and XGBoost, which both use decision-tree learning as the building block to predict outcomes, and Recurrent Neural Networks, which applies layers of intricate patterns overtop one another to model complex relationships between data inputs and outcomes.

We test these models using detailed data for 331,254 community college students in Virginia, all of whom initially enrolled between summer 2007 and summer 2012 as degree-seeking, non-dual-enrollment students. We focus on predicting “graduation,” which we define as the probability that a student completes any college-level credential within six years. Some 34 percent of students in our sample graduated within six years, either from a community college or a four-year school. This rich dataset includes hundreds of potential predictors, including student characteristics, academic history and performance, and financial aid information, among others.

We observe each student’s information for the entire six-year window after the term when they initially enroll. While in all of our models we use the full six years of data to construct the outcome measure, we test two different approaches to constructing model predictors.

Choosing the Student Sample. First, we construct a sample using all information from initial enrollment through one of two concluding events: either the term when the student first earned a college-level credential or the end of the six-year window, whichever comes first. As an alternative approach, we constructed a randomly truncated sample of students so the distribution of enrollment spells in the model-building sample matches the distribution for currently enrolled students.

Choosing Predictor Variables. Second, we investigate how using more and less complex predictors affects model performance. First, we test models that use simple data points like race and ethnicity, parental education, cumulative GPA, and the number of courses completed. Then, we use those same models but supplement the simple variables with more complex predictors, such as measures of students’ enrollment at institutions outside the Virginia community college system.

We then test how model performance is affected by the inclusion of predictors whose values vary over time. We include both simple term-specific predictors like GPA or credits attempted and separately test the inclusion of complex term-specific predictors, like how academically demanding students’ courses are in a given semester and the trajectory of students’ academic performance over time. Our overall aim is to compare how model accuracy varies based on our choices of sample and predictor construction and modeling method.

Our primary measure of model accuracy is the c-statistic, also known as concordance value. This “goodness of fit” measure determines whether a model is, in fact, predictive of the outcome of interest. In our study, the c-statistic assesses whether a randomly selected student who actually graduated has a higher predicted score than a randomly selected student who did not. A c-statistic of 0.5 indicates that the prediction is no better than random chance, while a value of 1.0 indicates that the model perfectly identifies students who will graduate. The higher the score, the better; often, a c-statistic value of 0.8 or above is used to identify a well-performing model.

Predictions versus Reality

Our analysis finds that it is possible to achieve strong model performance with a simple modeling approach, such as Ordinary Least Squares regression. However, doing so requires thoughtful approaches to sample and predictor construction. Alternatively, it is possible to achieve strong performance with basic predictors, but doing so requires more sophisticated modeling approaches.

Using the relatively simple Ordinary Least Squares model as a baseline, we look closely at the improved accuracy of predictions made using more or less complex sampling and data selection (see Figure 1). Applying Ordinary Least Squares to the entire sample results in a c-statistic value of 0.76. That grows to 0.81 when using the sample that is “truncated” to be more representative of currently enrolled students with respect to their time enrolled in college and 0.88 when also including more comprehensive predictors.

We apply the same truncated sample and set of comprehensive predictors to five additional modeling approaches to document the gains in accuracy from using more complex prediction algorithms (see Figure 2). The c-statistics are similar across the six models, ranging from 0.88 for the Ordinary Least Squares model to 0.90 for the more complex, tree-based XGBoost model. These fairly high values are not particularly surprising, given both the large sample size and detailed information we observe about students in the sample, but the fact that a basic model has nearly as high a score as a more complex model is notable.

To put this result in context, Figure 3 shows the number of students at a prototypical community college expected to be assigned a correct prediction across the different models we tested. Out of 33,000 students, Ordinary Least Squares would correctly predict the graduation outcomes of 27,119, or 82 percent. Three models perform a bit better: Logistic Regression, XGBoost, and Recurrent Neural Networks. XGBoost is the best-performing model and would correctly predict graduation outcomes for 681 more students than Ordinary Least Squares, a 2.1 percent gain in accuracy. The most computationally intensive model, Recurrent Neural Networks, presents the smallest gain over Ordinary Least Squares and would correctly predict outcomes for an additional 287 students.

A Question of Risk

One of the main purposes of predictive analytics is to identify at-risk students who may benefit from additional intervention. In predicting the likelihood of graduation for all students in our sample, each model also generates for each student a “risk ranking”—for example, that the student is at the 90th percentile among all students in terms of the probability of earning a degree. The higher the percentile value, the more likely a student is predicted to graduate relative to their peers. Students assigned lower predicted probabilities are therefore deemed at higher risk of dropout.

Colleges and universities may vary in which students they target for proactive outreach and intervention along the distribution of predicted risk. Some colleges may take the approach of targeting students at highest risk, while others may focus on students with more moderate predicted risk if they consider those students more responsive to intervention.

This raises a question about the relative accuracy of risk rankings. Regardless of where along the risk spectrum institutions choose to focus their attention, a desirable property is that different modeling strategies assign students similar risk rankings. How consistent are these rankings in practice from model to model?

We pair models together to compare where a student’s relative risk ranking falls. We divide the risk distribution into 10 equal groups, or deciles, and observe the extent to which students are assigned to different deciles across the two modeling approaches. For instance, among students whose predicted values from the Ordinary Least Squares model place them in the bottom 10 percent in terms of likelihood of graduation, we examine what percentage of those students are also assigned to the bottom 10 percent in the two other simple models. Some 86 percent of students in the bottom 10 percent based on Ordinary Least Squares are also in the bottom 10 percent from Logistic Regression. The same rate of consistency occurs between Logistic Regression and the third conventional model, Cox Proportional Hazard Survival Analysis.

However, discrepancies are more pronounced across all other model pairs. For example, half of students in the bottom 10 percent based on predictions from the tree-based Random Forest model are assigned to a different decile by the Recurrent Neural Network algorithm. We find even larger inconsistencies across models when considering students with lower predicted levels of risk. For example, across all model pairs, fewer than 70 percent of students assigned a risk rating in the third decile by one model were in that same decile by the other model.

If resource constraints prohibit colleges from intervening with all students predicted not to graduate, this instability in risk rankings means that the particular method of prediction used can significantly impact which students are targeted for additional outreach and support.

More Predicted Risk for Underrepresented Students

One common concern is that using predictive modeling in education may reinforce bias against subgroups with historically lower levels of academic achievement or attainment. In our sample, many historically disadvantaged groups—including Black and Hispanic students, Pell recipients, first-generation college goers, and older students—have significantly lower graduation rates than their more advantaged peers. At a conceptual level, including these types of demographic characteristics in predictive models could result in these subgroups being assigned a lower predicted probability of graduation, even when members of those groups are academically and otherwise identical to students from more privileged backgrounds.

This would likely result in students from disadvantaged groups being more likely to be identified as at-risk and provided additional supports. To be sure, if available interventions are effective, such identification could be a good thing. However, being flagged as “at risk” could be detrimental if it compromises students’ sense of belonging on campus, which is an important contributor to college persistence and success.

We examine how excluding demographic predictors affects model performance and student-specific risk rankings. It’s an intuitive approach to addressing this concern: without including demographics in predictive models, researchers and administrators might assume that students’ predicted outcomes would not vary by race, age, gender, or income. Furthermore, some state higher education systems and individual colleges and universities face legal obstacles or political opposition to including certain demographic characteristics in predictive models.

We compare the c-statistic values of models that include demographic characteristics to models that exclude this information and find their accuracy virtually unchanged. This occurs because many of the non-demographic predictors that remain in the model, such as cumulative GPA, are highly correlated with both student demographic characteristics and the probability of graduation. For example, Black students have a cumulative GPA of 2.13, on average, a half-grade lower than the 2.63 average of non-Black students. Even when race is not incorporated into prediction models explicitly, the results still reflect the factors that drive race-based differences in educational attainment. Institutions are therefore more likely to identify students of color as being at risk when using predictive analytics.

Questions to Consider

We believe there is a broad set of questions that are important for colleges and universities to consider when making decisions about using predictive analytics.

First, do the benefits of predictive modeling outweigh the costs? A back-of-the-envelope calculation can put this cost-benefit question in context. We find that using a more advanced prediction method like XGBoost would correctly identify graduation outcomes for an additional 681 students at a prototypical large community college that enrolls 33,000, compared to Ordinary Least Squares. If the cost to purchase proprietary predictive modeling services is estimated at $300,000, this implies an average cost per additional correctly identified at-risk student of $4,688. What other ways could institutions spend that money to boost completion rates? Are the potential benefits from sophisticated predictive analytics likely to be greater than those other investments?

Second, the instability in students’ relative risk ranking across models calls into question how strongly colleges should be relying on the “dropout risk” designation. In practical terms, this instability means that a student who is at substantial risk of dropping out may not get targeted for intervention, or a student who is predicted to have a higher probability of completion may get support they do not need. We encourage colleges and universities to advocate for greater transparency from their predictive analytics providers about the sensitivity of students’ relative risk rankings to different modeling choices. Choosing which prediction model to use may therefore depend, in part, on multiple factors, such as the intervention a college is developing, which set of students the college wants to target, and how closely the profile of students identified by a set of candidate prediction models comes to the target profile of students for intervention.

Third, students from underrepresented groups are likely to be ranked as less likely to graduate, regardless of whether demographic measures are included in the models. On the positive side, this could lead to institutions investing greater resources to improve outcomes for traditionally disadvantaged populations. But there is also the potential that outreach to underrepresented students could have unintended consequences, such as reinforcing anxieties students have about whether they belong at the institution. Colleges should weigh these considerations carefully.

Fourth, we see potential hazards regarding privacy and whether students are aware of and would consent to these uses of data. For instance, researchers at the University of Arizona constructed an experiment using machine learning to predict whether students dropped out before earning a degree with up to 90 percent accuracy based on their levels of campus engagement within the first few weeks of school. The source data: student ID swipes, which tracked their movements across campus—when they left their dorm rooms, checked out library books, or even bought a coffee. While this sort of data-gathering could have the potential to improve model accuracy, it also raises important privacy questions that higher education administrators need to actively consider.

A final question is whether predictive analytics is actually enabling more effective identification and support for at-risk students. Few studies to date have rigorously examined the effects of predictive analytics on college academic performance, persistence, and degree attainment; the few that do find limited evidence of positive effects.

However, it is easy to conflate the accuracy of predictive modeling with the efficacy of interventions built around its use. It could be that predictive models convey limited information about students, but it also may be the case that the resulting interventions were ineffective. While predictive analytics is intended to provide answers, we see further questions ahead.

Kelli A. Bird is research assistant professor at the University of Virginia, where Benjamin L. Castleman is Newton and Rita Meyers Associate Professor in the Economics of Education and Yifeng Song is data scientist. Zachary Mabel is associate policy research scientist at the College Board. This article is adapted from a study titled “Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education.”

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Bird, K., Castleman, B., Mabel, Z., and Song, Y. (2021). Big Data on Campus: Putting predictive analytics to the test. Education Next, 21(4), 58-64.

The Covid-19 pandemic that prompted a nationwide shutdown of schools and a shift to online instruction in spring 2020 also prompted a wave of articles calling this instructional change a “natural experiment” that could be used to study the effects of online education. Yet the pandemic disrupted so many aspects of children’s academic, social, emotional, and economic lives that its broad scope poses serious challenges to isolating the causal impact of any specific change, such as the switch to remote instruction.

Educators and policymakers should proceed with caution when interpreting studies that attempt to identify such specific effects. Instead, researchers should focus on helping education leaders understand the overall impact of the pandemic on students, putting particular emphasis on discovering which groups have suffered the worst effects. The best evidence to date indicates that Covid-19 has had a substantial negative impact overall and has disproportionately harmed the learning of disadvantaged students. Continued research in this direction could provide a sharper picture of which students have faced the most severe challenges under Covid-19, pointing the way toward how best to allocate resources to address learning losses.

Econometric Challenges

In econometrics, an “instrument” is a variable that has a direct impact on the probability that an individual is treated by a policy of interest. For example, some people have suggested that the pandemic could serve as an econometric instrument to study the effects of online learning, since the pandemic dramatically increased the number of students learning virtually in 2020. Major concerns arise, though, in using the pandemic as an instrument to study the impacts of such a policy change. These concerns relate to the fact that a rigorous causal study must have both internal and external validity. Internal validity requires that econometric analyses capture the true causal impact of only the policy change of interest—in this case, the shift from in-person to online instruction—rather than the possible effects of other contemporaneous changes. External validity requires that the estimated effects of a policy change induced by the pandemic would accurately predict the effects of similar treatments in other contexts, such as a typical school year. The nature of the pandemic presents serious challenges to both the internal and the external validity of most research designs, including the use of the pandemic as an instrument.

Challenges to internal validity. To understand why many studies of pandemic-induced policy changes could suffer from serious threats to internal validity, first consider a common research design with notably strong internal validity: a randomized experiment. If researchers were to use a randomized experiment to estimate the impact of remote instruction on student learning, they would need to assign students at random to the treatment group (students who receive remote instruction) or the control group (students who receive in-person instruction). Assigning the treatment at random ensures there are no systematic differences between the two groups other than the treatment itself. A randomized experiment has high internal validity because it clearly isolates the impact of a particular treatment.

Quasi-experimental research designs, or “natural experiments,” take advantage of variations in treatment status that occur as a result of policy changes or other “natural” phenomena outside of a researcher’s control. While natural experiments eliminate the need to actively assign individuals to treatment and control groups, they typically face greater threats to internal validity than do randomized experiments. One common type of natural experiment uses an econometric instrument to estimate the effects of a particular policy change. The onset of the pandemic holds obvious appeal as an econometric instrument for studying the effects of online instruction, since the crisis caused a sudden shift to remote teaching. A key assumption of this research design, however, is that the policy change of interest (the shift to online learning) does not coincide with other relevant changes. In econometrics, this is known as the “exclusion restriction,” which requires that the econometric instrument (the pandemic) affect the outcome of interest (student learning) only through the policy change of interest (the shift to online learning) and not through other channels.

While the Covid crisis did spur the shift to online instruction, it fails the exclusion restriction because of the many contemporaneous changes that likely also affected student learning. Students shifted to remote learning as their parents lost jobs, as their family members suffered Covid’s health effects, and as they lost the ability to leave the house and see friends, among other significant changes to their lives. If Covid-19 did affect student learning, it would be difficult to attribute the changes in outcomes to remote instruction rather than any of these other contemporaneous factors. We illustrate this in Figure 1, which shows the potential use of Covid-19 as an econometric instrument for remote instruction. The exclusion restriction requires that there be no “causal arrow” between the other channels affected by Covid-19 and student learning, an assumption that is certainly violated based on both common sense and prior empirical evidence. It may be possible to estimate the overall effect of Covid-19 on student outcomes, but attributing that effect to any one channel is likely impossible.

We can illustrate the violation of the exclusion restriction with an example from our own recent research into Covid’s impacts on household Internet-search behavior. In that work, we show that Covid-induced school closings caused parents to seek out online learning resources that might compensate for lost in-school instructional time. An example of this can be seen in the top panel of Figure 2, which shows a large increase in Google searches for “online learning” that corresponded precisely with the timing of the pandemic outbreak in the United States. This provides evidence that the Covid-19 crisis indeed represents a sudden shock to the demand for online learning resources.

At the same time, however, there were many other changes in students’ lives that are reflected in Internet search behavior. Data show, for example, that there were sudden and contemporaneous increases in Google searches for terms relating to the economic condition of households, such as “unemployment insurance” and “food stamps.” The pandemic changed students’ educational experiences but also generated a considerable economic shock to many households. These large, simultaneous changes make it difficult to separate the effects of one shock from another.

Challenges to external validity. The unprecedented circumstances surrounding the Covid-19 crisis also present serious challenges to external validity. Researchers typically examine whether the context and implementation of a policy shift are representative of potential future enactments of the same policy. Neither the context nor the implementation is likely to be representative in this case.

First, the learning environment during the pandemic is unlikely to generalize to typical school years because of the many changes in students’ lives that likely put a strain on their capacity to learn. For instance, students and young adults have reported substantial increases in anxiety and depression during the pandemic. Second, pandemic-induced policy changes were implemented in a way that is unlikely to resemble a more carefully planned implementation of the same policy in a typical year. For example, the pandemic-induced shift to online learning required teachers, with no advance warning, to quickly redesign lessons originally intended for in-person instruction. Under normal circumstances, teachers would have been afforded time to prepare lessons specifically for online instruction. Even in the fall of 2020, there was still substantial uncertainty around schooling logistics and instructional modality, making it difficult for educators to plan instruction effectively in advance. The unprecedented circumstances of the pandemic and the corresponding ad hoc policy shifts are therefore unlikely to generalize to well-planned policy changes in a typical school year.

The Overall Impacts of Covid-19 on Students

Although it is nearly impossible to disentangle the effect of any specific policy, the overall effect of the pandemic—including economic, health, social, and educational changes—is something we can attempt to assess. Moreover, as the pandemic continues to disrupt daily life more than a year after schools first closed, it is increasingly important to understand the impact of the pandemic itself. In addition to the short-run impacts on learning, a range of prior evidence suggests that the effects of health, social, and economic experiences in early childhood can persist into adulthood.

Emerging evidence on the short-run impacts shows that Covid-19 has caused substantial disruptions to students’ learning, particularly for disadvantaged students. Raj Chetty and colleagues find that student progress on Zearn, a popular online math platform, decreased by roughly 30 percent over spring 2020, with children in the lowest-income schools seeing progress drop by 50 percent and those in the highest-income schools quickly recovering to pre-pandemic levels. Nationwide evidence from fall 2020 MAP Growth assessments suggests that students lost ground in mathematics and that reading losses were concentrated among Black and Hispanic students in upper elementary grades. Recent work in Georgia suggests that students lost further ground as the school year progressed through the winter of 2020, with such losses larger among low-income, Black, and Hispanic students. Our research reveals one potential reason for these disparities: when the pandemic first struck, demand for online learning resources increased substantially less in low-income areas than in high-income areas of the United States (see “What Google Search Data Reveal about Learning During the Pandemic,” web only).

Education researchers predict that the pandemic will substantially increase achievement gaps between students from low- and high-income households, even beyond the 2020–21 school year. The best evidence to date shows that Covid-19 not only reduced the learning of the average student compared to typical school years, but that it also increased achievement gaps by disproportionately harming disadvantaged students.

What Comes Next?

Instead of framing the pandemic as a “natural experiment” for studying specific educational interventions, we propose that researchers and policymakers focus on measuring the overall effects of the pandemic itself. We believe it is possible to generate econometrically sound estimates of the overall social, emotional, and academic costs of the pandemic. The pandemic is, however, too large and unprecedented a shock to give us precise insights into individual aspects of children’s educational experiences that have changed. Too many things changed all at once.

After more than a year of pandemic-induced restrictions and shutdowns, there is reason for guarded optimism. Vaccinations have become widely available in the United States, Covid cases, hospitalizations, and deaths here have dropped rapidly, and the era of widespread school closures and fully remote instruction is ending. The educational effects of the pandemic are, however, likely to linger unless we identify the students who have been most adversely affected and provide additional resources to reverse these impacts. The best evidence to date shows that Covid has not only impeded the learning of the average student, but also widened achievement gaps by disproportionately harming the learning of low-performing students. Educators now face the challenge of not only making up for lost instructional time but also closing gaps that are even wider than usual. Though choices of how best to remedy these losses may be best left to individual states, districts, or schools, substantial resources should be devoted to these efforts. Without such investment, particularly among students who have experienced the greatest setbacks, we will likely enter an era of increased educational inequality persisting beyond the return of fully reopened schools.

Andrew Bacher-Hicks is assistant professor at Boston University Wheelock College of Education and Human Development, where Joshua Goodman is associate professor of education.

This article appeared in the Fall 2021 issue of Education Next. Suggested citation format:

Bacher-Hicks, A. and Goodman, J. (2021). The Covid-19 Pandemic Is a Lousy Natural Experiment for Studying the Effects of Online Learning: Focus, instead, on measuring the overall effects of the pandemic itself. Education Next, 21(4), 38-42.