Despite its popularity, “evidence-based” policymaking is often less rational than it sounds.
In an era of fierce debate over the relevance of facts and expertise in Washington, it is noteworthy—and curious—that evidence-based policymaking appears to be having a bipartisan moment. Consider a few examples from Congress: the Every Student Succeeds Act of 2015, with its emphasis on evidence; the Evidence-Based Policymaking Commission Act of 2016; and the pending Foundations for Evidence-Based Policymaking Act of 2017. For its part, the Obama Administration entrenched evidence-based policymaking as a key priority throughout the budget process, and the Trump Administration has continued to encourage that work.
Both Republicans and Democrats in Congress and the White House seem enthralled with the promise of evidence-based policymaking. At the same time, external observers from across the political spectrum, sometimes former policymakers themselves, have issued heady praise for what evidence-based policymaking can accomplish. Evidence-based policy purportedly provides a way to “avoid…ideological roadblocks,” “revolutionize America’s government,” and even “change history.”
How to explain this rare but consistent bipartisan agreement in support of evidence-based policymaking at a time when known facts are subject to political contestation and basic science is thought to be “under attack”? Will the push for evidence-based policymaking come close to meeting its proponents’ goals of cutting through ideological division and transforming American government?
It will not. “Evidence” is a word that garners bipartisan support because it sounds rational, serious, and limiting, but in many instances it does little to influence or restrain policymaking and implementation. Calls for evidence-based policymaking mask deep disagreements about what evidence is available and what it shows, as well as about value judgments for which research evidence has little to offer. This dynamic is especially true in social policy areas, where there is little shared consensus on what appropriate goals should be. Calls for evidence-based policymaking therefore allow policymakers to claim credit for their actions without taking any heat for specific contested policy choices. The assertions about what evidence-based policymaking can accomplish are simply overblown.
We typically think of evidence-based policymaking as an instruction to agencies to make a particular decision on the basis of the best available evidence. But many provisions are much more general. For example, the much ballyhooed evidence provisions in the Every Student Succeeds Act are not addressed to the U.S. Department of Education at all. Instead, the provisions give state grantees tremendous leeway in addressing low-performing schools; these provisions just say that when states decide how they want to intervene, they should use some sort of evidence-based practices. Such an instruction barely constrains government actors from selecting the policy interventions they prefer.
Even if the evidence requirements were more stringent, thinking that evidence-based policymaking will transform the facts on the ground assumes too much about the current state of research knowledge, especially in social policy areas. It is one thing to expect the U.S. Environmental Protection Agency to make decisions about climate policy based on the overwhelming scientific consensus about climate change, or the Centers for Disease Control and Prevention to make vaccination recommendations based on the overwhelming scientific consensus about immunization. But for many social policy areas, there is no clear consensus, let alone systemic findings. The Education Department’s “What Works Clearinghouse,” for example, contains reviews of 10,000 studies, but a recent meta-analysis indicated that only 29 interventions showed any significant effect. Yet schools have to continue to operate without waiting for research to tell them what to do.
The research evidence is often inconclusive because social science research is extremely hard to do, relying as it must on the complexities and diversities of human interactions. Implementing evidence-based social policies is even harder, as street-level bureaucrats interpret rigidly (or resist loosely) what they are told to do, often in unobserved ways that do not necessarily comport with what the research suggests should be done. Sociologist Peter Rossi captured some of these difficulties in an oft-cited paper with the wry suggestion that the “better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.”
Moreover, there is often disagreement in social policy research about the appropriate metrics by which to evaluate contested programs. For example, should we embrace charter schools only if they raise student test scores, or are test scores less important than charter schools’ effect on parent satisfaction? Should we evaluate welfare reform based on how many fewer people are on state welfare rolls, or should we examine instead the health and well-being of vulnerable families? To say we should make policy decisions based on evidence answers none of these questions.
At bottom, much policymaking, especially but not only in social policy, is about values. Evidence that the Deferred Action for Childhood Arrivals policy increases high school graduation rates and decreases teen pregnancy among its recipients will not convince anyone who believes that immigration is bad for the United States and that illegal immigration should be discouraged at all costs. Evidence that Medicaid work requirements result in fragile Americans losing their health insurance will not convince anyone who believes that work is the primary value government social programs should emphasize.
Members of the evidence community know all of this. They routinely remind policymakers that evidence can only inform decision-making; it cannot tell decision-makers what the right thing to do is. Advocates for evidence just want evidence to have a seat at the policymaking table.
Evidence should have that seat. But there are nonetheless dangers associated with making great claims for what evidence can accomplish. Policymakers can assert that a policy is or is not evidence-based to mask the fact that what is really going on is a decision based on values. Moreover, when policymakers wield seemingly conflicting research studies in ideologically weaponized ways, the whole research enterprise can seem like nothing more than a rationalization for political decisions.
This is not a call to reject evidence. By all means, we should seek to pursue evidence-based policymaking across ideological divisions. Let us fund research and ask agencies and grantees to make decisions based on evidence. Let us revise our policies when new evidence emerges to correct previous misconceptions. But let us also be skeptical about claims for what evidence-based policymaking will achieve. It will not—and should not—let us avoid difficult conversations and decisions based on value judgments.
This essay draws on Pasachoff’s recently published article in the Columbia Law Review.
