Policy for Evidence

Clear standards for producing and sharing data can encourage more effective policymaking.

Although the idea of rigorous evidence for policy has few detractors, a lack of clear and effective policies to support the development of evidence can pose real obstacles to getting the work done. Rigorous pilots and evaluation studies in many cases represent a departure from the status quo, which makes it all the more crucial to develop effective policies that mandate, encourage, or specify resources for piloting and evaluation to inform a broader learning agenda.

One way to encourage rigorous piloting and evaluation is by specifying it. For example, the Dodd-Frank Act—which created the Consumer Financial Protection Bureau (CFPB)—authorizes the agency to provide waivers in connection with trial disclosure programs as a way to promote agency innovation. As a result, the CFPB has engaged in many disclosure pilots, and its recently launched “disclosure sandbox” invites additional pilots.

In contrast to agency-specific provisions are law-specific evaluation requirements. These include both prospective mandates to evaluate proposed legislation as well as retrospective mandates such as those contained in the America Invents Act, which required multiple studies of the implementation of the law and related topics following enactment. In the field of criminal justice, the U.S. Sentencing Commission has a mission-specific mandate of analyzing the effects of sentencing laws through high-quality impact assessments.

But mandating evaluation, without providing resources or attending to the details, will not by itself lead to rigorous learning or improved outcomes. Policy support and clarity are needed to help ensure that evaluations are effective. To this end, the Pew-MacArthur Results First Initiative—and in particular, its excellent 2017 report—has collected examples of well-designed state statutes that create evidence definitions, develop program inventories, specify cost-benefit analyses, and require program effectiveness data to be reported in the budget process.

With or without a mandate, another hurdle to randomized evaluation and rigorous piloting is the perception that withholding a benefit or burden from some, but not others, is unfair and possibly illegal. Applying a rule randomly to one set of entities but not another might appear to violate the equal protection and due process guarantees enshrined in the U.S. Constitution. It might also seem to present impermissibly arbitrary or capricious agency action forbidden by the Administrative Procedure Act. However, as detailed in a recent paper that one of us authored, courts have by and large rejected such objections to specific pilots.

For example, the U.S. Supreme Court has stated that a statutory classification that neither burdens fundamental rights nor targets a suspect class is consistent with equal protection as long as it “bears a rational relation to some independent and legitimate legislative end.” In one of the leading cases on agency experimentation, Aguayo v. Richardson, the U.S. Court of Appeals for the Second Circuit upheld an experimental welfare work program because a governmental “purpose to determine whether and how improvements can be made in the welfare system is as ‘legitimate’ or ‘appropriate’ as anything can be.”

Certain cases have also established that a random distribution of burdens passes constitutional muster. For example, in Engquist v. Oregon Dept. of Agriculture, the U.S. Supreme Court denied a public employee’s claim that she was fired arbitrarily while other similarly situated employees kept their jobs. The Court held that government actions like personnel decisions enjoy a measure of discretion, and the dissenting justices agreed with the principle that “a random choice among rational alternatives does not violate the Equal Protection Clause.” That is a good thing in light of the widespread use of random audits by the government, as described in another essay from this series.

Aside from legal concerns, denying a benefit to a control group within a randomized controlled trial can present important ethical concerns. In some cases, issues of differential treatment can be resolved by offering the experimental treatment at the conclusion of the trial if it is shown to improve outcomes. Instituting preferences for less vulnerable populations or offering additional compensation can also mitigate such concerns. Requiring individuals or entities to opt into a pilot can also help address the concerns of risk disclosure. Many of these decisions lie in the details of the individual studies, so it is important to have the involvement of ethics boards and the benefit of evaluation guidelines to help researchers carry out projects in a legal and ethical manner.

Another distinct set of issues surrounds the evaluation of federal policies that are implemented by states. States may perceive that there are policies or prohibitions that prevent them from carrying out the monitoring needed to support iterative policymaking—such as privacy concerns, or a rule that program funds cannot be allocated to evaluation. Well-intended but inefficient and cumbersome federal reporting requirements can also take away time from rigorous evaluation or discourage it altogether.

Finally, government actors should develop policies that encourage the production of high-quality data and create reliable pathways for sharing it. One of the hardest—but most important—parts of rigorous evaluation is gathering data on outcomes. In a study based on interviews with 350 state officials by the Pew Charitable Trusts, the first, second, and third greatest challenges to state data work, other than staffing, were data accessibility, data quality, and data sharing. Administrative outcome data, if it exists at all, often sits in government silos with limited access or connection to other datasets. Existing frameworks for data sharing are antiquated, and poor data quality or inadequate documentation can make data unusable.

Policy can help. Some ingredients of a well-functioning policy infrastructure for evidence include requirements for clear data documentation, a centralized and digitized data clearinghouse, and the capacity to link data across agencies. Legislative mandates that respect privacy but specify data access to evaluators can also, in our experience, be incredibly helpful. If agencies are compelled to report or share data by law, they have to make doing so a priority. In a world of resource constraints this can represent the difference between reporting times of days and years.

Clear data sharing protocols can also help break down government data silos. One priority area for data sharing is criminal record expungement policy, which has as one of its goals the reduction of recidivism. Expunging criminal records can lead to better outcomes not only in criminal justice, but also in employment, education, health, and housing—but these outcomes can only be observed by connecting data across agencies.

Policy for evidence has an often overlooked but important role in supporting evidence for policy. Attending to some of the most important details underlying the creation of evidence-driven policy—data, permissions, and mandating continuous evaluation—can help clear the way for better policy research and reorient the policymaking process toward the adoption of programs and policies that have demonstrable success.