A recent Supreme Court decision could allow courts to rely on dubious science.
In its famous opinion in Daubert v. Merrell Dow Pharmaceuticals, the U.S. Supreme Court took a major step toward assuring that our legal system functions on the basis of sound scientific principles. The Court held that judges must apply criteria based on such principles when they decide whether to admit expert testimony. Daubert was a reaction to the well-documented problem of court decisions that are based on “junk science”—opinions offered in evidence by supposed experts but that are not supported by reliable data and analysis.
Daubert requires an expert witness to disclose the data and methodology they relied on as the basis for any opinion they propose to offer in court. The judge then decides whether the data and methodology are sufficiently reliable to support admitting the expert’s testimony. If the opposing counsel challenges the reliability of the data or methodology, the judge conducts a hearing during which opposing counsel has the opportunity to test the reliability of the data and methodology by cross-examining the witness.
Earlier this month, the Court issued a potentially infamous opinion that encourages agencies to rely on junk science. The facts of Biestek v. Commissioner of Social Security are simple. Biestek applied for Social Security disability benefits. At a hearing before an Administrative Law Judge (ALJ), Biestek claimed that he was so disabled that he could not perform the functions required by any job that is available in significant numbers in the U.S. economy.
In such a hearing, the government has the burden of proving the availability of a significant number of jobs that can be performed by someone with the age, education, experience, and health state of the applicant. The Social Security Administration (SSA) used a vocational expert as a witness, who testified that there were 120,000 jobs available for “sorters” and 240,000 jobs available for “bench assemblers” that Biestek could perform.
When asked to describe the basis for these numbers, the witness referred to two sources—the Bureau of Labor Statistics (BLS) and private surveys the witness had conducted for other clients. Since BLS does not report job availability statistics with the specificity required to support the number of jobs of a particular type the witness claimed to be available, the only possible basis for those numbers were the private surveys.
Biestek’s lawyer asked the witness to provide the private surveys. The witness refused on the basis that they were part of her confidential client files. Biestek’s lawyer then asked the witness to provide redacted versions of the surveys that excluded any client-specific information. The witness should have been prepared to comply with that request, since SSA’s Vocational Expert Handbook instructs witnesses to “have available at the hearing, any vocational resource materials you are likely to rely upon.”
We will never know whether the witness had a redacted version of the surveys with her because the ALJ stated that he would not require the witness to provide a redacted version of the surveys. The ALJ then relied solely on the testimony of the vocational witness as the basis for his decision to deny Biestek’s application for benefits based on his finding that there were a significant number of jobs available that Biestek could perform.
Of course, we will never know whether the private surveys exist. If they exist, we will never know whether they support the witness’s testimony. If they support the witness’s testimony, we will never know whether they are based on reliable data. If they are supported by reliable data, we will never know whether the methodology the witness used to draw inferences from the data was reliable.
Three Supreme Court justices expressed the view that the ALJ’s decision to deny Biestek’s application for disability benefits based solely on the unsupported testimony of the vocational expert should be reversed for all of the same reasons that any court would reverse the decision of a district judge in similar circumstances, under Daubert. Unfortunately, they had to express that view in a dissenting opinion.
The majority reasoned that the ALJ could rely on the unsupported opinion of the witness for two reasons: the evidence satisfied the substantial evidence test and the Federal Rules of Evidence (FRE) do not apply to SSA disability hearings. The first reason is based on a misunderstanding of the substantial evidence test, while the second is based on a misunderstanding of the reasons why Congress decided that the FRE do not apply to agency hearings.
The majority argued that it would make no sense to conclude, on the one hand, that an expert witness’s opinion qualifies as substantial evidence when no one asks the expert to provide the data and analysis to support the opinion—but that, on the other hand, it does not qualify as substantial evidence when someone asks for the supporting data and analysis and the witness refuses to comply with the request. That reasoning is inconsistent with the Court’s opinion in Universal Camera v. NLRB: “The substantiality of evidence must take into account whatever in the record detracts from its weight.” A refusal to provide support for an expert opinion clearly detracts from its weight.
The inapplicability of the FRE to agency hearings is unrelated to the wisdom of the Court’s decision in Daubert to keep juries from deciding cases based on junk science. The substantive reasoning in Daubert is equally applicable to agency hearings. The difference in context bears only on the mechanism that courts can and should use to implement the Daubert principle. In the context of a jury trial, the Daubert criteria must be applied at the time that an expert opinion is offered into evidence. In the context of an agency hearing, the Daubert criteria can be applied in the process of judicial review of the stated basis for an agency’s decision to rely on expert testimony as the basis for a finding.
The FRE are not appropriate for application in agency hearings because agencies must give reasons for their findings of fact. Thus, if an ALJ admits evidence that he later determines to be unreliable, he can decline to rely on it as the basis for a finding. If an ALJ erroneously relies on unreliable evidence as the basis for a finding, a reviewing court can detect the error and reject the finding as not supported by substantial evidence.
The Court’s opinion in Biestek has the potential to produce an administrative state in which many important decisions are based on junk science. As Jason Johnston has explained, many agencies have relied on junk science as the sole basis for decisions that have serious financial consequences. For instance, some decisions in which the U.S. Environmental Protection Agency (EPA) has required firms to incur billions of dollars in costs are based on findings from researchers who have refused to provide anyone, including EPA, with the data and analysis on which the findings are based. It is impossible to know whether those findings are supported by reliable data and analysis.
The Biestek majority refers to the credentials and reputation of an expert witness as an adequate independent basis to conclude that the opinions of the expert are sufficiently reliable even if the expert refuses to provide anyone access to the data and analysis that she relies on. That is a dangerous method of reasoning that is inconsistent with the Court’s reasoning in Daubert.
The Daubert Court did not indulge the naïve assumption that an expert should be believed simply because she has impressive credentials and an impeccable reputation. There is a mountain of evidence that contradicts that assumption. For instance, in just a two month period, Harvard, Cornell, Dartmouth, and Sloan Kettering were reportedly forced to announce that some of their top researchers had committed academic fraud in conducting some of the most important studies that those prestigious institutions have produced.
Attempts to replicate the findings of hundreds of important studies have produced powerful circumstantial evidence that academic fraud in conducting studies is a widespread problem. For instance, four attempts to replicate the findings of hundreds of studies published in prestigious journals found that only 36% to 67% of the findings could be replicated using the data and methodology that the researchers claimed to have used. It is fair to infer that the other 33% to 64% of the findings were the product of either academic fraud or incompetence.
Academic fraud can be extremely costly to society. The measles epidemic that now afflicts the nation is the result of an academic fraud that took place over twenty years ago. A team of twelve distinguished researchers published a study in 1998 in The Lancet, the most prestigious British medical journal, in which they reported the results of a study of the relationship between vaccines and autism. They found a correlation between taking a vaccine and being diagnosed with autism so high that the lead author used the finding as the basis for an international speaking tour in which he warned that vaccines cause autism.
The vaccine autism study was a fraud perpetrated by the lead author, who was well paid by attorneys who wanted to sue vaccine manufacturers. But the fraud was not detected and publicly disclosed until 2011. Scores of studies conducted since 1998 have found no relationship between vaccines and autism. The fraudulent 1998 study caused a dramatic decrease in the proportion of the population of parents who vaccinate their children, and a corresponding increase in the number of children who suffer from a variety of potentially lethal communicable diseases like measles.
Society must act aggressively to reduce the risk of damage caused by junk science. Daubert was an important step in the right direction. Biestek was an unfortunate step in the wrong direction.
The dissenting justices expressed hope that lower courts will not interpret the majority opinion as an invitation to uphold all uses of junk science by agencies. The reasoning in the majority opinion can be interpreted narrowly. At the end of its opinion, the majority stated that it was only rejecting a “categorical rule” that the petitioner’s lawyer had urged the Court to adopt. It went on to disavow any intent to decide in Biestek’s case “whether, in the absence of that rule, substantial evidence supported the ALJ in denying him benefits.” That is a strange statement that suggests that the majority did not do the one thing that every court is required to do—resolve the case before the court based on the facts of the case.
I hope the optimism of the dissenting justices proves to be accurate, but it would be easy for lower courts to interpret the majority opinion as an invitation to allow agencies to rely on junk science in virtually all cases.
