Tuesday, October 17, 2017

Truthiness in Numbers

In 1953, U.S. Supreme Court Justice Robert H. Jackson famously said of his place of work: "We are not final because we are infallible, but we are infallible only because we are final." Brown v. Allen, 344 U.S. 443 (1953) (Jackson, J., concurring). This week, ProPublica released the results of a study which examined Supreme Court opinions for factual errors. While the sampling of eighty-four cases from 2011-2015 is too small to draw sweeping statistical conclusions, the researchers did uncover factual errors, both large and small, in seven of the twenty-four sampled SCOTUS cases which contained "legislative facts." (The report also highlights five earlier opinions containing additional factual mistakes.)

ProPublica notes that the sources of the mistakes varied: some apparently originated with a justice's extrajudicial research, while other errors had been repeated from faulty filings and amicus briefs. The impact of the errors also varied – some were minor errors with insignificant effects, while other mistakes seemed to carry more weight on the Court's ultimate ruling. The report analyzes errors within six of the seven opinions in the sampling period; a seventh will be described in a separate article.

A sobering error within the sampling period involved Shelby County v. Holder, 133 S. Ct. 2612 (2013), which invalidated section 4(b) of the Voting Rights Act based on its outdated "coverage formula" for federal oversight of state voting laws. In support of the majority opinion, Chief Justice John Roberts included a chart on page 2626, comparing voter registration breakdowns by race in the six states which fell under the oversight coverage; the chart was intended to show that voter registration gaps between white and black citizens of those states had narrowed dramatically between 1965 and 2004. As ProPublica notes, Roberts's charts were skewed by a misinterpretation of Census Bureau race categories, using a category for the "White" column which included white voters of Hispanic ethnicity as well as non-Hispanic white voters.

Statistical data was on the Chief Justice's mind again earlier this month, in oral arguments concerning partisan gerrymandering. During questioning in Gill v. Whitford, the Chief Justice expressed concerns about using political science "efficiency gap" (EG) measures as a determining factor in the Court's opinion: "It is just not, it seems, a palatable answer to say the ruling was based on the fact that EG was greater than 7 percent. That doesn't sound like language in the Constitution […] [Y]ou're taking these issues away from democracy and you're throwing them into the courts pursuant to, and it may be simply my educational background, but I can only describe as sociological gobbledygook."

In response, the American Sociological Association released an open letter, defending the use of social science data and describing its benefits to society. The ASA also pointed out that, while "your alma mater would be disappointed to learn that you attributed your lack of understanding of social science to your Harvard education," the ASA would be willing to send representatives to meet with the Court and its staff.

While we don't all have the luxury of renowned social scientists providing in-person overviews of statistical basics, there are many resources available to improve statistical literacy. An accessible introduction to spotting common data misuse in the media can be found in Joel Best's Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists (HM535 .B47 2001 & online) and More Damned Lies and Statistics: How Numbers Confuse Public Issues (HM535 .B474 2004 & online). Shorter guides to spotting erroneous statistics can be found at the UK's Guardian newspaper and at the website Statistics How To. Although not every example contains numbers, you can test your ability to spot misleading statistics and news reports with the Factitious online game developed at American University.

For more academic overviews of statistical methods, the Empirical Collection on Level 3 of the library includes more than 150 titles on statistical methods, including An Introduction to Empirical Legal Research and Storytelling with Data: A Data Visualization Guide for Business Professionals. Purdue University's Online Writing Lab also offers many tips on accurately Writing with Statistics.

For help with locating information about Supreme Court opinions or statistical methods, be sure to Ask a Librarian.