Appellate Advocacy Blog

Editor: Tessa L. Dysart
The University of Arizona
James E. Rogers College of Law

Monday, November 18, 2019

Does it Add Up? Evaluating Numerical Data

This is a guest post by Albert Navarra. Mr. Navarra has been practicing law in California since 1999 and has a background in philosophy, education, and writing. He also has a passion for making complicated subjects simple.

You want to present the best argument, but how do your claims stack up? You may have numerical data to support your argument, but what do the numbers really say? Does your data prove what you say it does? Here are some things to consider the next time you are compiling statistics or evaluating your opponent’s data.

Are the Numbers Accurate?

Statistics can be very persuasive because they’re measurable. They create a sense of objectivity because they’re based on math. Two plus two equals four. Who can argue with that? So people cite numbers for everything you can imagine. The number of illegal immigrants, the number of people without health insurance, the number of people unemployed, and so on. But sometimes people cite numbers that are not accurate. Or it’s unclear exactly what the numbers measure. So sometimes numbers don’t live up to the hype. I once heard a political consultant state, authoritatively, that political action committees affiliated with the opposing party spent “hundreds of millions of dollars” on behalf of candidates without disclosing the donations to the Federal Election Committee. Sounds pretty bad, doesn’t it? But wait a minute. How does the consultant know the figure is “hundreds of millions” of dollars? Where’s the evidence for that? Especially if the donations were not disclosed! If donations were not disclosed, how can anyone know the amounts?

Author Sam Harris created some controversy in 2014 when he appeared on Bill Maher’s HBO show, “Real Time,” and said that about 20% of Muslims are either jihadists or Islamists who want to force their interpretation of Islam on the rest of society. Harris later appeared on CNN’s Fareed Zakaria GPS, where Zakaria questioned the figure of 20%. Harris explained there’s a difference between a jihadist and an Islamist, but that he had grouped them together. In that case, Harris could have avoided confusion by not grouping jihadists and Islamists together in the figure of 20%.

Dig to find out what the number really measures, and if it’s reasonably accurate.

What Do the Numbers Prove?

On January 24, 2012, President Barack Obama said the following in his State of the Union Address:

Right now—right now—American oil production is the highest that it’s been in eight years. That’s right—eight years. Not only that—last year, we relied less on foreign oil than in any of the past 16 years.”

The President was arguing that the U.S. imported less foreign oil because the U.S. produced more oil. Let’s assume for a moment that the numbers were accurate; American oil production increased, and reliance on foreign oil decreased. What do these numbers mean? Do they really prove that reliance on foreign oil decreased because American oil production increased? Or, is it possible that reliance on foreign oil decreased because a weak U.S. economy decreased demand for oil?

Six days after the President’s speech, reported that the cost of oil had been dropping, “after weaker-than-expected fourth-quarter U.S. economic growth also raised doubts about demand outlook.” And that same day, reported that U.S. oil demand dropped 1.3% according to “revised government data released Monday.” So even the governments own data indicated lower U.S. demand for oil!

Even if the numbers are accurate, question what they really indicate.

Watch Out for “Average” Arguments

If your real estate agent recommends you sell your house for $200,000 because that’s the “average” sale price for homes in the neighborhood, you’ll want to know what “average” means. The same goes for when a politician declares what the “average” voter wants, when a scientist states what the “average” temperature is; and pretty much anytime someone uses the word “average” in an argument. What does “average” mean? There are three main possibilities.

Mean average is an average of a set of numbers. Median average is the middle number in a set of numbers. No Clue average—sometimes the person using the word “average” has no clue what the average is. They’re just throwing out a number and hoping you like it. It’s an easy thing to say when you’re desperate to make a point: “The average [blah, blah, blah]!”

So, what does your real estate agent mean by “average”? Ask for neighborhood sale prices so you can do the math and calculate the mean and median averages. For example, neighborhood home sales of $198,000, $200,00, and $250,000 provide a mean average of $216,000 and a median $200,000. The difference is $16,000, a significant amount. If you’re selling, you’ll argue the mean average is fairer and more accurate, because you want a higher number. If you’re buying, you’ll argue the median.

Define what “average” means.

Is the Sample Big Enough?

Statistics often seek to learn something about a large group of things by examining a small sample of the group. For example, a public opinion poll may want to estimate the percentage of voters likely to vote for one presidential candidate over another. Rather than ask 200 million voters (the entire group or “population” of voters”), the polling company might ask 1,000 (a small sample of voters). Asking 1,000 is not as accurate as asking all 200 million, of course, but that’s how statistics work. “Sampling” is a relatively quick, cheap, and practical way to make predictions or estimates. So how many people should be polled? How large should the sample be? There are free calculators on the Internet that you can use to find out how large a sample should be.

After the Deepwater Horizon oil spill in the Gulf of Mexico, the federal government banned all deepwater drilling for oil in the Gulf of Mexico. The next month, a federal court invalidated the ban on deepwater drilling. The Judge explained that in deciding to ban all deepwater drilling, the government focused on only one incident: “Deepwater Horizon and BP [British Petroleum] only. None others.” The government assumed that “because one rig failed and although no one yet fully knows why, all companies and rigs drilling new wells over 500 feet also universally present an imminent danger.” The Judge summed up the error of hasty generalizations rather nicely: “If some drilling equipment parts are flawed, is it rational to say all are? Are all airplanes a danger because one was? All oil tankers like Exxon Valdez? All trains? All mines? That sort of thinking seems heavy handed, and rather overbearing.”

Check sample size.

Is the Sample Relevant?

I heard a story on the radio about a gentleman who hitchhiked across the country. He explained that he never had a bad experience. As a result, his faith in humanity increased. He was encouraged about the kindness of people. Can you see the problem here? The gentleman based his opinion only on those people who were kind enough to give him a ride! What about all the drivers who passed him by? How kind were they? The “sample” of people included only persons nice enough to give him a ride, and was not a fair representation of all of humanity.

Don’t be fooled by the numbers. When evaluating numerical data, ask yourself these questions. Are the numbers accurate? What do they mean? What is meant by “average”? Is the sample big enough? Is the sample relevant.

This article contains excerpts from The Joy of Argument by Albert Navarra.  You can learn more about The Joy of Argument by clicking here

Find The Joy of Argument on Amazon.

Follow The Joy of Argument on Facebook.

| Permalink


Post a comment