Nate Silver on the Art of Statistical Analysis

We are vastly more capable of spotting faint signals against immense background chatter thanks to Hadoop and other big data tools. But according to renowned statistician and journalist Nate Silver, having huge swaths of data and fancy algorithms to run against them cannot compensate for a well-trained analyst who has his feet grounded in reality.

Silver dropped in on the American consciousness in the fall of 2008, when he correctly predicted the outcome of the presidential election in 49 of 50 states. The Michigan native did it again in 2012, when forecasted the outcome in 50 out of 50 states and the District of Columbia, and nailed 31 of 33 U.S. Senate races. Today, the 35-year-old analyzes political data for The New York Times.

In a recent interview with Alex Krawchick of SAS, Silver discussed his thoughts on the value of experience in statistical analysis, the value of big data, Big Brother, the U.S. Census, and his preference in ties.

“As important as it is to have a good data set and good tools and technology, you also need people who really have some expertise, some experience, and that’s a skill set where supply is falling short of demand right now,” Silver says.

Being able to ask the right questions of data is a core, non-technical skill that can be tough to find, Silver says. “It’s not purely a case where conclusions just fall out of from the data naturally,” he says. “You have to know what you’re interested in looking for, and preserve the right balance of curiosity when you are poking around a bit, and skepticism, though, to understand that often what seems to be good in a model won’t resonate as well in the real world.”

There are several approaches to analytics that organizations would be wise to take, according to Silver. For starters, the organization needs to give analysts enough room to be wrong and to allow them to change their models. “They have to feel safe in presenting that information,” he says. “If you have a strategy that has a 70 percent chance of improving [some aspect of the business], it’s worth implementing and worth correcting if it doesn’t.”

While analysts should feel free to explore data and to (occasionally) be wrong, that doesn’t mean going off on wild tangents. Instead, analysts will benefit from being process oriented, Silver says.

“One reason why weather forecasters have gotten a lot better is because they’re making forecasts everyday in every city in the world, pretty much. You have a lot of feedback you’re getting. If you’re off, miscalibrated, then you correct,” he says. “In the short run, you want to be process oriented, which means people follow best practices, or doing things they know correlate with making good predictions in long term.”

While some considered the 2010 U.S. Census too intrusive, Silver applauded the project. “It’s probably important…to have projects like the census that occur at least once in a while,” he told Krawchick. “The reason being, the traditional telephone poll is having more and more problems getting an adequate sample. We’re now at only 10 percent of people responding to calls….So it’s not very representative of the bigger whole.”

There are techniques that statisticians can use to increase that number. “We have ways that get data from people that don’t rely on voluntary participation,” Silver says. “I’m not saying we should be Orwellian, but I think they can be at least useful from the standpoint of understanding what they are.”

At the same time, living in a big data world can be a bit disorienting at times. “It’s kind of crazy,” Silver says. “If I buy like a nice tie or something … my credit card’s like, ‘There’s no way Nate Silver is spending this much on a tie,’ and vetoes the transaction. It’s kind of creepy how they know you potentially.”

Related Items: