October 27, 2017

Dealing with Deep Learning’s Big Black Box Problem

Alex Woodie

Deep learning currently carries the torch for artificial intelligence, providing us with a glimpse of how powerfully intelligent machines may do our bidding in the future. But there’s a big problem with deep learning: Nobody really knows how it works.

That’s not to say that it’s a complete mystery. The machine learning algorithms at the heart of today’s neural networks are decades old and are well-defined and extensively documented in academia. These algorithms have been employed in regulated industries like banking and insurance for years without causing a major stir.

“If you look at so-called machine learning, which is basically linear mathematics, it’s pretty easy to explain,” Teradata CTO Stephen Brobst said during a panel discussion Tuesday at Teradata‘s PARTNERS conference. “When you get into multi-layered neural networks, it’s very non-linear mathematics. Expressing those relationships of ‘What were the different variables’ is much more opaque.”

The clarity challenges of neural networks stems from their basic architecture, which are complicated by design. By stacking many hidden layers on top of each other, we have essentially created a nearly unlimited number paths upon which for data to travel as it’s being trained within a network. And then by cycling the data through the pathways over and over again and then handing over weighting of the variables on each go-round to the machine itself, we found we can create machines that are better at giving us better answers than traditional machine learning approaches.

It’s a rough rendition of the neural pathways that we see running in the human brain — with the emphasis on “rough” since we still don’t have a good grasp on how the brain actually works. But we do know that brains work, and we do know that neural networks work — even if we can’t fully explain how either of them work. In a way, the opacity of neural networks isn’t a bug — it’s a feature.

Complexity is the secret sauce for deep learning (all_is_magic/Shutterstock)

It’s also a big area of research, because industries like financial services would love to get their hands on neural networks but are currently blocked from using them because they can’t sufficiently explain how they work to regulators.  “This is one of the biggest barriers to using deep learning in domains like risk scoring and things like that, which are highly regulated,” Brobst said. “For fraud detection and recommendation engines, you can kind of get away with it. But not in highly regulated areas.”

It’s enough for DataRobot customers in regulated industries to avoid deep learning frameworks, such as Tensorflow, even though DataRobot helps automate its use. “In some cases those models get excluded based on the model validation required,” says DataRobot COO Chris Devaney. “They’re not as defendable. Even though it’s not necessarily a black box, it acts like a black box because you can’t document every single thing that happened deep in that neural net algorithm.”

While TensorFlow could provide a good approach for quickly generating predictions on massive sets of data, DataRobot’s customers won’t touch it. Work that DataRobot is doing with Immuta to identify and reduce bias in machine learning could eventually help, but it’s not there yet. “For some of our heavily compliant customers, they could not consider that type of model if they had to defend it to an government agency,” he says.

Mike Gualtieri, a principal analyst and VP with Forrester Research, says there is a level of suspicion around deep learning approaches even among the companies that use them. “Companies that actually use these models…do not trust them,” he said this week at PARTNERS.

Companies that are starting to use deep learning have ways to deal with the uncertainty level, including adding humans to the mix and surrounding them with rules, he says.

“A model can make a prediction — it’s always a probability — but what if it makes a bad one?” he said. “You may surround it with a rule. You may say, ‘OK this is fraud,’ or it may say ‘It’s not fraud.’ And you may have a business rule that a human wants to put in there that says, you know what, I don’t care what the model says — I’m going to call this fraud.”

There is some work being done to shine a light on neural networks. One of these is called the Local Interpretable Model-Agnostic Explanations (LIME) framework, which was introduced by University of Washington computer science professor Marco Tulio Ribeiro and his colleagues Sameer Singh and Carlos Guestrin.

The LIME framework was designed to provide a higher level of understanding and accountability for predictions generated by otherwise opaque algorithms of all sorts. This includes traditional machine learning techniques like random forests and support vector machines (SVMs), as well as the neural networks that are becoming fashionable for today’s deep learning approaches.

Available as open source software on GitHub, the LIME framework shows promise in peeling back the onion of neural nets. “LIME is an efficient tool to facilitate such trust for machine learning practitioners and a good choice to add to their tool belts,” professors Ribeiro, Singh, and Guestrin write in their 2016 O’Reilly article on LIME.

However, LIME still has some work yet before it can be fully relied upon by industry, Teradata’s Brobst said. “The LIME framework mentioned — we’re not done with that yet,” he said. “It’s an active area of research…but it’s what I would call graduate student code. It works in pretty narrow use cases. But for your use case, going to have to customize it.”

Related Items:

Keeping Your Models on the Straight and Narrow

Scrutinizing the Inscrutability of Deep Learning

Why Cracking the ‘Brain Code’ Is Our Best Chance for True AI

 

Share This