Follow Datanami:
April 9, 2015

Why Cybersecurity Needs Big Data Tech, Especially Hadoop

When cyber criminals really started ramping up their activities several years ago, security experts realized they needed bigger and better tools if they had any hope of slowing them down. In the wake of 2014, or “The Year of the Data Breach,” it’s clear those efforts largely failed, which is why so many security experts today are banking their hopes on Hadoop.

Cybersecurity has become a big data problem because the size and complexity of the data has simply grown too big and too complex to analyze with using traditional security tools, including the security event and information management (SIEM) applications that many companies adopted over the past decade, says Pete Schlampp, vice president of products at Platfora.

“I used to work at Solera Networks [a network forensics company acquired by Blue Coat in 2013], and we actually invented own file system and our own database technology to be able to handle this,” Schlampp says. “What Hadoop has out of box is so far superior to that that companies are moving off of [SIEM tools] rapidly.”

In response to the demand, Platfora today unveiled a joint Hadoop solution with MapR Technologies that’s designed to help companies flush out security threats lurking on their networks. Called, Big Data Analytics for Security, the offering combines MapR’s Hadoop distribution, a collection of machine learning algorithms from MapR, MapReduce-based analytic jobs, and Platfora’s Hadoop application to collect, transform, and visualize vast amounts of poly-structured data.

The idea is to give “next-gen” security analysts the tools they need to detect threats in something close to real time, and without needing the skillsets of a data scientist or Hadoop administrator, Schlampp says.


Platfora’s new cybersecurity tool helps security analysts detect patterns hidden in network data

“What these companies are finding is that traditional security tools are architected in a very regimented way to deal with security data,” he tells Datanami. “Sure they understand VPN records and IPS [intrusion prevention system] logs and all these things. But if part of the investigation involves other data sets—such as a bank that needs to look at transactions out of accounts–you can’t mix those things together. One of the hallmarks of big data is variety, and being able to easily mix these data sets together to find these anomalies is pretty important.”

Fraud sucks up anywhere from $200 billion to $250 billion every year in the retail space alone, according to Forrester. The average company loses 5 percent of revenue to fraud. Today, much of that fraud is perpetrated over the Internet through massive data breaches. In 2014—which saw Home Depot, JPMorgan Chase, Michaels, Neiman Marcus, Orange, American Express, and Community Health Systems hacked–more than 1 billion data records were comprised, a 78 percent increase over the previous year, according to figures cited by Platfora.

Clearly that represents a massive failure by the current state of network security tools. And that’s why new startups, such as Niara, will be targeting this space.

Niara today announced that it has received $20 million in Series B funding to continue development of its Hadoop-based cybersecurity tool. Founded by former Juniper Networks vice president of engineering Sriram Ramachandran, Niara is responding to the growing demand for more powerful analytics to head off the increasingly brazen cyber criminals that ply the Net’s open byways, stealing data, not for fun but niara logo

“There are two things we saw at the highest level,” Ramachandran tells Datanami. “The first is the sophistication of threats has increased dramatically. Very colloquially put, the threats have gone from hacking for fame and glory to basically organized crime. For every dollar we spend fighting it in Silicon Valley, there are two to five dollars going into to [by the bad guys].

“That’s on the outside,” he continues. “On the inside, in enterprises today, there’s increasing openness and loss of control. Things like BYOD [bring your own device] and SaaS [Software as a service] and Cloud–the enterprise security folks are starting to lose control on the inside, and you put those things together, and that’s an explosive combination.”

Both Niara and Platfora point out that the average security threat today is discovered six to nine months after it made landfall behind the enterprises’ firewall. The sophistication of today’s malware—such as the BlackPOS that stole data from Home Depot’s point of sale systems—demands that enterprise look at each piece of data much more closely than they have previously.

Niara’s product, which is currently in beta and slated for release by the end of June, will combine traditional heuristic techniques with newer machine learning technologies to enable companies to detect the traces that cybercriminals and malware invariably leave when they make their way across private networks.

The combination of a parallel Hadoop cluster equipped with well-trained machine learning models is necessary because cybercriminals have become so good at blending into the background of day-to-day activities. It’s not about seeing black from white, but isolating the many shades of gray, Ramachandran says.

“Let’s say in your enterprise, if I compromise your user name and password, I can go in through the front door. No existing security tool is going to catch that,” he says. “So what you have to do, in conjunction with systems that catch the black and white type of threats, you have to develop a system that begins where these things leave off, which means it begins in the gray zone. Any signal we pick up is going to be extremely weak.”

By aggregating all that gray security data and watching how it evolves and changes over time, Ramachandran hopes to be able to isolate, correlate, and ultimately connect those weak signals with the sophisticated cybercriminal schemes behind them. “You have to build up enough of a picture that you can actually raise something to a minimum threshold, where you can find out if this is a sophisticated threat,” he says.

Platfora and MapR are essentially taking the same approach as Niara, using Hadoop and the power of machine learning to crunch petabytes worth of data in search of a cybercriminal’s signatures. They’re not the only ones, of course. All of the Hadoop distributors have worked on security implementations, and other big data software vendors, like Palantir, have built their own security systems.

The trend to this point has been toward roll-your-own Hadoop security Hadoop_copsystems. According to Ramachandran and Schlampp, anywhere from 25 percent to 33 percent of production Hadoop clusters are being used to detect security threats. We’ve obviously reached the point where companies want pre-built security applications to do the job for them.

“There’s a change in attitude and philosophy in the network security space,” Schlampp says. “We’re all going to admit the network is insecure in some ways. You are going to be breached. Whith that change in attitude, the security operations teams are not just building an army to keep people out of the network. But they they’re also building a police force to detect what’s going on on the network itself.”

Related Items:

Stomping Out Criminal Scams with Hadoop

Fighting Crime with Big Data

Eight Ways Analytics Powers Fraud Detection