Follow Datanami:
April 19, 2016

Super Scalable SIEMs Set to Tackle Big Security Challenges


The huge volume of network data that organizations need to chew through is putting pressure on security incident and event management (SIEM) software products that use a traditional rules-based approach. That’s opening the door for a new breed of SIEMs that were built from the ground up to run on big data architectures, including both NoSQL and Hadoop.

SIEMs emerged near the turn of the century as the firewalls, intrusion detection systems (IDS), and intrusion prevention systems (IPS) of the day struggled to keep up with big network data. As the data volumes have continued their skyward march, the SIEMs have proven ill-equipped to keep up with the scalability requirements, which has opened the door for yet another generation of security tools.

Many of these next-gen SIEMs are based on big data architectures, specifically NoSQL databases, such as those from Splunk (NASDAQ: SPLK)–which today unveiled a new release of its machine learning-based tool, called User Behavior Analytics (UBA)–and Sumo Logic, which unveiled a new release of its machine data analytics software last week. There is also momentum building behind’s Elastic open source ELK stack, which includes its ElasticSearch NoSQL-based search engine, the Logstash log aggregation tool, and the Kibana analytics tool.sumo logic_logo

With the latest release of its cloud-based SIEM service, Sumo Logic is aiming to combine the ingestion and analysis of structured metric data alongside unstructured data, such as time-series log data. The new data dimension will help to eliminate the need for security personnel to engage in “context switching,” the company says.

Sumo Logic president and CEO Ramin Sayar says the release represents another step forward for its 1,000-plus customers. What’s more, the additional data dimension “will result in an explosion of use cases that leverage machine learning” he says.

Not to be outdone, Sumo Logic’s competitor Splunk aims to drive machine learning use cases forward with a new release of tis Splunk Enterprise Security and Splunk User Behavior Analytics (UBA). The new UBA release brings a new “threat detection framework” that control how alerts are generated from anomalies.


SIEMs are called on to perform many functions

“Customers now gain insights across the entire enterprise and take action more quickly by leveraging the combined power of machine learning, anomaly detection, correlation and ad-hoc investigation in an integrated solution,” says Haiyan Song, senior vice president of security markets for Splunk.

There are also a number of next-gen SIEMs being developed on the Apache Hadoop stack. One of the newest ones is called Apache Metron, which was accepted as an incubating project at the Apache Software Foundation in December.

Metron integrates a variety of Hadoop ecosystem components to make it easier to separate the security wheat from the chaff. The software, which was originally developed at IT giant Cisco (NASDAQ: CSCO) under the name OpenSOC, was designed to help security operation center (SOC) personnel to detect a variety of security threats, including hackers and advanced malware.

The Metron architecture is composed of four main parts, including:

  1. A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates;
  2. A way to process the data in real-time, including support for so-called “enrichment” metadata like threat intelligence, geolocation, and DNS data;
  3. A storage system that works efficiently while retaining the capability to reconstruct and mine all collected logs and packets;
  4. A user interface (UI) layer that gives security investigators access to all necessary tools.

The project incorporates existing big data tools, including Apache Storm, which is used to facilitate the ingest, enrichment, and real-time scoring of the raw network input from logs and other telemetry sources. Also included are open source sensors, including the Data Plane Development Kit, a high-speed packet capture probe available under a BSD license; the Bro IDS deep packet inspection and sensor and intrusion detection system (IDS); YAF (Yet Another Flowmeter), which processes packet data from PCAP files; and SNORT, a network based intrusion detection system.

apache_metron_logo“With Metron, our goal is to tie big data into security analytics and drive towards an extensible centralized platform to effectively enable rapid detection and rapid response for advanced security threats,” writes James Sirota, who spearheaded the development of Metron at Cisco and is now the director of security solutions at Hortonworks (NASDAQ: HDP), which is one of the sponsors of the project.

By combining the storage, the monitoring, and the analytic components (including advanced behavioral analytics using machine learning) in one tool, the folks behind Metron say they can simplify life for SOC personnel. Security people are well-accustomed to using different tools, but manually integrating the tools can be burdensome.

Hortonworks unveiled its support for Metron at the Hadoop Summit Europe conference in Dublin, Ireland, last week. Joining the Hadoop distributor in support of Metron are data center operator Rackspace, national and cyber security experts ManTech, and B23, a Washington D.C.-based provider of big data services.


Big Bro IDS is watching

Kent Warren, president, ManTech Commercial Services, says Metron has “all of the right elements to enable at scale ingestion, processing, indexing, and visualization of information like key cyber data sets to facilitate the protection, monitoring, analysis, detection, and response to nefarious unauthorized actions.” The company is looking to use Metron with its customers, he adds.

Whether they’re based on HDFS or NoSQL, there appears plenty of demand for big data security tools. According to a study last year by the SANS Institute of more than 200 IT and security professionals, 55 percent of organizations that have deployed big data projects use the technology for log management. Among those who plan to deploy a big data platform over the next two years, nearly six out of 10 said that log management is a top priority.

Security is one of the most popular use cases for Hadoop implementations, whether it’s used for fraud detection, internal access control, or detecting malware (check out this link to read how Cisco uses a Hadoop cluster to inventory the world’s viruses, worms, and other assorted nasties). Among the vendors shipping Hadoop-based security tools are Argyle Data, MapR Technologies, Niara, and Platfora, among others.


Related Items:

Why Cybersecurity Needs Big Data Tech, Especially Hadoop

Fighting Crime with Big Data

Stomping Out Criminal Scams with Hadoop