Follow Datanami:
April 21, 2016

Why Machine Learning Is Our Last Hope for Cybersecurity

Mike Stute

(Bruce Rolff/Shutterstock)

Fraud detection. Customer recommendations. Search engine results.  These use cases–and so many more–all owe a debt to machine learning. By automatically discovering patterns that lead to insights and creating predictive models that drive actions, the technology has proven its value many times, and to many industries. More recently, machine learning has begun to make a name for itself in the field of cybersecurity.

There are multiple reasons that manual cybersecurity practices on their own are no longer sufficient to protect businesses, governments and other organizations. Here are a few:

  • There’s simply too much security-related data coming onto the network from an ever-widening array of connected devices;
  • There are too few qualified personnel to assure proper integration of the multiple systems that feed security data into Security and Event Management systems that aim to detect intruder events;
  • These conditions contribute to security analysts’ time being taken up wading through far too many false positives and fake alarms, detracting from their ability to find and combat real threats. The impact is staggering: Cyber-security attacks cost businesses as much as $400 billion a year, according to insurance company Lloyd’s;
  • Once a risk is recognized and prioritized by security analysts, there may be a lag in communicating the threat to the network engineers and systems administrators charged with tackling these issues.


All of these issues are further aggravated by the fact that threats are growing in scale and complexity. Close to three-quarters of respondents to ISACA’s 2015 Advanced Persistent Threat Awareness report, for example, say it is likely or very likely that their organization will be the target of one of these adaptive, multi-faceted and relentless attacks. Such attacks are dangerous in their very patient, slow and deliberate stealth. Often conducted by cyber-criminals, the bad guys behind these threats combine multiple attack methodologies and tools to breach critical data, continuously and mindfully monitoring and interacting with their targets and developing more advanced tools as they need them.

Can things get worse? Sure. This year more attacks will become fully automated, and so become ever more difficult to detect and mitigate, according to the Radware Global Application and Network Security Report. Its survey results indicated that last year ushered in “a greater use of automated, bot-based attacks that generate large volumes of attack traffic in a short period of time, and maintain an attack campaign over a long period of time, essentially creating an APDoS (Advanced Persistent Denial of Service).”

More advanced detection and mitigation than manual cyber-security solutions will be required in the face of nefarious human actors slowly penetrating defenses and Internet zombies aggressively pushing through APDoS, DDoS, burst, and volumetric pipe attacks. These days, the forces behind this latter category of attacks also often have as their prime target extracting corporate data rather than clogging up enterprise links. These attackers put their automated methods to work to steal security analysts’ attention so that they can launch follow-up multi-vector attacks that go by unnoticed on the bit of bandwidth they’ve left open.

Machine Learning Steps Up

As part of a larger cybersecurity solution, machine learning can help human security analysts when it comes to detecting real threats more quickly, so that an enterprise can act on them more swiftly. The technology can plumb the depths of historical security data to learn what attacks look like based on hidden variables and their relationships to each other, all in preparation for “seeing” the next attack when it hits. From the big data it accumulates and analyzes of normal network behavior, it can learn what is appropriate activity and speedily flag that which appears to be out of place. The sooner such problems are identified, the sooner communication can take place between security analysts and engineers to address threats.

data flowing to computers_abstract

The volume of security-related data demands machine learning-based automation

It’s important, however, to understand that not all cyber security solutions leverage the same level of machine learning capabilities. Many take a one-size-fits-all approach: Their systems detect anomalies that could indicate a threat at hand based on the training data they have received using a single learning and prediction model. That’s helpful, but not quite helpful enough: All of the networks that rely on it will be alerted only to those threats the system has determined exist based on using collective sample data – and similarly they will miss all the threats that the single approach has missed. It fails to account for the individuality of a company’s network, composed of its own unique patterns of operation, applications and supported devices and data running through it.

Multiple ML Models

There is more value in using multistage machine-learning analysis and actual data in an effort to determine which machine learning model will work best for detecting real security events on any one particular network. Processing data streams from various subsystems (data transmission frequency measurements over time, for instance, or protocols in a network stream that identify affiliated applications and infrastructure devices) using a variety of machine learning models, and then comparing the learned data to the original raw data, lets an enterprise grade each data stream to reveal which models provide the highest predictability of anomaly detection for that distinct network. Machine learning models may run the gamut from associated rules learning, to sparse dictionary learning, to Bayesian fields and artificial neural networks.big data blocks

Ideally, a data stream can be mastered using unsupervised learning techniques. This approach learns the features of a data set, and classifies it into a “cluster” of similar data–either normal or abnormal. This is in contrast to supervised learning, which requires that sample data for which the outcome already is known be used for training.

Machine learning’s ability to automatically detect changes over time that inform network behavioral profiles of what is and isn’t normal traffic also makes it well-suited to helping the enterprise adapt to new forms of attacks without requiring human intervention. In conjunction with neural network machine learning models and their evolutionary programming adaptation process it is possible to iteratively create networks that become stronger at adapting to new problems, including aggressive automated invasions.

The industry really is just at the start of applying machine learning to the growing cyber-security challenges of detecting and analyzing increasingly sophisticated and targeted threats. The future will see neural networks trained in one data set become the input to others, thereby creating deep networks by extending the knowledge of high-level networks. The industry also will increase its use of hard AI–the simulation of biologic thinking in computers–in detection engines.

Frankly, there’s no choice but to push ahead here, unless everyone is okay with the idea of ceding their domains to the cyber criminals, state-sponsored groups, hackers, and hacktivists that can’t wait to get their hands on a government’s or business’ intellectual property or sensitive employee or customer data.


About the author: Mike Stute is Chief Scientist at Masergy Communications and is the chief architect of the Unified Enterprise Security
network behavioral analysis system. As a data scientist, he is responsible for the research and development of deep analysis methods using machine learning, probability engines, and complex system analysis in big data environments. Mike has over 22 years’ experience in information systems security and has developed analysis systems in fields such as power generation, educational institutions, biotechnology, and electronic communication networks.


Related Items:

Super Scalable SIEMs Set to Tackle Big Security Challenges

Why Cybersecurity Needs Big Data Tech, Especially Hadoop

Fighting Crime with Big Data