Data Tools Aid Open Source Intelligence

U.S. intelligence agencies and the military are increasingly leveraging analytics platforms based on machine learning to sift through data sources like social media. In the vernacular of the Pentagon, these efforts are generally referred to as open source intelligence initiatives.

While the U.S. intelligence community is spending billions of dollars on geospatial intelligence—the analysis and exploitation of imagery and geospatial information—open source efforts focusing on unstructured data like web pages, emails, instant messaging and social media are augmenting those efforts. The result is what the practitioners of geo-intelligence tradecraft often refer to as “human geography.”

One of the biggest challenges for intelligence analysts is the soaring volume of unstructured open source data as the bad guys resort to Facebook and Twitter to communicate and recruit. Hence, efforts are underway to automate open source intelligence gathering through machine learning and other emerging data analysis techniques.

One challenge centers on the fact that “some of the application services using in the ingestion phase need to be retooled to accept the open flow format and don’t have the capability to understand and therefore analyze these complex data sources,” noted John Kostak, senior director of marketing at Digital Reasoning, developers of a cognitive computing platform that seeks to automate the analysis of open source intelligence.

The company’s cognitive computing platform dubbed Synthesys scans unstructured open source data to highlight relevant people, places, organizations, events and other facts. It relies on natural language processing along with what the company called “entity and fact extraction.” Applying “key indicators” and a framework, the platform is intended to automate the process of deriving intelligence from open source data, the company claims.

The platform then attempts to assemble and organize relevant unstructured data using similarity algorithms, categorization and “entity resolution.”

Finally, using graph analysis along with temporal and geospatial reasoning, the machine learning system attempts to come up with actionable open source intelligence based on “opportunities, risks and anomalies” identified by users.

Underlying the platform is Hadoop distributed processing and distributed storage running on HBase and Cassandra databases, the company said.

Other analytics companies are taking different approaches to open source intelligence gathering. For example, Basis Technology of Cambridge, Mass., is focusing on text analytics software that its says can extract names and places from 55 languages. The output of its Rosette suite of analytics software can be fed into visualization and link analysis applications or alerting systems, the company said.

Another approach comes from Opera Solutions, which last year rolled out an algorithm call SignalSensor that uses machine intelligence to examine data streams to identify threats. The tool analyzes social networks, online forums and other open source commentary using proprietary algorithms that help identify threats.

The software is touted as capable of processing over 200 million online elements in over 50 languages, and using ontology of 80 million terms and 420 million relationships. Individual threats are then identified and ranked according to their severity, the company said.

Recent items:

 Using Open Source Data to Identify Security Threats

Bring Dark Data to Light