Follow Datanami:
December 1, 2014

NSA Releases Data Management Tool

The National Security Agency said it is expanding its open source portfolio by releasing the source code for software designed to manage data network interoperability.

NSA said Nov. 25 it was releasing the open source tool, “Niagarafiles,” or “Nifi,” through the Apache Software Foundation. The spy agency said the release was the first in a series of open source software tools from its technology transfer program that shares agency-developed technologies with academia, industry and research organizations.

The spy agency said the new technology “automates data flows among multiple computers, even when data formats and protocols differ.” It said companies could use Nifi to “quickly control, manage and analyze the flow of information from geographically dispersed sites—creating comprehensive situational awareness.

The Nifi release follows earlier open source efforts such as the agency’s distribution of code for its “Accumulo” project through the Apache Software Foundation. Accumulo is a NoSQL database store supervised by the software foundation. Apache’s Accumulo distribution is a high-end data storage and retrieval tool.

Nifi “provides a way to prioritize data flows more effectively and get rid of artificial delays in identifying and transmitting critical information,” according to Nifi lead developer Joseph Witt.

Added Linda Burger, director of NSA’s tech transfer program: “We use open source releases to move technology from the lab to the marketplace, making state-of-the-art technology more widely available and aiming to accelerate U.S. economic growth.”

The agency said it is focusing on data management technologies as the volume of data grows and legal frameworks for handling data in different countries grows in complexity. Hence, it is trying to leverage open source development as a way to promote data management innovations.

While some observers welcomed the NSA open source effort, security experts have alleged that the spy agency has in the past placed possible backdoors in encryption software that weakened government security standards overseen by the National Institute of Standard and Technology.

Others have suggested that NSA has a vested interest in promoting new data management tools since the rise of big data has made it harder for the spy agency to separate the desired signal from the noise created by the growing volume of stored data.

The spy agency’s tech transfer program includes a portfolio of patented methods covering data management, encryption and other technologies. The project office brokers patent licensing deals along with research and development agreements.

Among the tech transfer program’s other patented data tools are a method for connecting data in relational databases using SQL along with a text sorting and string searching technique that is said to increase efficiency when analyzing large datasets.

Another NSA visualization tool is based on Java and is said to capture user-defined process flows into a single step.

Meanwhile, the Apache Software Foundation said last week that the Nifi project, which it described as a “dataflow system based on the concepts of flow-based programming,” had entered the “incubator” phase.

Recent items:

Feds Strive to Balance Data Sharing and Risk

DARPA Seeks to Unlock Big Data Mechanisms