Follow Datanami:
August 25, 2015

Hortonworks Boosts Streaming Analytics, IoT Plays with NiFi Deal

Hortonworks today announced a definitive agreement to acquire Onyara, the company behind the data routing and streaming technology called Apache NiFi. The Hadoop heavy also announced NiFi will be the basis for its second major product line, Hortonworks DataFlow, which will underlie real-time streaming analytics and Internet of Things (IoT) applications.

The National Security Agency created Apache NiFi eight years ago to address its real-time data collection needs. Short for Niagrafiles (a play on Niagara Falls), the technology was designed to automate the flow of large volumes of data among multiple computers.

The software–which the NSA was released as open source late last year, formed the basis for the creation of Onyara, and became a top-level Apache project in July–allows customers to build big data flows, and includes provisions for two-way communication between data generation and collection points, built-in security, metadata tracking, and a graphical user interface.

“It’s about managing and collecting the data, conducting the workflow as that data is flowing through the pipes, so to speak, and then curating it into whatever form factor it needs to be in, either for real-time streaming analytics or storage in in a Hortonworks platform or other downstream data platforms,” says Hortonworks vice president of corporate strategy Shaun Connolly.

As the overall “command and control” product for big streaming data, NiFi will form the glue that holds together the various other streaming data products that Hortonworks customers may be playing with, including Apache Kafka and Apache Storm.

“It effectively gets rid of a lot of ad-hoc, roll-your-own, get-me-a-box-of-Lincoln-Logs-and-try-and-assemble-it-yourself,” he says. “It’s more a command and control solution that you can run all those things under one umbrella. And they’re all orchestrated.”

While NiFi doesn’t do the analytics itself, it will form the plumbing that allows users to feed streaming data into analytics engines. “We do see that there will be a raft of use cases [for NiFi] that will funnel data into Kafka, Storm, or Spark,” Connolly says.

Hortonworks DataFlow doesn’t run on Hadoop and there are no plans to do so. As is the case with many of the real-time streaming products storing the data for later use is not a priority. “When you net it out, you don’t need HDFS or a storage platform to run it,” Connolly says. “It’s all about the stream.”

The new products will give Hortonworks a boost in the IoT market, which the company refers to as Internet of Anything (IoAT). Connolly says that the IoAT differentiates not only from “moving iron,” such as sensors on planes, trains, and automobiles, but data generated from all the other sources, such as mobile devices and closed-caption TV signals.Hortonworks DataFlow Powered by Apache NiFi

One early adopter of NiFi, a security outfit called Prescient Edge, is using the technology to manage thousands of streaming feeds sources from around the world for the purpose of alerting clientele to potential trouble spots. The company uses NiFi to control the flow of data into an analytic application running in SAP HANA  that automatically sends alerts to clients to avoid geopolitical hotspots.

“NiFi’s well designed, mature API has made our integration process remarkably straightforward,” Prescient Edge Chief Systems Architect Mike Bishop says in a press release. “With it, we’re able to track the origin, transformation, and persistence of data throughout our analytic processes.”

The maturity, security, and built-in intelligence of the NiFi product attracted Hortonworks to the product and Onyara, Connolly says.

“The nice thing about NiFi is it can run in low-resource, small machines, as well as larger machines,” he says. “It’s really about the multidirectional and point to point problem that IoAT has. It isn’t just about acquiring data and shipping it in a single direction. A conversation can actually be had with the NiFi end points.”

That built-in intelligence allows the data sources to vary the amount of data they’re sending down the pipe, which could suddenly have a bandwidth spike that makes sending a full-definition feed too costly.

For a Hadoop distributor, the launch of Hortonworks DataFlow is unique in that it doesn’t run on Hadoop at all. “We’re expanding from single product to a multiple-product company,” Connolly says. “If you’re able to do management of the data stream and do real-analytics on that stream, and then you’re able to store it on a platform like HDP for rich historical analysis–if you look at cyber security application or other IoTA applications, they need both to build a robust predictive real time application. So this positions us really well today.”

Terms of the deal were not disclosed. The entire Onyara team has come over to Hortonworks as part of the deal, Connolly says.

Related Items:

NSA Releases Data Management Tool

Hortonworks Revenue Up 154 Percent

Datanami