Revelytix today announced that it will provide big data software and services for the United States Department of Defense (DOD) during 2013, and has completed a Hadoop-based pilot for a top tier U.S. healthcare company.

The DOD has relied on Revelytix for three years to create its data architecture and provide support to allow it to establish common architecture and semantics across all military service branches.

Revelytix CEO Mike Lang stated, "The management team at Revelytix has been working on complex data processing and data management problems for the federal government for the past 12 years. Our first company, Metamatrix, now part of Red Hat, produced data processing software used in the intelligence community and the DOD. Revelytix has been working for the DOD for the past four years specifically on the problem of processing and managing highly distributed sets of data. The resulting Revelytix technology is now in full production.

"A year ago we started incorporating our core intellectual property into a new product running on Hadoop clusters. The resulting product is Loom Dataset Management for Hadoop. It incorporates state of the art data processing and data management capabilities in Hadoop that have been forged in the real world on large sets of distributed data.

"In 2013, we will push ahead with broad implementation of the data management capabilities now in place at DOD. In fact, recent federal legislation is mandating that requests for DOD funding use the approach we've partnered with DOD to implement. There will be significant gains in the manageability of so much huge, important data."

The Hadoop pilot for healthcare sought to establish the scalability and flexibility of Hadoop in supporting the development of new analytic models. Once data sets were loaded into Hadoop, the approach "Start small, Scale quickly" was used. Beginning with samples of the data sets, queries were iteratively developed and tested. Users wrote queries in standard query languages (e.g. SQL) which the Hadoop-integrated query engine compiled to MapReduce. With each cycle the queries became progressively complex and were then deployed across the full data sets.

The pilot also sought to examine the data from new perspectives and required substantial data integration to create views which seamlessly span multiple data sets. The healthcare company also wanted to create transparency of data with industry partners.

The Revelytix approach was to first create a domain model that described the data sets using a desired semantic set of terms. The data was then mapped to that model and the Hadoop-integrated query engine was used to pull results from all data sets, thereby providing a complete picture – in this case, of consumer behavior.

CEO Mike Lang commented, "There are two important things we learned in this Hadoop Healthcare pilot. First, Hadoop is extraordinarily well suited to build the analytic environment for mega-scale healthcare data such as this. Without question, Hadoop will definitely play a big role in big data for healthcare.

"Second, our experience again confirmed the need for the Revelytix Loom solution for managing the huge complexity of Hadoop datasets such as this. The power of Hadoop spawns files from many perspectives and analytic approaches – keeping track of the data history of each approach is a significant and ongoing challenge. Having the pre-release of the Loom data management software and workbench allowed us to keep this pilot on track. We're looking forward to going further with this work in healthcare as well as other vertical markets such as financial services, telecommunications, life sciences and oil and gas."