Follow Datanami:
June 28, 2016

MapR Takes Aim at Hadoop Complexity

As Hadoop settles into the enterprise, deployments are getting bigger and more complex. To help its customers stay on top of things, MapR Technologies today unveiled Spyglass, a new operations console that’s customizable. The company also rolled out the first deployment pack that separates the core components of the distro from fast-moving projects like Spark and Drill.

MapR started the Spyglass initiative in response to input from MapR customers who were watching the complexity level increase in their MapR deployments. While they aren’t necessarily struggling now, they expressed concern that, if the trends continued, that monitoring the cluster would become increasingly difficult.

Many MapR customers are running multiple applications on their clusters, and keeping on top of everything can be difficult, according to MapR’s senior director of product marketing Dale Kim. “We’re seeing that many of our customers are running multiple use cases–over 50 use cases on a single deployment,” Kim says. “Having that kind of environment can be complex. Having a system like the Spyglass initiative will allow the customers to be able to better understand what’s going on in the system.”

Spyglass is based on core open source technologies, including ElasticSearch for managing log files, OpenTSDB, and Kibana and Grafana visualization tools. The software, which runs atop the MapR-DB, exposes metrics that MapR tracks internally to the visualization layer. Users can choose which visualization tools they want to use (Kibana is the default) and customize their dashboards. MapR also provides customers a way to share their dashboards with others in the community.

The key to the product, which competes with Apache Ambari, is extensibility and flexibility, explains Anil Gadre, senior VP of product management. “This gives the user choice and flexibility to decide what they want to see. You don’t have any choice in Ambari,” he says. “It’s a big deal here to be able to create your own dashboards and more importantly, share it with other admins. You get to decide. You have the freedom to pick and choose ad to create dashboards, or use the few that we provide to start you with.”

It will become more difficult to manage Hadoop clusters as organizations use it to run more applications, and more critical applications. “The minute it becomes more critical to the business, the cluster administrator wants to know that that engine, so to speak, is going to hum along as efficiently as possible,” Gadre says.

As Hadoop clusters grow into the hundreds or thousands of nodes, having intuitive management tools becomes more critical. “This is something that’s going to make the cluster admin more productive,” he says. “It’s going to give them more insight into how to plan for growth and managing multi-tenant, which is increasingly showing up as a major asset we have, and dealing with the needs of multiple business units.”

MapR is also exposing an API that will allow customers to plug their favorite visualization tool into Spyglass and be able to tap into the many cluster metrics that MapR is exposing to clients. While there’s a bit of data science behind the scenes in terms of aggregating the data and making it presentable, the intended users for Spyglass are IT professionals, not data analysts or scientists.

The company also used Hadoop Summit to announce the GA of its Ecosystem Pack. The goal of the Ecosystem Pack is to separate the fast-moving projects, such as Drill and Spark, from the slower moving projects, while maintaining version compatibility.

“We’re the leaders in the decoupling model of Hadoop, to separate platform upgrades from open source project upgrades,” Gadre says. “This is the next evolution of that, where we can make interoperability among multiple projects in a given pack so customers can get the latest version of projects, but also get a certified set of projects that will work together.”

Kim says separating the platform from the open spruce projects makes sense from a compatibility point of view. “Customer are going to the latest versions of some projects even though they may not necessarily need it, and find some incompatibilities with other open source projects,” he says. “What we’re trying to say here is now you get a certified view of these various projects that will work together.”

It’s a similar approach that MapR’s competitor, Hortonworks (NASDAQ: HDP), is now taking with its platform. The big difference is that Hortonworks is synching its releases to the ODPi‘s release cycles. MapR is still not buying what the ODPi is selling. “We’re constantly monitoring that,” Kim says. “I don’t think the ODPi has changed in anyway.   Some of our objections from the start still remain. It’s not clear if it’s resolving anything.”

The ODPi and MapR have different goals, he adds. “The ODPi is trying to gain more visibility for some projects that haven’t really been as popular in the market as certain vendors would expect,” he says.