A Bet Against Hadoop That Hopes to Deliver Big Gains

Two years ago, Greenplum veteran Ben Werther made a bet against Hadoop that led him on a journey to improve how enterprises could utilize and provide access to the data housed in the platform. This week, the company he founded, Platfora, launches the in-memory analytics platform that is the culmination of that bet.

“I just felt that betting Hadoop was going to magically become fast was not the right bet,” explains Werther in discussing the genesis of his company Platfora, and the creation of its flagship platform, which was released into general availability this week.

According to Werther, the challenge that organizations most often face when getting underway with a Hadoop cluster is that it is often bottlenecked between analysts and database administrators, and complications that arise in trying to prioritize and run batch jobs in competition with each other.

 “The question I asked after leaving Greenplum and DataStax is ‘what is the native stack for Hadoop going to look like?’ The endpoint can’t be that ‘look you have to make this work, you have to plug into the old traditional environment and reintroduce all of that inflexibility and long cycle time.” And so Werther got to work to create a new way for users to gain access to company data, endeavoring to learn the lessons of past models.

There had to be a better way, rationalized Werther, who sought to find a way to compliment the flexibility of Hadoop’s big batch engine by providing a scale-out, in-memory “lens” that could offload cluster data and allow business analysts to explore as much (or as little) of the data as they wish, while freeing up the Hadoop cluster for the next analyst.

“The key then is as they visualize, explore, and use that data, if they want to go through and get more detail – if they want to find some interesting set of users or population that they want to examine in some different way – they can always change what they’re interested in just a few clicks, and it will drive Hadoop automatically and redistill the additional detail, bring it into the in-memory layer, and then maybe 10 minutes later they’re back to fully interactive work again against that refined view.”

This approach, says Werther, puts the data back in the hands of the business analysts, and keeps them from having to get back in line with the database administrator for their next pull from the Hadoop system. The net result, says Werther, is higher productivity, and shorter decision-making cycles as business analysts are able to work in near real-time with the company’s data.

Werther cites Platfora client, Riot Games, as an example of this shortened discovery and decision-making cycles. Riot produces one of the most played online PC games in the world, League of Legends, which some analysts estimate logged over 1 billion hours of user play last year, generating an enormous amount of data.  The game, says Werther, is very focused on the user experience, being a free-to-play game.

Werther says that Riot lands all of their data into Hadoop, and then perform analysis against the data to help them create new ways to engage the user.  The value that Platfora offers Riot, says Werther, is the ability to quickly experiment and move into production new ideas and game concepts.

“If they built some new part of the game with some new game mechanics, for them to understand how that would work and to do development under previous models, it might take them weeks to build out that piece of analysis,” claims Werther. “Using Platfora, they can move more quickly to experiment and refine the game, look for problems, look for different types of behavior, and tune the game to reflect the data and improve the user experience.”

Platfora helps reduce the complexity as the game evolves, claims Werther, who explains that as the game evolves, new types of data are being fed into Hadoop constantly.

“With Platfora, now for the first time, as new data goes into the cluster, they don’t have to do new programming to extract it – they don’t have to do the things that previously would take weeks to get going. They’re able to immediately map the new data sets that are being created into Platfora literally the same day and start to visualize and understand behavior – look for problems, look for opportunities, look for different characteristics, etc.”

Using Platfora, says Werther, Riot’s developers are now able to be more creative with how they explore the company’s data. “Now they can sit down in an afternoon and try a number of different hypotheses around the data, and just go where their intuition takes them rather than being forced to justify a business case at every step which may miss opportunities.”

Related Items:

Self-Service Data Mining, Hold the Bottlenecks 

Platfora Cuts Through Big Data Hype and Delivers Hadoop Value 

How 8 Small Companies are Retooling Big Data