DataTorrent
Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan


August 14, 2012

Cloudera CTO Reflects on Hadoop Underpinnings


“All of this data felt like an ocean and all these technologies only offered a small straw. We don’t want to use a straw, we want to jump in and swim in the ocean,” remarked Cloudera’s Co-Founder and Chief Technology Officer Amr Awadallah.

The Hadoop distro company’s CTO and co-founder discussed his company’s beginnings at the Cloud 2012 conference in Honolulu, Hawaii. He related his company’s beginnings to the challenges of big data as well as talking about the improvements Hadoop has made over the several years along with the improvements his company, Cloudera, has made themselves to Hadoop with the recent release of CDH4.

Awadallah related Hadoop to a simple computer operating system, nothing that an operating serves two purposes: to store files and to run applications on top of those files. According to Awadallah, Hadoop does that as well but obviously on a much larger scale. Further, Hadoop offers what Awadallah calls ‘scalability of humans.’ What he means by that is that developers who generate queries for two or three nodes can run those exact queries on hundreds of nodes as a result of Hadoop.

According to Awadallah, there were four ‘headaches’ whose problems they were looking to alleviate: data growing rapidly, the many forms of data, data’s dynamic schemas, and the increased cultural need to “leverage data as an asset.” When he looked at the high performance computing landscape in 2006 while working for Yahoo, he realized that none of their offerings fit his needs. He thought the answers were in the cloud, and that this was the dawning of the “cloud era.”

Alas, “Big data,” Awadallah says “does not lend itself well to external clouds today. It might change in the future but there are too many logistic boundaries.” So they shifted their focus to the superior Hadoop technology but kept the now-recognizable “Cloudera” moniker.

Hadoop is also very good at preventing system failures. Awadallah says it has “so much smarts in it that predicts and routes against failure of hardware, failure of network, so that only one or two system admins are needed for every thousand nodes.” Combining the ability to run local queries on a system of a thousand nodes with self-servicing and self-diagnosing capabilities such that only one or two people are required to maintain it indeed scales a large computing cluster to a handful of humans.

Awadallah notes the removal of the single name node failure in talking about recent improvements his company has made to Hadoop with CDH4. Namely, those improvements include even more scalability (up to tens of thousands of nodes and breaking the 4,000 node barrier), faster performance (up to 100% faster in latency intensive jobs), and higher availability. Specifically, Awadallah mentions HBase replication which will help head off catastrophic data loss in the (less likely) event of a systems failure.

While Awadallah noted that there remained challenges in virtual infrastructure like Hadoop, namely that it relies on an inefficient centralized structure, he is as optimistic for the future of Cloudera and Hadoop as is he is pleased with Hadoop’s progress so far.

Related Stories

Cloudera Plots Enterprise Invasion

Six Super-Scale Hadoop Deployments

How 8 Small Companies are Retooling Big Data

Share Options


Subscribe

» Subscribe to our weekly e-newsletter


Discussion

There are 0 discussion items posted.

 

Most Read Features

Most Read News

Most Read This Just In



Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia

NVIDIA

Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
StampedeCon
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014
ISC'14
Leipzig
Germany

» View/Search Events

» Post an Event