Leverage Big Data
Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan


July 19, 2012

Research Lifts Cluster Weight Off Graphs


If one wanted to analyze the graph of a large social networking site such as Twitter, it would be an uphill climb without the weight of a significant cluster. And in many cases, even then, the result would not be reached for several hours.

Carlos Guestrin of Carnegie Mellon University thinks that is all about to change. Guestrin, the codirector of CMU’s Select Lab, and his team have developed GraphChi, software that stores graphs in a computer’s hard drive as opposed to its RAM. Delving into the hard drive for graph storage has apparently led to some remarkable results.

The researchers behind the project say that GraphChi can run massive graph computations on a single machine via the novel algorithm for processing the graph from disk (SSD or hard drive). Programs for GraphChi are written in similar vertex-centric model as the parent project, GraphLab.

GraphChi runs vertex-centric programs asynchronously (i.e changes written to edges are immediately visible to subsequent computation), and in parallel. GraphChi also supports streaming graph updates and changing the graph structure while computing.

According to a recent description of how some are seeking to solve graph problems, “a Mac Mini running GraphChi can analyze Twitter's social graph from 2010—which contains 40 million users and 1.2 billion connections—in 59 minutes.” Guestrin puts that into perspective, adding, “The previous published result on this problem took 400 minutes using a cluster of about 1,000 computers.”

Typically, hard drives are slow to read and write data, making them less than ideal for handling graphs. The lynchpin of GraphChi appears to be a system designed by Aapo Kyrola which accesses the hard drive with efficiency and order. If the hard drive problem has been solved, the power of the individual computer could indeed be unleashed. “PCs don't have enough RAM to hold an entire Web graph, but they do have hard drives, which can hold a lot of information,” says Guestrin.

This new infrastructure is hinted at but not explained in detail on GraphChi’s site. “GraphChi runs vertex-centric programs,” explains the site, “asynchronously (i.e changes written to edges are immediately visible to subsequent computation), and in parallel. GraphChi also supports streaming graph updates and changing the graph structure while computing.”

Even if GraphChi may not yet be able to handle the big data demands of the BI world, Guestrin predicts that researchers will find plenty of uses for the software, especially when it comes to cleaning large amounts of data. “A researcher in computational biology could do large-scale computations on their PC; a developer working on a data-center algorithm can test it on their laptop before pushing it to the cloud… Tools like GraphChi will let many companies and startups solve all their graph-computing needs on a single machine.”

Related Stories

Cray Puffs 70k Handout for Graph Computing Prowess

The Science of Social Sentiment

Big Data Cloud Delivers Military Intelligence to U.S. Army in Afghanistan

Share Options


Subscribe

» Subscribe to our weekly e-newsletter


Discussion

There are 0 discussion items posted.

 

Most Read Features

Most Read News

Most Read This Just In

ISC'14

Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia

NVIDIA

Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
StampedeCon
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014
ISC'14
Leipzig
Germany

» View/Search Events

» Post an Event