Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan
Leverage Big Data'14

April 05, 2012

Data Mining for the Masses

If we were all granted an hour-long crash course in data mining for every bit of information we’ve passed on about ourselves this week alone, something says that we’d be seasoned experts by now.

There seems to be growing awareness about the pervasiveness of data mining with a rash of recent news items and profile pieces covering the topic.

Arguably, the case of the pregnant teen that was ratted out by Target’s data mining oracle brought it to light first, but just the other afternoon the topic was front and center on popular talk programs, including NPR’s Diane Ream Show.

What many of these conversations overlooked was the actual process behind the data wrangling. In fact, some might say that the popular media has started “demonizing” the term “data mining”—associating it with covert corporate prying or at its worst, government espionage on its own citizenry.

This week, the long-standing, general interest magazine, The Atlantic, seemed to address this issue by publishing a rather detailed article that describes data mining as a concept, science and practice—and informed the masses why this is one technology they should fully understand in this world of sensors, signals and connectivity.

Without sentimentalizing in either direction on the topic, Alexander Furnas, a master’s candidate at the Oxford Internet Institute and author of the data mining cheat sheet, removes the judgment and explores what data mining is, isn’t, and what it might be in the future.

Even though it seems to many outside of tech circles that  data involves some kind of strange arcane magic, “the functioning of data mining algorithms is quite complex….[but] the users and capabilities of these approaches are, in fact, quite comprehensible and intuitive.”

 “For the most part, data mining tells us about very large and complex data sets, the kinds of information that would be readily apparent about small and simple things. For example, it can tell us that "one of these things is not like the other" a la Sesame Street or it can show us categories and then sort things into pre-determined categories. But what's simple with 5 datapoints is not so simple with 5 billion datapoints.”

In his condensed “everything you need to know” article, he manages to provide succinct definitions that put the core concepts of pattern recognition in context. He offers definitions of anomaly detection, association learning, cluster detection, classification and regression—using examples from businesses and common practices, including Amazon’s recommendation engine and spam filters for email clients.

He says that the tools and techniques of data mining are increasing in sophistication, not just because of the natural course of discovery and new approaches to data mining come to light, but also because there are so many data sources with so much subtlety, the tools behind the new wave of “big data” are essential.

As he writes, “At these scales patterns are often too subtle and relationships too complex or multi-dimensional to observe by simply looking at the data. Data mining is a means of automating part this process to detect interpretable patterns; it helps us see the forest without getting lost in the trees.”

We give kudos to Furnas for writing an overview guide to a complex topic for a general audience and using a Sesame Street reference without sounding patronizing. As he writes—right to the very heart of where the big data problem fits in—the world of data mining is becoming more complex as more data becomes available and the tools to manage it are overextended.

This is critical information for ordinary citizens since understanding how far even a little data can go can help people make smarter decisions about how their share and protect their information.

The problem is, while it’s nearly impossible to live off the grid and outside of the purview of data mining from governments and companies alike, too few people understand what goes on behind the scenes of both of these entities with information we’ve unwittingly or unthinkingly given up.

Related Stories

The New Era of Computing: An Interview with "Dr. Data"

How BI is Becoming Crime’s Biggest Enemy

TRUSTe CEO Addresses the New Privacy Ecosystem

Share Options


» Subscribe to our weekly e-newsletter


There are 0 discussion items posted.


Most Read Features

Most Read News

Most Read This Just In

Cray Supercomputer

Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia


Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014

» View/Search Events

» Post an Event