DataTorrent
Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan


January 22, 2013

A Meta-Analysis of MetaData


Meta-fiction and meta-commentary exist as self-referential forms of the larger topic as a whole. When a scandal breaks in the sports world, for example, most journalists will report on and analyze the events while a few will embark on meta-journalism and discuss how the scandal reflects how the world reports on things.

While these forms can be interesting ways in forming what is relevant in narrative versus what is not, they are relatively uncommon. Metadata, on the other hand, is not uncommon in technology—it exists everywhere. For example, the computer upon which this article is being written is compiling metadata, such as the word document’s time of creation (a meta-reference). And yet, the subject of metadata is somewhat underrated in the context of big data. Embarcadero’s Director of Product Management Henry Olson discussed with Datanami the issues of metadata efficiency and outlined the company’s top-down approach to help alleviate those issues.

“A lot of organizations have invested in [metadata] governance and have gotten shelfware out of it. Many others don’t know where to start,” Olson said of the problems companies have today in efficiently handling their metadata. “Our approach is to work with our customers to build an enterprise information map.”

Someone who wants to index the entire archive of Twitter such that researchers can efficiently and effectively search the archive (like the Library of Congress is doing) has to deal with (and ultimately rely on) the metadata. While the text in a given tweet may be useful to provide one variable, a second or third variable will likely come from one of the 50 metadata fields attached to a given tweet, with tweet time and location being common examples of those fields.

The fields most likely to be of use to historical researchers are that of time and location. A large scale retailer looking through social media data for insights may also latch onto other fields, such as a user’s interests and followers along with time and location, to properly personalize upcoming marketing pushes and evaluate existing campaigns.

Finding the relevant information by using the underlying metadata as a guide, especially in the context of large organizations with several databases, is one of the larger problems Embarcadero is working on, according to Olson. “In large organizations, we’ve found that there’s an enormous amount of metadata, let alone data, and metadata volumes get pretty vast. So we are continually looking to improve the selectivity of our search algorithm so that when somebody is searching on a business concept, they tend to quickly find what is most relevant.”

The key is to identify which processes are what Olson calls “business-critical”—something for which large organizations with big data requirements already have a decent idea. This top-down concept identifies and organizes the oft-used and most essential of the data sources, from which point Embarcadero goes to work. “This approach focuses on what is going to be most critical in terms of revenue, cost, and risk profile. Those are the processes that we’re most interested in.”

“The critical thing about the top-down approach is that at the end of the day, it’s the processors and the performance of the processors that dictates the business outcome,” said Olson.

Their approach is to provide the underlying architecture between the information storage, management units and the governance units that determine notable metrics, as noted in the image below.

The next step is to discover which databases contain the critical information and, more importantly, which can be left undocumented and unorganized. To explore this aspect, Olson discussed a situation where a company needed to determine efficient distribution methods.

“If you’re searching in an analytic context, you need to get at that production data or its representation in the data warehouse. We can quickly eliminate lots and lots of those databases that are associated with the CRM system or the logistic system or other systems and focus on the ones that have value in the context of a particular search.”

According to Olson, Embarcadero asks which business processes are most critical and moves from there. Taking a bottom-up approach to metadata would be akin to making thousands of barrels, each containing one possible ice cream flavor, before opening an ice cream shop in which only about ten are requested with any regularity.

 “A lot of times when organizations deal with enterprise architectural governance initiatives, they get so lost in the forest. They’d be taking along detailed documentation of every database and attribute as an organization.” In this sense, Embarcadero’s mission is to lower the ice cream maker’s scope such that he can make the most efficient use of his production time to fit his clientele.

However, limiting scope of the metadata at which one is looking may eschew some important fields. With that being said, this approach fits the current climate where storage ability outstrips processing power.

“By starting from the standpoint of the most critical business processes, we automatically exercise scope control, leading to a more pragmatic approach. If we were to tell our customers that step one is to document every database in their organization, it’d be a non-starter.” The monetary and time costs, Olson argues, would not be worth the ensuing efficiency kick.

Olson noted that some companies have developed their own systems with the same “breadth and depth” as that of Embarcadero’s. However, Olson argued that those systems are mostly self-serving and geared toward their own products. “There are a few others that have that breadth and depth, but most of them are focused on their particular product line,” said Olson. “For example, Microsoft probably covers all of these steps but they’re very focused on Microsoft technologies. We’re Switzerland, we have a history of working with all of the leading database products and the ability to deliver information to variety of different places.”

Related Articles

Building the Library of Twitter

Greenplum, Kaggle Team to Prospect Data Scientists

IEEE Tags Top Tech Trends of 2013

Share Options


Subscribe

» Subscribe to our weekly e-newsletter


Discussion

There are 0 discussion items posted.

 

Most Read Features

Most Read News

Most Read This Just In



Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia

NVIDIA

Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
StampedeCon
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014
ISC'14
Leipzig
Germany

» View/Search Events

» Post an Event