Language Flags

Translation Disclaimer

HPCwire Enterprise Tech HPCwire Japan
Leverage Big Data'14

October 23, 2012

Hadoop Illuminates Smart Meter Data

The advent of the smart meter has dramatically changed the landscape of the energy industry. Instead of utility companies relying on one readout per month for billing purposes, they are now provided with several readings per day on multiple measurements.

Smart meters represent a significant opportunity for utility companies, allowing them to predict and respond to energy spikes and equipment failure.

However, with that great opportunity comes great challenge, especially from the data perspective. SunGard held a webinar with several of their experts and CommodityPoint’s Patrick Reames to address these challenges through identification, problem-solving, and cost-cutting strategies, many of which point to Hadoop from a data analytics standpoint.

According to SunGard’s Melista Anderson, there will be a 500% growth in smart meter deployment from 2010 to 2015. This growth would represent a huge spike in the amount of data that utilities are collecting.

“For a mid-sized utility of a half-million customers,” Anderson said “the deployment of smart meters will result in an explosive growth of customer data, increasing the number of reads from six million per year to almost 18 billion per year. That’s a 3000-fold increase in the amount of data that then must be captured, ordered, stored and analyzed in near real-time.”

Real-time analysis is critical in the energy industry and it’s not difficult to see why. Utilities need to respond as quickly as possible to energy spikes to avoid costly shortages and even blackouts. But, of course, there exist other less dramatic examples, such as correlating weather changes to energy usages in certain areas.

One challenge not to be underestimates is the increased federal regulation that arose as a result of increased data access. According to SunGard’s Stephen Nimmo, utilities are almost required to process and analyze their data to show their intent to regulators while also keeping their raw data intact. “The solution to this problem, Nimmo remarked, “is to stop throwing data away.”

While conceptually simple, that response requires a significant data investment on the part of utilities. That’s before the utilities even start using the data to do work and deliver any sort of usable insight.

According to Reames, the smart meters are a big reason as to why traditional relational databases are being overrun. Instead, it is time for utilities to turn to Hadoop-type frameworks for storing and running their data.

 “Big data sets go up to multiple petabytes in size and this requires new tech and approaches such as Hadoop,” said Reames, “a framework for processing these massive data sets.”

There are several reasons that Reames and SunGard’s Neil Palmer point to as for why a Hadoop-based system could prove advantageous to the energy industry, including the ability to handle both structured and unstructured data, high levels of availability, and its open-source nature which, according to Reames, allows for “near-constant improvement.”

Most important, however, is its scalability, answering the fundamental question of utilities wondering how to store and process the petabytes of data they’re receiving from their smart meters. “To some extent,” Palmer said “Hadoop is replacing the initial pit stop for all data coming into an organization.” Palmer cautions that this may not always be wise and that existing BI infrastructure should not be discarded out of hand, but that Hadoop does offer the scalability necessary in the energy industry.

In this context, Hadoop has its limitations. For example, while physically accessible to those who want to invest in it, it is relatively inaccessible from a functional standpoint in that few know how to work confidently in it. “Adoption by the business users is critical,” said Palmer. “Disruption to their day-to-day lives is not acceptable and they need new ways to be able to understand and mine the data that’s out there.” However, Reames feels that as the Hadoop market advances and matures, so too will the ability of vendors to cater to the business users.

For Palmer, Hadoop belongs to one of four big data analysis subsets which will eventually be crucial to the energy industry. Other platforms, such as graph databases like Neo4J, could help utilities eventually model their entire grid in what would be a huge step for predicting shortages. Per Palmer, “Graph databases such as Neo4J are excellent for mining relationships. If you want to understand the connections between e-traders and counterparties and accounts, a graph database is an excellent place to start.”

As meters get smarter, so too must utilities. Those hanging around Hadoop World in New York City right now may be able to help them with that.


Related Articles

Mortar Takes Aim at Hadoop Usability

Utility Companies Riding Data Current

Six Super-Scale Hadoop Deployments


Share Options


» Subscribe to our weekly e-newsletter


There are 0 discussion items posted.


Most Read Features

Most Read News

Most Read This Just In

Sponsored Whitepapers

Planning Your Dashboard Project

02/01/2014 | iDashboards

Achieve your dashboard initiative goals by paving a path for success. A strategic plan helps you focus on the right key performance indicators and ensures your dashboards are effective. Learn how your organization can excel by planning out your dashboard project with our proven step-by-step process. This informational whitepaper will outline the benefits of well-thought dashboards, simplify the dashboard planning process, help avoid implementation challenges, and assist in a establishing a post deployment strategy.

Download this Whitepaper...

Slicing the Big Data Analytics Stack

11/26/2013 | HP, Mellanox, Revolution Analytics, SAS, Teradata

This special report provides an in-depth view into a series of technical tools and capabilities that are powering the next generation of big data analytics. Used properly, these tools provide increased insight, the possibility for new discoveries, and the ability to make quantitative decisions based on actual operational intelligence.

Download this Whitepaper...

View the White Paper Library

Sponsored Multimedia

Webinar: Powering Research with Knowledge Discovery & Data Mining (KDD)

Watch this webinar and learn how to develop “future-proof” advanced computing/storage technology solutions to easily manage large, shared compute resources and very large volumes of data. Focus on the research and the application results, not system and data management.

View Multimedia

Video: Using Eureqa to Uncover Mathematical Patterns Hidden in Your Data

Eureqa is like having an army of scientists working to unravel the fundamental equations hidden deep within your data. Eureqa’s algorithms identify what’s important and what’s not, enabling you to model, predict, and optimize what you care about like never before. Watch the video and learn how Eureqa can help you discover the hidden equations in your data.

View Multimedia

More Multimedia


Job Bank

Datanami Conferences Ad

Featured Events

May 5-11, 2014
Big Data Week Atlanta
Atlanta, GA
United States

May 29-30, 2014
St. Louis, MO
United States

June 10-12, 2014
Big Data Expo
New York, NY
United States

June 18-18, 2014
Women in Advanced Computing Summit (WiAC ’14)
Philadelphia, PA
United States

June 22-26, 2014

» View/Search Events

» Post an Event