Follow Datanami:
May 30, 2013

Harnessing the Power of Big Data to Reverse Retail Fortunes

Monte Zweben, CEO, Splice Machine

For many U.S. consumers, the twin-barreled retail holidays of Black Friday and Cyber Monday in 2012 offered the first chance to clear the cobwebs off of their credit cards. Early returns showed strong growth over 2011, especially for e-commerce, with IBM touting a 30.3 percent increase in online sales figures on Cyber Monday and comScore reporting a 26 percent increase in online sales on Black Friday.

With 57 million shoppers visiting online retailer sites on Black Friday alone, there is a vast amount of data being generated with every click. Smart retailers have been tracking, analyzing and adjusting based on in-store and online customer data for years. Every time a website serves up a recommended item, it’s dipping into the data pool for the perfect complement to what you (or even someone like you) have browsed, previously purchased or left behind in your online cart. It may seem like Big Brother, but it’s the precursor to Big Data-driven retail.

There are many definitions of Big Data depending on the application. In the retail and e-commerce world, Big Data could be defined as the merging of multiple, substantial data sources to create new insights. An example of this would be taking point of sale (POS) data and merging it with analysis of social media data to look for a correlation between book sales and online buzz. If a retailer could make better assumptions about what was going to be a top seller days in advance, it could increase promotion of the item, stock more of the item to prevent empty shelves and order more of the item to prevent stockouts.

Better use of data can boost distribution efficiencies and in-store, POS effectiveness. With brick-and-mortar retail sales lagging, this double-barreled combination is vital to success in the online age where the remains of retailers like Borders, Circuit City and Blockbuster litter the road. The changing landscape has lead major stores like Gap, Sears, and Abercrombie & Fitch to close hundreds of surplus stores in order to cut costs and tighten their real estate portfolio. After all, if there’s no longer a Gap store at the local mall, you can rest assured that is open all day, every day. This paradigm shift is just one of many the retail industry has undergone in the last 50 years.

During the 50s and 60s, retail success was tied to availability and proximity because saturation had not occurred and retailers could compete simply by being the closest option. In the 70s and 80s, retail outlets saturated the suburbs and competition shifted to price. Big box stores drove down prices – and drove small retailers out of business. Then, in the 90s and 00s, the Internet boosted price and selection competition to an extreme, crippling and even crippling and even killing big box chains like Circuit City such as Barnes and Noble’s eReader gambit, the NOOK.

Leading retail companies in this era, like Zappos and Apple, have proven that responsiveness and personalization matter to consumers – a service that Big Data can help every retailers with. Now, Big Data offers the opportunity to save retail by enabling companies to provide mass customization and exceptional service, both of which increase differentiation and margins. The question then becomes, how can your business harness its data to reach the peak of the retail industry?

The Era of Downtown Department Stores and Suburban Shopping Malls

The average consumer may not realize that they’re not the first person to buy what they’ve purchased. What ends up on a store shelf or in an e-tailer’s website is traditionally determined by a merchandise buyer who has to weigh myriad factors to determine what items to buy and at what quantity. If the economy is forecast to be slow, the buyer may shy away from big-ticket items, fearing they won’t sell unless deeply discounted. Or, if the weather forecast is colder for the winter season, it might be prudent to order more show shovels and fewer umbrellas.

In a perfect world, every item purchased by a store would sell at its intended full price and no customer would ever have to look twice for an item due to shortage. But as any “Tickle Me Elmo” line or 75% clearance sale illustrates, retail buying has long been an imperfect science, ruled by trend-spotting and risk-taking individuals.

A less than ideal buy at a corporate or store level naturally leads to a less optimal selling situation in a store. Looking back in time to big-city department stores and then suburban malls, retailers had a finite consumer base and imprecise means to reach them. Their biggest weapon – advertising – could bring a small number of products to a large number of people with each impression, which works for clearing the inventory of popular merchandise, but not for finding the couple who wants that yellow sofa the store has five too many of.

It was a time before the data-driven analysis of stocking, store layouts and point of sale data, but it was also a time that was not equipped to capitalize on these insights. Without a dynamic supply chain, a wider base of customers and the ability to target more products to more people, a retailer with Big Data in the 1950’s or 1960’s would be frustrated. So why, now that this framework is in place, are retailers still unable to optimize their stock and sales process?   

The Era of Multi-Channel Retailers and Supply Chain Optimization

The rise of the big box store and its effects on the mom-and-pop shop is a story that’s been written many times before. To recap, supply chain efficiencies, multi-channel flexibility and better analytics enabled cutthroat pricing, seemingly infinite inventory options and superior customer reach that drove the big box store revolution.

When we look at these three causes, we can begin to see how data played its part and how the application of Big Data is leading to new solutions.

The Supply Chain

In today’s retail world, every transaction is a signal into another system. It communicates which shelves need to be stocked, how much inventory needs to be delivered from the distribution center, and even how many more units need to be created, generating yet another cascade for every part and process.

This interconnection creates a more efficient supply chain that can place inventory where it’s needed in hours or days, instead of weeks and months. The role of the store buyer gives way to inventory managers who can look at shorter timeframes and quickly see results from their decisions. This ability to read, react and read again enables the delivery of a better retail experience. The whole concept of data-driven recommendation engines and personalized shopping falls down if the supply chain cannot keep pace, especially in brick and mortar locations.

Multi-Channel Flexibility

Built on top of this intelligent supply chain is a multi-channel selling environment that provides more choice to the consumer as well as more flexibility and insight to the retailer. The addition of e-commerce to storefronts allows retailers to reach consumers where they are and when they want to shop, but it also gives retailers the ability to broaden their inventory with less risk of dead stock weighing down their balance sheet.

But e-commerce is more than just a place to sell online – it’s a place to collect mountains of data about each customer. With cookies nearly ubiquitous on websites, a visitor can be identified, analyzed and served with targeted ads. The success or failure of these measures can be calculated, and the methods refined to generate more sales. Now, the advent of social media has created yet another layer of data and another avenue for retailers to reach their audiences.

Better Analytics

The approach of worrying about each shopper instead of every shopper is a construct of the e-commerce age where retailers often know who is on their site, what they like and what they don’t like. This enables them to not only feature products based on what is most likely to be purchased, but also to personalize the online advertising and email communications, a foundation of driving sales. 

However, the old generation of technology makes it difficult to perform analysis with the speed needed to impact change on the next day, let alone in the same day. If retailers could combine real-time point of sale data, inventory levels, trending topics from Twitter and tomorrow’s weather forecast, they could deliver timely offers to their customers – online, and even in-store – in ways that the store buyers of the past could never imagine.

NEXT — The Era of Big Data and the Personalized Shopping –>

The Era of Big Data and the Personalized Shopping Experience

So where does retail go next? It goes Big…and it goes personal. With new levels of intelligence gleaned from multiple data streams and then analyzed in real-time, retailers will be able to get personal with their online, mobile and in-store customers like never before. These are ideas that I wrote about in The Perfect Message at the Perfect Moment for the Harvard Business Review in 2005, but now the scale of data is overwhelming and truly enables these next generation approaches.


Online personalization is already in place, but the sea change will come when retailers can process all of their data streams in real-time and return even better results in an instant. This is especially critical during the high-volume holidays, when shoppers are inundated with deals. It’s the antithesis of the traditional, big box Black Friday approach where a cheap flat screen TV was used as a draw to bring in the crowds. Big Data will enable online retailers to improve their advertising effectiveness and leverage a personalized product to attract each customer, going beyond price comparison to a comparison of service.


As smartphones become commonplace, the average shopper has a 24-hour storefront in their pocket where they can receive email ads, do research, read reviews and make purchases. The shift is already underway as, according to IBM’s Cyber Monday 2012 report, “more than 18 percent of consumers used a mobile device to visit a retailer’s site, an increase of more than 70 percent over 2011.”

While some retailers might see mobile as yet another channel to fret over, others are grasping the full opportunity that a location-aware device provides. Mobile apps and sites that are powered by a Big Data platform can generate unique content for every user, depending not only on who they are, but also where they are. This integration will also be a large part of in-store personalization in the years ahead.


Personalization is not just for online interactions. Already, high-end retailers “clientele” by asking their customers to identify themselves in return for better service and personalized offers. When the salesperson says, “let’s look up your wife’s size,” she also gets an automated hint of a suggested cross-sell based on what you or your wife bought recently. Imagine taking this to the next level – where every retailer offers you self-serve clienteling – and all you have to do is click on a smartphone app or, even easier, send a short code text message that results in 1) identifying the customer, 2) locating them at a store, and 3) providing compelling offers and messages.

The ultimate Big Data approach will be able to process everything in real-time and then return new results in an instant. In a time of retail conglomerates with multiple brands under one roof, it could be possible to buy a pair of shoes in one store and then have the next store recommend a matching belt. The velocity of data needed to turn these science-fiction ideas into reality is achievable with the right solutions.

Recommendations for Retailers

Technology is going to play an increasing role in monitoring, mapping and molding consumer behaviors, and determining where companies that best harness the wealth of data at their fingertips will stand now and in the years ahead.

While “privacy advocates” are sensitive to the methods that can be used to transform the shopping experience, many customers now appreciate being marketed to directly instead of being one in the crowd. If executed correctly, Big Data personalization solutions should never be confused with Big Brother, but rather, leave a shopper thinking: “that salesperson really knows me.”

Retailers should be using the data points of the 2012 holiday season as a stepping-stone to 2013. By collecting, combining and analyzing more data, smart retailers can use Big Data platforms to boost their sales year round and create their own data revolution.

About the Author:

Monte Zweben is the co-founder and CEO of Splice Machine, a San Francisco-based company that provides a leading SQL-compliant database designed for Big Data applications. A technology industry veteran, Monte was the founder and CEO of Blue Martini Software – the leader in e-commerce and multi-channel systems for retailers – that is now part of Red Prairie. Monte also worked for NASA, Red Pepper Software, PeopleSoft, SeeSaw Networks and Clio Music prior to starting Splice Machine. Monte is the co-author of Intelligent Scheduling, has published articles in the Harvard Business Review, including The Perfect Message at the Perfect Moment on relationship marketing, and has participated in various computer science journals and conference proceedings.



Related Items:

Watson Moves Into the Call Center 

Alteryx Driving Towards Line-of-Business Analytics 

Big Transaction Data Creating Big Value