Big Data and the $465 Billion Xmas Prize

Between now and December 25, Americans are expected to spend nearly half a trillion dollars on gifts and goodies for themselves and loved ones, according to the National Retail Federation. Retailers will use all the tricks in the book to get their share, including the latest in big data analytic technologies.

There’s a lot on the line for retailers, especially with the annual year-end shopping extravaganza set to officially kick off on Friday. This year, retailers have been especially eager to get things rolling early and have succumbed to “Christmas Creep.” In some retail sectors, all of last week was Black Friday.

Aside from directly manipulating the calendar, there’s still a lot that retailers can do to maximize their sales and profits this holiday season. Retailers have more data than ever about the wants and desires of their customers. Combine that with advanced analytics, and you have a potentially winning combination.

A Product Prediction Machine

A big part of the game is predicting which products will be winners and which will be duds. This buying process has traditionally been one part retail science, one part black art, and one part luck. But increasingly, retailers are increasing their odds of having a winning hand by doubling down on data science.

A company called First Insight is using a combination of predictive analytics, crowdsourcing, and gamification to help retailers like Abercrombie and Fitch, The Limited, PacSun, and David’s Bridal optimize their product mix.

First Insight targets its games at the informed consumer and product enthusiasts who closely follow a particular market, such as shoes for teenage girls or baseball bats. The games take several shapes. In some, the game encourages users to look at sketches or CAD models of new products, and guess which ones will be popular (think “Farmville” for retail), while another asks them to guess how much a product will sell for (think “The Price is Right”).

FirstInsight encourages consumers to show their merchandising expertise with online games.

The company uses data science to closely analyze who takes part in these games and what they do within the games, says First Insight CMO Jim Shea. “We’re looking for a subgroup of experts in the crowd,” he tells Datanami. “In any given crowd there’s going to be certain people who are more predictive of what’s going to happen next season and next year and others who just don’t know.”

Not all of the products the players see are new. The control for the study comes in the form of older products with known histories. The players who are better at guessing past winners and losers are more heavily weighted for the new product predictions. “In the end, it’s a very simple dashboard output to product developer or merchant buyer–these are the winners and these are the dogs,” Shea says.

For any given customer, it takes about 300 or so players to get an accurate assessment of a product. In addition to getting the “thumbs up” or “thumbs down” on products, retailers and brands get forward-looking price curves, price elasticity curves, and optimal entry points for pricing of new products. The data also lets retailers set different markdown cadences, and can even inform the retailers on how much merchandise to stock. Overall, this approach gives a three to nine percent boost in sales using this technique, the company claims.

Real-Time Pricing

It’s too late for retailers to pick which products will be a big hit this Christmas. Those decisions were made long ago. But increasingly, big data analytics is helping retailers react to consumer trends as they occur.

One company on the cutting edge of real-time analytics for retail operations is BeyondCore. The San Mateo, California company built a large distributed cluster that uses MapReduce techniques to crunch millions of its customers’ data points. It’s not used exclusively in retail, but when used in this industry can help optimize the placement and pricing of products.

In the old days, major retailers would set up war rooms on Black Friday with the goal of keeping ahead of customer sentiment over the holiday weekend. “The merchandising team would run lots of analyses very quickly trying to figure out which products are selling and where they should send the products,” says BeyondCore CEO Arijit Sengupta. “It’s a very complex analytics project. This is where a lot of mistakes happen because you’re under such pressure and you have to analyze it so quickly to find these patterns that even experts get tired and make mistakes.”

BeyondCore’s analytics infrastructure can crunch that data faster and more accurately than humans can ever hope to do it, Sengupta says. “BeyondCore does that analysis really quickly without any human error,” he says. “It distinguishes between what’s statistically sound and the

BeyondCore CEO Arijit Sengupta

things that are not and presents it back to the analyst in the way they expect it so they can act on it.”

The massive scalability of clustered analytic systems like BeyondCore’s is enabling retailers to squeeze more insight than ever out of their data. That is allowing retailers to analyze data related to merchandising, marketing, and logistics together instead of analyzing it separately.

“If you combine all of those variables together, the computational complexity of that analysis goes way beyond what a human can do,” Sengupta says. “If I have 100 variables, there are a million variable combinations…But at BeyondCore, we can take the logistical, merchandising, and marketing information and optimize it all. That is extremely difficult to do without big data techniques.”

This approach allows retailers to respond much more quickly to evolving consumer trends. Instead of waiting until Cyber Monday to see how things shook out over the weekend, they can start looking at the sales returns early tomorrow morning. (Black Friday has merged with Thanksgiving for many retailers this year.)

“When the analysis can be done in minutes across millions of variable combinations, you can actually say, I will learn the behavior on Thursday morning and change my merchandising and promotion decisions on Thursday,” Sengupta says. “You couldn’t do that without automated systems that are learning the behavior of the people.”

Related Items:

How Big Data Analytics Is Shining a Light on Anonymous Web Traffic

Mining for YouTube Gold with Hadoop and Friends

Congratulations Hadoop, You Made It–Now Disappear