Follow Datanami:
January 31, 2018

Learning IoT Lessons the Big Data Way


Spending on Internet of Things (IoT) projects is expected to triple over the next few years, to more than $450 billion globally, as the technology matures and use cases emerge. As organizations embark upon exciting new IoT projects, we’ll likely run into some of the same challenges – and learn some of the same lessons – that early big data practitioners ran into.

That’s the prediction from Bill Schmarzo, the chief technology officer of the Big Data Practice of EMC Global Services, who is researching IoT and talking about it these days. Schmarzo has been critical of the inflated expectations the industry as a whole brought to the big data conversation, and in particular how people seemed to believe that if you adopted big data technologies and ran enough data through them, that they would somehow magically generate good results.

Schmarzo was ahead of the curve in popping the bubble that had built up around the whole Hadoop ecosystem. And while he’s a supporter of Hadoop as a distributed data store, he’s on the record for encouraging businesses to put more forethought into their big data projects (such as his simple SAM test for big data projects) and just being aware of the risks that new technologies bring.

Now, as spending around IoT ramps up, Schmarzo is concerned that we’re doomed to repeat the same mistakes that businesses made at the peak of the big data craze. “As an industry we tend to do a really bad job of relearning our lessons,” Schmarzo told Datanami in an interview last week. “We’ll break our pick. We’ll go and chase lots of [IoT] technology…The use case will not be well-defined, so the business returns will be marginalized.”

Like anything in life, careful planning in IoT increases the chance of success

While it may be possible to put a sensor in something and start pulling data from it, Schmarzo cautions business decision-makers about moving too fast with IoT. Instead, we should slow down and think it through before jumping into the IoT fray. But Schmarzo admits that’s not likely to happen.

“I predict, like we have in the big data space, that we’ll stub our toe, we’ll over promise, we’ll over hype,” Schmarzo said. “And then we’ll realize that Stephen Covey had it right the whole time, ‘Begin with an end in mind’ [which is the second habit of highly effective people]. What are you trying to achieve? Let that drive what IoT solutions you want to get and what architecture you’re going to put in place, and all the other thigs that are required to support that particular use case.”

Taking the long-term view with IoT solutions will be difficult because the short-term payoff can actually be pretty good. People are enticed to move quickly with IoT projects because they have a compelling return on investment, Schmarzo says. “If I can buy this thing to optimize energy usage and it pays for itself in 18 months, that’s a no brainer,” he said.

However, as companies keep embarking upon new IoT projects, that short-term ROI will come at the expense of long-term costs associated with adopting disparate technologies that do not, as yet, work well together, Schmarzo said.

Schmarzo recommends that organizations take a three-pronged approach to IoT projects if they want to achieve long-term success with them. The three stages are integration, scalability, and operationalization.


What the IoT industry really needs – what organizations need from the industry, really — is an IoT reference architecture that vendors can build towards, Schmarzo says. Without a standards-based way of integrating all the technologies and data that come from IoT solutions, it significantly lowers the odds that organizations can achieve long-term success with IoT.

“I’ve seen organizations that are making decisions to go out and buy bunch of one off, point IoT solutions, and the challenge is, instead of data silos, they have architectural silos, because these architectures don’t really play well with each other,” he said. “It’s very hard after the fact to try to figure out how do I tie architecturally all these IoT solutions together.”


Once you have a successful proof of concept for an IoT solution, scaling it up can present problems, both architecturally as well as cost-wise, he says.

“Once you have these things, how do I scale across an organization?” Schmarzo says. “I don’t want to have to have separate energy optimization solution for each of my business units. I want to have one that scales across all of them.”


Before one can celebrate a successful IoT solution, one needs to think through what kind of day-to-day management and maintenance the IoT solution will require. If an IoT solution can generate good results, but keeping it running is not economically feasible, then it’s probably not worth pursuing in the long run.

The Dean of Big Data, Bill Schmarzo. Follow him on Twitter at @Schmarzo

“How do I make certain that what I put in place I can manage, that I can get maintenance work, that I know the upgrade path?” Schmarzo says. “I can very easily see organization get into an environment where you have all these different applications on different upgrade schedules, and trying to manage that across the enterprise could become an absolute IT nightmare.”

These are all some of the same challenges that have plagued big data projects. While some organizations have worked through the problems associated with managing huge amounts of data, operationalizing the data science work, and then automating the data engineering work, many more have floundered and failed.

Schmarzo’s IoT advice parallels his big data advice: Let the use cases tell you what you need to do. Identify something that works, then validate that it’s real, and vet that it’s feasible. Build out a roadmap of other use cases, and then prioritize the order that they will be executed.

But of course, many organizations won’t be nearly so methodical with their IoT approaches. The will be doomed to learn the big data lessons all over again.

“[IoT] is a new source of data and I think you’re going to see organizations that are very interested, motivated, and fanatical about how to monetize the data,” Schmarzo says. “You’re going to see organizations saying, ‘We’ve got IoT, we’ve got sensors kicking off data, what are we doing to monetize it?’ You know that half of the data doesn’t mean [dink]. What are you going to do with it?”

While some IoT adopters will swing for the fences and try to find use cases that make a big impact, there’s a good business case to be made for projects that yield small 1% to 2% gains. If you string up enough of those small gains together, you can have something that’s worthwhile, Schmarzo says.

But IoT for its own sake isn’t worth the time or effort they require. “I see them putting sensors on toilets and mirrors. How’s that going to benefit me? How does that help me make a better decision?” Schmarzo says. “If it’s not helping me, I don’t want it. You’re just adding complexity where something could very well go wrong.”

Related Items:

Hadoop Has Failed Us, Tech Experts Say

Big Data’s Relentless Pace Exposes Old Tensions and New Risks in the Enterprise

One Deceptively Simple Secret for Data Lake Success