Power Laws and the Never-Ending Big Data Boom

(spainter_vfx/Shutterstock)

We’re about a decade into the big data boom, which may seem like a lifetime for those who have worked through it. But in many ways, we’re just getting started with what big data and AI technology is capable of giving us, Forrester principal analyst Brian Hopkins said at Hortonworks DataWorks Summit yesterday.

Forrester’s respected analyst gave a sneak peek into his vision for the future of business and technology during his keynote address before an estimated 2,100 people yesterday at the DataWorks Summit, which continues today and tomorrow at the San Jose McEnery Convention Center.

Hopkins had some practical advice for data practitioners. For instance, he encouraged customers to reconsider their big data architectures. Instead of building batch-oriented data lakes and relying on DataOps techniques to run it, he told the audience they were probably better served by investing in big data fabrics, real-time edge processing, and “NoOps” serverless cloud implementations.

Metcalfe’s Law was proven with Facebook data

But it was Hopkins’ views on where technology is headed and how businesses can prepare for it that really resonated.

Hopkins observed that several computing “laws” are currently working themselves out, even if we don’t notice them on a day-to-day basis. The first of these, Moore’s Law, is probably the best known. And indeed, we’re told on a regular basis that Moore’s Law is petering out as processors reach physical limits.

But other laws are picking up where Intel founder’s Gordon Moore’s leaves off. That includes Metcalfe’s Law, which states that the effect of a telecommunications network is proportional to the square of connected users. Metcalfe’s Law was first postulated in 1980 by Robert Metcalfe, who founded 3COM (now part of HPE) and co-invented Ethernet. But it wasn’t until 2013, when Metcalfe had access to Facebook social networking data, that he was able to prove it, Hopkins says.

Another law working itself out is Bezos Law, which says the cost of cloud computing power decreases by 50% every three years. Bezos Law, of course, was named after Amazon founder Jeff Bezos, who is now the richest man in the world. “We’re just getting started with that,” Hopkins says about Bezos Law. “We really don’t know. We haven’t measured that enough to know if it’s true.”

Exponential Growth

It’s possible there are other power laws operating out there that haven’t been discovered yet. Hopkins notes that the futurist Ray Kurzweil, author of the 2005 book The Singularity Is Near: When Humans Transcend Biology, predicted that other Moore’s Law-like laws could be expected to exist in exponential systems.

“In fact he said that Moore’s Law is not the exception. Rather we should expect Moore’s Law to be the rule,” Hopkins says.

Forrester principal analyst Brian Hopkins speaking at DataWorks Summit

Hopkins was also influenced by Kurzweil’s 2001 paper, “The Law of Accelerated Return,” which described what happens when one technology builds on another and accelerates the returns. “You can see that, for instance, in Hadoop. Then it was Spark. Then it was ML and AI and on and on and on. And it’s not going to stop.”

It’s easy to be overwhelmed by the pace of technological progress, but if Kurzweil’s predictions are true — that technology will force the next 100 years of human history to be the equivalent of 20,000 years of human evolution — then we really haven’t seen anything yet.

Hopkins says he initially doubted that prediction. “But at the end of the day, you can really kind of see it happening before our very eyes,” he says. “The pace at which things are growing and doubling.”

Insight-Driven Firms

When faced with the rapid pace of technological innovation, some of us are content to let the bleeding edge “bleed out” before we jump in. That may seem like a safe strategy, Hopkins says, but it has an inherent flaw in a world where these exponential power functions are helping to further accelerate the pace of technological change.

Waiting for new tech to mature before adopting it is akin to having a “drafting strategy,” Hopkins says. “What’s the problem with a drafting strategy?” he asks. “When the guy in front of you can accelerate faster than you — that’s the problem.”

Hopkins says Forrester research shows that companies that aggressively adopt data technology, or what the analyst group calls “insight-driven firms,” have better business results than those that don’t.

“One of the things we noticed once we began to identify these firms is they were growing about 30% annually,” he says. “Big public companies don’t grow like startups…. GE was growing at about 3.5%, which means these insight-driven firms were growing simply because they were taking customers from firms that weren’t insight-driven.”

Clearly, winning companies are the ones that can more adeptly navigate and adopt big data tech despite the breakneck acceleration we’re currently experiencing, and which we can expect to continue. Companies are spending hundreds of millions of dollars to digitally transform themselves. Some are even spending billions, Hopkins says, to be in the winning side of the ledger.

The risks and the rewards don’t get much bigger, and technology is viewed as the key differentiator between success and failure. “And it’s dangerous for you because, if you don’t understand and make the right technology moves going forward,” he says, “you could get left behind when you step on the accelerator and the engine sputters out.”

Bronze Age of Big Data

The good news is there’s reason to believe we’re still in the earliest days of big data. If Kurzweil’s prediction about 20,000 years of human evolution being crammed into the next 100 years, then there’s a huge amount of technological growth still ahead of us — and lots of room to catch up (provided you don’t adopt a drafting strategy).

Twenty thousand years ago, our human ancestors were still in the Stone Age, but on the cusp of entering the Bronze Age, which was a major step in our technological progress. That led Hopkins to theorize we’re not far from entering the Bronze Age of Big Data.

“When you think about what’s going on and the sustained business acceleration, I think we are absolutely just getting started,” he says.

Related Items:

Machine Learning is Everywhere: Preparing for the Future

Jeff Dean Thinks AI Can Solve Grand Challenges–Here’s How