Follow Datanami:
March 13, 2017

Will Overconfidence Kill Big Data?

Here’s an alarming disconnect:  Noted research firm Gartner predicts 80% of big data projects will fail by 2018, due largely to complexity and integration challenges. Yet in a just-published survey by Dimensional Research, 87% of data professionals feel confident that they can build a self-service big data infrastructure – despite the fact that almost all of them (88%) describe themselves in an ‘early stage of adoption’.

What does this mean? A lot of big data professionals don’t know what they don’t know – and they are going to find out the hard way. Big data failures are common because of an interesting paradox:  big data technology is attractive because it ‘scales out’. As you add more compute/storage you get a corresponding increase in output. Unfortunately the data teams supporting these big data platforms don’t. As you add more people, complexity and management challenges makes every individual less effective. Eventually you get to a point of diminishing returns where adding people makes no perceived difference (or maybe makes things worse). That makes building a highly scalable data platform that can serve hundreds or thousands of users/applications extremely difficult.

How will you achieve big data achieve success in 2017?

1:  Emulate the pioneers.
You know most of these companies because they’ve been very public about their use of self-service data platforms that truly scale:  e.g., Facebook, Uber, LinkedIn, Twitter. They developed not only the technologies required to scale but more importantly the processes, culture, best practices and tools to scale.

2:  Adopt a self-service approach to data and analytics.  
Lot’s of vendors talk about this – but almost none of them explain how. That’s largely because most of the changes are not about technology – they are about people, process and culture. The emerging DataOps trend is a tip off.  Work differently to enable a more agile, collaborative relationship between all the stakeholders involved in big data deployments.

3:  Learn from your peers.
Nearly everyone in big data is trying to figure out the same things. Trading best practices (or worst practices to avoid) and sharing perspectives and experiences can get you up the learning curve quicker and avoid repeating mistakes that others made before you. Whether it’s online, in local meetups or brader forums, aggressive peer-to-peer networking is a worthwhile way to mitigate risk.

4:  Automate. Automate. Automate.
Automate as many of the tedious, sometimes monotonous data engineering / DataOps tasks. Automation offsets loss of productivity as your big data deployment grows, and reduces considerable complexity. Qubole’s “secret sauce” is intelligent automation, a key component in building a scalable modern big data platform. In fact, we make the work of the data team easier, simpler and even fun.

Want to unlock all four doors to success, you should attend Data Platforms 2017, May 24-26 in Phoenix, AZ. This is a first-of-a-kind conference focused specifically on data professionals and their data platforms. There is an amazing lineup of speakers and attendees – including the leaders who built the data platforms a Facebook, eBay, LinkedIn, Twitter, Expedia and more.  Spend 2+ days immersed with your peers to learn from each other and get into technical detail on what they’ve done and how. Learn more about the conference

Data Platforms 2017