The Data Nerds Have Won
The data is in and the results are clear: The data nerds have won. That’s the consensus emerging from Looker’s user conference in San Francisco this week, where hundreds of data professionals congregated to extend their winning streaks with data.
“The data nerds have won,” Looker chief product officer Nick Caldwell declared during his keynote address at JOIN 2019 yesterday. “That is the way the world is today. The reason for that is…people don’t need to be convinced the data matters. They’re not asking why. They’re asking how to get it done.”
In the recent past, data was the provenance of technical experts who could get stuff done with data by writing elaborate Spark routines or programming in MapReduce. That was Caldwell at a previous job, where his boss thought he was a hero because he could join disparate data sets.
“This technology is advancing so quickly,” Caldwell said. “Five years ago I was learning MapReduce and Hadoop jobs, and I never would have imagined a world where I could pump 20TB of data into something like BigQuery where I could be running machine learning clustering algorithms on it and it only takes five seconds to run.”
But times have changed, and these days the level of technical aptitude required to manipulate large amounts of data has dropped considerably. The maturation of data warehouse offerings in the cloud, like Google BigQuery, Snowflake, and Databricks, is also a big part of that change.
This has helped unleash an army of data nerds, those regular employees who may not have any formal data training, but have gotten a taste for the power of data, and want more of it. Increasingly, the data nerds are demanding that all the applications they use provide access to data and ways to manipulate it.
That collective demand from data nerds presents challenge. “Maybe the data infrastructure is getting easier,” Caldwell said. “But demand certainly is picking up. That’s where the real challenge is. You now have widening demand from people who aren’t analysts, who all need data in some way to get their jobs done.”
One of the companies that’s benefiting from the democratization of data is Namely, a provider of SaaS-based HR solutions. As Namely’s business intelligence manager, it’s Alex Jia’s job to get data into the hands of Namely customers, as well as internal Namely users.
“It used to be a lot harder to use data correctly,” Jia said in an interview with Datanami at Looker’s JOIN 2019 last week. “Data used to be a privilege only afforded to large companies because it was so hard. You needed an on-premise server. You needed a DBA. You needed Vertica. It was so hard. Even with Hadoop, it was so hard. Now it’s to the point where I alone can just stand up the entire stack.”
Jia has set up analytic stacks for three companies over his career. But instead of building it from scratch, as data engineers would have done in the past, Jia is buying subscriptions to cloud-based solutions that have already been built. The solutions are largely pre-integrated, which dramatically lowers the level of technical complexity required to unleash data.
As Namely’s resident data person, Jia must be able to communicate with various constituencies. “You have to be able to go to the marketing team and speak their language,” he said. “I say ‘funnel conversion.’ They’re like, ‘yeah funnel conversion!’ I have to be able to go to engineering and say ‘We need to partition it this way because.’ You have to speak everyone’s language.”
Josh Temple, senior data engineer at MilkBar, supports about 30 to 40 internal users who want more data to support the fast-growing confectionary. Keeping these users flush with data and analytics would have kept a handful of full-time analysts busy. The company, which sells via a chain of stores and online, looked for alternative approaches.
“I knew MilkBar wasn’t going to be buying a 10-person data team any time soon, so I needed something that could act as a force multiplier,” Temple said. “In Looker, you could really give your business users the keys to the kingdom and let them build and run their own queries.”
Whether you want to call them data nerds or not, these users at MilkBar and Namely are fairly typical of the current marketplace dynamic for data. The explosion of data, coupled with the lowering of technical barriers, is changing the data expectations of today’s business user. They want more data and better data, and they won’t be stopped.
“We are an industry in transition,” Looker CEO Frank Bien said. “We’ve seen the shift to the cloud. It’s probably as much a fragmentation as anything else. We’ve seen the traditional tools that we used to have aren’t really carrying over. We also see that users don’t want the same thing. They want new stuff. Better stuff.”
Looker is clearly hoping that it can deliver the better stuff that today’s data users — today’s data nerds — are expecting. To that end, the company is emphasizing the application development aspects of the Looker platform, as well as its embedded analytic capabilities.
Keeping this new group of users satisfied will not be easy, particularly with the explosion of data silos and SaaS applications. Looker says the average enterprise today uses more than 1,100 SaaS applications, create a potential nightmare for integration.
This explosion of data and SaaS apps will pose challenges to any data strategy, partiucary with new regulations requiring IT professionals to secure and govern data. Trying to cope with this new data reality using yesterday’s data management and analytics tools is a non-starter, according to Caldwell.
“IT is right to be concerned. If there’s more data, it exposes them to risk,” he said. “But the question is, if there’s a proliferation of SaaS application, a proliferation of connections, and everyone is coming at you, those old systems aren’t going to work. It’s going to take too long to connect all those in a safe way if you use the older approaches.”
So far, this strategy seems to be working for Looker. The approach obviously has the support of Google, which is acquiring Looker for $2.6 billion. According to Looker execs, the Web giant is not looking to change Looker’s strategy, which appears to have the support of data nerds.