Google Adds In-Memory BI Option to BigQuery
Response times for interactive queries in Google BigQuery are about to get better thanks to a new in-memory BI option added by the Web giant. Google is also adding a new migration service for moving data from AWS S3 and RedShift, as well as a new Flex Slot usage option.
The new BigQuery BI Engine option will deliver blazing fast responses to ad-hoc queries driven by third-party BI tools like Qlik, Tableau, and Google’s own Looker, among others, says Debanjan Saha, the vice president of engineering and general manager of data analytics at Google Cloud.
“Today if you want to do interactive analytics, it takes a second or a couple of seconds to get the responses back,” Saha tells Datanami. “Here the response time is going to go to milliseconds.”
The BI Engine option uses Google’s own in-memory processing technology, rather than the disk-based processing method that BigQuery typically uses. Google decides how much memory the computation is going to require, and reserves the RAM on the servers that BigQuery uses, but none of that scheduling complexity is exposed to users.
It’s all about giving customers a faster and better user experience with their analytics, Saha says. “If you have a have a dashboard and you saw something and you want to drill down and make the next iterative set of queries, those requests will be very quick,” he says of the in-memory option. “So the interaction that you have will be much faster.”
The BigQuery BI Engine will work with most BI clients that customers use with the Google Data Warehouse. The company recently acquired Looker, but it hasn’t developed any special capabilities using the Looker BI or data modeling tools, Saha says. “Of course we are happy to now have Looker in our family,” he says. “We’re looking forward to close integration for Looker.”
Similarly, the new Flex Slots pricing mechanism is all about giving customers more flexibility to scale their BigQuery environment up and down quickly. Flex Slots gives customers the flexibility to reserve BigQuery resources down to one-second intervals. But it also gives customers the security of knowing they don’t have to use the resources and that they’ll get a refund for unused portions (beyond 60 seconds).
According to a blog post published today, Google is selling each flex slot for $30 per slot per month. The slots are available in increments of 500 slots (meaning you have to buy at least 500). Customers can reserve Flex Slots in the BigQuery Reservations environment, and combine them with other usage patterns, such as Pick Slots.
“Flex Slots is like Pick Slots, but instead of committing for a year or a month, you can commit for a second,” Saha says. “That gives people the experience they wanted: predictability and flexibility.”
BigQuery is gobbling up lots of enterprise data warehouse workloads these days, from on-premise systems like Teradata and Oracle Exadata, as well as from competing cloud solutions like AWS RedShift. The company has worked with partners to build migration services for those on-premise warehouses, and now it’s launching its own migration program to move AWS customers from S3 and RedShift to BigQuery Storage or Google Cloud Storage (GCS).
The new Google service automates much of the work of moving the data from one cloud to the other, Saha says. Customers enter their credentials, and Google initiates the data movement out of RedShift and S3 into BigQuery Storage or GCS. Most customers store their data in RedShift in an open format, like Parquet or ORC, Saha says. Google BigQuery can also support those formats – in addition to its own internal data format, called Capacitor — which makes the migration easier.
Thanks to query federation features that Google added last October, the location you choose to store your data in Google Cloud doesn’t matter as much any more. “One of the things that BigQuery does very well is integration with data lakes,” Saha says. “BigQuery can have data internally [in Capacitor] or BigQuery can access data in GCS. Sometimes people want to join data in GCS as well as BigQuery data.”
Enterprises are scrambling to move data into the cloud to take advantage of flexible processing capacity and advanced analytic tools. In years past, Google has struggled to gain traction with enterprises compared to its larger cloud competitors, AWS and Microsoft Azure. But with accounts like KeyBank, HSBC, Home Depot, Zulily, 20th Century Fox, AccuWeather, Wayfair, and Vodafone, it would appear that Google is making progress in the enterprise.
“We are seeing both enterprise customers as well as born-in-the-cloud customers,” Saha says. “We have made a number of announcements with respect to new customers and existing customers who have increased their use of BigQuery. BigQuery is one of the reasons that people come to GCP, and that momentum is growing.”