The New Economics of the Separation of Compute and Storage
No one can predict the future analytical needs of their business. The right storage needs today might not be the right storage needs next year, or even next week. An organization’s workload requirements might grow five-fold by next year, or next quarter, or next month. With exploding data volumes, organizations are making significant investments in HDFS or S3 data lake infrastructure for storing “cold” data more cost-effectively. Given these volatile and rapidly changing scenarios, how do organizations ensure they can handle dynamic workloads and evolving data analytics requirements, as well as safe-guard their data lake investments?
Moving data in and out of various data stores is time consuming. Provisioning storage and compute resources for worst-case scenarios is costly. To avoid these hassles, organizations should ensure that their analytical platform provide freedom from underlying infrastructure, maximize investments in data lakes, and support their pursuit of cloud economics as well as on premise and hybrid deployments, while unifying the analytics that support them all.
Optimizing analytics for the cloud involves placing data on reliable and durable shared storage, separating storage from compute (along with the right amount of local cache), and matching the performance requirements of existing workloads. This allows organizations to tie data platform costs directly to business needs by provisioning just the right amount of compute resources for queries and just the right amount of storage resources for data. The ability to scale up and down the cluster elastically to accommodate analytic workload peaks and troughs results in lowered infrastructure cost and maximum business value. When separation of compute and storage is architected with cloud economics in mind, it results in greater operational simplicity and workload isolation to meet SLA and business stakeholder objectives – a win-win for any IT organization.
Separation of Compute and Storage in Action
Cloud economics and the separation of compute and storage bring even greater flexibility and financial viability to organizations looking to capitalize on the promise of big data and advanced analytics. Advanced analytics is changing the way companies across every industry operate, grow, and stay competitive. Let’s look at a few use cases that can benefit from this architecture.
Consider a medical equipment manufacturer that is going to market with a smart medical device embedded with more sensors than ever before. Before formally launching, they need to run a trial at their key customer accounts. By separating compute and storage, the manufacturer can optimize its resources to analyze the early-release testing data so that they benefit from maximum reliability upon the mass release.
Retailers who separate compute and storage can also strengthen their abilities to analyze seasonal sales patterns. Though analytic workloads in retail settings vary based on volume, they still require maximum dashboard performance – regardless of the number of concurrent users – so that key members can gain access to important sales metrics.
Engagement analysis can be streamlined with the separation of compute and storage, too. For instance, think of a gaming company that is about to introduce a brand-new game. Prior to launch, they must increase their analytical capacity so that they can evaluate success based on customer insights. This architecture would enable them to deliver real-time analytics during critical stages of the product launch, such as a special event or tournament, so that they can swiftly make modifications that better appeal to their community.
Understanding the Financial Benefit
These real-world examples can really be applied to any organization that relies on analytics. To truly grasp the immense cost savings by leveraging the separation of compute and storage capabilities, let’s elaborate on the retail use case.
Retailers struggle to accommodate the busiest shopping season of the year in the U.S., which begins late on Thanksgiving Thursday, continues through Black Friday and into the weekend, and concludes on Cyber Monday. The retailer must increase their compute capacity just for this seasonal peak – maybe a partial peak after Cyber Monday until about ten days after New Years once the returns are all in. For the rest of the year, the retailer is operating within a nominal compute capacity. Let’s break it down:
Assuming that the retailer needs:
- 20-node cluster for the Thanksgiving weekend and Cyber Monday
- 12-node cluster until after New Year, and
- Six nodes for the other months of the year
Scaling capacity to workload (figure) offers significant cost savings. A traditional solution (provisioned for peak workload year-round) vs. separating compute and storage will deliver savings of approximately 66 percent just in compute costs.
Perhaps the workload is uniform, just periodic. For short-term projects, separating compute and storage enables “Hibernating.” Hibernating lets you shut down all compute nodes until you need them again. Come out of hibernation when your project starts up again by simply creating a new cluster and reviving your database.
Lastly, most organizations have use cases requiring consistent compute requirements, as well as use cases that have variable compute need. Businesses need to support the right use cases today, knowing with certainty that they may change in the future. To achieve that end, seek options to buy compute power only when needed and reduce storage costs based on use case requirements. But, deployments in the cloud are always likely part of a larger, more hybrid strategy that needs to be kept in mind.
With so many existing as well as emerging SQL and NoSQL databases, query engines, and data warehouses options coming to market, businesses are feeling the pressure to understand how their data analytics platform can maximize cloud economics, while, naturally, deriving timely insights. This is even more important as the key attributes driving cloud adoption are operational simplicity, just-in-time deployments, and elastic scaling of resources. The separation of compute from storage provide maximum flexibility and enables organizations to minimize resource burden and maximize business value.
As cloud vendors continue to lower the price of storage and compute services, organizations have even more incentive to move their data to the cloud, especially for variable workloads and use cases that require heavy compute for finite periods of time. This is particularly attractive for New Economy businesses that have not yet earmarked the capital outlays and built out data centers to handle peak compute and storage needs – however rare or infrequent these peaks happen. In contrast, cloud economics enable organizations to pay for only what they need, when they need it. If this scenario sounds familiar, then the question that your organization should ask itself is: “How do I take advantage of cloud economics without being penalized for fluctuating requirements and dynamic workloads?”
About the author: Jeff Healey is the Senior Director of Vertica Product Marketing at Micro Focus. Jeff leads the product marketing team and digital property for the Vertica Advanced Analytics Platform. With more than 20 years of high-tech marketing experience and deep knowledge in messaging, positioning, and content development, Jeff has previously led product marketing for Axeda Corporation (now PTC). Prior to Axeda, Jeff held product marketing, customer success, and lead editorial roles at MathWorks, Macromedia (now Adobe), Sybase (now SAP), and The Boston Globe.
July 8, 2020
- Circonus Announces Free 45-Day Trial of its Kubernetes Monitoring Solution
- Talend Donates Nearly $3M in Data Skills Courses, Technologies to Higher Education
- HNI Corporation Taps Ascend.io to Fuel Operational Analytics
- GridGain Announces Nebula Managed Service For Apache Ignite and GridGain In-Memory Computing Platforms
- Data Mining System Unearths US Counties Most at Risk for COVID Deaths
- Researchers Create Online Resources for Data Exploration, Visualization, and Discovery for the Pan-Cancer Project
- Looker Now Available in the Microsoft Azure Marketplace
- Exasol Launches Exasol V7
- AIMART Enhances Investigative Intelligence Portfolio with Siren Partnership
- StataCorp Contributes Data Science Software to COVID-19 Research Database for Academic and Medical Research
- Petro.ai, AWS Partner to Operationalize Modern Analytics and Reduce Legacy Infrastructure Costs
- University of Kentucky Joins National Data Collaborative for COVID-19 Research
- Logical Clocks Joins European Initiative to Bring AI to 5G Networks with Hopsworks Feature Store
- Matillion Exchange Enables Faster Data Innovation with Community Contributed Pre-Built ETL Workflows
- Yellowbrick and Sonra Partner to Modernize Data Warehouses
July 7, 2020
- Dynatrace Expands AI-Powered Observability for Kubernetes Environments
- AtScale Expands COVID-19 Insights Model to Help Global Organizations Understand the New Normal
- Qubole Expands Integration with Informatica’s AI-driven Data Engineering Integration
- Virtual Esri User Conference to Explore How GIS Interconnects the World
- Dremio to Host Cloud Data Lake Conference
Most Read Features
- Big Data File Formats Demystified
- Nvidia Destroys TPCx-BB Benchmark with GPUs
- How to Build a Better Machine Learning Pipeline
- BI Tools — Are They Enough to Build a Data-Driven Culture?
- How COVID-19 Is Impacting the Market for Data Jobs
- Databricks Brings Data Science, Engineering Together with New Workspace
- Understanding Your Options for Stream Processing Frameworks
- SAS Provides Big Data Solutions for… Bees?
- MongoDB Steps Up Game with MongoDB Cloud
- Is Python Strangling R to Death?
- More Features…
Most Read News In Brief
- New Report Ranks Countries by COVID-19 Safety
- Spark 3.0 Brings Big SQL Speed-Up, Better Python Hooks
- IBM Brings Back a Netezza, Attacks Yellowbrick
- Blurred Lines: SAS and Microsoft To Go Deep in Analytics Partnership
- New Map Shows Hundreds of Counties in the COVID-19 Endgame — and Thousands on the Uptick
- U.S. Special Ops Launches $600M Analytics Effort
- NIH Launches Massive Initiative for COVID-19 Patient Data Analytics
- Data Prep Still Dominates Data Scientists’ Time, Survey Finds
- War Unfolding for Control of Elasticsearch
- Bitnine Looks to Scale PostgreSQL
- More News In Brief…
Most Read This Just In
- HSBC Joins Data Privacy Firm Privitar’s Series C Financing Round with $7M Investment
- D2iQ Unveils KUDO for Kubeflow to Accelerate Enterprise-Grade Machine Learning on Kubernetes
- SAS Debuts Tools to Gauge Risks and Impacts of Reopening
- Databricks Introduces Delta Engine, Acquires Redash
- Cloudera Debuts its Cloudera Data Platform Private Cloud
- The Linux Foundation Cloud Engineer Bootcamp Announced
- Technology Aims to Provide Cloud Efficiency for Databases During Data-Intensive COVID-19 Pandemic
- BP Invests $5M in Geospatial Analytics Software Company Satelytics
- MariaDB Announces the General Availability of MariaDB Community Server 10.5
- Alation Launches Data Governance Initiatives
- More This Just In…