Why Integration and Governance Are Critical for Data Lake Success
This is the final article in a three-part series exploring what it takes to build a data lake capable of meeting all the requirements of a truly enterprise-scale data management platform. While earlier installments focused on enterprise-scale data management in Hadoop, data onboarding into the data lake, and security, this article will focus on two things: Integrating the data lake within the broader enterprise IT landscape, and data governance.
As more lakes are deployed, we see patterns emerge for how data lakes are positioned relative to existing databases, data warehouses, analytic appliances, and enterprise applications in larger organizations.
Data Lakes: Here to Stay
Some data lakes are deployed from the outset as centralized system-of record data platforms, serving other systems in an enterprise scale, data-as-a-service model. As a centralized data lake builds momentum, collecting more data and attracting more use cases and users, its value grows as users collaborate on improving and reusing the data.
Other projects start at the edge of the organization to deliver data and meet the analytic needs of a specific business group. A localized data lake often expands to support multiple teams or spawn additional separate data lake instances to support other groups who want the same improved data access as the first group got.
Regardless of what pattern the data lake takes as it lands and expands in the organization, the data lake’s increasing role in the organization brings with it new requirements for enterprise readiness.
To be enterprise-ready, the data lake needs to support a set of capabilities that allow it to be integrated within the company’s overall data management strategy and IT applications and data flow landscape.
Here are some requirements to keep in mind:
- It must be possible to automate and embed the process of interacting with the data lake so that jobs to update the lake with new data or deliver data out of the lake can be automatically called and executed in a lights-out production mode. This means that the data lake needs to provide a RESTFul API that can be called by other scripts or schedules in the environment and which exposes all the functionality needed to interact with the data lake in a production environment.
- The data lake needs to be able to export data and associated metadata in multiple formats so that data from the lake can be easily integrated with other applications or downstream reporting/analytic systems.
- The data lake needs to support development, test, and production environments and allow for the easy promotion of data ingest, data preparation, and similar assets developed in the data lake environment from one environment to the next.
- The data lake needs to make it easy for parts of the lake to be shared across separate Hadoop clusters so that in a large organization with multiple data lakes, data, metadata and related assets can be easily and consistently shared.
- It must be possible for metadata collected and generated in the data lake to be exchanged with other enterprise standard metadata repositories.
Governing the Lake
In addition to streaming the integration of your data lake, you must prepare the lake to support a broad and expanding community of business users.
As more users begin working with the data lake directly or through downstream applications or reporting/analytic systems, the importance of having strong data governance grows. This topic — data governance — is the final dimension of enterprise readiness.
By bringing together typically hundreds of diverse data sets in a large repository and giving users unprecedented direct access to that data, data lakes create new governance challenges and opportunities.
The challenges have to do with ensuring that data governance policies and procedures exist and are enforced in the lake. Enterprise-ready data governance in the data lake starts with a clear definition of who owns or has custodial responsibility for each data asset as it enters the lake and as it is maintained and enhanced through the data lake process. In addition, the data lake needs to include well-documented policies regarding the required accuracy, accessibility, consistency, completeness, and updating of each data source.
To monitor and enforce application of these policies, the data lake environment must automatically profile each data source on ingest with respect to the data quality, character, and completeness. Additionally, the data lake should automatically track and record any manipulation of data assets (cleansing, transformation, preparation) to provide a clear audit trail of all users and activities occurring in the lake.
Finally, when it comes to enterprise-scale data governance in a data lake, it is essential that restrictions are in place to ensure that people only see the data they should be allowed to see. (See part two of this series for more on the importance of authentication, authorization and data access controls.)
Virtuous Cycles at Enterprise Scale
But data governance’s role in a truly enterprise-ready data lake isn’t only to reduce risk and enforce controls. It can also create added value and enable richer broader collaboration around data across users and groups.
If designed properly, data lakes are unique in their ability to allow large populations of non-technical business users to access, explore, and enhance data as they move it along the evolutionary path, from raw source system data to business user ready information.
Good data governance abets this process by helping business users enhance data with crowd-sourced business metadata and tagging that adds context, business definition, and meaning to the data. Combined with data governance policies that selectively cull and promote the best of this crowd-sourced insight to “gold standard” data in the organization, participation of a growing group of business users in the enterprise scale data lake can create a virtuous cycle in which user participation enhances data, bringing more users, more enhancement, and ultimately more value to the lake.
About the author: Dr. Paul Barth is founder and CEO of Podium Data, creator of the Podium big data management platform. Paul has spent decades developing advanced data and analytics solutions for Fortune 100 companies, and is a recognized thought-leader on business-driven data strategies and best practices. Prior to founding Podium Data, Paul co-founded NewVantage Partners, a boutique consultancy advising C-level executives at leading banking, investment, and insurance firms. In his roles at Schlumberger, Thinking Machines, Epsilon, Tessera, and iXL, Dr. Barth led the discovery and development of parallel processing and machine learning technologies to dramatically accelerate and simplify data management and analytics. Paul holds a PhD in computer science from the MIT, and an MS from Yale University.
September 24, 2021
- AWS Announces General Availability of Amazon QuickSight Q
- IDC’s 3rd Platform Industry Spending Guides Provide In-Depth Sub-Industry Forecasts for Technology Investments Across Nine Industries
- Scality Awarded US Patent for Hyperscale Data Protection
September 23, 2021
- AtScale Expands Semantic Layer Solution for Microsoft Excel
- CNCF End User Technology Radar Provides Insights into DevSecOps
- At Annual OCEANS 2021, Sofar Ocean Debuts First-of-Its-Kind Maritime Open Standard, Bristlemouth
- Elastic Announces the General Availability of Elastic App Search Web Crawler, New Features for Elastic Enterprise Search
- Securonix Achieves FedRAMP In-Process Authorization
- EDJX and Cubic Corporation Partner to Launch the Internet of Military Things Edge Platform
September 22, 2021
- GigaOm Names Moogsoft an Industry Leader in “Radar for AIOps Solutions” Report
- Clearsense Acquires Plug-and-Play AI Analytics Firm
- Purdue University Global Launches Master of Science in Data Analytics
- Dihuni OptiReady CognitX Deep Learning Servers and Workstations Powered by NVIDIA Ampere Architecture-based GPUs
- Scality Awarded New U.S. Patent for Breakthrough Technology in Hyper-Scale Data Protection
- MicroAI to Bring AI Training to Renesas MCUs
- Recent Gartner VP Analyst Sanjeev Mohan Joins Okera as a Strategic Advisor
- C3 AI Reinvents Enterprise Software UX With C3 AI Data Vision
September 21, 2021
- Healthcare Analytics Summit 21 Virtual Kicks Off Today
- Tesco Selects Teradata Vantage to Drive Enterprise-Wide Analytics at Scale
- Ketch Secures $20 Million in Series A1 Funding, Accelerating its Rapid Growth
Most Read Features
- One on One with Google Cloud Product Director Irina Farooq
- Big Data File Formats Demystified
- Tabular Seeks to Remake Cloud Data Lakes in Iceberg’s Image
- What’s the Difference Between AI, ML, Deep Learning, and Active Learning?
- Who’s Winning In the $17B AIOps and Observability Market
- SambaNova Brings Custom Silicon To Bear on High-End AI Workloads
- In Search of the Modern Data Stack
- How the Coronavirus Response Is Aided by Analytics
- COVID-Driven Cloud Surge Takes a Toll on the Data
- Rethinking Education in an AI-First World
- More Features…
Most Read News In Brief
- LinkedIn Open Sources Tech Behind 10,000-Node Hadoop Cluster
- Data and AI Salaries Continue Upward March, O’Reilly Says
- Data Prep Still Dominates Data Scientists’ Time, Survey Finds
- Gartner Shuffles the Technology Deck with Latest ‘Hype Cycle’ Report
- Who’s Winning in Open Source Data Tech
- Bigeye Observes $45 Million in Funding
- Hands-Off: Manual Data Integration Tasks Plummeting, Gartner Says
- Can Apple Right its Privacy and Security Cart?
- Apollo CEO Bullish on GraphQL’s Potential in the Enterprise
- Why Is SAS Going Public?
- More News In Brief…
Most Read This Just In
- TIBCO NOW 2021 Showcases Limitless Power of Data
- Toloka Launches Data Research Grants, Announces First Eight Recipients
- Anaconda Announces Support for Pyston, Hiring Lead Developers Kevin Modzelewski and Marius Wachtler
- Kinetica Fuses Streaming and Contextual Analysis At Scale
- Aporia Launches Self-Serve Machine Learning Platform Open to Public
- MariaDB Announces SIS Provider Campus Cloud Services Migration to MariaDB SkySQL
- Transaction Processing Performance Council (TPC) Launches an Artificial Intelligence Benchmark (TPCx-AI)
- DataRobot Launches “DataRobot AI Cloud” Platform
- OneStream Previews New AI and ML Capabilities at Splash 2021
- JetBrains Launches Public Early-Access Program for JetBrains DataSpell IDE
- More This Just In…
Sponsored Partner Content
October 5 - October 7
October 12 - October 14
October 19London United Kingdom
October 27 - October 28
November 29 - December 3
December 6 - December 10San Diego CA United States