Data Mesh 2.0: Realizing the Promise of Decentralization

(Oleksii Lishchyshyn/Shutterstock)

It’s fair to say that “data mesh” is having a moment.

Even a cursory search via Google News turns up dozens of related results, and there’s considerable variety in the mix. Sure, this happens with many new technology advances. Also, we all know how even disciplines with real promise can flame out as fast as they emerged, while some survive without matching their original splash. This is a dynamic arena, and rapid changes are an essential element in the bloodstream. But even with all those caveats, this evolving architecture might be worth the buzz.

Even by the normal levels of change around data-driven business practices, it seems apparent that data mesh offers true potential for meaningful change. In our constant journey toward a genuinely data-centric culture—an environment where every strategic decision and business operation is guided by actionable intelligence built by analyzing mountains of raw data in real time—it’s likely that data mesh will be a significant milestone.

The problem is that such a critical journey involves many milestones, just as no single technology solution can do it all. Data mesh is a solid foundation, but how can it be combined with other approaches to deliver greater benefits? If data mesh is so good, what will data mesh 2.0 look like?

First, a little context. While data mesh is often put in the same category as data lakes and data warehouses, it doesn’t really belong there. Those are essentially technologies used to store or move the data; data mesh instead focuses squarely on the data itself. The architecture more easily enables data ownership to be distributed across multiple business-centric domains and similar constituencies rather than a single, centralized authority. This democratization brings with it numerous related benefits, from analytics spanning heterogenous infrastructures to scaling as dictated by business needs.

At its best, this represents a remarkable vision. Within a data mesh, data is no longer hoarded by a new generation of elite specialists and stewards; it is accessed by business professionals to help them do their work. This is how it was always supposed to be.

(Aleutie/Shutterstock)

But the best is yet to come. The broad promise of a decentralized architecture is far from being realized. There have been some implementations and initiatives, of course, but they’re relatively limited in scope. The ‘data mesh success stories’ column is mostly bare.

We can do better—and it starts with a few core principles.

First, the concept of domain-based data ownership needs to be understood in its full context. More than just a change from a centralized power structure, what this means is that the data no longer belongs to the specific applications or the technologists who oversee those applications, but instead to the business itself. This is best achieved with emerging technologies that decouple data from the apps that create or store that data—and those solutions are now available. This is a fundamental change with major ramifications, and it will require significant changes in long-held best practices.

Second, enterprises need the ability to manage data as a product. This is no longer the incidental output of sophisticated technologies—data is instead a specific, identifiable and discrete product that can be separately owned and managed by one or more business domains. More accurately, we’re talking data products, not just one but many, all created to support both analytics and operational systems. This is revolutionary even though it’s another realization of our longtime vision of data as a corporate asset that can take its place alongside other, more tangible resources.

(Chombosan/Shutterstock)

Next, imagine the benefits of a self-servicing data platform. This constitutes a metadata-driven data browser that business users and technologists alike can use to collaborate—as in, discover, access, change, create and even originate data, all without creating new silos. There’s no question that data integration is the bane of the digital era: The mountains of data stored in house are housed (or rather locked away) in silos where they’re allegedly secure but isolated. Collating related data strands from different silos is unquestionably painful. This is so basic, and so common, that the numbers can still come as a surprise, but integration-related tasks can drain half the IT budget. A data mesh architecture boosted with a next-generation data platform goes a long way toward eliminating this issue.

Finally, there’s the issue of federated computational governance. In an environment where a range of rules and regulations related to security and compliance mandate stronger governance, this forward-facing solution delivers the ability to embed governance policies from the data product and business domain owners inside the data itself. This entails a huge operating advantage: No matter how end users experience the data—regardless of the device or other touchpoint, no matter what application is involved, and from any location—the permissions, controls, policies and privacy are guaranteed to remain consistent.

While these principles are distinct from the core of a data mesh approach, they are perfectly complementary. The allure of a decentralized architecture that enables true data democratization is irresistible—just as business users rather than the IT department now decide what hardware to buy, what software to use, and which applications to download, lines of business and the executives increasingly have the power to decide what data to analyze, and what custom data products to develop in order to meet business priorities.

Data mesh has been with us long enough to see its benefits, and appreciate the need for data mesh 2.0. There will be more advances down the road, for sure. But for now, the next iteration of this architecture will take us further down the road toward a data-centric universe than ever before.

About the author:  Karanjot Jaswal is co-founder and CTO of Cinchy, a data collaboration company. He spent over 10 years at leading global financial institutions where he was responsible for developing and launching leading enterprise solutions. He’s a regular contributor to public sector initiatives including Finance Canada’s Open Banking consultations, Ontario Data Privacy, and Smart City projects. He also speaks regularly at leading technology events such as Strata Data NYC.

Related Items:

Data Mesh Creator Takes Next Data Step

How to Maximize the Value of Data with Data Mesh

Data Mesh Vs. Data Fabric: Understanding the Differences