In recent decades, there has been a data revolution, and most companies have realized that data can deliver a great deal of value. Some of them have adopted a data-driven approach, spending a large part of their budget on building a data lake infrastructure with a central data team, aiming to capture that value for their companies.
Consider how a data lake infrastructure works today. On one side, we find the plane of operational or transactional data: the data generated and used by applications. In the middle, we have the analytical plane, or data lake, which stores the data for analysis. Finally, on the other side, we would find the data scientists and data analysts, the people responsible for diving into the data lake to find the data they need for their reports or models. Between these planes and people sits a large number of data extraction, transformation, and loading pipelines; typically, data engineers are responsible for implementing and maintaining this part.
martinfowler.com/articles/data-mesh-principles.html
Today, however, companies have realized that this monolithic, centralized infrastructure has some problems. The central data team, made up of data engineers, analysts, and scientists, has become a bottleneck for many companies. This team starts to struggle to answer all the questions coming from management and other departments as quickly as needed. That is a huge problem for the data-driven philosophy, since timely decisions must be made to stay competitive.
The central data team wants to answer all these questions quickly. In reality, they have to spend a lot of time fixing errors when trying to add new data sources, creating new extraction, transformation, and loading pipelines that fit this structure. On the other hand, they struggle to serve all the consumers of this data with truly high-quality data, because the central data team has to understand each new domain to answer questions in a meaningful way. Acquiring the knowledge of each domain is a difficult task. As a result, these problems make it impossible to realize the real value that the data-driven philosophy can deliver.
In addition, companies have invested in domain-driven design, autonomous domain teams, and decentralized service architecture (you can read our blog post on serverless). These domain teams own and understand their domain, including the business's information needs. They design, build, and run their web applications and APIs on their own. Even with all this knowledge, these teams have to go to the central data team to get data-driven insights.
datamesh-architecture.com
As the company grows, the situation between the domain teams and the central data team gets worse. The solution to this problem, and the core idea of data mesh architecture, is to shift responsibility for data from the central data team to the domain teams. Domain-oriented decentralization for analytical data. A data mesh architecture enables domain teams to perform cross-domain data analysis on their own and interconnects data, much like APIs in a microservices architecture.
Principles of data mesh architecture
Data mesh is a term coined by Zhamak Dehghani in 2019, and it is based on four core principles:
- Domain ownership is a principle that assigns responsibility for each domain to the team allocated to it. Under this principle, analytical data should be organized around domains, just as the team is limited to the bounded context defined by the system. Following domain-driven distributed architecture (DDD), responsibility for analytical and operational data lies with the domain team, rather than the central data team.
- The data as a product principle seeks to apply the product thinking philosophy to analytical data. This principle tells us that there are consumers for the data beyond the domain. The domain team is responsible for meeting the needs of other domains (consumers) by providing high-quality data. In other words, domain data should be treated like any other public API.
- Self-serve data platform seeks to apply the platform thinking philosophy. A team dedicated to the data platform provides independent features, tools, and systems to build, run, and maintain interoperable data products for all domains. With its platform, this team enables domain teams to consume and create data products seamlessly.
- The federated governance principle enables interoperability of all data products through standardization, which the governance group promotes across the entire data mesh. The main goal is to create a data ecosystem that respects the organization's rules and industry regulations.
Components of data mesh architecture
A data mesh architecture takes a decentralized approach, allowing each domain's team to perform cross-domain data analysis on its own. The domain team ingests operational data and builds analytical data models to perform its own analyses. It uses the analytical data to build data products based on the needs of other domains.
At the center of the data mesh are the domains, which are related to one another. A domain can consume a data product from another domain and/or serve a data product to another domain.
Data product
A data product is usually a published dataset that other domains can access. For example, a table or view in a SQL database, a JSON file with event data stored in AWS S3, a marketing report with KPIs and charts in PDF format, or even a movie recommendation model.
To make the data product discoverable, accessible, and usable, it is described with metadata, including ownership and contact information, data location and access, update frequency, and a specification of the data model.
The domain team is responsible for operating the data product throughout its entire lifecycle. The team must continuously monitor and ensure data quality and availability. For example, keeping the data free of duplicates or reacting to missing entries.
Data platform
So that domain teams don't have to worry about implementing and maintaining the data infrastructure required to add a new data product, you need a team responsible for building a self-serve data platform.
The core functions of a self-serve data platform can be divided into two groups: analytical functions and data product functions. Analytical functions allow the domain team to build an analytical data model and perform analyses to make data-driven decisions. In addition, the platform needs functions to ingest, store, query, and visualize data on a self-serve basis.
A more advanced data platform for the data mesh also provides additional domain-agnostic data product capabilities to create, monitor, discover, and access data products.
The self-serve data platform should support domain teams so they can quickly build a data product and run it in production in their isolated area. The platform should support the domain team in publishing its data products so other teams can discover them. Discovery requires a central entry point for all decentralized data products. A data catalog can be implemented in different ways: as a wiki, a git repository, or even through existing vendor solutions for a cloud-based data catalog, such as Select Star, Google Data Catalog, or AWS Glue Data Catalog.
Actually using data products, however, requires a domain team to access, integrate, and query other domains' data products. The platform should support, monitor, and document cross-domain access to and use of data products.
Global governance
For all these relationships between domains to run quickly and with as little complexity as possible, you need to define a set of global policies: the rules that will define how new data products are created. The global governance body will be responsible for this task.
Interoperability policies are the starting point. They allow other domain teams to use data products consistently. For example, global policies could define that the standard way to provide data is as a CSV file in AWS S3, in a bucket owned by the corresponding domain team.
Next, there needs to be some form of documentation to discover and understand the available data products. A simple policy for this could be a wiki page with a predefined set of metadata, such as the data product owner, the location URL, and descriptions of the CSV fields.
A uniform way to access the actual data product securely could be role-based access in AWS IAM, managed by the domain team.
Global policies such as privacy and compliance are also common. Think of protecting personally identifiable information (PII) or industry-specific legal requirements.
martinfowler.com/articles/data-mesh-principles.html
Enabling team
Finally, to help domain teams and act as ambassadors for this architecture, you need a team of specialists with extensive knowledge of data analytics, data engineering, and the self-serve data platform to provide consulting and teach best practices to domain teams.
A member of the enabling team temporarily joins a domain team for a limited period, such as a month, as an internal consultant to understand the team's needs, set up a learning environment, train team members in data analysis, and guide them in using the self-serve data platform. They don't create data products themselves.
___________________________________________________
