Data Products

Introduction

With the introduction of Data Products as first-class citizens of Blindata, Blindata offers a comprehensive solution for managing your data platform. This advancement facilitates seamless sharing and collaboration throughout your organization. Data Products play a pivotal role in enabling organizations implementing the Data Mesh architecture pattern to efficiently organize, manage, and govern their data domains.

The transition towards domain-oriented decentralized data ownership and architecture presents both challenges and opportunities from organizational and technical perspectives, as well as from a data governance standpoint. Key aspects include redefining data ownership paradigms, establishing well-defined data contracts between producers and consumers, enhancing discoverability of data assets across the organization, ensuring clear data quality and monitoring responsibilities, and visualizing dependencies through embedded lineage of data products within your platform. This paradigm shift necessitates a re-engineering of data governance processes, integrating them as integral components rather than afterthoughts.

Features of Blindata’s Data Products Catalog include the ability to define data product properties, schemas, SLAs, and contracts between producers and consumers. Furthermore, Blindata enables tracking of data product dependencies, expansion of column-level lineage with automated SQL lineage modules, monitoring of key quality indicators, and sharing them across the organization through the Blindata data quality module. Additionally, Blindata facilitates defining roles and responsibilities for data products, enabling close monitoring of team members’ workloads and adjustments as necessary. Collaboration is streamlined through embedded issue management and collaboration features, allowing feedback and collaboration from all stakeholders, even across teams and channels.

Incorporating federated computational governance policies into the data product lifecycle establishes a quality gateway for ensuring integrity, security, and compliance. These policies provide guidelines and automated checks, monitoring data development processes and enforcing governance controls consistently. By integrating these policies, Blindata enhances data quality assurance throughout the product lifecycle, fostering trust and reliability in decentralized data environments.

Blindata offers various methods for utilizing its governance solutions, including utilizing governance features embedded within the Data Mesh module, integrating with third-party tools through Metadata Activation Features such as Web Actions and Web Hooks, and developing custom solutions using Blindata’s Open API to implement custom connectors and integrate them into data management CI/CD pipelines.

Main Features

  1. Data Products Catalog

    • Access a comprehensive catalog of available data products, facilitating easy discovery and exploration of relevant datasets.
    • Utilize the catalog to find datasets, dataset schemas, contracts all integrated with the other Blindata features.
  2. Data Products Domains

    • Define responsibilities, team members, and documentation within specific domains to streamline data product development and usage.
  3. Governance Policies

    • Gain insights into governance policies that govern the development and usage of data products, including documentation of policies, issue management, dashboarding, and support for automation.
    • Implement governance policies to ensure compliance and promote standardization across distributed data platforms

Glossary Of Common Terms

Term Description
Data Products Data Products encapsulate structured data, along with associated metadata like schemas, service level agreements (SLAs), and data contracts. They serve as the fundamental building blocks for data discovery, sharing, and collaboration within a Data Mesh architecture. Data Products facilitate well-defined ownership and governance, enabling teams to efficiently manage and utilize data assets across the organization.
Data Contracts A formal agreement that outlines the terms and conditions for data exchange between a data producer (the team providing the data) and a data consumer (the team using the data). This agreement ensures clear communication and expectations regarding the data product.
Computational Policy Within Blindata’s Data Mesh framework, these are automated rules or guidelines enforced throughout the data product lifecycle. They act as a quality gateway, ensuring data integrity, security, and compliance with regulations. These policies are typically expressed as code, enabling automated checks and enforcement during data development processes.
Input Port Data entry point where information is received or imported into a system or process
Output Port Data exit point where processed or analyzed information is delivered from a system or process
Schema The structure or blueprint defining how data is organized within a data product. It specifies elements like columns, data types, relationships between data points, and constraints.