This section dives into Blindata’s approach to data governance through computational policies. These policies automate checks and enforcements throughout the data product lifecycle, ensuring data quality, standardization, and compliance.

Why Use Computational Policies?

Modern data governance needs to keep pace with the realities of distributed data models and “shift-left” approaches, where governance is embedded early in the data lifecycle. This paradigm shift demands robust procedures to control and oversee governance processes at every stage. However, manually enforcing policies throughout this lifecycle can be cumbersome and error-prone.

Blindata’s computational policies offer a solution to these challenges by providing the tools to automate governance tasks. They offer several advantages:

• Reduced Manual Work: Automation eliminates the need for manual verification of data products, freeing up time for data teams to focus on strategic initiatives.
• Improved Standardization: Automated enforcement ensures consistent policy application across all data products, minimizing the risk of human error and the divergence of data management standards.
• Enhanced Transparency and Trust: Policy definitions and evaluation logs provide clear visibility into governance practices, fostering trust and collaboration.
• Scalability: Computational policies can efficiently manage a large number of data products, making them ideal for organizations with growing distributed data ecosystems.

How to Utilize Computational Policies in the Data Product Lifecycle

Blindata provides a robust set of tools within its policies repository to streamline policy documentation, definition, implementation, result gathering, and non-compliance management.

These policies can be integrated into the lifecycle of data products, forming a crucial part of the data ops pipeline. Let’s walk through the typical stages of a data product’s lifecycle:

1. Data Product Version Publication: This stage marks the creation or update of a data product. Here, the data product descriptor and its contract undergoes validation, and its metadata is stored in an internal registry.

2. DevOps Activity: Following version publication, DevOps activities are initiated. Tasks are executed on the infrastructure provider, deploying the data product to the designated environment, whether on-premises or in the cloud.

3. Deployment Validation: After deployment, we analyze the results to ensure that the data product functions correctly in its production environment.

Throughout these stages, policies are evaluated to ensure compliance and adherence to governance standards. These policies can cover various aspects:

• Metadata Policies: Analyze compliance regarding the presence of metadata in the data product descriptor or data contract.
• Stage Transitions: Govern the transition of data products between different stages and environments.
• Cloud Provisioned Resources: Ensure compliance with regulations such as data location or the cloud region used to store data.
• Post Deployment Policies: Verify that data contract expectations are met before moving to production.

These processes can be implemented as custom solutions or leveraged through the Open Data Mesh Platform (ODM) . ODM serves as a central quality gateway and control plane for managing distributed data architecture, facilitating efficient policy enforcement and governance across data products.

Define Policies in Blindata

Blindata provides a centralized location to browse and explore all available computational policies. Each policy has a clear and concise definition that outlines its purpose and requirements as well as complete set of tools to ease the management and the assessment.

• Documentation: Documentation provides additional details and best practices related to the policy. Documentation of policies is facilitated through a widget-rich text editor, which allows users to create, edit, and share policy documents seamlessly. This editor provides a user-friendly interface with formatting options such as bold, italic, bullet points, and headings, making it easy to structure and style the content.
• Adoptions: The comprehensive list of resources that adopt a specified policy. Adopting a policy refers to the process of integrating a specific set of rules, guidelines, or standards into your data product or business domain. When you adopt a policy, you commit to following and enforcing the principles outlined within that policy.
• Evaluation Logs: These logs track the validation process of the policy on specific data products. The evaluation of a policy on a data product in Blindata involves assessing whether the data product complies with the rules or guidelines defined by the adopted policy. This evaluation process can be automated via API or performed manually.
• Policy as Code: For automated policies, Blindata serves as an implementation registry, allowing you to efficiently manage and organize your policy implementations.

Types of Policies

Blindata offers a variety of policy types to support different governance needs:

• Documentation Only Policies: These outline best practices and guidelines that data product owners should be aware of, but don’t require formal verification. These policies cannot be adopted or evaluated.
• Adoption Only Policies: These policies require data product owners to acknowledge and adhere to specific requirements without undergoing formal verification. This ensures they are aware of critical guidelines. These policies can not be evaluated but can be adopted. Adoption requires the user to specify the resource type to which the policy applies e.g. data product, domain, etc…
• Manual Verification Policies: For complex policies or areas where automation might not be feasible, manual verification is still an option. Blindata’s issue management module seamlessly supports these workflows. These policies can be adopted and evaluated.
• Automated Verification Policies: These policies leverage automated tools and scripts to streamline verification. For these policies, evaluations are automatically uploaded without requiring any user intervention.

Adopting a Policy for Your Data Product

Data product owners can easily adopt relevant computational policies within Blindata. The adoption process typically involves selecting applicable policies from the available list and associating them with their data products. This ensures the data products adhere to the chosen governance standards.

When it comes to automated policies, the adoption process is smooth and does not require any manual intervention. Once a data product receives an evaluation result, the relevant policy gets automatically adopted without any additional steps needed from the user. Blindata takes care of the adoption process seamlessly in the background, so users don’t have to worry about it.

Visualize all the policies and their evaluation status for a data product

To adopt policies manually click on the policies card in the data product detail page. The card shows the number of adopted policies and the number of successful evaluations. Not all the policies require validation. Documentation and adoption-based policies do not require any kind of verification process.

List of policies adopted by a data product

On this particular page, you will find a comprehensive list of all the policies that have been adopted by the data product. Not only that, but you will also be able to see the evaluation status of each policy, providing you with a clear and transparent view of the data product’s compliance with its policies.

Adopt a new policy for a data product

Within the interface, you’ll notice a button labeled “All policy” situated in the top right-hand corner. This button serves the purpose of displaying a comprehensive list of all the policies that have been defined within the system. Notably, any policies that have yet to be adopted by the data product in use will be displayed with a subtle grey background to differentiate them from those that have been adopted.

By clicking on a policy item, a user can create an adoption for the specific data product.

Evaluating Policy Compliance

Blindata allows you to evaluate policies for your data products on the adoption list page. By clicking on a policy item, you can view adoption information and the evaluation log for the selected data product.

To create a new evaluation record, simply press the plus button located at the top right corner of the evaluations log card. This will open an interactive dialog box that will prompt you to select the validation status, either Verified or Failed, along with the date of the control. You can also add an optional comment if you wish.

To access current and past evaluations, you have several options available:

• Policy Metrics Cards: These cards offer a concise summary of the number of policies linked to a data product and their verification status. You can quickly gauge whether policies are successfully verified or require attention.

• Evaluations Log: Detailed logs provide in-depth insights into the specific checks conducted by automated policies and their outcomes. This feature enables you to pinpoint areas where a data product may not be fully compliant, facilitating targeted remediation efforts.

• Policy Detail Page: For a comprehensive view from the policy perspective, the policy detail page displays the list of resources that have adopted the policy, along with their evaluations logs. This allows for a more granular examination of policy adherence and evaluation results.

Policy as Code: Implementation Registry

For automated policies, Blindata serves as an implementation registry, allowing you to efficiently manage and organize your policy implementations. Stay tuned for this exciting feature, coming soon!