The data product lifecycle

How stages are defined inside the descriptor

The lifecycle stages of a Data Product are explicitly defined within its descriptor under the internalComponents.lifecycleInfo section. This section provides a structured representation of the various phases the Data Product undergoes, from development (dev) to production (prod). Each stage includes key metadata related to DevOps automation, pipeline configurations, and execution parameters.

Structure of Lifecycle Stages

Each lifecycle stage (e.g., dev, prod) is represented as an array of tasks. These tasks contain essential configuration elements:

• Service Reference (service): Specifies the external DevOps tool (e.g., Azure DevOps, GitHub Actions) used for managing the deployment process.
• Pipeline Template (template): Defines the execution template for the pipeline. It includes:
  • specification and specificationVersion for version control.
  • definition containing the organization, project, pipeline ID, and branch.
• Configurations (configurations): Stores settings and execution parameters for the pipeline. Key elements include:
  • stagesToSkip: Lists stages that should be bypassed (e.g., “Test”, “Deploy”).
  • params: A dictionary of key-value pairs representing dynamic parameters, which may include:
    • Static values.
    • Referenced outputs from previous tasks (e.g., ${dev.results.task1.outputValue}).
    • Callback references for external event notifications.

Example of a Lifecycle Stage Definition

"internalComponents": {
    "lifecycleInfo": {
        "dev": [
            {
                "service": {
                    "$href": "azure-devops"
                },
                "template": {
                    "specification": "spec",
                    "specificationVersion": "2.0",
                    "definition": {
                        "organization": "organization",
                        "project": "opendatamesh",
                        "pipelineId": 3,
                        "branch": "main"
                    }
                },
                "configurations": {
                    "stagesToSkip": ["Test", "Deploy"],
                    "params": {
                        "paramOne": "value1",
                        "paramTwo": "value2",
                        "callbackRef": "http://localhost:8002/api/v1/pp/devops"
                    }
                }
            }
        ]
    }
}