What is an Executor?

An Executor is a microservice that performs a specific task execution as part of an automated workflow in a Data Platform. It acts as a processing unit that receives a structured task request, executes it, and returns the result.

Executors are particularly useful in DataOps pipelines, where tasks such as data transformation, validation, deployment, or other operations must be automated and monitored.

Each executor exposes a standard API that allows the DevOps module to trigger execution requests and retrieve results.

For a broader understanding of executors, visit the Open Data Mesh Executor Overview .

Registering and Configuring an Executor

When registering an executor, you have two options:

• Use an existing executor: The Open Data Mesh initiative provides a collection of pre-built executors in its GitHub repository. These executors cover common use cases and can be readily integrated into your Policy Service. For more details on existing executors, refer to the Executor Adapters for Azure DevOps .
• Create a custom executor: If your requirements are more specific or you need a tailored solution, you can develop your own executor by following the steps outlined in the “Implementing an Executor” section. This allows for greater flexibility and control over the execution process.

Registering an Executor in ODM

To register a new executor, define it inside the DevOps properties file in your ODM project:

odm:
  utilityPlane:
    executorServices:
      azure-devops:
        active: true
        address: http://localhost:9003
        checkAfterCallback: false

How It Works

• The address field specifies where the executor service is running.
• ODM will lookup the registered service when a task requests execution.
• If the service is available, execution is delegated to it.
• If the service is not found or unavailable, the request fails, and the task enters the FAILED state.

Implementing an Executor

To create an executor, you need to expose an API that allows external systems to trigger executions:

• Endpoint to expose:

POST /api/v1/up/executor/tasks

• Request Body

{
 "taskId": "123456",
 "taskType": "data-processing",
 "callbackRef": "http://localhost:8002/api/v1/pp/devops/tasks/123456",
 "parameters": {
   ...params
 }
}

• Response

{
 "taskId": "123456",
 "status": "PROCESSING"
}

The executor is responsible for processing the task. After completing execution, it should send a PATCH request to the provided callbackRef with the execution result.

Example Callback Request

URL for a callback:

PATCH http://devops-platform-address/api/v1/pp/devops/tasks/123456

Body of the patch:

{
 "taskId": "123456",
 "status": "PROCESSED",
 "results": {
   "message": "Task completed successfully",
   "executionTime": "5s"
 }
}

How to Call an Executor

Once the executor is registered and running, tasks can also be triggered manually by clicking the Start Activity button from the Activity Detail Page:

Upon receiving the request, the executor processes the task, marking it as PROCESSING.

When execution completes, the executor sends a PATCH request to update the task status.

Key Considerations

• Technology Choice: Executors can be implemented using any technology stack (e.g., Java, Python, Node.js).
• Scalability: Executors should be stateless and scalable to handle concurrent task execution.
• Error Handling: Executors should implement retry mechanisms and provide meaningful error messages.

By implementing a robust executor, you ensure seamless automation and execution of tasks within the DataOps ecosystem.