Set up the endpoint

This page covers what you prepare once, before any producer sends an event: where the agent must be reachable, who the producers sign in as, and the job definition that each producer addresses.

What you need

Agent reachability

Producers push to the agent, so the agent address must be reachable from where they run. Typical choices, from the narrowest:

• Inside your network: Spark, Airflow, Trino and Flink clusters that already reach the agent.
• A self-hosted CI runner in the agent’s network, for pipelines that publish events from a repository.
• An exposed agent ingress, over HTTPS only.
• An agent hosted for you.

Exposing the agent widens its attack surface: use TLS, and give producers only the application user described below. The endpoint requires a tenant header and AGENTS_EDITOR on every request, and it limits the size of a request body (see Request format ).

The Blindata user of the producers

Create an application user with API access enabled and give it the permission AGENTS_EDITOR, the same permission that runs agent jobs. Producers sign in with its user name and API key, as described in API key authentication . A caller without it gets 403.

AGENTS_EDITOR lets the user send events, and also create, edit and run any agent job definition of the tenant. Give the user to producers only, and keep its API key in a secret store. The user does not need any permission to write metadata: the job writes with its own connection, described next.

Who writes to Blindata

Every write of a job run (Systems, entities, fields, documents, data flows, routines, quality and profiling metrics, and the cleanup ) uses the job’s BLINDATA connection, named by outputOptions.targetConnectionName, whatever producer sent the events. The connection’s service principal holds the Blindata permissions:

When Blindata answers 403 to the cleanup, nothing is deleted and the run ends with a warning (CLEANUP_PERMISSION_DENIED). See the permissions list .

The agent records who sent each event, as identity only: the Blindata user uuid of the caller and its user name (X-BD-User, or the uuid when the request carries a Bearer token). The run report counts the events of each sender. The agent never stores the sender’s API key or token, and the sender’s own permissions are not used to write.

What a producer’s events can change is bounded by its job definition: its allowedJobNamespaces, its System policies and its removal cap (cleanup.maxRemovalsPerRun). Keep them tight, because the connection’s service principal writes for every producer of the definition.

The events stay on the agent, per tenant, until they are processed. Keep the agent’s database protected like any other agent data.

Create the job definition

Each producer addresses its own job definition by uuid, so mapping rules, System policies, snapshot mode, allowed job namespaces, dry run and quality options are tailored per producer and scenario. A producer sharing a definition with another shares all its rules.

1. Open the agent, go to Schedules (see How to Schedule Jobs ), and create a schedule with job type OPENLINEAGE.
2. Paste the job configuration. The only required key is allowedJobNamespaces. Start with the defaults, which make the job a dry run: see Configure the OpenLineage job .
3. Set the cron expression. For a BATCH_SNAPSHOT definition it is a safety net, see When events are processed .
4. Save, and copy the uuid of the schedule. The agent API lists the definitions with their uuids at /agent-api/v1/jobs/schedules (permission AGENTS_VIEWER).

The endpoint

The address of a producer is the agent address, the path of the OpenLineage receiver, and the uuid of its job definition:

https://<agent>/agent-api/v1/openlineage/{jobDefinitionUuid}/api/v1/lineage
https://<agent>/agent-api/v1/openlineage/{jobDefinitionUuid}/api/v1/lineage/batch

The two paths are the operations of the OpenLineage HTTP API, POST /lineage (one event) and POST /lineage/batch (an array of events), version 2.0.2 of the OpenAPI event schema specification published with OpenLineage 1.55.0. The tail api/v1/lineage is the default endpoint of the OpenLineage clients.

Authentication

Send three headers with every request. In an OpenLineage client they are the custom headers of the HTTP transport.

The user needs AGENTS_EDITOR. A caller that already holds a Blindata Bearer token can send it in Authorization instead of X-BD-User and X-BD-ApiKey, together with X-BD-Tenant; the API key headers are the practical choice for clients and pipelines, because they do not expire.

Why the clients’ own authentication options do not work. The OpenLineage clients offer api_key (a static Authorization: Bearer token), jwt and oauth2 (client credentials against a token endpoint). Their tokens are not Blindata tokens, and the receiver does not accept them: oauth2 would need client credentials issued by the Blindata OAuth server. Use custom_headers (or headers, depending on the client). A managed integration that does not allow custom headers cannot send events to this endpoint.

Request format

• Body: JSON. A request to /lineage carries one event (an object), a request to /lineage/batch an array of events.
• gzip: Content-Encoding: gzip is accepted. Set compression: gzip in the client, which is worth doing for large batches.
• Size limit: 50 MB, measured on the received body and again on the decompressed body. A larger request is rejected with 400 and Request body too large. Split a large batch only if you do not need it as a snapshot : a snapshot is a single request.
• Content type: any.

Answers to /lineage

Answers to /lineage/batch

The body of a 200 follows the OpenLineage specification:

{
  "status": "partial_success",
  "summary": {
    "received": 2,
    "successful": 1,
    "failed": 1,
    "retriable": 0,
    "non_retriable": 1
  },
  "failed_events": [
    {
      "index": 1,
      "reason": "Missing required field: eventTime",
      "retriable": false
    }
  ]
}

A client or a pipeline step should fail on any non-2xx answer and on any failed_events. The reasons are:

Retries are safe. If a client sends again an event, or a whole batch, that is still waiting to be processed (for example after a timeout that lost the answer), the agent answers as if it were new but does not store a second copy. A batch counts as a repeat only as a whole, so a snapshot is never stored in part. An event sent again after it was processed is processed again, which does not change the result.

A batch with a rejected event is not a snapshot. Its valid events are stored and processed, but they only describe themselves: nothing is deleted because it is missing from that batch. See Removal of deprecated metadata .

When events are processed

Receiving and processing are separate. The receiver validates each event and stores it on the agent. An agent job run turns the stored events into metadata.

Good to know:

• One run at a time per definition. A run that starts while another run of the same definition is in progress is skipped, but it asks the running one to read the inbox again before it ends (at most limits.maxDrainsPerRun times, 3 by default). A batch that arrives during a run is therefore processed by that run. Keep a cron schedule on a BATCH_SNAPSHOT definition anyway, as a safety net if the agent stops between receiving a batch and starting its run.
• Events are deleted after processing. A run that ends without error, with a warning or not, deletes the events it processed. A run that ends in error (Blindata unreachable, for example) keeps them for the next run, and so does a run that had to discard a row of its upload. Events of a run that has no terminal event yet also stay, until it ends.
• Old events are dropped. Events older than limits.maxEventAgeDays (7 by default) are deleted with a warning, and a definition accepts at most limits.maxPendingEvents (100,000) unprocessed events.
• A dry-run definition also discards its events after processing. To validate a pull request without publishing, use a definition of its own with dryRun true, and read the report in the agent job history.

Check your setup

1. Create the definition with the default configuration, adding only allowedJobNamespaces.
2. Send one event with curl (see Send a batch with curl ) and check for 204.
3. For a BATCH_SNAPSHOT definition, or after the cron time, open the Runs tab of the agent and read the report of the run: events read and processed, systems resolved, gaps and, in a dry run, what would be created.

Updated on 04 Nov 2026