Configure the OpenLineage job

This page describes the JSON configuration of an agent job of type OPENLINEAGE, key by key. You create one job definition per producer, as described in Set up the endpoint .

The configuration is a JSON object. Unknown keys are ignored without an error. Every key is optional except allowedJobNamespaces, and a missing key takes the default shown. Without outputOptions, the job is a dry run.

Start here

Two configurations cover the first steps. Everything else has a default and is described key by key below.

1. A dry run, to see what the producer’s events would become. It writes nothing:

{
  "allowedJobNamespaces": ["github://my-org/ds-quality"]
}

2. Live, once the report of the dry run looks right. Add the BLINDATA connection that writes to Blindata:

{
  "allowedJobNamespaces": ["github://my-org/ds-quality"],
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

Then decide, with systemPolicies , which Systems the job may create and clean, and, with snapshotMode , whether a batch is a complete snapshot. The Recommended setups give a starting point per kind of producer.

Top-level keys

A glob in this page is a pattern where * matches any sequence of characters (also none and /), and everything else matches literally. A pattern matches the whole namespace: s3://* matches every S3 bucket, github://my-org/* every repository of the organization.

snapshotMode

How the definition treats a batch.

A batch is a snapshot only if every event of it was accepted. See Removal of deprecated metadata .

allowedJobNamespaces

The job namespaces this definition accepts, as globs: ["github://my-org/ds-quality"], ["airflow-prod"], ["spark-*"]. An event whose job namespace matches none is rejected (Job namespace not allowed: <namespace>, non-retriable). A job namespace belongs to one definition: scopes then never overlap between producers, which is what makes the removal safe.

Rules:

• The job namespace is the one of the job, not the one of its datasets. Dataset events carry no job and are not checked.
• A job namespace that contains | is always rejected, and one longer than 200 characters too.
• For a BATCH_SNAPSHOT definition, use one pattern without *: it is the condition for removing dataset entities by cutoff (see cleanupMode ).

lineageReplacement

A JobEvent is always authoritative for its job. See Removal of deprecated metadata .

openLineageSystem

It is looked up, or created, only when a job namespace goes to it.

jobSystems

Places the job entities and routines of a job namespace in a System chosen by rules, so that, for example, the jobs of Airflow production and those of each repository are in Systems of their own. The rules are read in order: the first whose match matches the job namespace decides. A namespace that no rule matches goes to openLineageSystem . All the jobs of a namespace are in the same System, in the schema named after the namespace.

The placeholders, for two example namespaces:

A System that holds OpenLineage jobs carries the additional property openLineageJobSystem, with the uuid of the definition that created it or first used it with shared. A rule may use any System that carries it. An existing System without it is used only with shared set to true; otherwise the jobs of the namespace go to openLineageSystem and the run reports JOB_SYSTEM_NOT_SHARED. A System that is missing and may not be created, or an unknown systemUuid, sends the jobs to openLineageSystem too, with JOB_SYSTEM_NOT_FOUND.

Changing a rule moves the jobs of a namespace: the next run creates the job entities and routines in the new System and removes them from the former one (see Job Systems ). Descriptions and documents that people added to the former entities are not moved.

"jobSystems": [
  { "match": "airflow-prod", "systemName": "Airflow production" },
  { "match": "github://my-org/*", "nameTemplate": "{lastSegment}", "technology": "GitHub" }
]

namespaceMappings

Maps dataset namespaces to existing Systems. The first entry that matches is used, before anything else.

An entry whose System is not found is skipped. A mapped System is not changed (no property is added to it).

To map a namespace once, for every producer, add the additional property openLineageNamespace with the namespace as value to the System in Blindata (for example redshift://my-dwh.eu-west-1:5439 on the Redshift System that your crawler already maintains). The property is tenant-wide: every definition resolves that namespace to that System. Resolution order is in Metadata mapping .

systemPolicies

Rules per dataset System, matched by namespace. The first policy whose match fits the namespace is used.

Creation needs care. A System created for a store that a crawler also covers gives two Systems for the same data. Use creation: NEVER for warehouse schemes that you crawl (redshift://*, postgres://*) and IF_MISSING for object stores. One System per S3 bucket: the namespace s3://my-bucket is one System, named after the bucket. When a System with the planned name already exists, it is used and stamped with the namespace. A created System is stamped with openLineageNamespace and with openLineageOwner, the uuid of the definition that created it.

cleanupMode says what the job may delete from the System’s entities and fields:

CUTOFF is kept only when both hold, otherwise it is changed to DAYS and the run reports SHARED_SYSTEM_DOWNGRADED:

1. The System carries openLineageOwner equal to the uuid of this definition. A System with no owner, or owned by another definition, is not owned. Only a System created by the definition has the stamp.
2. The definition’s allowedJobNamespaces is exactly one pattern without *.

Why: an entity that only another repository’s events refer to was not touched by this run, and a cutoff cleanup would delete it. See Removal of deprecated metadata .

ignoredNamespaces

Globs of dataset namespaces that are not catalogued. An event dataset that matches is skipped and counted (IGNORED_NAMESPACE). The default skips OpenSearch and Elasticsearch, since a search engine does not generate data lineage. Setting the key replaces the default list.

quality

See OpenLineage to Blindata quality .

cleanup

limits

outputOptions

The user of the connection needs the permissions listed in Who writes to Blindata , SYSTEMS_ADMIN included for the cleanup. The producers’ own users need only AGENTS_EDITOR to send events.

A dry run still discards its events after processing. Events are never replayed, so a dry run is a way to validate what a producer sends, not to preview and then publish the same events. To go live, change dryRun and send the events again. A dry run sends nothing to Blindata, cleanup included: the cleanup calls are planned and reported, with zero counts, so a dry run does not tell you what a real run would remove. Because it reads nothing from Blindata, it treats every System and entity as new, so its counts of what it creates are an upper bound.

Full example

A BATCH_SNAPSHOT definition for one repository: warehouse tables are crawled and only mapped, S3 buckets are created and cleaned.

{
  "snapshotMode": "BATCH_SNAPSHOT",
  "allowedJobNamespaces": ["github://my-org/ds-quality"],
  "lineageReplacement": "NONE",
  "openLineageSystem": { "name": "OpenLineage" },
  "processLayer": true,
  "namespaceMappings": [],
  "systemPolicies": [
    { "match": "redshift://*", "creation": "NEVER", "entityCreation": false, "cleanupMode": "NONE" },
    {
      "match": "s3://*",
      "creation": "IF_MISSING",
      "subtype": "DATABASE",
      "technology": "Amazon S3",
      "entityCreation": true,
      "replaceUserDocumentation": false,
      "cleanupMode": "CUTOFF"
    }
  ],
  "ignoredNamespaces": ["opensearch://*", "elasticsearch://*"],
  "includeIndirect": false,
  "quality": { "enabled": false, "metricsEnabled": false },
  "cleanup": { "enabled": true, "staleJobDays": 30, "maxRemovalsPerRun": 100, "batchSize": 500 },
  "limits": { "maxEventsPerRun": 50000, "maxPendingEvents": 100000, "maxEventAgeDays": 7, "lockStaleMinutes": 120, "maxDrainsPerRun": 3 },
  "outputOptions": { "targetConnectionName": "blindata-prod", "dryRun": false }
}

With these outputOptions, every write, cleanup included, is made by the user of the connection blindata-prod, which needs the write permissions and SYSTEMS_ADMIN.

Here the Redshift System must carry the property openLineageNamespace with the namespace used in the events (for example redshift://my-dwh.eu-west-1:5439), because creation is NEVER: otherwise the datasets are reported as UNMAPPED_NAMESPACE.

Recommended setups

Each row has a step-by-step guide in How-to guides .

Validation errors

The job validates the configuration before doing anything. Messages start with Configuration cannot be parsed:.

If a run starts while another run of the same definition holds the lock, it logs OpenLineage run skipped: another run holds the lock; re-run requested (information, not a warning) and the running one reads the inbox again before it ends. If that run reaches maxDrainsPerRun, it logs OpenLineage run read the inbox <n> times; <m> events wait for the next run.

If a run finds that another run took its lock over (it was slower than lockStaleMinutes), it stops with OpenLineage run stopped: the inbox lock was taken over: it writes and deletes nothing more, and its events stay for the run that holds the lock.

Read the run report

Every run returns a report in the job history of the agent:

A run that logs a problem that changes the outcome (a gap, a skipped cleanup) ends in WARNING.

Updated on 04 Nov 2026