OpenLineage to Blindata lineage

The connector publishes lineage as data flows between the entities and fields of your Blindata catalog. Each flow names the OpenLineage job that moved the data, so you see not only where data comes from but which step produces it.

Evidence precedence

An OpenLineage event can describe the same lineage in several ways. For each job and output, the job uses the finest proven evidence and never invents a pair:

Rank	Evidence	Gives	Source
1	SQL analyzed by Blindata	Table and column flows	The job’s sql facet, analyzed as a routine .
2	Declared field lineage	Table and column flows	The lineage job facet: entries per output dataset, with entity inputs and fields.<target>.inputs[].
3	Column lineage facet	Table and column flows	The columnLineage facet of an output dataset.
4	Inputs and outputs	Table flows	The job’s inputs and outputs, when there is no lineage facet.

• Table flow fallback. A table flow is always published. Column flows come only from analyzed SQL or declared field lineage; when none exists, or the SQL analysis fails, the table flow remains and the gap is reported. This is how a producer that cannot give field mappings still gives lineage.
• No invented pairs. When a job has several inputs and several outputs and no lineage facet, the job cannot know which input feeds which output. It publishes a table flow for every input and output pair and reports the gap UNCORRELATED_IO when there is more than one output.
• Only direct transformations make column flows. Field inputs with DIRECT transformations become field flows. INDIRECT ones (filters, joins, group by) and inputs for a whole dataset are not flows: with includeIndirect: true they are described in the description of the flow.
• Nested fields resolve to the dotted physical fields (window.z_score). Analyzed SQL stays at top-level columns.
• A DatasetEvent with a lineage facet gives table flows between datasets, with no job.

How to write a producer’s events for column lineage

The step is	Send
SQL (including a INSERT ... SELECT)	The sql facet with the statement as executed and its dialect, and inputs and outputs. No field lineage needed.
Not SQL	The lineage facet: one DATASET entry per output, entity-level inputs, and fields.<target>.inputs[] with the transformation only when you are sure. Otherwise send entity inputs only: the table flow is published.
Mixed	Treat it as non-SQL; the read query can go in the documentation.
COPY or UNLOAD on Redshift	Non-SQL.

A complete example is in Configure your producers .

What a flow looks like

• Name: the ends, then @<job namespace>/<job name>. A name longer than the limit is shortened and ends with a hash.
• Description: the evidence (declared, observed, analyzed), the producer, and the descriptions of the transformations. Filters and joins are in the description when includeIndirect is true.
• Scope: see below.

Scopes

Every data flow created for an OpenLineage job carries a scope, the key of its owner:

Flows	Scope
Data flows (table and column)	ol:<job namespace>|<job name>
Orchestration flows	ol:<job namespace>|<job name>#orchestration

The scope has at most 230 characters, and is shortened with a hash past that, keeping the namespace intact. Scopes are how the job finds what to remove : it acts on scopes, never on flows created by crawlers or other jobs. You can filter data flows by scope in Blindata. A job namespace never contains |, so the scopes of two namespaces never overlap.

Orchestration flows

Some events say how jobs depend on each other rather than how data moves. They become flows between job entities, so the process layer shows the orchestration next to the data:

Event content	Flow
parent run facet (a run started by another job)	Parent job to child job
jobDependencies run facet	Upstream job to downstream job
airflow job facet (downstream_task_ids of each task)	Task to task

Orchestration flows are marked as orchestration: the scope ends with #orchestration, the name with @orchestration, and the description starts with “Orchestration”. They are published only from completed runs (and the Airflow task graph from the job description), and only when processLayer is true. A target job that is not a job entity yet is reported as UNRESOLVED_ORCHESTRATION_TARGET. An explicit lineage facet with JOB entries or inputs gives job to job flows as data flows, since that is what the producer declared.

SQL in the events

The sql facet is cataloged as a routine on the job System and analyzed in a query bucket, named by queryBucketName (OpenLineage SQL <job definition uuid> by default). The routine is analyzed only when its SQL changed, and the analysis result may become visible only in a later run, so a new or changed SQL job can report QLP_PENDING first. The statements and flows that the analysis creates belong to the routine and are removed with it, never by the scope cleanup of the job: this is why the bucket name must not start with ol:.

If the analysis fails, the table flow from the job’s inputs and outputs remains and the run reports QLP_ANALYSIS_ERROR. Constructs that the SQL parser does not understand stay at table level. See Automated SQL lineage for the parser and its configuration , including dialect pre-processing.

Why is my lineage missing?

Every run counts gaps by code in its report (gaps). The ones that explain missing lineage:

Gap	Meaning	What to do
UNMAPPED_NAMESPACE	The dataset namespace maps to no System and creation is not allowed.	Stamp the System with openLineageNamespace, add a namespaceMappings entry, or allow creation. See Namespace to System .
IGNORED_NAMESPACE	The namespace matches ignoredNamespaces.	Expected for OpenSearch and Elasticsearch.
TARGET_NOT_CATALOGUED	A flow end is an entity that does not exist and cannot be created (entityCreation is false, or the System is crawled).	Crawl the store, or allow entityCreation for it.
FIELD_NOT_CATALOGUED	A column flow names a field that is not in the catalog. It is checked on Systems created by OpenLineage.	Send the schema facet of the dataset, or crawl it.
UNCORRELATED_IO	Several outputs and no lineage facet: all input and output pairs were published.	Add the lineage facet.
QLP_PENDING	The routine is waiting for its analysis.	Wait one run.
QLP_ANALYSIS_ERROR	The analysis failed. Table flow kept.	See SQL lineage .
UNRESOLVED_ORCHESTRATION_TARGET	An orchestration flow names a job that is not catalogued.	Make sure that job’s events reach a definition with processLayer on.
DANGLING_FLOW	A flow had an end that could not be resolved and was dropped.	Fix the dataset gaps above.
JOB_SYSTEM_NOT_FOUND	A jobSystems rule names a System that does not exist and may not be created. The jobs went to openLineageSystem.	Fix the uuid or the name, or set creation to IF_MISSING.
JOB_SYSTEM_NOT_SHARED	A jobSystems rule names an existing System that does not hold OpenLineage jobs. The jobs went to openLineageSystem.	Use a System for the jobs, or set shared on the rule.
MALFORMED_FACET	A facet is not valid for its schema. It is ignored; the event is still read.	Fix the producer.
PRODUCER_NO_LINEAGE	The producer says an operator has no lineage extraction (Airflow unknownSourceAttribute).	Producer-side limit.
PRODUCER_PARTIAL_LINEAGE	The producer could not handle everything (Spark spark_unknown, extractionError).	Producer-side limit.

Lineage is anchored: a flow whose end is not in the catalog is not published, and the gap says why. Gaps that concern Systems and cleanup are in Configure the OpenLineage job and Removal of deprecated metadata .

Updated on 04 Nov 2026