Removal of deprecated metadata

OpenLineage has no event that says “this job no longer exists”. A job that is deleted, renamed or rewritten simply stops being described. The connector removes what the producers no longer describe in two ways, chosen by the snapshotMode of the job definition:

	Touched in this run (cutoff)	Day-based
For	BATCH_SNAPSHOT definitions, and jobs replaced by an authoritative description	ADDITIVE definitions
Removes	What the run fully described and did not describe again	What was not updated for N days
Speed	In the same run	After the number of days you set

Requirements

• The user of the job’s BLINDATA connection, which runs the cleanup, needs SYSTEMS_ADMIN. See Who writes to Blindata .
• cleanup.enabled must be true (the default).
• The job must not be a dry run: a dry run only plans the calls (see outputOptions ).

The idea: touched in this run

Every object the job uploads gets its last-update time set to the time of the upload, whether it changed or not. The job uses that:

1. At the start of the run, the agent reads the time from the Blindata API clock (never its own), the cutoff.
2. The run uploads again everything its scopes still own: all flows (data and orchestration), all job entities, all routines, and the dataset entities and fields it refers to on Systems it owns. Unchanged objects are re-sent on purpose.
3. After every upload succeeded, anything in those scopes that was last updated before the cutoff was not described by this run: it is deprecated and is deleted.

This is why a snapshot run sends more data than an additive run, and why one cleanup call per set is enough.

What a run fully describes

Only fully described scopes get the cutoff cleanup.

Run content	Scopes that get the cutoff
A /lineage/batch to a BATCH_SNAPSHOT definition, with every event accepted	Every job of every job namespace present in the batch.
A JobEvent, or a completed run when lineageReplacement is JOB, for a job	The two scopes of that job.
Anything else (everything an ADDITIVE definition receives)	None: the day-based cleanup applies.

A snapshot is a batch stored whole: if any event of the batch is rejected, it is not a snapshot (see Answers to /lineage/batch ). A run also reads a snapshot whole: it reads past limits.maxEventsPerRun rather than cut the last batch it started.

What is owned and what is removed

Object	Owner	Removed by
Data flows	The scope ol:<job namespace>|<job name>	The flows call.
Orchestration flows	The scope ol:<job namespace>|<job name>#orchestration	The same call: the scope prefix of the namespace covers both.
Job entities and routines	The job System of the namespace, in the schema of the job namespace	The job Systems calls (snapshots only). A routine is deleted together with its statements and their flows.
Dataset entities and fields	Systems created by the job, when their policy allows	The dataset Systems calls (see below).
Quality checks	The job’s suite	Disabled by the agent, never deleted. History is kept.

The job never acts on objects owned by crawlers or by other jobs.

The calls

A run makes one dry run, then one real call, for each set:

Call	Covers
Flows, on the scope prefix ol:<namespace>| of each snapshot namespace, or on the scopes of each replaced job	Data and orchestration flows across all Systems.
Each job System, level schema, for the snapshot’s job namespaces it holds, and each former job System for all of them (snapshot only)	Job entities and routines of removed jobs, and of namespaces moved to another System, with their statements and flows.
Each dataset System with cleanupMode CUTOFF that the batch refers to	Entities no longer referenced, and fields of the entities whose schema the run carried.

For a single job replaced by a JobEvent, only the flows call is made: its entity is uploaded again anyway. A routine that the job no longer has (the sql facet was removed) is left to the age-based cleanup of its job System.

Job Systems

The job entities and routines of a namespace are in its job System . When a jobSystems rule changes, a namespace can move to another System. Each run therefore also visits the former job Systems of the definition: the openLineageSystem, when it exists, and the Systems whose openLineageJobSystem property holds the uuid of the definition. There, the job entities and routines of the namespace were not uploaded by the run, so they are removed like any other stale object: in the same run for a snapshot, after staleJobDays for an additive producer. The removals count towards maxRemovalsPerRun.

A System that another definition created, and that this definition left after a rule change, is not visited: remove its job entities by hand.

Ownership of dataset Systems

The cutoff is safe on a dataset System only if this definition is the only producer that refers to it: an entity that only another repository’s events refer to was not touched by this run, and would be deleted. The cleanupMode of the System policy decides:

cleanupMode	Meaning
CUTOFF	The System is owned by this definition (a bucket only this repository writes). Snapshot runs clean it with the cutoff.
DAYS	Shared or unknown ownership. Age-based cleanup only, after cleanupDays days.
NONE	The default. Mapped and crawled Systems are never cleaned by this job.

A System created by a definition carries openLineageOwner, the uuid of that definition. CUTOFF is kept only when the System is owned by this definition (a System without an owner is not owned) and the allowedJobNamespaces of the definition is exactly one pattern without *. Otherwise the System is cleaned with DAYS and the run reports SHARED_SYSTEM_DOWNGRADED.

Day-based cleanup

For ADDITIVE definitions, and for Systems with DAYS, the job uses the age-based cleanup of the standard Blindata system cleanup:

• The job Systems, current and former: job entities and routines of the job namespaces seen in Blindata that match allowedJobNamespaces, not updated for cleanup.staleJobDays days (30 by default).
• Flows: on the scope prefix of each job namespace seen, with the same number of days.
• Dataset Systems with cleanupMode DAYS: after cleanupDays days.

Since an additive run does not upload again what it did not receive, an object is removed only after it has not been described for the whole period: set staleJobDays longer than the interval at which your jobs run.

Guards

A broken input must not wipe lineage. The job makes no cleanup when:

• The run failed, the batch is empty, or any upload of the run failed. A row whose upload failed is “not touched” and would be deleted, so the cleanup is skipped for the whole run, and the events stay for the next run.
• The dry-run counts exceed cleanup.maxRemovalsPerRun (500 by default), summed over the calls: no real call is made, the run ends in warning, and the counts are in the report. Lower the cap on small repositories, so that an empty or truncated batch cannot remove much.
• The user that runs the cleanup is refused (403, CLEANUP_PERMISSION_DENIED): nothing is deleted and the run ends in warning.
• The job is a dry run: only the planned calls are reported.

Rows that the upload drops because a flow end is missing are not touched either, and are deleted: they were invalid. The report counts them as DANGLING_FLOW.

The report has, in cleanup: the mode, the cutoff time, the skippedReason when no cleanup ran, the calls with their dry-run and real counts per object type, and the removedJobs by name.

Example

A repository publishes its two jobs etl.build_daily_kpi and features.export_anomalies as a snapshot. The next batch contains only etl.build_daily_kpi.

• The second job’s flows and its job entity are not uploaded again, so the cutoff cleanup deletes them in that run. Its quality checks are disabled.
• The S3 entity it wrote, anomalies, is deleted in the same run if its System is owned by this repository (CUTOFF), and otherwise after 30 days without references (DAYS).
• If the batch had been empty, or the number of objects to delete were above maxRemovalsPerRun, nothing would be deleted.

Updated on 04 Nov 2026