Google Dataform connector

Dataform to Blindata catalog

How Dataform repositories, actions and the BigQuery tables they build appear in the Blindata catalog

This page explains where each Dataform and BigQuery object lands in the Blindata catalog, so you know what to look for after a run.

How Dataform objects map to Blindata

The rule of thumb:

  • A BigQuery table or view that a Dataform action builds, reads or declares becomes one physical entity on the System of its BigQuery project, in the schema named after its dataset. Its columns become physical fields. This is the same entity a BigQuery crawl, Power BI or query-history lineage uses, so Dataform enriches it instead of duplicating it.
  • What has no table of its own (a repository, an operation that builds nothing, a notebook) becomes a physical entity in a single Google Dataform System. You can switch this layer off.
  • The SQL of every action becomes a system routine on the same Google Dataform System, with the file that defines it. See System routines .
  • Data moving between tables becomes data flows. See Data lineage .
  • Assertions become quality checks. See Assertions as quality checks .

Systems

One System per BigQuery project

Every BigQuery project that the pipelines read or write gets a System, including source projects such as bigquery-public-data that you did not list in the job.

Attribute Value
Name The pinned name, or the one built by projectSystemNamePattern (default: the project id)
Subtype DATABASE
Technology Google BigQuery
Property bigQueryProjectId: the project id

The job looks for the System of a project in this order, and creates one only when none is found:

  1. The System you pinned in systems.projectSystems .
  2. A System that already carries bigQueryProjectId with that project id.
  3. A System named as projectSystemNamePattern says.

When it reuses a System that has no bigQueryProjectId, it adds the property and nothing else: it never renames a System and never changes its description, technology or other properties. A System that carries a different project id is left as it is. If two Systems carry the same project id, the first by name is used and the run reports AMBIGUOUS_PROJECT_SYSTEM.

The Google Dataform System

Attribute Value
Name Google Dataform, or systems.dataformSystem
Subtype SERVICE
Technology Google Dataform

It always exists, because it holds the routines. capabilities.processLayer controls only the repository and process entities below: with it off, the System holds routines only. A table is never duplicated here.

Physical entities

On the BigQuery project System

Dataform object Schema Name Type
Table, incremental table Dataset Table name The BigQuery type (TABLE, …), else TABLE
View, materialized view Dataset View name The BigQuery type, else VIEW or MATERIALIZED_VIEW
Operations with an output table Dataset Table name The BigQuery type
Data preparation destination Dataset Table name The BigQuery type, else TABLE
Declaration (a source table) Dataset Table name The BigQuery type
Assertion view, only with assertionViews Assertion dataset View name VIEW

Names keep the case BigQuery uses. The type written for an entity that already exists is never replaced by a default: when the job cannot read the BigQuery type, the stored type is kept.

A wildcard declaration such as events_* is one entity with that name and the property dataformWildcard. Its fields come from the newest shard whose suffix is all digits (dataformWildcardSampleShard names it). If the dataset has more tables than limits.maxTablesListedPerDataset, or no such shard, the entity has no fields and the run reports WILDCARD_SCHEMA_UNAVAILABLE.

A declaration is written only if its metadata could be read or if it is not in Blindata yet. So a declaration of a table you already crawl never degrades it.

On the Google Dataform System

Dataform object Schema Name Type
Repository (every repository read, even with no environment) Project id Repository id DATAFORM_REPOSITORY
Operations without output, and operations that create a routine <projectId>.<repositoryId> Dataform schema.name of the action DATAFORM_OPERATIONS
Notebook <projectId>.<repositoryId> Dataform schema.name of the action DATAFORM_NOTEBOOK

The system routines of the SQL are on this System too. BigQuery unit tests are counted in the run report only.

Physical fields

Source Blindata field
Top-level column of a BigQuery table Name, GoogleSQL type such as STRING or TIMESTAMP, position (starting at 1), and description
Column documented in the Dataform action The description of the matching field. Matched by name, ignoring case.
Nested column documented in Dataform, customer.email One line `customer.email`: <description> added to the description of the parent field customer
Column with policy tags The property bigqueryPolicyTags on the field

Only top-level columns are fields. The fields of a STRUCT or RECORD are not.

When BigQuery metadata is off or cannot be read, the job writes the descriptions of fields that already exist. It creates the documented columns without type or position only on tables it created in the same run, and never on a table another job cataloged. Column lineage is skipped for tables whose columns are not known with their position (COLUMNS_UNAVAILABLE).

System routines

The connector loads all the SQL that Dataform runs as system routines, so it is all cataloged and you can find it from its source file. Routines live on the Google Dataform System, except the BigQuery routines that Dataform operations create, described at the end of this section.

Routines of the Dataform SQL

Each SQL text of an action becomes one routine, for every action of every environment read, enabled or not:

Attribute Value
Catalog <projectId>.<repositoryId>
Schema The dataset of the action’s target table
Name <schema>.<name><suffix>@<environment>, with the schema and name of the action’s target (see the table below for the suffix)
Type SCRIPT
Definition The SQL, as described in Every SQL is a routine
Description Dataform <kind> SQL of <action id>, environment <environment>. Blindata keeps the first description it stored.

The name has no file path in it: moving or renaming a .sqlx file never recreates the routine. The environment is part of the name because the same SQL exists once for each environment.

dataformSqlKind SQL Suffix of the name
RELATION The query of a table, view or materialized view none
INCREMENTAL The incremental query of an incremental table #incremental
DATA_PREPARATION A data preparation written in SQL none
OPERATIONS Each query of an operations action, including scripts #<i>, from 0
PRE_OPERATIONS Each pre-operation of a table or view #pre<i>, from 0
POST_OPERATIONS Each post-operation of a table or view #post<i>, from 0
ASSERTION The select of an assertion none

For example, the incremental query of ga4_demo_output.base_ga4_events in the production environment is ga4_demo_output.base_ga4_events#incremental@production.

Every routine carries these additional properties. Only non-empty ones are written, and the set is replaced on every run, so a property the job stops writing disappears:

Property Meaning
dataformActionId Stable identity of the action, as on the tables and process entities
dataformFilePath Path of the file that defines the action in the repository. Omitted when Dataform gives none.
dataformRepository Resource name of the repository
dataformEnvironment The environment (release configuration) whose compilation the SQL comes from
dataformSqlKind The kind in the table above
dataformTags The action’s tags, sorted and comma-separated, as on the tables

Whether a routine is also analyzed for column lineage depends on its kind and on the job settings: see Every SQL is a routine . A routine that is only cataloged is still uploaded on every run. The upload never changes the analysis state of a routine.

Find the routines of a source file

Routines are listed in the routines page of Blindata, and are not in the global search. To find all the SQL of a file such as orders.sqlx, open the routines list and filter by the additional property dataformFilePath, with the file name as the value. The filter exists only after an administrator has defined the property as a searchable custom property of the System Routine type: see Make routines searchable . Then the filter matches values that contain the text, whatever the case, so orders.sqlx finds definitions/staging/orders.sqlx. The result lists the routines of every action defined by that file and of every environment. One file can give several actions (assertions declared in the same file, a .js file that publishes tables in a loop), and one action gives several routines (incremental query, pre and post operations), so a file is not one routine.

The same filter works with the other properties, once defined the same way, for example dataformActionId for one action, or dataformEnvironment for one environment.

BigQuery routines

An operation that creates a procedure or function (CREATE [OR REPLACE] PROCEDURE | FUNCTION) becomes a system routine on the BigQuery project System, with the name, definition and description Dataform and BigQuery give it. It is added only when the routine is not already in Blindata, so it never overwrites a routine a JDBC crawl cataloged. The routine is not a table and is not analyzed for lineage. TEMP routines are counted only.

Properties

Each entity carries additional properties that tie it back to Dataform. Only non-empty properties are written.

On tables and views

Property Meaning
dataformRelationType TABLE, INCREMENTAL_TABLE, VIEW, MATERIALIZED_VIEW, OPERATIONS, DATA_PREPARATION, DECLARATION, ASSERTION
dataformRepository Resource name of the repository that builds the table
dataformActionId Stable identity of the action: dataform:<project>/<location>/<repository>:<database>.<schema>.<name>. It is the process of the data flows.
dataformFilePath Path of the file that defines the action in the repository
dataformTags The action’s tags, sorted and comma-separated
dataformEnvironments The environments of that repository that build this table
dataformGitCommit, dataformCoreVersion The commit and Dataform core version of the compilation
dataformDisabled true for a disabled action
dataformUniqueKey, dataformProtected Incremental tables: the unique key columns, and true when a full refresh is disabled
dataformPartitionBy, dataformClusterBy Partition expression and cluster columns
dataformTableFormat, dataformStorageUri For Iceberg tables
dataformLoadMode For data preparations
dataformAssertions The assertions on the table, as <kind>:<assertion name> separated by ;
dataformDeclaredBy For declarations, the repositories that declare the table
bigqueryLabels BigQuery labels as key=value, sorted
dataformRepositoryUrl Link to the repository in the Google Cloud console
dataformLastRunState, dataformLastRunTime, dataformLastRunFailure, dataformLastBigQueryJobId The last run of the action. Only with runMetadata .

On repositories and process entities

Property Meaning
dataformRepository, dataformDisplayName, dataformLocation The repository, its display name and its location
dataformGitUrl, dataformDefaultBranch The Git remote and its default branch
dataformRepositoryKind A Dataform repository, or a BigQuery pipeline
dataformReleaseConfigs One line per release configuration: its id, the Git reference, the schedule, disabled, and the overrides (default database, schema, location, suffixes, prefix)
dataformWorkflowConfigs One line per workflow configuration: its id, the release configuration, the schedule and what it selects (targets, tags, dependencies)
dataformLastCompiledCommit, dataformCoreVersion The commit and Dataform core version of the last compilation read
dataformFilePath, dataformTags, dataformDisabled, dataformEnvironments On operations and notebooks, as on tables
dataformQueryCount On operations: the number of SQL queries
dataformDependencies On notebooks: the actions they depend on

The connector sets no tags and creates no tasks or logical entities.

Example

Take a Dataform repository ecommerce-demo in the project acme-analytics, built on the BigQuery public dataset bigquery-public-data.thelook_ecommerce. It declares four e-commerce tables and builds four staging views and two reporting tables, with three built-in assertions. Deployed with a production release configuration, it produces:

Dataform action System Schema Entity Type Key properties
Declaration bigquery-public-data.thelook_ecommerce.users bigquery-public-data thelook_ecommerce users TABLE dataformDeclaredBy with the repository
View staging.stg_customers acme-analytics staging stg_customers VIEW dataformRelationType = VIEW, dataformTags = staging, the description from the .sqlx file
Table reporting.customer_orders, with unique key and row conditions assertions acme-analytics reporting customer_orders TABLE dataformTags = daily, dataformAssertions with both assertions
Assertions on customer_orders none none none none Two quality checks in the suite of the production environment, linked to customer_orders
Repository Google Dataform acme-analytics ecommerce-demo DATAFORM_REPOSITORY The Git remote, the release configuration and the schedules

Three Systems appear (bigquery-public-data, acme-analytics and Google Dataform), and no table exists twice. The Google Dataform System also holds one routine for each query, for example staging.stg_customers@production for the view stg_customers, with dataformFilePath set to the .sqlx file of the view. The assertions have routines too, which are cataloged and not analyzed.

Living with a BigQuery crawl

If you already crawl BigQuery, run the two jobs in any order. The Dataform job follows these rules, so it never makes a crawled entity worse:

  • It uses the Systems your crawl created (see how a System is found ).
  • It never blanks a type or a position that is stored. If it cannot read a value, it writes the stored one back.
  • It adds its own properties by name and leaves the others as they are.
  • It writes descriptions documented in Dataform. When Dataform has no description for an object, the stored description is kept.
  • It never removes an entity, a field or a routine of a BigQuery System.
  • It adds a BigQuery routine only when your crawl has not already cataloged it.

Re-runs and changes

You can run the job again as often as you like. Each run updates what is already in Blindata instead of adding a second copy.

What changes in Dataform or BigQuery What happens in Blindata
A table, a column or a description changes The entity or field is updated in place, with no duplicates. Data flows are updated in the same way.
An action or a table is renamed The new name is a new entity. The old one stays until you remove it: the job never deletes entities of a BigQuery System.
An action is deleted Its entity stays in Blindata. Process entities in the Google Dataform System are removed only by the optional cleanup . Its routines, and the statements and flows they produced, are removed by the next run that reads the environment completely.
A SQL file is moved or renamed The routines keep their name and are updated: dataformFilePath changes, and the analysis is not repeated.
The SQL of an action changes Its routine is updated and analyzed again, and the new column lineage appears one run later.
An action is disabled It stays in the catalog with dataformDisabled = true, and its lineage is no longer published (ACTION_DISABLED). Its routines stay, cataloged and not analyzed.
The same table is built by several environments One entity, whose dataformEnvironments lists them. One flow lists the environments it appears in.
The same table is built by two repositories One entity: the last repository wins (repositories are taken in the order of their resource names), and the table is listed in targetCollisions in the run report.
The service account cannot read a table The entity is still published, with the values already stored, and the run reports METADATA_UNAVAILABLE.
A compilation has errors The actions it contains are read, but nothing is cleaned up for that environment, and its stale routines and statements are not removed, until a run reads it without errors.

A System’s description and technology are written only when the System is created. Edit them in Blindata afterwards and later runs keep your changes.

Note

Cleanup runs only after a complete run, so a partial run never removes content by mistake. The conditions are listed under cleanupOptions .