Google Dataform connector

Prepare Google Cloud

Create the service account and grant the Google Cloud roles the Dataform connector needs, capability by capability

The connector signs in to Google Cloud as a service account, never as a user, and only ever reads. What it can catalog depends on the roles you give that service account. This page explains each role, what it allows the connector to do, and which features need it, so you can grant exactly what you want to use.

Permissions at a glance

# What you set up What it allows Features that need it Required
1 A service account (or Application Default Credentials ) The connector gets a Google access token. Everything Yes
2 APIs enabled : Dataform and BigQuery The connector can call the two services. Everything Yes
3 roles/dataform.viewer on the projects that hold repositories Reading repositories, release configurations, compilations and workflow runs. Everything Yes, on each project you list
4 roles/bigquery.metadataViewer on the datasets the pipelines read or write Reading table and routine metadata: columns, types, descriptions. Typed columns, column lineage, table types Recommended. Turn off bigQueryMetadata without it
5 Impersonation : token creator on a target service account Using the roles of another service account instead of granting them to the key owner. Only if you configure impersonateServiceAccount Optional
6 Blindata permissions of the target connection Writing the catalog, lineage and quality checks to Blindata. Upload, routines and their properties, quality checks, routine analysis Yes, for a live run

The service account never needs write roles, and it never needs BigQuery job permissions: the connector does not run queries and does not compile your Dataform code.

Create the service account

Allows: the connector to authenticate. Needed by: every feature.

  1. In the Google Cloud console , open IAM & Admin > Service accounts in a project of your choice and create a service account, for example blindata-dataform-reader.
  2. Do not grant it any role at creation: you grant the roles below on the projects and datasets it will read.
  3. Create a JSON key for it and download the file. You will attach it to the connection . The key is a long-lived secret: store it only in the agent, and rotate it according to your policy.

If your organization forbids service account keys, use Application Default Credentials or impersonation instead.

With these credentials the connector requests one access token for the Google Cloud platform scope (https://www.googleapis.com/auth/cloud-platform) and uses it for both services it reads:

Service What the token allows Used for
Dataform API Listing locations and repositories, reading release configs, workflow configs, compilations and runs Discovery, actions, dependencies, assertions, run results
BigQuery API Reading table and routine metadata (tables.get, tables.list, routines.get) Columns and types, routines, wildcard table schemas

Enable the APIs

Allows: the connector to call the services. Needed by: every feature.

Enable these APIs:

  • Cloud Dataform API (dataform.googleapis.com)
  • BigQuery API (bigquery.googleapis.com), unless you turn off capabilities.bigQueryMetadata

Enable them in the project that owns the service account. If a call is refused with the reason SERVICE_DISABLED, the run report names the project: enable the API in that project too. See Check your setup .

You do not need the Data Lineage API: the connector does not read Dataplex lineage.

Dataform Viewer

Allows: reading everything the connector needs from Dataform. Needed by: every feature.

Grant roles/dataform.viewer to the service account on each project that holds Dataform repositories you want to read. You can grant it on the project, or, to be stricter, on individual repositories.

The role lets the connector:

Dataform resource The connector reads… …which enables
Locations and repositories The repositories of each project and location, with their labels and Git remote The repository entities, and the scope of the run
Release and workflow configurations Release configs, their schedules and the compilation each one produced The deployed environments, and the schedules on the entity
Compilation results The compiled actions, dependencies, columns documentation and assertions The catalog and the table lineage
Workflow invocations The runs of a workflow and the state of each action Assertion results, last run properties, the fallback environment

The role is read-only. In particular it does not allow creating a compilation, so the connector can never trigger a compile or install packages in your project.

Dataform repositories are regional. Grant the role on the project and it applies to every location.

BigQuery Metadata Viewer

Allows: reading table and routine definitions, never their data. Needed by: typed columns and column lineage.

Grant roles/bigquery.metadataViewer to the service account on the datasets (or the projects) that your pipelines read or write. That includes the datasets of source projects that your repositories declare as sources, even when you do not list those projects in the job.

This role lets the connector:

  • read each table’s type, columns, GoogleSQL types, order, descriptions and policy tags;
  • read the definition of routines created by Dataform operations;
  • list the tables of a dataset, only to find the schema of wildcard tables such as events_*.

Without it, the connector degrades instead of failing:

  • tables it cannot read are reported as METADATA_UNAVAILABLE;
  • fields that already exist in Blindata keep their stored type and position;
  • new tables get fields from the columns documented in Dataform, without type or position, so column lineage is skipped for them (COLUMNS_UNAVAILABLE).

If you cannot grant the role at all, set capabilities.bigQueryMetadata to false. The connector then makes no BigQuery call, and relies on the columns a BigQuery crawl already cataloged. See Configure the job .

Application Default Credentials

When the agent runs on Google Cloud (Compute Engine, GKE with Workload Identity, Cloud Run) or on a host where the Google Cloud environment is configured, the agent can use the identity it already has instead of a key file. This is Application Default Credentials (ADC).

  • Turn on useApplicationDefaultCredentials in the connection and attach no key file.
  • Grant the roles above to the identity the agent runs as.
  • ADC works only where the agent process can find Google credentials. On other hosts, the connection test fails when it asks for a token.

Impersonation

Impersonation lets the agent borrow the identity of a target service account that holds the roles above, so that no key for that account is ever created. The agent signs in with its own credentials (a key file or ADC) and asks Google for short-lived tokens of the target.

  1. Grant roles/dataform.viewer and roles/bigquery.metadataViewer to the target service account.
  2. Grant the identity the agent signs in with roles/iam.serviceAccountTokenCreator on the target service account.
  3. Make sure the IAM Service Account Credentials API (iamcredentials.googleapis.com) is enabled in the project of the target.
  4. Set the target’s e-mail in impersonateServiceAccount in the connection . It must end in .gserviceaccount.com.

Regional endpoint

Dataform offers regional endpoints for data residency, for example https://dataform.europe-west3.rep.googleapis.com. To use one, set dataformEndpoint in the connection . Dataform calls then go to that endpoint. BigQuery calls always use the global endpoint.

Blindata permissions

The agent writes to Blindata with the connection named in outputOptions.targetConnectionName. Besides the permissions to upload metadata that any agent job needs, the Blindata user behind that connection needs:

Permission Needed for
QUERY_PARSER_EDITOR Query buckets and the statements of their analysis, the base of column lineage. Also needed to ask for the analysis of routines.
SYSTEMS_EDITOR Setting the BigQuery project id on a System, and uploading the routines of the Dataform SQL and their additional properties (dataformFilePath and the others). Analyzing routines needs both this and QUERY_PARSER_EDITOR.
QUALITY_ADMIN Creating and updating quality suites.
QUALITY_EDITOR Creating and updating quality checks, and uploading their results.

The platform must also be recent enough to accept additional properties on routines (the routine properties upload): with an older version the upload of the routine properties fails and the run is reported in error. Update Blindata before you update the agent job.

Only the quality round degrades gracefully: when the two quality permissions are missing, that round stops and is reported once as QUALITY_PERMISSION_MISSING, and the catalog and the lineage are still published. A refused call in any other step is reported as a publication error and ends the run in error. If you turn quality checks off, the two quality permissions are not needed.

Blindata configuration: make routines searchable

The connector stores the dataform… properties on every routine, but the routines list can filter by a property only if an administrator has defined it as a custom property of the System Routine resource type and marked it searchable. Without the definition the values are still stored and shown in the routine’s detail page (Properties tab, with the other properties), and you cannot filter by them. Do this once per Blindata tenant, before or after the first run: the values already uploaded become filterable as soon as the definition exists.

You need the SYSTEMS_ADMIN permission. For each property in the table:

  1. Open Settings > Custom Properties and add a new custom property.
  2. Set Resource Type to System Routine.
  3. Set Name to the exact property name, for example dataformFilePath. It must match, and it cannot be changed afterwards.
  4. Set Label to the text users will see, for example Dataform file.
  5. Set Type to STRING.
  6. Enable Render as a search filter, and save.
Property Define it? Why
dataformFilePath Yes Finds all the routines of a source file, such as orders.sqlx.
dataformActionId Yes Finds the routines of one action, in every environment.
dataformEnvironment Yes Limits the list to one environment, such as production.
dataformSqlKind Yes Separates, for example, incremental queries from assertions.
dataformRepository Optional Useful when one Blindata tenant catalogs several repositories.
dataformTags Optional Finds the routines whose action has a tag. Tags are comma-separated in one value.

To search, open the routines list, open the filters, pick the property by its label and type part of the value. The filter matches values that contain the text, whatever the case, so orders.sqlx finds definitions/marts/orders.sqlx. For how custom properties work in general, see Custom Properties .

Check your setup

Run the job as a dry run : it reads Google Cloud and counts what it would upload, without writing to Blindata. The connection test checks only that the agent can obtain a token, so it cannot tell you whether a project is readable. The run report does.

Open the run and look at crawledDataform.scopeGaps and at the log:

What you see What it means What to do
A scope gap ACCESS_DENIED with reason SERVICE_DISABLED The Dataform or BigQuery API is not enabled in the project named in the message. Enable the API in that project.
A scope gap ACCESS_DENIED with reason IAM_PERMISSION_DENIED or PERMISSION_DENIED The service account cannot read that project or location. Grant roles/dataform.viewer on the project. Check impersonation, if used.
A scope gap RESOURCE_UNAVAILABLE The project or location does not exist, or has no Dataform repositories there. Check the project id in projects, and the values in locations.
Log METADATA_UNAVAILABLE, and the gap on the tables The service account cannot read the metadata of some BigQuery tables. Grant roles/bigquery.metadataViewer on those datasets.
Log QUALITY_PERMISSION_MISSING The Blindata user cannot write quality suites or checks. See Blindata permissions .
Unable to obtain a Google access token The key, ADC or impersonation setup is wrong. The message never contains key material. Check the connection and the token creator role.

A scope that cannot be read never stops the run: the connector reports it and goes on with the other projects and locations.