Configure the Dataform connector

This page walks you through the connection, the job, and then every field of the job configuration, from top to bottom. The job type is DATAFORM.

Before you start, complete Prepare Google Cloud : you need the service account key file (or Application Default Credentials), and the ids of the Google Cloud projects that hold your Dataform repositories.

Create the Google Cloud connection

Create the connection in the Blindata Agent, on the agent that will run the job. Open the agent, go to Connections, create a connection of type Google Cloud (see Manage Connections ), and fill in the properties below. Use the connection name as the value of connectionName.

The connection is generic: it holds how to sign in to Google Cloud, not which project to read. The projects are listed in the job. It uses shared (service principal) credentials.

In the connection wizard, search for Google Cloud (searching for “Dataform” or “BigQuery” finds it too). The wizard fields map to the properties below:

• Authentication section: the Use Application Default Credentials switch (useApplicationDefaultCredentials) and, only while the switch is off, the Service account key file upload (keyFile), asked for in the credentials step.
• Impersonation section: Impersonated service account (impersonateServiceAccount).
• Regional endpoints section: Dataform endpoint (dataformEndpoint).

Leave impersonateServiceAccount and dataformEndpoint empty unless you need them. Blindata treats the key file as a secret: it is stored encrypted and cannot be viewed after it is saved.

How the agent authenticates

For each run, the agent builds its Google credentials from the connection:

1. Key file: the service account in the key file. ADC: the identity the agent runs as, found the way Google client libraries find it.
2. With impersonation, it then asks Google for short-lived tokens of the target service account, using the first identity.
3. It requests an access token for the platform scope (https://www.googleapis.com/auth/cloud-platform) and sends it to the Dataform and BigQuery REST APIs. The token is refreshed when it expires, and once when Google answers 401.

Calls that Google answers with 429, 500, 502, 503 or 504 are retried, up to limits.retryMaxAttempts attempts, waiting for the time Google asks for or, otherwise, a few seconds more each time (up to 30).

The connection test only checks that a token can be obtained, including through impersonation. It does not name a project, so it cannot tell whether a project is readable: use a dry run for that.

If the connection is not valid, the job stops before any Google call, with one of these messages. They never contain the key:

The connection is used only to read Google Cloud. Writes go to Blindata through a separate connection, named in outputOptions.targetConnectionName, which must be a Blindata connection (type BLINDATA). That connection is used only when dryRun is false. See Blindata permissions for what its user needs.

Create the job

1. Open the agent, go to Schedules, and create a schedule with job type DATAFORM. Set the cron expression and the time zone.
2. Paste the job configuration (JSON, below) as the job’s configuration.
3. Save the schedule. Run it, then check its log in the agent’s Runs tab.

The job configuration is a JSON object. Unknown fields are ignored without an error (see Common mistakes ).

Job configuration parameters

The job configuration has these top-level fields. Only connectionName, projects and dataFlowScope are always required.

Omitting outputOptions, or leaving dryRun out of it, makes the job a dry run.

Project ids are the ids, not the names or numbers: 6 to 30 characters, lowercase letters, digits and hyphens, starting with a letter. A domain-scoped id such as example.com:my-project is accepted.

queryBucketPattern accepts the placeholders <projectId>, <location>, <repository> and <environment>, and must contain <repository> and <environment>. Bucket names are unique in Blindata, so keep <projectId> in the pattern when two projects can have a repository with the same id.

repositories

Patterns are Java regular expressions. They are case-sensitive, and they must match the whole id (prod-.* matches prod-orders, but orders does not match prod-orders). A repository that is filtered out is reported as skipped, with the reason FILTERED.

Repositories are classified by their labels. Regular Dataform repositories and BigQuery pipelines are read. Standalone BigQuery Studio assets (saved queries, notebooks, data canvases and data preparations that are not part of a repository or pipeline) are skipped and counted in the run report.

environments

An environment is a repository plus the release configuration that produced the compilation the job reads. It is named after the release configuration id.

Workspace and user-scoped compilations are never read. A repository whose release configurations are all left out by releaseConfigs is skipped, with the reason NO_SELECTED_ENVIRONMENT. A selected release configuration that has not produced a compilation yet gives the gap NO_COMPILATION_RESULT for that environment: the repository is still cataloged.

Each environment can write different tables, because release configurations can add suffixes or prefixes, or change the default project or dataset. Reading development environments adds their tables and lineage to the catalog. Most teams read production only: see Recommended setups .

systems

The System of a BigQuery project is found in this order: a projectSystems entry; a System that already carries the property bigQueryProjectId with that project id; a System with the name given by projectSystemNamePattern; otherwise a new System is created. A systemName that does not exist is created. A systemUuid that does not exist stops the run with an error (in a dry run it is planned as a new System).

Every project that the pipelines read or write gets a System, including source projects you did not list in projects. See Dataform to Blindata catalog .

capabilities

runtimeLineage

Runtime lineage from Dataplex is not available yet. The other keys of this block are accepted and have no effect.

quality

After the first import, only runs newer than the check’s last result are imported. See Assertions as quality checks .

limits

outputOptions

A dry run reads everything and reports what the run would create. Because it reads nothing from Blindata, it treats every System and entity as new, so its counts are an upper bound, and links from quality checks to tables are not resolved. The run report has dryRun: true.

cleanupOptions

cleanupOptions is the standard Blindata system cleanup, the same object the other agent connectors use. After a successful run, it deletes content of the Google Dataform System that was not updated for daysFromLastUpdate days. Cleanup runs only when all the following are true:

• daysFromLastUpdate is greater than 0.
• The run is not a dry run.
• The run is complete: no project or location was unreadable, every environment was read without compilation errors and with all the BigQuery metadata it needed, and lineage finished for every environment.
• The Google Dataform System exists, that is, capabilities.processLayer is on.

When a condition fails, the log says which one.

The flags are sent to Blindata as you write them: only the ones you set to true delete anything. The job sets the cleanup level to SYSTEM, the System to the Google Dataform System and the data flow scope to your dataFlowScope.

Good to know:

• BigQuery project Systems are never cleaned: they may belong to a BigQuery crawl. A table that is deleted in BigQuery stays in Blindata until your crawl removes it.
• Flows of query buckets are not in your dataFlowScope. The job removes stale statements, and with them their flows, itself, on every run, for each environment it read completely. It also removes the script routines it no longer plans, together with their statements.
• Script routines that were not analyzed for a while. When a script is held back (for example, because a table it reads has no columns in Blindata), its routine is not updated. With systemRoutinesDeletionEnabled, cleanup deletes it after daysFromLastUpdate days, and the next run removes its statements and the lineage they produced. The lineage comes back when the script can be analyzed again.

Full example

This configuration lists every field with its default value, except the pinned projectSystems entry and the project, which are examples. Anything you leave out takes the default shown.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "locations": [],
  "dataFlowScope": "dataform",
  "repositories": { "includePatterns": [], "excludePatterns": [] },
  "environments": {
    "releaseConfigs": [],
    "fallback": "INVOCATION_THEN_COMPILATION"
  },
  "systems": {
    "projectSystems": [
      {
        "projectId": "my-raw-project",
        "systemName": "BigQuery Raw"
      }
    ],
    "projectSystemNamePattern": "<projectId>",
    "dataformSystem": { "name": "Google Dataform" }
  },
  "capabilities": {
    "bigQueryMetadata": true,
    "processLayer": true,
    "columnLineage": true,
    "qualityChecks": true,
    "assertionViews": false,
    "runMetadata": false
  },
  "runtimeLineage": { "mode": "OFF" },
  "quality": { "firstImportLookbackDays": 7 },
  "queryBucketPattern": "<projectId>/<repository>/<environment> (Dataform)",
  "limits": {
    "maxParallelism": 4,
    "retryMaxAttempts": 5,
    "invocationRequestsPerMinute": 50,
    "maxInvocationsPerRepository": 500,
    "maxInvocationsQueriedPerEnvironment": 50,
    "maxCompilationResultsListed": 200,
    "maxTablesListedPerDataset": 10000
  },
  "outputOptions": {
    "dryRun": false,
    "targetConnectionName": "blindata-prod"
  },
  "cleanupOptions": { "daysFromLastUpdate": 0 }
}

Recommended setups

Each example shows only the fields that differ from the defaults.

1. First run: dry run. Use the defaults and a dry run. Check the run report before you enable the upload.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "outputOptions": { "dryRun": true }
}

2. Nightly upload of production only. Read the production release configuration of every repository, upload, and remove content that was not updated for 30 days. Repositories without release configurations are still read through the fallback.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "environments": { "releaseConfigs": ["production"] },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" },
  "cleanupOptions": {
    "daysFromLastUpdate": 30,
    "dataFlowsDeletionEnabled": true,
    "physicalEntitiesDeletionEnabled": true,
    "physicalFieldsDeletionEnabled": true
  }
}

3. Reuse the Systems of a BigQuery crawl. Pin the BigQuery projects to the Systems your crawl already created, so Dataform enriches them instead of creating new ones. A System that already has the property bigQueryProjectId is found without this.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "systems": {
    "projectSystems": [
      {
        "projectId": "my-dataform-project",
        "systemName": "BigQuery Analytics"
      },
      { "projectId": "my-raw-project", "systemName": "BigQuery Raw" }
    ]
  },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

4. Enrichment only, without the process layer. Add Dataform context and lineage to your BigQuery entities, with no Google Dataform System and no quality checks. Scripted operations are not analyzed in this setup, because they are analyzed as routines of the Dataform System.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "capabilities": { "processLayer": false, "qualityChecks": false },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

5. No BigQuery access. The service account can read Dataform but not BigQuery. The job does not call BigQuery, and relies on the columns your BigQuery crawl already stored. Tables with no stored columns are published without column lineage.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "capabilities": { "bigQueryMetadata": false },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

6. One team’s repositories. Read two locations and only the repositories whose id starts with sales-, except the sandbox.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "locations": ["europe-west3", "europe-west8"],
  "dataFlowScope": "dataform-sales",
  "repositories": {
    "includePatterns": ["sales-.*"],
    "excludePatterns": [".*-sandbox"]
  },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

7. Table lineage and catalog only. No SQL analysis and no quality: the lightest run.

{
  "connectionName": "google-cloud",
  "projects": ["my-dataform-project"],
  "dataFlowScope": "dataform",
  "capabilities": { "columnLineage": false, "qualityChecks": false },
  "outputOptions": { "dryRun": false, "targetConnectionName": "blindata-prod" }
}

Validation errors

The job rejects the configuration before it calls Google Cloud. Every message starts with Configuration cannot be parsed: , followed by:

A configuration that is not valid JSON fails with Configuration cannot be parsed.

A systemUuid in systems.projectSystems that Blindata does not know stops a live run with systems.projectSystems[<project>] not found.

Read the run report

Every run ends with a report. Read it after a dry run, and whenever something is missing.

A run that degraded (an unreadable project, a table without metadata, a quality permission that is missing) finishes with a warning and still publishes the rest. Only an invalid configuration, a System you pinned that does not exist, and a refused Blindata upload end it in error.

Common mistakes

• Unknown fields are ignored. The job does not fail on a misspelled key, so a setting you typed wrongly is simply not applied. Check the run report after the first run.
• dryRun defaults to true. A job with outputOptions that sets only targetConnectionName is still a dry run. Set "dryRun": false.
• Repository patterns are whole-name and case-sensitive. sales does not match sales-orders. Use sales-.*.
• Development environments add tables. By default every enabled release configuration is read, so a dev configuration with a table prefix adds its tables and lineage to the catalog. List the ones you want in environments.releaseConfigs.
• Source projects are cataloged without being listed. projects bounds which repositories are read, not which projects get a System. If a source project’s datasets are not readable, its tables are reported as METADATA_UNAVAILABLE.
• Changing queryBucketPattern creates new buckets. The job reconciles only the buckets it uses in the run: the old bucket keeps its statements and flows until you delete it.
• Steward edits to quality checks do not last. The checks are managed by the job. See Assertions as quality checks .
• dataFlowScope is shared by cleanup. Use a value that no other job uses, or cleanup of this job can remove the flows of the other.

Updated on 23 Oct 2026