Apache Spark SQL
Catalog Spark SQL tables and views through the Spark Thrift Server (HiveServer2 protocol).
Capabilities
| Capability | Support | Notes |
|---|---|---|
| Tables, views & columns | Supported | |
| Keys, constraints & indexes | Not available | |
| Procedures & functions | Not available | |
| Data lineage | Partial | |
| Lineage from query history | Not available | |
| Business glossary links | Supported | |
| Data quality probes | Supported | |
| Data profiling | Supported | |
| Data classification | Supported | |
| Chat with your data | Supported |
Connection settings
Create the connection on the Blindata Agent from Settings > Agents > Connections with the connection wizard, choosing Apache Hive in the technology step.
| Driver class | org.apache.hive.jdbc.HiveDriver |
|---|---|
| JDBC URL | jdbc:hive2://localhost:10000/default |
| Authentication |
Service principal (shared technical user) User principal (personal credentials, BYOC) Ephemeral credentials (asked at each run)
|
| Crawling strategy | JDBC |
The Blindata Agent does not include this driver: download the vendor JDBC driver and copy the .jar into the agent drivers folder, as described in
Add JDBC drivers.
Driver properties can be added under Advanced settings in the connection wizard.
Prerequisites & permissions
- A Blindata Agent installed with network access to Apache Spark SQL.
- The vendor JDBC driver installed on the agent (see Add JDBC drivers).
- A user allowed to list databases and describe tables through the Thrift Server.
Set up metadata crawling
- Create the connection to Apache Spark SQL on the agent (see Connection settings) and use Test to verify it.
- Create or open the system that represents Apache Spark SQL in the catalog and link it to the connection.
- In the CRAWLING tab add a crawling job with strategy JDBC and, if needed, replace the default queries with the ones suggested below.
- Run a TEST to check the number of extracted objects, then RUN or schedule the job.
The job options and the column aliases of every extraction query are described in the Generic JDBC reference.
Lineage through OpenLineage
The metadata crawled through the Thrift Server describes tables, not how they are produced. Spark applications can describe the transformations they run with the OpenLineage Spark listener, and push the events to the Blindata Agent. The OpenLineage connector turns them into lineage on the tables you crawl.
What the integration emits and Blindata uses:
- Inputs and outputs of each job give table lineage, and the
columnLineagefacet gives column lineage. spark_applicationDetails(application name and id, master, Spark version) is stored as properties of the job.spark_unknown(plan nodes the integration could not handle) is reported as the gapPRODUCER_PARTIAL_LINEAGE.spark_properties,spark_jobDetails,spark.logicalPlananddebugare ignored.
Configuration of the Spark listener and its transport: Configure your producers . Facet by facet: Facet support reference .