TIBCO Spotfire
Import Spotfire library reports and generate lineage from source tables and columns to reports, by querying the Spotfire repository database.
Capabilities
| Capability | Support | Notes |
|---|---|---|
| Tables, views & columns | Supported | Library items of type dxp are imported as REPORT physical entities, with their folder path as schema. |
| Keys, constraints & indexes | Not available | |
| Procedures & functions | Not available | |
| Data lineage | Supported | Dataflows from source columns to reports from LIB_RESOLVED_DEPEND and LIB_PROPERTIES. |
| Lineage from query history | Not available | |
| Business glossary links | Not available | |
| Data quality probes | Not available | |
| Data profiling | Not available | |
| Data classification | Not available | |
| Chat with your data | Not available |
Connection settings
Create the connection on the Blindata Agent from Settings > Agents > Connections with the connection wizard, choosing Template of the repository database (Oracle, SQL Server, PostgreSQL) in the technology step.
| Driver class | oracle.jdbc.OracleDriver / com.microsoft.sqlserver.jdbc.SQLServerDriver |
|---|---|
| JDBC URL | JDBC URL of the Spotfire repository database |
| Authentication |
Service principal (shared technical user) User principal (personal credentials, BYOC) Ephemeral credentials (asked at each run)
|
| Crawling strategy | SQL |
The PostgreSQL and SQL Server drivers are included in the Blindata Agent. For an Oracle repository, install the Oracle JDBC driver on the agent. See Add JDBC drivers. Driver properties can be added under Advanced settings in the connection wizard.
Prerequisites & permissions
- A Blindata Agent installed with network access to TIBCO Spotfire.
- Read-only access to the Spotfire repository tables
LIB_ITEMS,LIB_ITEM_TYPES,LIB_RESOLVED_DEPENDandLIB_PROPERTIES.
Set up metadata crawling
- Create the connection to TIBCO Spotfire on the agent (see Connection settings) and use Test to verify it.
- Create or open the system that represents TIBCO Spotfire in the catalog and link it to the connection.
- In the CRAWLING tab add a crawling job with strategy SQL and, if needed, replace the default queries with the ones suggested below.
- Run a TEST to check the number of extracted objects, then RUN or schedule the job.
The job options and the column aliases of every extraction query are described in the Generic JDBC reference.
Suggested extraction queries
Spotfire stores its library in a repository database (Oracle, SQL Server or PostgreSQL). Create a connection to that database and a crawling job with the SQL strategy: the following queries extract Spotfire reports and generate dataflows from the tables and columns of a source system to the reports in the Spotfire system.
Reports query (physical entities)
SELECT distinct
li.TITLE as table_name,
'REPORT' as table_type,
left((
coalesce (folders5.title || '/', '') ||
coalesce (folders4.title || '/' , '') ||
coalesce (folders3.title || '/', '') ||
coalesce (folders2.title || '/', '') ||
coalesce (folders.title, '')
), 255) as table_schema,
coalesce (li.DESCRIPTION, '') as description
FROM LIB_ITEMS li
left join lib_items folders
on li.parent_id = folders.item_id
left join lib_items folders2
on folders.parent_id = folders2.item_id
left join lib_items folders3
on folders2.parent_id = folders3.item_id
left join lib_items folders4
on folders3.parent_id = folders4.item_id
left join lib_items folders5
on folders4.parent_id = folders5.item_id
WHERE li.ITEM_TYPE = (SELECT LIB_ITEM_TYPES.TYPE_ID FROM LIB_ITEM_TYPES WHERE LIB_ITEM_TYPES.LABEL = 'dxp')
Data flows query
SELECT distinct
'insert_dataflow_scope_here' as scope,
left((
coalesce(SchemaName.Property_Value || '.', '') ||
coalesce(TableName.Property_Value || '.','') ||
coalesce(ColumnName.Property_Value,'') ||
'->' ||
coalesce(SchemaName.ReportFolder || '/', '') ||
coalesce(SchemaName.ReportName, '')
),
255
) as name,
'insert_data_flow_description_here' as description,
'insert_source_system_uuid (optional)' as from_system_uuid,
'insert_source_system_name' as from_system_name,
SchemaName.Property_Value as from_physical_entity_schema,
TableName.Property_Value as from_physical_entity_name,
ColumnName.Property_Value as from_physical_field_name,
'insert_spotfire_system_uuid (optional)' as to_system_uuid,
'insert_spotfire_system_name' as to_system_name,
left(coalesce(SchemaName.ReportFolder, ''), 255) as to_physical_entity_schema,
SchemaName.ReportName as to_physical_entity_name
FROM
( SELECT A.ReportName, D.* FROM (SELECT TITLE ReportName, ITEM_ID
ReportItemID
FROM lib_items) A
JOIN LIB_RESOLVED_DEPEND B ON A.ReportItemID = B.DEPENDENT_ID
JOIN LIB_RESOLVED_DEPEND C ON B.REQUIRED_ID = C.DEPENDENT_ID
JOIN LIB_PROPERTIES D ON D.ITEM_ID = C.REQUIRED_ID
WHERE D.PROPERTY_NAME = 'table') TableName
JOIN (SELECT A.ReportName, D.*
FROM (SELECT TITLE ReportName, ITEM_ID ReportItemID
FROM lib_items ) A
JOIN LIB_RESOLVED_DEPEND B ON A.ReportItemID = B.DEPENDENT_ID
JOIN LIB_RESOLVED_DEPEND C ON B.REQUIRED_ID = C.DEPENDENT_ID
JOIN LIB_PROPERTIES D ON D.ITEM_ID = C.REQUIRED_ID
WHERE D.PROPERTY_NAME = 'column') ColumnName
ON TableName.ITEM_ID = ColumnName.ITEM_ID AND
TableName.ReportName = ColumnName.ReportName
JOIN (SELECT A.ReportName, D.*
FROM (SELECT TITLE ReportName, ITEM_ID ReportItemID
FROM lib_items) A
JOIN LIB_RESOLVED_DEPEND B ON A.ReportItemID = B.DEPENDENT_ID
JOIN LIB_RESOLVED_DEPEND C ON B.REQUIRED_ID = C.DEPENDENT_ID
JOIN LIB_PROPERTIES D ON D.ITEM_ID = C.REQUIRED_ID
WHERE D.PROPERTY_NAME = 'catalog') DatabaseName
ON ColumnName.ITEM_ID = DatabaseName.ITEM_ID AND
DatabaseName.ReportName = ColumnName.ReportName
JOIN ( SELECT A.ReportName, D.*, left(
(
coalesce (folders5.title || '/', '') ||
coalesce (folders4.title || '/' , '') ||
coalesce (folders3.title || '/', '') ||
coalesce (folders2.title || '/', '') ||
coalesce (folders.title, '')
), 255) as ReportFolder
FROM ( SELECT TITLE ReportName, ITEM_ID ReportItemID, parent_id
FROM lib_items ) A
JOIN LIB_RESOLVED_DEPEND B ON A.ReportItemID = B.DEPENDENT_ID
JOIN LIB_RESOLVED_DEPEND C ON B.REQUIRED_ID = C.DEPENDENT_ID
JOIN LIB_PROPERTIES D ON D.ITEM_ID = C.REQUIRED_ID
left join lib_items folders
on A.parent_id = folders.item_id
left join lib_items folders2
on folders.parent_id = folders2.item_id
left join lib_items folders3
on folders2.parent_id = folders3.item_id
left join lib_items folders4
on folders3.parent_id = folders4.item_id
left join lib_items folders5
on folders4.parent_id = folders5.item_id
WHERE D.PROPERTY_NAME = 'schema' ) SchemaName
ON DatabaseName.ITEM_ID = SchemaName.ITEM_ID AND
DatabaseName.ReportName = SchemaName.ReportName