Configure the Reltio Data Pipeline for Databricks for Azure
Learn how to configure Databricks to receive data from your Reltio tenant in the Azure cloud.
Ready to move your data with your Reltio Data Pipeline for Databricks? Configure the pipeline to keep your Delta Lake tables and views in sync with your Reltio data model.
- Configure Databricks pipeline for Azure using Console UI - simpler UI-based configuration with automated steps.
- Configure Databricks pipeline for Azure using APIs - API-based configuration with many manual steps.
Before you start
Before you start configuring the Reltio for Reltio Data Pipeline for Databricks, ensure you have the necessary permissions and information at hand. You may find it helpful to consult this page for easy reference.
| Prerequisite | Required information | Your details |
|---|---|---|
| Configure Azure cloud storage for Databricks | ||
| The service requires the object storage to be publicly accessible over the internet. | ||
| Azure storage account management permissions | You are an Azure administrator OR Ask your Azure administrator to perform these tasks | |
| Integrate Azure cloud storage with Databricks | ||
| Azure storage account management permissions | You are an Azure administrator OR Ask your Azure administrator to perform these tasks | |
| Databricks account administrator permissions | You've been assigned these roles:
OR You've been assigned a role that contains these roles OR Ask your Databricks administrator to perform these tasks | |
| Databricks Unity Catalog (when used) |
| |
| Configure the Reltio Data Pipeline for Databricks | ||
| Reltio tenant | Tenant Environment Name | |
| Tenant ID | ||
| Support request | Reltio Data Pipeline configuration request for Databricks | |
| Validate and sync with the Reltio Data Pipeline for Databricks for Azure | ||
| Reltio administrator permissions | You have one of these roles:
OR Ask your Reltio administrator to perform these tasks. | |
Take note
As you work through the configuration, you'll want to make a note of some values you'll need in later steps and stages. You may find it helpful to download and make a copy of this page and record your information as you go along.
| Stage/section | Entry field | Your details |
|---|---|---|
| Determine mode for running pipeline | Mode of Delta Live Tables pipeline | |
| Configure Azure cloud storage for Databricks | ||
| Create an Azure resource group | Resource group | |
| Create an Azure storage account | Storage account name | |
| Create an Azure storage account container for Staging with a lifecycle rule. | Staging Storage container name | |
| Create an Azure storage account container for Target | Target Storage container name | |
| Get the Azure subscriptionId under which all the storage accounts are created | subscriptionId | |
| Configure Azure Event Notification for Staging Container [Only Required if using File Notification Mode] | ||
| Create an Azure storage account queue | Storage queue URL | |
| Create an Azure event grid subscription | Event subscription name | |
| Permission setup for Data Pipeline Hub service | ||
| If auth method is Client Credentials: | Application display name | |
| Application (client) ID | ||
| Directory (tenant) ID | ||
| Client secret | ||
| If auth method is SAS Token: | SAS Tokens | |
| If auth method is Keys: | Storage Account Key | |
| Create Azure access controls for Staging | Staging Custom role name | |
| Permission setup for Databricks | ||
| Databricks host URL | Databricks URL | |
| Create secret scope and add keys to Databricks | Secrets Scope Name | |
| Create secret scope and add keys to Databricks | Secrets Key Name for Staging Container | |
| Create secret scope and add keys to Databricks | Secrets Key Name for Target Container | |
| Manage service principals, a Databricks guide | Service Principal Token | |
| Create Azure access controls for Target container | Target Custom role name | |