Accelerate the Value of Data

Configure a Snowflake pipeline for Azure in the Console

Learn how to use the Reltio Console to configure a Snowflake pipeline for Azure.

Before using the Reltio Data Pipeline for Snowflake on the Azure cloud platform, you must configure it in Reltio.
This topic describes how to configure this data pipeline using the Reltio Console cloud data warehouse pipelines application. If you prefer to configure this data pipeline using Reltio REST APIs, see topic Configure the Reltio Data Pipeline for Snowflake for Azure.
To configure a new Snowflake data pipeline for Azure:
  1. In the Console, select Cloud data warehouse pipeline. You will see the Cloud data warehouse pipeline page that lists the pipelines already configured.
  2. Select + NEW PIPELINE and from the displayed options select Snowflake.
    The Configure Snowflake pipeline page is displayed.

  3. In the Name field, enter the name of the pipeline, which is used to refer to the pipeline in the UI. The name must be alphanumeric, can be 3 to 20 characters in length, and should not contain any spaces.
  4. In the Authentication section, from the Cloud provider dropdown, select the Azure.
  5. Select Authenticate using client credentials to authenticate requests using client credentials.
    1. In the Client Credentials field, enter the unique client credentials to be used for authentication.
    2. In the Storage account name field, enter the name of the Azure storage account.
    3. In the Bucket Name field, enter the name of the bucket on Azure.
    4. In the Client ID field, enter the Azure active directory client ID.
    5. In the Tenant ID field, enter the Azure active directory tenant ID.
    6. In the Staging Bucket Region field, enter the name of the region if the bucket region is not from the United States.
  6. Select Authenticate using client key to authenticate requests using client key.
    1. In the Key field, enter the client key used for authentication.
    2. In the Storage account name field, enter the name of the Azure storage account.
    3. In the Bucket name field, enter te name of the bucket on Azure.
    4. In the Staging Bucket Region field, enter the name of the region if the bucket region is not from the United States.
  7. Select Authenticate using client token to authenticate requests using client token.
    1. In the Token field, enter the access token used to for authentication.
    2. In the Storage account name field, enter the name of the Azure storage account.
    3. In the Bucket name field, enter te name of the bucket on Azure.
    4. In the Staging Bucket Region field, enter the name of the region if the bucket region is not from the United States.
Verify configuration settings in your tenant
  1. In the Data delivery options section, from the Attribute format dropdown, select the format of the attributes stored.
    1. Select Data filtering to enable filtering for adapter.
    2. Select Transmit OV values only if you want to sync only operational values to the Snowflake.
    3. Select Serialize initial sources in Crosswalks to enable sending initial source in crosswalk.
Verify configuration settings in your tenant
  1. Select Save & Create at the top of the page to save and create the Snowflake pipeline for Azure.
Troubleshoot your Reltio configuration
Follow these steps if data is not publishing to your Azure cloud storage:
  1. Make sure you have:
    • access to edit the Reltio physical tenant configuration.
    • permission to view and modify Reltio streaming and pipeline settings.
    • access to view your Snowflake settings.
    • the ability to run API endpoints to validate your configurations and connections.
  2. Ensure streaming is enabled in the Reltio physical tenant configuration.
    • "streamingEnabled": true

    • "streamingAPIEnabled": true

    "streamingConfig": {
          "streamingEnabled": true, <<<===Make sure that it is set to true
          "streamingAPIEnabled": true, <<<===Make sure that it is set to true
          "analyzeOvChanges": false,
          "emptyStartEndRelationCrosswalks": false,
          "largeObjectsSupport": false,
          "skipPayload": false,
          "payloadType": "SNAPSHOT",
          "ovOnly": false,
  3. Ensure the pipeline queue is defined and enabled in the messaging.destination section of the streamingConfig
    • "enabled": true

     {
                      "payloadType": "SNAPSHOT",
                      "type": "queue",
                      "provider": "dph",
                      "name": "<environment>-datapipeline-events_<tenant>",
                      "dtssQueue": false,
                      "groupingQueue": false,
                      "dataPipelineQueue": true,
                      "enabled": true,  <<<<=====  Make sure that it is set to true
                      "format": "KRYO"
                  }
  4. Validate the data pipeline with this API and confirm the status is successful (200):
    • Check the external queue to ensure that events are processing.
      POST https://<environment>-data-pipeline-hub.reltio.com/api/tenants/<tenant>/adapters/Snowflake/validate
      
    • Validate the processed event, using a specific entity, across the pipeline.
      GET https://<environment>-data-pipeline-hub.reltio.com/api/tenants/<tenantid>/monitoring/_entityMonitoring?entityId=<entityID>