Accelerate the Value of Data

Enable attribute filtering for the Reltio Data Pipeline for Snowflake

Learn how to enable filtering for the Reltio Data Pipeline for Snowflake.

You need to have ROLE_ADMIN_TENANT or ROLE_ADMIN_CUSTOMER and access to the tenant business configuration.
By default, the JSON file contains all the attributes defined in the tenant business configuration. Enable attribute filtering to send only specified attributes for entity, interaction, and relationship object types to the data warehouse.
Note: If dataFilteringEnabled is set to true but dataPipelineConfig sections are not defined in the business configuration then no data will go to the storage and all data will be filtered out.
To enable attribute filtering for the Reltio Data Pipeline for Snowflake:
  1. Enable the following parameter in your tenant during the initial configuration of your Reltio Data Pipeline for Snowflake:

    "dataFilteringEnabled": true

  2. In your tenant business configuration, locate the relevant object type section, add a data pipeline configuration, and specify the attributes you want to include:
    • without object inheritance

      Entities attribute filteringIn this example for the Employment entity type, only these attributes will be sent to the cloud data warehouse: Title, IsCurrent, and Commenters.
      {
        "uri": "configuration/entityTypes/Employment",
        "dataPipelineConfig": {
          "enabled": true,
          "attributes": [
            "configuration/entityTypes/Employment/attributes/Title",
            "configuration/entityTypes/Employment/attributes/IsCurrent",
            "configuration/entityTypes/Employment/attributes/Commenters"
          ]
        }
      }
      Interaction attribute filteringIn this example for the ExhibitionEventNoMembers interaction type, only these attributes will be sent: EXHBT_NAME , EXHBT_URL, and EXHBT_DAYS.
      {
        "uri": "configuration/interactionTypes/ExhibitionEventNoMembers",
        "dataPipelineConfig": {
          "enabled": true,
          "attributes": [
            "configuration/interactionTypes/ExhibitionEvent/attributes/EXHBT_NAME",
            "configuration/interactionTypes/ExhibitionEvent/attributes/EXHBT_URL",
            "configuration/interactionTypes/ExhibitionEvent/attributes/EXHBT_DAYS"
          ]
        }
      }
      Relation attribute filteringIn this example for the Supplier interaction type, only these attributes will be sent: Type, Primary, and Primary.
      {
        "uri": "configuration/relationTypes/Supplier",
        "dataPipelineConfig": {
          "enabled": true,
          "attributes": [
            "configuration/relationTypes/Supplier/attributes/Type",
            "configuration/relationTypes/Supplier/attributes/Primary",
            "configuration/relationTypes/Supplier/attributes/Primary"
          ]
        }
      }
    • With object inheritance (for entities only)

      Override filteringParent entityChild entity
      DefaultIn this example, attribute filtering is enabled on the parent entity (Individual) and by default applies to the child entity, so the Name, FirstName, and LastName attributes of the child entity (HCP) are sent to the cloud data warehouse.
      {
          "uri": "configuration/entityTypes/Individual",
          "label": "Individual",
          "description": "Invividual",
          "typeColor": "#99CCEE",
          "dataPipelineConfig": {
            "enabled": true,
            "attributes": [
              "configuration/entityTypes/Individual/attributes/Name",
              "configuration/entityTypes/Individual/attributes/FirstName",
              "configuration/entityTypes/Individual/attributes/LastName"
            ]
          }
        }
      {
          "uri": "configuration/entityTypes/HCP",
          "label": "HCP",
          "description": "Health care provider",
          "typeColor": "#99CCFF",
          "extendsTypeURI": "configuration/entityTypes/Individual",
          "lifecycleActions": {}
        }
      Disable filtering in the child entityIn this example, the child entity type (HCP) overrides the data filtering of the parent entity type (Individual) by disabling filtering, so all HCP attributes are sent to the cloud data warehouse.
      {
          "uri": "configuration/entityTypes/Individual",
          "label": "Individual",
          "description": "Invividual",
          "typeColor": "#99CCEE",
          "dataPipelineConfig": {
            "enabled": true,
            "attributes": [
              "configuration/entityTypes/Individual/attributes/Name",
              "configuration/entityTypes/Individual/attributes/FirstName",
              "configuration/entityTypes/Individual/attributes/LastName"
            ]
          }
        }
      {
          "uri": "configuration/entityTypes/HCP",
          "label": "HCP",
          "description": "Health care provider",
          "typeColor": "#99CCFF",
          "extendsTypeURI": "configuration/entityTypes/Individual",
          "lifecycleActions": {},
          "dataPipelineConfig": {
            "enabled": false
          }
        }
      Child entity filters a different attribute than the parent entityIn this example, the child entity type (HCP) overrides the data filtering of the parent entity type (Individual) by filtering only the name attribute, so only that HCP attribute is sent to the cloud data warehouse.
      {
          "uri": "configuration/entityTypes/Individual",
          "label": "Individual",
          "description": "Invividual",
          "typeColor": "#99CCEE",
          "dataPipelineConfig": {
            "enabled": true,
            "attributes": [
              "configuration/entityTypes/Individual/attributes/Name",
              "configuration/entityTypes/Individual/attributes/FirstName",
              "configuration/entityTypes/Individual/attributes/LastName"
            ]
          }
        }
      {
          "uri": "configuration/entityTypes/HCP",
          "label": "HCP",
          "description": "Health care provider",
          "typeColor": "#99CCFF",
          "extendsTypeURI": "configuration/entityTypes/Individual",
          "lifecycleActions": {},
          "dataPipelineConfig": {
             "enabled": true,
             "attributes": [
              "configuration/entityTypes/HCP/attributes/Name"
            ]
          }
        }