Configure chain cleansing

Learn how to configure chain cleansing in cleanseConfig to run cleanse functions in sequence, where each function runs based on the result of the previous one.

Chain cleansing uses the chain function of cleanseConfig in tenant configuration to run cleanse functions in sequence. Use chain cleansing when a primary cleanse function must run first and a fallback cleanse function must run only when the primary function does not return a complete result.

Before you configure a chain function, review the parameters that control its behavior and understand how the platform decides whether a cleanse step has produced a complete result.

Chain cleansing parameters and behavior

Each cleanse function in a chain runs in sequence, and the platform checks the result before continuing to the next function.

The following table describes the parameters in chain cleansing.


Parameter	Description
`sequence`	Specifies a list of cleansing chains that are applied in sequence during cleansing.
`chain`	Specifies the list of cleanse functions that are applied in sequence in the chain.
`proceedOnSuccess`	Specifies whether the chain proceeds to the next cleanse step if the current cleanse function succeeds. The default value is `true`.
`proceedOnFailure`	Specifies whether the chain proceeds to the next cleanse step if the current cleanse function does not succeed. The default value is `true`.
`filter`	An optional condition on a cleanse function. It controls whether the function runs for specific entity attribute values. Using a filter prevents unnecessary processing and restricts the function to relevant data.
`params`	Specifies additional parameters for the cleanse function.

In a chain, each cleanse function runs only when the previous function allows the chain to continue.

proceedOnSuccess controls whether the chain continues after a successful cleanse.
proceedOnFailure controls whether the chain continues after an unsuccessful cleanse.

After each cleanse step runs, the platform checks the outputMapping. If all fields marked "mandatory": true have values, the step succeeds and the chain applies proceedOnSuccess. If one or more mandatory fields are empty or missing, the step fails and the chain applies proceedOnFailure. For address cleansing, the Address Verification Code (AVC) is typically marked as mandatory, if a cleanse step does not return an AVC, the platform marks the step as failed.

The following JSON shows an outputMapping configuration with AVC set as mandatory.

"outputMapping": [
  {
    "attribute": "configuration/entityTypes/Location/attributes/AVC",
    "mandatory": true,
    "allValues": false,
    "cleanseAttribute": "AVC"
  }
]

Note:

If an upstream cleanse function leaves the input value unchanged and proceedOnFailure is set to false, the next cleanse function does not run. This scenario occurs when a string cleanse function is configured to remove special characters but the input is already plain text. In that case, the function does not write a new value or a crosswalk. To keep the chain running, set params.isForce: true on the upstream cleanse function so it always generates a value.
If you use filter in a chain, place the most restrictive filter first.

Sample Address

The following example shows the StringFunctionCleanser and AddressCleanser running in sequence, with the string cleanser configured before the address cleanser.

{
 "Address1": "12021 Wilmington (Ave) 1st$ Fl",
 "City": "Los Angeles",
 "State": "CA",
 "Country": "United States",
 "Pincode": "90059"
}

First, the StringFunctionCleanser removes special characters. Then, the AddressCleanser standardizes and verifies the address against geolocation data.

Address Output after Applying the String Cleanser

Applying the StringFunctionCleanser to the sample address produces this output:

{
 "Address1": "12021 Wilmington Ave 1st Fl",
 "City": "Los Angeles",
 "State": "CA",
 "Country": "United States",
 "Pincode": "90059"
}

Address Output after Applying the Address Cleanser

Applying the AddressCleanser to the address, after the string cleanse, produces this output:

{
 "PremiseNumber": "12021",
 "DeliveryAddress": "12021 Wilmington Ave Fl 1",
 "Address": "12021 Wilmington Ave Fl 1<BR>Los Angeles CA 90059-3019",
 "DeliveryAddress1": "12021 Wilmington Ave Fl 1",
 "Locality": "Los Angeles",
 "AdministrativeArea": "CA",
 "CountryName": "United States",
 "PostalCode": "90059-3019",
 "PostalCodePrimary": "90059",
 "PostalCodeSecondary": "3019",
 "Latitude": "33.923550",
 "SubBuilding": "Fl 1",
 "Premise": "12021",
 "Address1": "12021 Wilmington Ave Fl 1",
 "Address2": "Los Angeles CA 90059-3019",
 "ISO3166-2": "US",
 "ISO3166-3": "USA",
 "GeoAccuracy": "P4",
 "GeoDistance": "0.0",
 "ISO3166-N": "840",
 "Thoroughfare": "Wilmington Ave",
 "Longitude": "-118.239390",
 "AVC": "V44-I55-P7-100",
 "PremiseNumberStatus": "fsVerifiedNoChange",
 "LocalityStatus": "fsVerifiedNoChange",
 "SubAdministrativeArea": "Los Angeles",
 "AdministrativeAreaStatus": "fsAdded",
 "ThoroughfareStatus": "fsVerifiedNoChange",
 "MatchRuleLabel": "1a"
}

Recommendation

To achieve the desired result, we recommend you use different resultingValuesSourceTypeURI (crosswalk) for StringFunctionCleanser and AddressCleanser in cleanseConfig. For example:

{
  "cleanseConfig": {
    "mappings": [
      {
        "uri": "configuration/entityTypes/Location/cleanse/mappings/address",
        "outputMapping": [
          {
            "attribute": "configuration/entityTypes/Location/attributes/VerificationStatus",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "VerificationStatus"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/VerificationStatusDetails",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "VerificationStatusDetails"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/StateProvince",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "AdministrativeArea"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/City",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "Locality"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/Country",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "ISO3166-2"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/Zip/attributes/Zip5",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "PostalCodePrimary"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/Zip/attributes/Zip4",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "PostalCodeSecondary"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/GeoLocation/attributes/Latitude",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "Latitude"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/GeoLocation/attributes/Longitude",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "Longitude"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/GeoLocation/attributes/GeoAccuracy",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "GeoAccuracy"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/Organization",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "Organization"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/DeliveryAddress1",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "DeliveryAddress1"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/AddressLine1",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "DeliveryAddress1"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/AddressLine2",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "DeliveryAddress2"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/ISO3166-2",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "ISO3166-2"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/ISO3166-3",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "ISO3166-3"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/DeliveryAddress",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "DeliveryAddress"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/PremiseNumber",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "PremiseNumber"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/ISO3166-N",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "ISO3166-N"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/Unmatched",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "Unmatched"
          },
          {
            "attribute": "configuration/entityTypes/Location/attributes/AVC",
            "mandatory": false,
            "allValues": false,
            "cleanseAttribute": "AVC"
          }
        ]
      }
    ],
    "infos": [
      {
        "uri": "configuration/entityTypes/Location/cleanse/infos/other",
        "useInCleansing": true,
        "sequence": [
          {
            "chain": [
              {
                "cleanseFunction": "Loqate",
                "resultingValuesSourceTypeUri": "configuration/sources/ReltioCleanser",
                "proceedOnSuccess": true,
                "proceedOnFailure": false,
                "mapping": {
                  "inputMapping": [
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine1",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "Address1"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine2",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "Address2"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/Country",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "Country"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/StateProvince",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "AdministrativeArea"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/City",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "Locality"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/Zip/attributes/Zip5",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "PostalCode"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/Organization",
                      "mandatory": false,
                      "allValues": false,
                      "cleanseAttribute": "Organization"
                    }
                  ],
                  "outputMappingRef": "configuration/entityTypes/Location/cleanse/mappings/address/outputMapping"
                },
                "params": {
                  "verificationStatusMapping": {
                    "Verified": [
                      "V(4|5).*"
                    ],
                    "Partially Verified": [
                      "V(1|2|3).*",
                      "P.*"
                    ],
                    "Unverified": [
                      "U.*"
                    ],
                    "Ambiguous": [
                      "A.*"
                    ],
                    "Conflict": [
                      "C.*"
                    ],
                    "Reverted": [
                      "R.*"
                    ]
                  }
                }
              },
              {
                "cleanseFunction": "StringFunctionCleanser",
                "resultingValuesSourceTypeUri": "configuration/sources/StringCleanser",
                "proceedOnSuccess": true,
                "proceedOnFailure": true,
                "mapping": {
                  "inputMapping": [
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine1",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "AddressLine1"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine2",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "AddressLine2"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/City",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "City"
                    }
                  ],
                  "outputMapping": [
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine1",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "AddressLine1"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/AddressLine2",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "AddressLine2"
                    },
                    {
                      "attribute": "configuration/entityTypes/Location/attributes/City",
                      "mandatory": true,
                      "allValues": false,
                      "cleanseAttribute": "City"
                    }
                  ]
                },
                "params": {
                  "casing": "Title"
                }
              }
            ]
          }
        ]
      }
    ]
  }
}

String and address cleanser parameters

Here's a table that explains how the configuration in cleanseConfig works:

Table 1. String Function Cleanser and Address Cleanser parameters
Cleanse Configuration Parameter	Description
`"resultingValuesSourceTypeUri": "configuration/sources/StringCleanser"`	The `StringCleanser` crosswalk is used for string cleansing. The output obtained is the address cleansed for the special characters by the `StringFunctionCleanser`.
`"resultingValuesSourceTypeUri": "configuration/sources/ReltioCleanser"`	The `ReltioCleanser` crosswalk uses the cleansed value from the `StringCleanser` crosswalk as the source. The `AddressCleanser` further cleanses the entire address for verification and Geo-location.

Note: To apply the AddressCleanser before applying the StringFunctionCleanser, place the AddressCleanser first in the config sequence, followed by the StringFunctionCleanser. We recommend configuring the first cleanser in the chain with proceedOnSuccess set to true and proceedOnFailure set to false. This ensures the process continues only if the first cleanser succeeds, preventing errors from affecting subsequent cleansing steps.

Unify and manage your data