Define Data Validation Functions

Learn how to define Data Validation Functions.

The Data Validation Function (DVF) ensures data accuracy, improves data quality, and validates the data ingested in the Reltio platform. Use DVFs to validate data ingested from external sources or manual entry against a set of validation functions and then report the correctness of the data.

Create Data Validation Functions (DVFs) for entity and relationship type attributes. Each DVF can contain these attributes: name, expression, action, and message. DVFs can be defined only on single or nested attributes .

Note: The system takes up to five minutes to reflect DVF changes after creation or update.

Use either of these methods to define Data Validation Functions:

Defining Validation Functions in Reltio UI
Defining Validation Functions using the API

Best practices for defining DVFs

Use a short and informative Data Validation Function name that doesn't contain the name of the attribute. Informative names will give you better results in the Data Quality charts.
The same Data Validation Function name can be applied to multiple attributes across different entity types. For example, in the following chart, the names of DVF functions are the same:
To drill down records relevant to the chart, select a metric on the chart to navigate to the Advanced search page, where you can manage profiles with data validation warnings. For more information, see Exploring the Advanced Search and Profiles with invalid data.
In the Hub profile view, you can see validation function warnings in Attribute Validations.
Use Java 8 compliant Regular expressions (Regex) in the validation functions.
Escape backslashes in expressions with an extra backslash. For example, regexp(attributes.FirstName.value, '^\s+.*$') must be regexp(attributes.FirstName.value, '^\\\\s+.*$').

Important: DVF placement when Cleanse is involved

When a DVF expression references fields that are populated by Cleanse, you need to be aware of its lifecycle placement. In the Reltio UI flow, validation executes in the validate hook that runs before the beforeSave hook, so Cleanse has not run yet. As a result, Cleanse-generated fields may not be available during validation.

Fields such as VerificationStatus, DPVConfirmedIndicator, DPV fields, Geocode fields, and other standardized address outputs may be empty during validation.
If a DVF checks these fields in the validate hook, it can incorrectly raise ERROR or WARNING results.
If a DVF depends on Cleanse-generated fields, you need to configure it to run only in the beforeSave hook.
Use the validate hook only for rules that evaluate raw user input and do not depend on Cleanse output.

Table 1. Common DVF use cases
Use case	DVF name	DVF Expression	Validation function summary message
Finding entities with attributes missing needed values	The needed value is missing	`missing(attributes.Date.value)`	The attribute type needs at least one value.
Finding entities with attributes having no values	Exist but Empty	`regexp(attributes.FirstName.value, '^\s*$')`	Attribute value must not be empty.
Finding entities with attributes having space or any non-readable character like \n \t \r as values.	Has Non-Readable Characters	`regexp(attributes.FirstName.value, '(?s)^.(?:\s\|\t).$')`	Attribute value can’t contain space and non-readable characters.
Finding entities with attributes having space as the prefix of the value. For example, `“ text“ ()`.	Space as prefix	`regexp(attributes.FirstName.value, '^\s+.*$')`	Attribute value can’t start with space.
Finding attributes exceeding the 21 character value length.	Character length limit exceeded	`not(regexp(attributes.FirstName.value,'^.{0,21}$'))`	Attribute value has exceeded the 21 characters limit.
Finding entities with values containing special characters	Only English letters and numbers are enabled	`not(regexp(attributes.FirstName.value, '^[a-zA-Z0-9]+$'))`	Only English letters and numbers can be used in the value field. All other languages and characters aren’t enabled.

Defining Validation Functions in Reltio UI

Manage validation functions for your tenant in the Console Data Validation Functions Editor, which provides a function builder to transform business requirements into data validation functions. For more information about defining validation functions using the Reltio UI, see topics Create DVFs in the Data Modeler and Create a revalidate job.

Defining Validation Functions using the API

This table describes validation function attributes you specify in API calls. For more information on using the API to create, update, retrieve, and delete validation functions, see the Data Validation Function API on the Developer Portal.

Table 2. List of Attributes in Validation Function
Attributes	Required/Optional	Description
uri	Optional	This is a unique identifier that is automatically generated by the system. If a particular URI is provided then that validation function is updated.
name	Required	This is the name of the validation function.
label	Optional	The label for the validation funcIf the label is not provided, then the value provided for the `name` attribute is considered for the `label` attribute.
description	Optional	This indicates the brief description of the validation function. If the description is not provided, then the value provided for the label attribute is considered for the description attribute.
expression	Required	This indicates the actual expression of the validation function containing the operation or condition name, and the validation input parameters. It can contain one or more operations or conditions with one or more logical operators (`AND`/`OR`). For more information, see Understanding the Expression Attribute.
action	Optional	This indicates the next course of action when the data validation function is violated, either ERROR or WARNING (the default). Note: If parent entity, child entity, or relationship attributes encounter an error, the entire profile won't load.
validationEvent	Optional	This indicates the value of the Validation event field in the Create new validation function page: ALL (the default): if you select Create & update profile. CREATE: If you select Create profile. UPDATE: If you select Update profile. For more information, see topic Create DVFs in the Data Modeler.
message	Required	This indicates the message that is displayed when a validation warning is reported.
attribute	Required	In case of multiple attribute expression, this indicates the attribute to which the warning message must be added.
createdBy	System generated response field	This indicates the user who created the validation function and is automatically picked up while creating the function.
createdTime	System generated response field	The time at which the validation function was created.
updatedBy	System generated response field	This indicates the user who updated the validation function and is automatically picked up while creating the function.
updateTime	System generated response field	The time at which the validation function was updated.
status	System generated response field	The status of the validation function: ACTIVE (default): The validation function is active when no status is specified INACTIVE: Use this status to deactivate the validation function.
applyOn	Optional	This parameter determines the level at which the DVF is applied: `OV` (default): Applies validation to the Operational value. `CW`: Applies validation to the specified crosswalk. `Both`: Applies validation to both OV and Crosswalk.
sourceTypes:	Required only if applyOn is set to `CW` or `Both`	This parameter specifies the list of source systems the validation function should apply to when applyOn is set to `CW` or `Both`.

Sample Validation Function

[
  {
    "name": "HCP.FirstName.01",
    "expression": "regexp(attributes.FirstName.value, '[a-zA-Z0-9]+')",
    "message": "FirstName cannot have special characters.",
    "applyOn": "CW",
    "sourceTypes": "DnB",
    "attribute": "configuration/entityTypes/HCP/attributes/FirstName"
  },
  {
    "name": "HCP.CreditScore.01",
    "expression": "(equals(attributes.Status.value, 'Active') AND gte(attributes.CreditScore.value, '150') ) OR  Not equals(attributes.Status.value, 'Active')",
    "message": "HCP credit score should be greater than 150 for active profiles.",
    "attribute": "configuration/entityTypes/HCP/attributes/CreditScore"
  }
]

DVF responses

When a record is successfully saved, the value of the successful field is set to true in the response.

{
"index": 0,
"successful": true, //successful=true means the profile is saved successfully.
"object": {
"uri": "entities/09Owxbk",
"type": "configuration/entityTypes/HCP",
"createdBy": "thirupathi.reddy@reltio.com",
"createdTime": 1574831965997,
"updatedBy": "thirupathi.reddy@reltio.com",
"updatedTime": 1574831965997,
"attributes": {
"FirstName": [
{
"type": "configuration/entityTypes/HCP/attributes/FirstName",
"ov": true,
"value": "AAA2",
"uri": "entities/09Owxbk/attributes/FirstName/1BkE09Cq"
}
],
. . .
},
"isFavorite": false,
"crosswalks": [
{
"uri": "entities/09Owxbk/crosswalks/1BkE0ppQ",
"type": "configuration/sources/Reltio",
"value": "09Owxbk",
. . .
}
],
"analyticsAttributes": {},
"label": "AAA2 BBB2",
"secondaryLabel": ""
}
}

When a record is rejected, the value of the successful field is set to false in the response, which contains one or more validation errors or warnings.

{
"index": 0,
"successful": false, //successful=false means the profile is not saved due to some errors.
"errors": {
"severity": "Error",
"errorMessage": "The profile has failed against one or more DVF Functions.",
"errorCode": 31010,
"errorDetailMessage": "The profile has failed with 1 DVF error(s) and 6 DVF warning(s). Please review the
below validation errors and correct the data accordingly.",
"foundErrors": [
{
"severity": "ERROR", 
"errorMessage": "Identifier is required one", 
"errorCode": 31010,
"errorDetailMessage": "Identifier missing description", //3. Description from validation function
"exception": "Identifier missing" //4. Validation function name
},
{
"severity": "WARNING",
"errorMessage": "desc missing",
"errorCode": 31010,
"errorDetailMessage": "",
"exception": "desc"
}
],
"innerErrorData": {..}
},
}

The value in the following fields depend on values in the in the Edit validation function page:

severity: Depends on the value in the Validation level field.
errorMessage: Depends on the value in the Display error with message field.
errorDetailMessage: Depends on the value in the Description field.
exception: Depends on the value in the Name field.

Unify and manage your data