Define Data Validation Functions

Learn how to define Data Validation Functions

The Data Validation Function (DVF) ensures data accuracy, improves data quality, and questions the data ingested in the Reltio platform. Use DVFs to validate data ingested from external sources or human entry against a set of validation functions and then report the correctness of the data.

Create Data Validation Functions (DVFs) for entity and relationship type attributes. Each DVF can contain these attributes: name, expression, action, and message. DVFs can be defined only on single or nested attributes .

Note: After adding or editing a validation function, the system takes five minutes of refresh time to update the changes.

Use either of these methods to define Data Validation Functions:

Defining Validation Functions in Reltio UI
Defining Validation Functions using APIs

Best practices for defining DVFs

Use a short and informative Data Validation Function name that doesn't contain the name of the attribute. Informative names will give you better results in the Data Quality charts.
The same Data Validation Function name can be applied to multiple attributes across different entity types. For example, In the following chart, the names of DVF functions are the same:
To drill down records relevant to the chart, select a metric on the chart to navigate to the Advanced search page, where you can manage profiles with data validation warnings. For more information, see Exploring the Advanced Search and Profiles with invalid data.
In the Hub profile view, you can see validation function warnings in Attribute Validations.
Use Java 8 compliant Regular expressions (Regex) in the validation functions.
Escape backslashes in expressions with an extra backslash. For example, regexp(attributes.FirstName.value, '^\s+.*$') must be regexp(attributes.FirstName.value, '^\\\\s+.*$').

Table 1. Common DVF use cases
Use case	DVF name	DVF Expression	Validation function summary message
Finding entities with attributes missing needed values	The needed value is missing	`missing(attributes.Date.value)`	The attribute type needs at least one value.
Finding entities with attributes having no values	Exist but Empty	`regexp(attributes.FirstName.value, '^\s*$')`	Attribute value can’t be empty.
Finding entities with attributes having space or any non-readable character like \n \t \r as values.	Has Non-Readable Characters	`regexp(attributes.FirstName.value, '(?s)^.(?:\s\|\t).$')`	Attribute value can’t contain space and non-readable characters.
Finding entities with attributes having space as the prefix of the value. For example, `“ text“ ()`.	Space as prefix	`regexp(attributes.FirstName.value, '^\s+.*$')`	Attribute value can’t start with space.
Finding attributes exceeding the 21 character value length.	Character length limit exceeded	`not(regexp(attributes.FirstName.value,'^.{0,21}$'))`	Attribute value has exceeded the 21 characters limit.
Finding entities with values containing special characters	Only English letters and numbers are enabled	`not(regexp(attributes.FirstName.value, '^[a-zA-Z0-9]+$'))`	Only English letters and numbers can be used in the value field. All other languages and characters aren’t enabled.

Defining Validation Functions in Reltio UI

Manage validation functions for your tenant in the Console Data Validation Functions Editor, which provides a function builder to transform business requirements into data validation functions. For more information about defining validation functions using the Reltio UI, see topics Create DVFs in the Data Modeler and Create a revalidate job.

Defining Validation Functions using APIs

This table describes validation function attributes you specify in APIs. For more information on using the Data Validation Function APIs to create, update, retrieve, and delete validation functions, see the Developer portal.

Table 2. List of Attributes in Validation Function
Attributes	Required/Optional	Description
uri	Optional	This is a unique identifier that is automatically generated by the system. If a particular URI is provided then that validation function is updated.
name	Required	This is the name of the validation function.
label	Optional	If the label is not provided, then the value provided for the `name` attribute is considered for the `label` attribute.
description	Optional	This indicates the brief description of the validation function. If the description is not provided, then the value provided for the label attribute is considered for the description attribute.
expression	Required	This indicates the actual expression of the validation function containing the operation or condition name, and the validation input parameters. It can contain one or more operations or conditions with one or more logical operators (`AND`/`OR`). For more information, see Understanding the Expression Attribute.
action	Optional	This indicates the next course of action when the data validation function is violated, either ERROR or WARNING (the default). Note: If parent entity, child entity, or relationship attributes encounter an error, the entire profile won't load.
validationEvent	Optional	This indicates the value of the Validation event field in the Create new validation function page: ALL (the default): if you select Create & update profile. CREATE: If you select Create profile. UPDATE: If you select Update profile. For more information, see topic Create DVFs in the Data Modeler.
message	Required	This indicates the message that is displayed when a validation warning is reported.
attribute	Required	In case of multiple attribute expression, this indicates the attribute to which the warning message must be added.
createdBy	System generated response field	This indicates the user who created the validation function and is automatically picked up while creating the function.
createdTime	System generated response field	This indicates the time at which the validation function was created.
updatedBy	System generated response field	This indicates that the user updated the function and is automatically picked up while updating the function.
updateTime	System generated response field	This indicates the time at which the validation function was updated.
status	System generated response field	This indicates the status of the validation function: ACTIVE (default): if you don't specify a value, the DVF status is active. INACTIVE: Specify this to make the DVF status inactive.

Sample Validation Function

[
  {
    "name": "HCP.FirstName.01",
    "expression": "regexp(attributes.FirstName.value, '[a-zA-Z0-9]+')",
    "message": "FirstName cannot have special characters.",
    "attribute": "configuration/entityTypes/HCP/attributes/FirstName"
  },
  {
    "name": "HCP.CreditScore.01",
    "expression": "(equals(attributes.Status.value, 'Active') AND gte(attributes.CreditScore.value, '150') ) OR  Not equals(attributes.Status.value, 'Active')",
    "message": "HCP credit score should be greater than 150 for active profiles.",
    "attribute": "configuration/entityTypes/HCP/attributes/CreditScore"
  }
]

DVF responses

When a record is successfully saved, the value of the successful field is set to true in the response.

{
"index": 0,
"successful": true, //successful=true means the profile is saved successfully.
"object": {
"uri": "entities/09Owxbk",
"type": "configuration/entityTypes/HCP",
"createdBy": "thirupathi.reddy@reltio.com",
"createdTime": 1574831965997,
"updatedBy": "thirupathi.reddy@reltio.com",
"updatedTime": 1574831965997,
"attributes": {
"FirstName": [
{
"type": "configuration/entityTypes/HCP/attributes/FirstName",
"ov": true,
"value": "AAA2",
"uri": "entities/09Owxbk/attributes/FirstName/1BkE09Cq"
}
],
. . .
},
"isFavorite": false,
"crosswalks": [
{
"uri": "entities/09Owxbk/crosswalks/1BkE0ppQ",
"type": "configuration/sources/Reltio",
"value": "09Owxbk",
. . .
}
],
"analyticsAttributes": {},
"label": "AAA2 BBB2",
"secondaryLabel": ""
}
}

When a record is rejected, the value of the successful field is set to false in the response, which contains one or more validation errors or warnings.

{
"index": 0,
"successful": false, //successful=false means the profile is not saved due to some errors.
"errors": {
"severity": "Error",
"errorMessage": "The profile has failed against one or more DVF Functions.",
"errorCode": 31010,
"errorDetailMessage": "The profile has failed with 1 DVF error(s) and 6 DVF warning(s). Please review the
below validation errors and correct the data accordingly.",
"foundErrors": [
{
"severity": "ERROR", 
"errorMessage": "Identifier is required one", 
"errorCode": 31010,
"errorDetailMessage": "Identifier missing description", //3. Description from validation function
"exception": "Identifier missing" //4. Validation function name
},
{
"severity": "WARNING",
"errorMessage": "desc missing",
"errorCode": 31010,
"errorDetailMessage": "",
"exception": "desc"
}
],
"innerErrorData": {..}
},
}

The value in the following fields depend on values in the in the Edit validation function page:

severity: Depends on the value in the Validation level field.
errorMessage: Depends on the value in the Display error with message field.
errorDetailMessage: Depends on the value in the Description field.
exception: Depends on the value in the Name field.

Unify and manage your data