Identifying Operational Values (OV) for Attributes

When an entity or relationship is the result of previous merges, it contains the aggregation of attributes and attribute values from the contributing objects. As a result, any attribute, whether it be a simple, nested, or reference, may contain multiple values. For example, after merging with two other entities, the first name attribute of an entity could contain three values: 'Mike', 'Mikey', and 'Michael'.

Through Advanced Search, you can use the has all option to search for Source System Names for which to add values to attributes for the crosswalk. From the values you specify, the system will choose the best value from these recent values. Furthermore, although multiple values are shown, you have the option to select the configuration to use to not calculate survivorship based on all of the system sources but to calculate survivorship only on certain sources. All rules work on the entire set of crosswalks that exist for the record.

If you don't want all of the survivorships to be calculated based on all of the records or all of the crosswalks that exist on any records, then you need to set Survivorship Rules from the Sources View of any entity. For more information, see Configuring Survivorship in the Hub.

While it is important to store all the contributing values in the attribute for audit purposes, ultimately, the 'best value' or set of values for the attribute must be determined so that they can be returned to Hub users and calling applications in a request. These 'best values' are called the Operational Values, or winner values, and referred to as the OV of the attribute.

In the Hub, the OV is primarily shown next to the attribute label. The Hub provides an indicator if additional, yet non-OV values exist. The indicator is a blue oval with a + and a number in it. The number indicates how many additional unique values are held within the attribute. Clicking on the oval will navigate the user to the Sources view, where all source crosswalks and all contributed values can be seen for each attribute.

Each attribute can have 0, 1, or multiple values that have been marked as OV. The OV flag is a boolean property used by the Hub to determine which attribute values must be shown to the user. The OV flag of each attribute value is calculated just-in-time whenever the entity's values are requested by either the Hub or a calling application.

Survivorship strategy is configurable for each entity type. Survivorship strategy can be changed on the fly and will take effect immediately. This ensures that you have the agility to change the rules for calculating the OV flags at any time, and the new definition will affect the very next payload returned from the database. Survivorship rules can be configured via the Hub or via the Configuration API.

Survivorship rules can be set for simple, nested, sub-nested, and referenced attributes. However, survivorship rules cannot be set for sub attributes of referenced attributes because survivorship rules for sub attributes are taken from the referenced entity/relation and can't be overridden on the sub attribute level. For example, if an address attribute has sub attributes such as AddressLine1, AddressLine2, and City, the survivorship rules for these sub attributes will be determined by the survivorship rules that are set for the Location entity. However, sub attributes can be used as a link in additional fields of strategy (primaryAttributeUri, comparisonAttributeUri).

Supported survivorship strategies are described in the following table:

Survivorship rules are defined for object attributes by survivorship mappings. All rules are defined separately for each attribute and are calculated independently of each other (the only exception is the OtherAttributeWinnerCrosswalk strategy which depends on a primary attribute). A survivorship group consolidates a set of survivorship mapping.

An entity type can have multiple survivorship groups, and one of them must be marked as default. A default group is used in most cases; otherwise, non-default groups are applied for users with specific roles only. Roles are also defined in a group setting. However, if two users are assigned different roles with each role having a different set of survivorship rules, then the OV values displayed to these users may be different, depending on the rules applied. Survivorship groups can be associated with user roles by using the Rulesets.

If some attribute isn’t specified in NOT default mapping, the mapping for this attribute is taken from the default group.

All survivorship rules are defined in a business configuration for object types.

It's impossible to specify survivorship rules for referenced attributes. The OV values of referenced attributes are calculated by rules of a corresponding relation or a referenced entity which also has its own attributes survivorship mappings.

Survivorship Mapping Parameters

Survivorship mapping parameters are provided in the following table.

Some customers store values of integer, double, boolean and other types in fields that have type String in business configuration. Default implementation of Min/Max OV strategies in such cases provides only comparison of these values with lexicographical order. To provide the ability to compare these values for OV calculation with cast to target types the existing behavior of Min/Max calculation strategy was extended by additional parameter sortAs.

Configuration

Example of new Survivorship Group Configuration:

{
  "attribute": "configuration/entityTypes/HCP/attributes/ID",
  "sortAs": "Int",
  "survivorshipStrategy": "MinValue"
}

Examples

Let's consider an attribute ID that has type as String and two values with survivorshipStrategy and MaxValue.

Metadata validation

For more information, see the table with validation error codes at https://docs.reltio.com/infoandconfig/validationerrors.html.

Common Rules

Survivorship calculation rules include:

• Survivorship strategy: Applies specific rules to determine winner values.
• Filtering settings: Splits all attribute values by defined criteria. Each split set of values has its own winner values, and all calculations are performed independently between split sets.
• Fallback strategy settings: Indicates what strategy must be used to determine the winner if a main survivorship strategy couldn’t determine them or found a lot of them.
• Restrictions of sources for OV: Set by the sourcesForOv field. If these restrictions exist, only crosswalks (and values linked to them) from permitted source-systems are used to determine winners, all other crosswalks and values are ignored. So if there are no values from permitted sources, there are no winner values.
• Pinned value: Survivorship rules are not applied to attribute values if one of those values is pinned. All pinned values become OV’s and all attribute’s survivorship rules are just ignored.
  Note: If sub-nested attribute is pinned but its parent has OV set as false this sub-nested attribute also will have the OV set as false.
• Ignored value: Ignored values do not participate in OV calculation.
• Crosswalks with end date (delete date): Do not participate in OV calculation just as those crosswalks do not exist.
• Default strategy: If an attribute does not have its own survivorship mapping, the default survivorship Last Update Date (LUD) is applied.

If all crosswalks of value were filtered out because they are end-dated or they aren’t from sourcesForOv, such value is ignored and does not participate in OV calculation.

Filtering

Filtering settings split all values from an attribute to different sets of values using a filter. These sets of values can intersect with each other if the value meets the filter condition for different filters. For each set, the winner values are calculated separately. Each filter condition is specified in separate attribute mappings. If a value is contained in multiple subsets, it can be a winner value in one of the subsets.

If a value does not match any filter condition in the attribute mappings, it goes to the set which will be calculated by default strategy and also separately from other sets. Separate mapping without filtering property must be specified for this strategy; otherwise, LUD is applied.

When filtering values, use these conditional operators:

• equals - matches values that are equal to the specified value.
• ne - not equals; matches values that differ from the specified value.
• lt - less than; matches values that are lower than the specified value.
• lte - less than or equal to; matches values that are lower than or equal to the specified value.
• gt - greater than; matches values that are higher than the specified value.
• gte - greater than or equal to; matches values that are higher than or equal to the specified value.
• in - matches values that are within a specified list of values.
• notin - matches values that are outside a specified list of values.
• missing - matches cases where the specified URI attribute does not exist.
• exists - matches cases where the specified URI attribute is present.

Use these in conjunction with the and and or logical operators to filter values.

You can build complex filters using these operators. Additionally, use the following argument with the above mentioned operators:

• NOW - denotes the current timestamp when filtering date and time attributes and can be used with all operators except missing and exists.

The response, when using these operators, is always a boolean value. The survivorship rule to be executed depends on this boolean value. In other words, if the answer is true, a particular survivorship rule is executed.

For the specified value, all values of the attribute where the operational values are set to true or false are searched.

Here's an example:

"filter": {
  "or": [
    {
      "uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Type",
      "operator": "equals",
      "value": "Mobile"
    },
    {
	"uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Type",
	"operator": "equals",
	"value": "Work"
    },
    {
	"or": [
	  {
	    "uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Number",
	    "operator": "gt",
	    "value": "112233"
	  },
	  { 
	    "and": [
	    {
	      "uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Rank",
	      "operator": "lte",
	      "value": "1"
	    },
	    {
		"uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/CountryCode",
		"operator": "ne",
		"value": "US"
		}
	    ]
	  }
	]
     }
  ]
})

The above code translates to this expression:

((Type = "Mobile") OR (Type = "Work") OR ((Number is greater than 112233) OR ((Rank is less than or equal to 1) AND (CountryCode is not US))))

In this example, the rule applies if any of the following conditions are true:

• Type is "Mobile"
• Type is "Work"

• Either:

  • Number is greater than 112233, or

  • Both:

    • Rank is less than or equal to 1
    • CountryCode is not "US"

Here's an example with the in operator:

{
  "filter": {
    "in": [{
      "uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Type","value": "Mobile"
      },{
      "uri": "configuration/entityTypes/Individual/attributes/Phone/attributes/Type","value": "Work"
    }]
  }
}

The above code translates to this expression:

Type IN ("Mobile", "Work")

In this example, the rule applies if any of the following conditions are true:

• Type is "Mobile"
• Type is "Work"

And here's another example:

{
  "attribute": "configuration/entityTypes/Shop/attributes/Addresses",
  "comparisonAttributeUri": "configuration/entityTypes/Shop/attributes/Addresses/attributes/StartDate",
  "survivorshipStrategy": "MaxValue",
  "filter": {
    "or": [
      {
        "uri": "configuration/entityTypes/Shop/attributes/Addresses/attributes/EndDate",
        "operator": "missing"
      },
      {
        "uri": "configuration/entityTypes/Shop/attributes/Addresses/attributes/EndDate",
        "operator": "gt",
        "value": "NOW"
      },
      {
        "uri": "configuration/entityTypes/Shop/attributes/Addresses/attributes/Country",
        "operator": "exists"
      }
    ]
  }
}

and this translates to:

(EndDate is missing) OR (EndDate > NOW) OR (Country (exists)

In this example, the rule applies if any of the following conditions are true:

• Addresses.EndDate is missing.

• Addresses.EndDate is greater than the current date (NOW).

• Addresses.Country exists.

A URI for matching any attribute can be used, for example:

• A value from a neighbor attribute on the same level (FirstName -> LastName, Identifiers/ID -> Identifiers/Type).
• Not a neighbor value from anywhere (Identifiers/ID -> Address/AddressLine1) This makes sense only for different entities: OV for Identifiers/ID from one entity with one address will be calculated with one rule, but for another entity with other address it will be calculated with another rule.

If multiple values exist on URI, one required value is enough for the filtering match.

Fallback Strategies

Fallback strategy options are determined by the fallbackUsingCriteria and fallbackStrategies properties on a survivorship mapping.

There are two fallbacks using the criteria:

1. MORE_THAN_ONE (default): If a strategy returns two or more winner values, its fallback strategy will use only the winner values and winner crosswalks from the strategy.
2. ZERO_OR_MORE_THAN_ONE: If a strategy returns zero, two, or more winner values, the logic for determining values used by the fallback strategy is based on the retrieved number of winners.

   If there are zero winners, then all values from the strategy are considered by its fallback strategy. If there are two or more winners, then only winner values and winner crosswalks from the strategy are considered by its fallback strategy.

Example JSON for Fallback Strategies

{
   "sourcesUriOrder":[
      "configuration/sources/TWITTER",
      "configuration/sources/FB"
   ],
   "mapping":[
      {
         "attribute":"configuration/entityTypes/Individual/attributes/FirstName",
         "survivorshipStrategy":"SRC_SYS",
         //step 1: Check the main strategy. According to the ZERO_OR_MORE_THAN_ONE fallback criteria, 
         //If there are zero or more than one winners, then all the values (if zero values won) or only the winner values (if there is more than one winner) are passed to the fallback section.
         "fallbackUsingCriteria":"ZERO_OR_MORE_THAN_ONE",
         "fallbackStrategies":[
            {
               "attribute":"configuration/entityTypes/Individual/attributes/FirstName",
               "survivorshipStrategy":"LUD",
               //step 2: The strategy used depends on the number of winners from the previous step. If the LUD strategy returns more than one winner, the next fallback strategy will take part in the Operational Value (OV) calculation.
                    "fallbackUsingCriteria": "MORE_THAN_ONE",
                    "fallbackStrategies": [
                        {
                            "attribute": "configuration/entityTypes/Individual/attributes/FirstName",
                            "survivorshipStrategy": "MinValue" //this step is required only if the LUD strategy returns more than one winner value (in accordance with the MORE_THAN_ONE fallback criteria).
                            ...
                        }
                    ]
                }
            ]
        }
    ]
}

Example

Consider the following mapping:

• Main strategy: OtherAttributeWinnerCrosswalk.
• Fallback strategies chain:
  • Record-Values Based Source Priority with winner source type A.
  • Record-Values Based Source Priority with winner source type B.
  • MaxValue.

Then, you can start calculating winners step by step.

1. Apply Other Attribute Winner Crosswalk. It returns five winners.
   • If the fallback using criteria is MORE_THAN_ONE, take five winners and filter their crosswalks to use only winner crosswalks. Then, use them in the next step.
   • The same thing for ZERO_OR_MORE_THAN_ONE criteria.
2. Apply Record-Values Based Source Priority (source A). It returns 0 (zero) winners.
   • If the fallback using criteria is MORE_THAN_ONE, stop the calculation. The result will be 0 (zero) winners.
   • If the fallback using criteria is ZERO_OR_MORE_THAN_ONE, take the previous five winners with their winner crosswalks, and use them in the next step.
3. For ZERO_OR_MORE_THAN_ONE only, apply the second Record-Values Based Source Priority. Consider the different results:
   • If it returns 0 (zero) winners, take and use the 5 previous winners again in the next step.
   • If it returns one winner, mark it as the final winner and stop the calculation.
   • If it returns two or more winners, take them (only them) and use in the next step.
4. For ZERO_OR_MORE_THAN_ONEonly, apply Max Value. As this is the last strategy in the chain, no matter what the result is, it will be used as the final result (with 0 (zero) or one or more winners) anyway.

Strategy Examples

Example of custom survivorship strategy definition in tenant's configuration

"survivorshipStrategies": [
    {
      "uri": "configuration/survivorshipStrategies/TwitterSRCSysStrategy",
      "parentStrategyURI": "configuration/survivorshipStrategies/SRC_SYS",
      "label": "Twitter Custom  SRC_SYS Strategy",
      "mainSourceType": "configuration/sources/TWITTER"
    }
  ]

Example of record-values based source strategy definition in tenant's configuration

"survivorshipStrategies": [
{
    "uri": "configuration/survivorshipStrategies/CleanserWinsStrategy",
    "label": "Record values based source priority OV rule (based on ReltioCleanser)",
    "winnerSourceType": "configuration/sources/ReltioCleanser",
    "winnerSourceAttributes": [
        "configuration/entityTypes/Location/attributes/AddressLine1"
]
}
      ]

Example of attribute mapping with OtherAttributeWinnerCrosswalk strategy

{
      "attribute": "configuration/entityTypes/HCP/attributes/LastName",
      "survivorshipStrategy": "OtherAttributeWinnerCrosswalk",
      "primaryAttributeUri": "configuration/entityTypes/HCP/attributes/FirstName"
}

Survivorship Groups (Changing Default Survivorship Strategy)

The section survivorshipGroups could be added to every type in the configuration. Each survivorship group has a mapping of attribute URIs to survivorship strategies. One group for a type could be set as default from the API. That would mean that if the survivorship strategy of a particular attribute type is defined in the survivorship group, which is default for the type and attribute, a value would be selected in accordance with the survivorship group. For example, a type could have a survivorship group, as follows:

{
   "default" : true,
   "mapping" : [
   {
     "attribute" : "configuration/entityTypes/Reltio.System.Individual/attributes/MiddleName",
     "survivorshipStrategy" : "LUD"
   }
   ]
}

Add a new Survivorship Group

The request creates a survivorship group in configuration type.

POST {TenantURL}/configuration/entityTypes/{entityTypeName}/survivorshipGroups
{
   "default" : true,
   "mapping" : [
   {
     "attribute" : "configuration/entityTypes/Reltio.System.Individual/attributes/MiddleName",
     "survivorshipStrategy" : "LUD"
   }
   ]
}

In the above example, the new survivorship group would be created that overrides survivorship strategy of the MiddleName attribute to LUD, and the group would be considered as the default.

POST {TenantURL}/configuration/entityTypes/{entityTypeName}/survivorshipGroups
{
  "sourcesUriOrder": [
    "configuration/sources/TWITTER",
    "configuration/sources/FB"
  ],
  "mapping": [
    {
      "attribute": "configuration/entityTypes/Individual/attributes/FirstName",
      "survivorshipStrategy": "SRC_SYS",
      "fallbackStrategies": [
        {
          "attribute": "configuration/entityTypes/Individual/attributes/FirstName",
          "survivorshipStrategy": "MinValue"
        },
        {
          "attribute": "configuration/entityTypes/Individual/attributes/FirstName",
          "survivorshipStrategy": "LUD"
        }
      ]
    }
  ],
  "default": "true"
}

In this example, the new survivorship group would be created that overrides survivorship strategy of the FirstName attribute to SRC_SYS and sourceUriOrder for the group. The Fallback strategies section means that in case of multiple OV values for this attribute, the system will try to re-calculate OV (among values already marked as OV) in order to get the only one.

Update the Survivorship Group

This request updates the present survivorship group in configuration type.

PUT {TenantURL}/configuration/entityTypes/{entityTypeName}/survivorshipGroups/{groupShortUri}
{
   "default" : false,  
   "mapping" : [
   {
     "attribute" : "configuration/entityTypes/Reltio.System.Individual/attributes/MiddleName",
     "survivorshipStrategy" : "LUD"
   }
   ]
}

In the above example, survivorship group uri {groupShortUri} would be updated, which overrides survivorship strategy of the MiddleName attribute to LUD; therefore, group would not be considered as default.

PUT {TenantURL}/configuration/entityTypes/{entityTypeName}/survivorshipGroups/{groupShortUri}
{
    "sourcesUriOrder": [
        "configuration/sources/TWITTER",
        "configuration/sources/FB"
    ],
    "mapping": [
        {
            "attribute": "configuration/entityTypes/Individual/attributes/FirstName",
            "survivorshipStrategy": "SRC_SYS",
            "sourcesUriOrder": [
                "configuration/sources/FB",
                "configuration/sources/TWITTER"
            ]
        }
    ],
    "default": "true"
}

In this example, survivorship group uri {groupShortUri} would be updated, which overrides survivorship strategy of the FirstName attribute to SRC_SYS and sets sourceUriOrder for the group.

This request allows you to change both the information about a survivorship group and a strategy for the specified attribute. Unlike the group information that is replaced with the attributes from the request, there is no need to specify all the strategies from the mapping section. Only mappings that are set in the request will be updated in the metadata configuration.

When the strategy in the request requires some extra attributes (for example, OtherAttributeWinnerCrosswalk requires primaryAttributeUri), they are initially searched in the metadata configuration. If these attributes are found, their values are taken from the configuration. If there are no such values, validation error occurs.

The SRC_SYS strategy uses the sourceUriOrder attribute, that can be located on a group/strategy/group and strategy levels, so there are some features referring to updating this attribute. If sourceUriOrder is set on the strategy level in L3, but is absent on the same level in the request, it is taken from L3. If sourceUriOrder is set on the group level in L3, but is absent on the same level in the request, it is removed from the configuration.

Filters in Survivorship Group Mapping

With the Complex OV rule type, you can define survivorship strategy for a nested attribute based conditionally on values of a sub-attribute. This is accomplished using the optional filter property for survivorship group mapping. Therefore, a survivorship strategy, which is defined in the survivorshipStrategy property of the mapping, will be applied only for attributes which match the filter criteria. In this way, several survivorship strategies can be leveraged to treat different sub-attribute types. The resulting winners for the nested attribute are the aggregation of winners emerging from each strategy.

Examples

Example 1 The developer states that for all identifiers, where the Identifier.type = NPI, the survivorship strategy must be MinValue. However, for all identifiers where the Identifier.type = ME, the survivorship strategy must be OldestValue. Lastly, use the default LUD strategy for all other types.

{
  "default": true,
  "mapping": [
    {
      "attribute": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/ID",     
      "survivorshipStrategy": "MinValue",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/Type",
            "value": "NPI"
          }
        ]
      }
    },
    {
      "attribute": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/ID",     
      "survivorshipStrategy": "OldestValue",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/Type",
            "value": "ME"
          }
        ]
      }
    }
  ] 
}

Example 2 For Identifiers of type ME, there must be one OV with MinValue of ID. For identifiers of type NPI, select the OldestValue. For all others, use SRC_SYS.

{
  "default": true,
  "mapping": [
    {
      "attribute": "configuration/entityTypes/HCP/attributes/Identifiers",
      "survivorshipStrategy": "MinValue",
      "comparisonAttributeUri": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/ID",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/Type",
            "value": "ME"
          }
        ]
      }
    },
    {
      "attribute": "configuration/entityTypes/HCP/attributes/Identifiers",
      "survivorshipStrategy": "OldestValue",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Identifiers/attributes/Type",
            "value": "NPI"
          }
        ]
      }
    },
    {
      "attribute": "configuration/entityTypes/HCP/attributes/Identifiers",
      "survivorshipStrategy": "SRC_SYS"
    }
  ]
}

Example 3

OldestValue survivorship strategy for the FirstName attribute for all HCPs from NY or TX states and Aggregation for all others.

{ 
  "default": true,
  "mapping": [
    {
      "attribute": "configuration/entityTypes/HCP/attributes/FirstName",
      "survivorshipStrategy": "OldestValue",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Address/attributes/State",
            "value": "NY"
          }
        ]
      }
    },
    {
      "attribute": "configuration/entityTypes/HCP/attributes/FirstName",
      "survivorshipStrategy": "OldestValue",
      "filter": {
        "equals": [
          {
            "uri": "configuration/entityTypes/HCP/attributes/Address/attributes/State",
            "value": "TX"
          }
        ]
      }
    },   
    {
      "attribute": "configuration/entityTypes/HCP/attributes/FirstName",
      "survivorshipStrategy": "Aggregation"
    }
  ] 
}

Example of the OV Calculation of a Winner Entity

If Entity 21 and Entity 3 are merged with the specified winner as Entity 3, then the merged entity will have the FirstName attribute value of FirstName3. The operational value is set to true and the FirstName1 and FirstName2 operational values are set to false.