D&B Batch Integration

The D&B batch integration processes that Reltio supports are Match and Append and Recertification.

Reltio supports the following two D&B (Dun and Bradstreet) batch integration processes:

  • Match and Append: The match and append process is used for receiving the potential matches and append the organization’s data with D&B data when the organization's DUNS (Data Universal Numbering System) number is not available. It is also used periodically to get details of new organizations that have been created in a Reltio tenant.

  • Recertification: The recertification process is used to refresh the existing data with the latest D&B data when the organization's DUNS number is available.

    In obsolete Batch processes using FTP, we have dedicated file formats and entities with DUNS numbers that are automatically routed for recertification.

    In Direct+ Batch process the connector automatically sends entities with DUNS numbers to the D&B jobs with processId=data. Entities without DUNS numbers are routed to the job with processId=extmatch for matching.

If an organization in the Reltio tenant does not have a DUNS number, then the match and append process is used for integrating the organization entity with D&B, otherwise the recertification process is used.

D&B Batch Integration Steps

Overview

The Reltio D&B Batch Connector utilizes two REST API requests, one to send data to D&B and the other to retrieve response data from D&B. The API endpoints are provided and can be utilized manually or integrated into ETL tools or scheduled jobs. Reltio recommends a minimum frequency of not less than 48hrs, as this is the SLA (Service Level Agreement) for D&B to fulfill the request.

Sending Data to D&B (PUT Endpoint)

Data is sent to D&B using the API request given below. When the request is successful, a job ID is returned as a response. This process kicks off an export task for the specified tenant, so that export job can be monitored using APIs or Reltio Console. The email address associated with the service account will also receive standard Reltio export notification emails.

Note: In order to update the distribution list for notification emails, you must contact D&B directly. The admin can update the Reltio service account for customer's email address as necessary.

After the export task has been completed, the data will be transformed and uploaded to D&B SafeTransport. Standard SafeTransport notification emails are received after successful upload. Due to some reason, if the job fails to complete successfully (D&B notification emails are never received), you must first check the export task and then file a support ticket. Make sure to include the following details: task ID, environment, and tenant ID.

Request

POST https://dev-dnbconnector.reltio.com/b2bIntegration/dnb/batch

Request Body

You can modify the filter to suit your requirements. Records that satisfy the query will be sent to D&B. The most basic filter that can be used is: “equals(type,configuration/entityTypes/Organization)”. For more details of the filters that can be used, see Filtering Entities.

{
"filter": "equals(type, 'configuration/entityTypes/Organization') and 
equals(attributes.IsActive,'true') and 
equals(attributes.BuyerLifecycleStatus.BuyerLifecycleStatus,'Customer') 
or equals(attributes.BuyerLifecycleStatus.BuyerLifecycleStatus,'Prospect') 
or (equals(attributes.BuyerLifecycleStatus.BuyerLifecycleStatus,'Lead') 
or equals(attributes.BuyerLifecycleStatus.BuyerLifecycleStatus,'Out of Business') 
or equals(attributes.BuyerLifecycleStatus.BuyerLifecycleStatus,'Internal Use Account')"
}            

Request Response

{
    "jobId": <job-id-value>,
    "message": "Scheduled"
}
Table 1. Parameters
Parameter Required Description/Value
Headers Content-Type Yes Should be Content-Type: application/json.
Reltio-Batch-Type Yes Should be Reltio-Batch-Type: put.
D&B-Ftp Yes ftp.dnb.com.
D&B-Path Yes /puts.
D&B-User Yes {{SafeTransport Username}}.
D&B-Password Yes {{SafeTransport Password}}.
EnvironmentUrl Yes Tenant Envirenment. Value: {{environmentUrl}}/reltio/api/.
TenantId Yes Tenant ID, data will be extracted from this tenant. Value: {{tenantID}}.
Reltio-User Yes Service account. Data will be exported using this user. Value: {{Reltio Service Account Username}}.
Reltio-Password Yes {{Reltio Service Account Password}}.
s3Bucket No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage.
s3Path No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage. Value: DnB_Connector/DnB_Response_Files/{{directory}}.
awsAccessKey No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage. Value: {{AccessKey}}.
awsSecretKey No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage. Value: {{SecretKey}}.

Checking Export

D&B Email Notifications

Export notifications emails will be sent to the email address associated with the service account.

  1. Receipt of the file.

  2. File acceptance.

  3. Service Request.

Retrieving Data from D&B (GET Endpoint)

After receiving the fourth and final D&B notification, service fulfillment is completed.

You can now retrieve the data using the API request given below.

Request

POST https://dev-dnbconnector.reltio.com/b2bIntegration/dnb/batch

Request Body

None

You can view when the file has been loaded into Reltio using the dashboard or activity log.

Connector updates will appear as updates from the service account. The number of records in the data file, plus the number of records displayed on the dashboard is a good indication of job progress. See the parameters below for reloading specific response files, if required.

Note: If you have any technical difficulties with the endpoint/data load, create a support ticket that includes all required information.

Request Response

{
    "jobId": <job-id-value>,
    "message": "Scheduled"
}
Table 2. Parameters
Parameter Required Description/Value
Headers Content-Type Yes Should be Content-Type: application/json.
Reltio-Batch-Type Yes Should be Reltio-Batch-Type: get.
D&B-Ftp Yes ftp.dnb.com.
D&B-Path Yes /gets.
D&B-User Yes {{SafeTransport Username}}.
D&B-Password Yes {{SafeTransport Password}}.
EnvironmentUrl Yes Tenant Envirenment.

Value: {{environmentUrl}}/reltio/api/.

TenantId Yes Tenant ID, data will be extracted from this tenant.

Value: {{tenantID}}.

Reltio-User Yes Service account. Data will be exported using this user.

Value: {{Reltio Service Account Username}}.

Reltio-Password Yes {{Reltio Service Account Password}}.
s3Bucket No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage.
s3Path No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage.

Value: DnB_Connector/DnB_Response_Files/{{directory}}.

awsAccessKey No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage.

Value: {{AccessKey}}.

awsSecretKey No Currently this is not used by PUT process, but is a required attribute. Reserved for future usage. Value: {{SecretKey}}.
mergeOrgsWithSameDUNS No Allows the same D&B crosswalk to be added to multiple entities in Reltio resulting in automatic merge by crosswalk at the platform level. This happens exclusive of match rules and cannot be changed. Remember that all documents are merged/aggregated when Reltio entities merge. So be sure this does not impact other areas of your solution.

Default value is false.

minConfidenceCode No The minimum confidence code of matches to be written back to the tenant. Any match below this confidence code will be treated as a ‘Low Confidence’ match.

Default value is 7.

s3File No When specified, you can ignore the D&B SafeTransport files. Instead, load a specific file from S3 storage. Can be use for testing or re-loading of files.

Value: {{FileName(GLB)}}

s3SkipLines No When specified, ignores a number of lines in the specified s3File. Can be used in conjunction with s3File parameter for testing or re-loading of files.

Value: {{Number of lines to skip}}

AWS Usage

  1. If you have provided AWS details such as awsAccessKey, awsSecretKey, s3Bucket, and s3Path, the connector uploads files from ftp to S3 with the same names as in ftp. All four parameters must be specified to activate S3 feature.
  2. If you have additionally provided s3File, the connector will not use ftp and will directly load file from S3.
  3. If you have specified s3SkipLines, the specified number of lines will be skipped from the head of the file.

S3 Parameters Description

In this screenshot:
  • s3Bucket is snaplogic.testbucket
  • s3Path is batch
  • s3File is o0_1KK.glb

D& Connector Monitoring

Summary

Connector writes logs to stackdriver. The GET and PUT processes always starts from the event process_started and completes with the event process_finished. Get process also provides special event lines_completed with total lines already processed.

Table 3. Events
Event Name Additional Information Comment
process_started context put and get processes
process_finished context put and get processes
lines_completed total lines count Only in get process
put_process_export total_objects, task_id put process
put_process_skip_uri skipped uris list put process
put_process_file file name, total entities in file/sink put process
get_process_file file name, size, timestamp get process, only *.glb and *.ref
get_process_skip_file file name, size(optionally), timestamp(optionally) get process skip all files with extension is not match glb or ref
get_process_entity entity uri get process

GBQ Reports

If you do not have access to stack driver or you prefer SQL language, you can use the view based on stack driver logs. Main process view is called report.pmon and contains summary information about processes. You can use it with the following SQL query:

select * from reports.pmon order by ProcessStartTs desc limit 5

Result is similar to the following:

You can view the following fields:

  • Unique Process Identifier: This can be used as a key for other views
  • User Name
  • Tenant Id
  • Process Type
  • Process Status
  • Process Start Timestamp
  • Process Finish Timestamp
  • Process Last Operation Timestamp

Process Status Calculation

  • If process_finished is present in GBQ -> Status is Finished.
  • If process_finished is not present, but (current timestamp - last operation timestamp) is less than two minutes -> Status is In Progress.
  • If process_finished is not present and the last operation timestamp was long time ago -> Status is Crashed.
Note: Crashed status: Each connector node has an internal buffer for stack driver messages with default size of 2. This means that in most cases, if there is no activity on node after process finished, that messages will not appear in the stack driver and GBQ relatively. In most cases, it is the cause of false positive Crashed status. You must be aware of this. Worse case, when the endpoint contains more than one instance (all production endpoints), all new activities can be routed to different nodes.
Note: Sorting by Timestamps:Sometimes, when you sort events by timestamp you may notice that process_finished is not the last event in the process. It happens because in one single timestamp, we usually have more than one event.

Details view reports.pdet2: View based on stack driver with very little formatting. You can use it with the following query:

select * from reports.pdet2 where ProcessIdentifier = '96eaaacb-a2ac-4110-9ec3-0a408a3a0a79'
order by timestamp desc

Use ProcessIdentifier from report.pmon view. The result is as shown below: