Synchronize data between Reltio and Snowflake
Learn how to synchronize your entities, relations, matches, interactions, and merges data from Reltio to Snowflake.
- Start the synchronization by running the syncToDataPipeline:
POST https://{{tenantEnvironmentName}}.reltio.com/reltio/api/{{tenantId}}/syncToDataPipeline
where:-
<tenantEnvironmentName>: is the name of the environment on which your Reltio tenant is hosted, which forms part of the fully qualified URL, for example:
-
https://dev-data-pipeline-hub.reltio.com
-
https://test-data-pipeline-hub.reltio.com
-
https://361-data-pipeline-hub.reltio.com
-
-
tenantId: is the ID of the Reltio tenant you configured the Reltio Data Pipeline for Snowflake
-
- Run these APIs to confirm the request completed successfully:
Get Active Tasks for Tenant to check the current tasks for the tenant: GET {ApplicationURL}/{tenantId}/tasks
Get Tasks History to check finished tasks for all tenants GET https://{{tenantEnvironmentName}}.reltio.com/reltio/{{tenantId}}/tasks/history
Get Task by ID for Tenant to check tasks with the given ID for the specified tenant GET https://{{tenantEnvironmentName}}.reltio.com/reltio/{{tenantId}}/tasks/{{taskId}}
where:-
taskId: is the ID of the Reltio object type processing task.
For more information, see topics Get Active Tasks for Tenant, Get Tasks History, and Get Task by ID for Tenant.
-
- Run the Tenant Queue Status to verify that Snowflake received Reltio events:
GET https://{{tenantEnvironmentName}}-data-pipeline-hub.reltio.com/status/tenant/{{tenantID}}/details
For more information, see topic Tenant Queue Status API.
- When the task is complete (count is zero), validate that the number of objects in Snowflake is the same as in Reltio.Note: When the synchronization completes, a row is added in the landing table with details of the sync. A separate row is created for each sync, with the time of the sync recorded in the
Timestamp
field. For more information, see topic Landing table datasets for Snowflake.If it is not, check the following issues:-
If there is a discrepancy in the count of events in Reltio and in Snowflake, use the Event Monitoring to check if the events are being loaded to your cloud storage:
GET https://{{tenantEnvironmentName}}-data-pipeline-hub.reltio.com/api/tenants/{{tenantID}}/monitoring/_eventMonitoring
For more information, see topic Event Monitoring API.
-
If events are being passed to your cloud storage, run the COPY_HISTORYcommand to view a copy history table that identifies whether an attempt was made to partially load your Snowflake landing table or if the load failed. For more information, see Loading history in the Snowflake function reference.
In the displayed data loading history within the selected time period, view these columns to check whether files were loaded or not and the reason if it didn't load correctly:-
STATUS : Whether or not a particular set of files was loaded, partially loaded, or failed to load.
-
FIRST_ERROR_MESSAGE The reason a load attempt was only partially completed or if didn't happen at all.
If a set of loaded files has multiple issues, this FIRST_ERROR_MESSAGE column only indicates the first error encountered. Run the COPY INTO <table> command with the VALIDATION_MODE copy option set to RETURN_ALL_ERRORS and reference the set of files you'd attempted to load using Snowpipe to view all errors in the files.SELECT * FROM TABLE(information_schema.copy_history( table_name=>'<LANDING_TABLE_NAME>', start_time=>dateAdd(days, -2, current_timestamp())));
Note: The VALIDATION_MODE copy option instructs a COPY statement to validate the data to be loaded and return results based on the validation option specified. No data is loaded when this copy option is specified.For more information, see COPY INTO <table> in the Snowflake function reference.
-
-
- If events are not being passed to your cloud storage, run the Event Monitoring to observe the specific transfer stage that a particular entity event is at in Reltio and check against your Snowflake data.
Take these steps depending on the value in the
eventState
field:-
DATAPIPELINE_PROCESSED
: recheck your steps in this Configure the Reltio Data Pipeline for Snowflake for Azure section. -
INCOMPLETE
: Raise a Support ticket for further investigation.
For more information, see topics Event Monitoring API and Need some help?
-