External queues overview

Learn about streaming events between Reltio and external systems using Tenant Management external queues.

Do you need to stream data between the Reltio platform and your external systems in near real time? If so, use the External queue in the Tenant Management application to transmit your data automatically as Reltio processes it.

Note: Alternatively, you can export data using batch jobs or APIs to obtain the mastered, cleansed, and enriched data from Reltio. For more information, see topic Events API.

Use the Reltio event streaming (or message streaming) service to process Match and CRUD (Create/Read/Update/Delete) events from internal to external queues or topics in JSON format. For more information on supported CRUD and MATCH events, see topic Message Streaming Format.

Use these key features to configure event streaming using External queues:

Multiple cloud providers:
- Amazon SQS/SNS
- Google Cloud Pub/Sub (Publisher/Subscriber)
- Microsoft Azure Service Bus
Message filters:
- Filter by Event type - Receive messages only for selected events.
- Filter Objects - Receive messages only for objects that meet the filter criteria.
Different message payload types:
- Full Snapshot (all fields)
- Snapshot with selected fields
- Delta changes - updated part of the object in a before-after format (very similar to the Activity Log)
- Snapshot combined with Delta changes
Different Payload formats:
- JSON
- zipped JSON
Destination flexibility:
- Configure multiple queues to the same destination.
- Configure the same events to multiple destinations.

Event streaming

Understanding how Reltio event streaming works will help you understand options when you Add an external queue configuration.

This diagram illustrates the Reltio event streaming process for three Reltio events and two external queue configurations.

Let's walk through this process flow:

Reltio creates message process events:
1. When you add an entity in the Hub (or through an API), Reltio generates a process event.
  In this diagram, we see that adding new entity A generates an ENTITY_CREATED event.
2. If you’ve defined a match rule to merge a new entity with an existing entity, Reltio generates two process events.
  In this diagram, we see that merging existing entity B with new entity A generates an ENTITY_CHANGED event for entity B and an ENTITY_MERGED event for entity AB.
Reltio processes these generated events through a CRUD or MATCH internal queue.
The Reltio message streaming service sends CRUD and MATCH events in JSON format to the external message queue, applying any configured filters.
In this diagram, we see that Reltio streams messages to the following external queues:
- External Queue Configuration 1: Contains a Type filter for ENTITY_CREATED events.
  Reltio streams the ENTITY_CREATED EVENT for entity A into this external queue.
- External Queue Configuration 2: Contains a Type filter for NOT_ENTITY_CREATED events (that is, events for which an entity wasn’t created) and an Object for the Name attribute.
  Reltio streams the ENTITY_CHANGED event for entity B and the ENTITY_MERGED event for merged entity AB to this event queue only if the value of the Name matches what is defined in the Object filter.

Accelerate the Value of Data

External queues overview

Event streaming

Footer