Add an external queue configuration

Learn how to add an external queue to stream data between Reltio and external systems in the Tenant Management application.

Do you want to send message events and payloads from Reltio to an external message queue service that can consume and propagate them to downstream systems? Add an external queue configuration in the Tenant Management console, and you'll be ready to roll.

Reltio plays nicely with the message streaming services from our supported cloud providers: Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure (Azure). For more information, see topic Cloud Providers Overview. We'll help you out further in the steps below by identifying any options that are specific to a particular cloud provider.

To add an external queue configuration in Tenant Management:

  1. Open the External queues page:
    1. In the Reltio Console, select Tenant Management.
    2. On the left navigation pane in the Manage section, select External queues.
  2. At the top of the displayed External queues - destination configuration page, select Add configuration.

  3. In the Service Provider field, select the appropriate message streaming service from your cloud provider:
    Option Description
    Amazon SNS/SQS One of:
    • Amazon Simple Queue Service (SQS)
    • Amazon Simple Notification Service (SNS)
    Google Pub/Sub Google Cloud Publisher/Subscriber
    Microsoft Azure Microsoft Azure Service Bus
    Note: The fields in the Add configuration page cascade, so some values are displayed only when a particular checkbox is selected or cleared. Some of the following field values depend on the value that you select in this Service Provider field. We've indicated these with AWS, GCP, or Azure in the Option column of the following steps. You know the drill – an asterisk (*) means a required field.
  4. Specify the authentication method for your cloud provider:
    Option Description
    Authenticate using roleAWS
    • Select this checkbox to authenticate using an AWS Identity and Access Management (AIM) role:
      • Role: The AWS IAM user role.
      • External ID: The AWS account external ID.
        Note: Enter the External IDs in Universally Unique Identifiers (UUID) format. Using an AWS IAM Role is a more secure way to provide access to files in an S3 bucket and it’s the preferred approach.

        Example of a UUID:

        07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d.

        For more information, see Universally unique identifier.

    • Clear this checkbox to authenticate using AWS account key credentials:
      • Key*: The AWS key ID.
      • Secret*: The AWS secret access key.

      Note: Encode special characters in the AWS Key to enable Reltio to parse the URI. For more information, see the Encodings table in topic Message Streaming Provider.

    For more information on these authentication options, see the AWS documentation on Identity and Access Management and Account and Access Keys.

    Queue nameGCP Specify the name of your queue. The name must start with a letter, but it can't start with the string goog. It can include a combination of alphanumeric characters and the special characters hyphen (-), percent (%), period (.), plus sign (+), tilde (~), and underscore (_).

    TypeAzure Select one of the following:
    • Queue: for point-to-point communications.
    • Topic: For publisher or subscriber.

  5. Provide connection details for your cloud provider:
    Option Description
    Use ARNAWS Specify the following:
    • Select this checkbox to provide queue details using the Amazon Resource Name (ARN) of your AWS queue in the following format:
      arn:partition:service:region:account-id:resource-id

    • Clear this checkbox to enter specific queue details:
      • Queue name*: The name of the queue. The name can include a combination of alphanumeric characters and the special characters hyphen (-) and underscore (_). It can’t contain a period (.) since that character isn't valid in SNS/SQS.
      • Type*: The type of AWS queue, one of SQS or SNS.
      • Region*: The AWS region for the queue.

    GCP Specify one of:
    • Upload Service Account Key File: Upload your Google Developer service account key file in JSON format. Generate this in the Google Developer Console.

    • Enter project details:
      • Project ID*: Enter the project ID for the queue.
      • Private key*: Enter the private key, without the first and last extra lines or any new line breaks.
      • Client email*: Enter the email address for the client.

    Azure Specify the following:
    • Queue/Topic name: The name of the queue or topic. The name can be up to 260 characters and must start and end with a letter or a number. It can contain letters, numbers, and the special characters hyphen (-), period (.), slash (/), and underscore (_).
    • Namespace name*: The name of the Azure namespace. The namespace must be between 6 and 50 characters. It must start with a letter and end with a letter or a number. It can contain only letters, numbers, and the special character hyphen (-).
    • Authorization policy rule name*: The name of the Azure Service Bus authorization policy rule.
    • Access key*: The key that defines the Azure Service Bus authorization policy rule primary or secondary key.

  6. In the Format field, specify the Relation message format for streaming messages into a queue:
    Option Description
    JSON Select this to use a standard JSON file. This is the default.
    JSON_ZIP_BASE64 Select this to use a GZIP file in Base64 encoding containing a long JSON string. This format is useful for long messages whose size prevents them from being sent in standard JSON format. For more information on size limits, see the FAQs below.

  7. In the Object filter field, optionally specify a Relation object filter expression for a message event object to be sent to this external queue. This filter query parameter uses the same expression structure as the Search and Export filters:
    filter=({Condition Type}[AND/OR {Condition Type}]*)

    For more information, see topic Search and Create and run an export job.

  8. In the Type filter field, specify any Relation event type collections you want to stream using this external queue. By default, all event types are streamed; clear any event types you want to have ignored during transmission to the external queue. You can select CRUD events and MATCH events. For more information, see topics CRUD events and Match events.
  9. In the Payload type field, specify the payload type for messages to be streamed to this queue, one of:
    Option Description
    Snapshot (All fields) Stream all the fields of the objects that have changed due to the event that is being transmitted. The queue adds the object Version and the commit Time fields to the message meta data. You can use these fields to order events during data consumption. This is the default payload type.

    Snapshot (Selected fields) Stream selected/limited fields of the objects that have changed due to the event that is being transmitted. In the displayed Event payload field, specify the attributes you want to stream.
    Note: You can only select simple attributes in this field. Nested and reference attributes aren’t supported as of now. For more information on attribute types, see topic Understanding the Reltio Attribute Type.
    • You can select the following fields:
      • Uri
      • type
      • created By
      • createdTime
      • objectVersion
      • commitTime
      CAUTION:
      If you clear these fields, you'll get an error message, and you won't be able to save the queue details.

      The queue adds the objectVersion and commitTime field to the message metadata. It adds the remaining attributes to the payload.

    • From the dropdown list, select any additional attributes that you want to stream. Use the Search field to locate a known attribute.

    Delta Stream only the changed fields of the objects that have changed due to the event that is being transmitted in a format similar to the Activity Log.

  10. Enable streaming: Specify whether or not to publish events to this queue:
    • Select this checkbox to publish events. This is the default value.
    • Clear this checkbox not to publish events. This is useful if you aren’t ready to start streaming yet, or if you want to turn off streaming temporarily.

  11. Select Save.
    Your external queue is ready to stream data between Reltio and external systems.

Something didn’t go to plan? Let's see if we can give you a hand!

Q. Is there a size limit for messages streaming through external queues?
A. The provider limits the size of messages that can be streamed through its message queues.
Table 1. Message streaming size limits
Queue type Maximum even size
AWS SNS/SQS 256 kb
GCP Pub/Sub ~10Mb
Microsoft Azure Service Bus 1 MB
You can bypass this size limits using these properties in your Reltio Tenant Configuration (let us know if you need a hand with this):
  • largeObjectsSupport: Ensure that this property is set to true and then set the exceededQueueSizeLimit attribute to build the consumer logic to work with larger messages in either of the following ways:
    • False: Get the complete message, including the object data. This is the default.
    • True: Get the object URI instead of the message data. Then, get the object data from Reltio using the GET by URI API.
Q. How is the order of messages streaming through external queues managed?

A. Reltio doesn't stream events in First In, First Out (FIFO) order. Instead, multiple nodes process multiple requests in parallel. This enables fast and efficient horizontal scaling. But, it also can result in processing a request that arrives a few milliseconds after another faster and putting it into the queue first.

You can use the objectVersion and commitTimeattributes in streamed message headers to manage the order of messages in the consuming system. Identify the version of the object. If you already have a newer version or timestamp, don’t persist or propagate the consumed message downstream.

For an ENTITIES_MATCHES_CHANGED event, the commitTime indicates the time when the match event occurred and doesn’t indicate the time when the object is updated.

Q. Can all Reltio event types be streamed to external queues?
A. No, you can't stream the following event types:
  • AS_MATCHES_RESET
  • AS_MATCHES_SET
  • NOT_MATCHES_RESET
  • NOT_MATCHES_SET
  • POTENTIAL_MATCHES_FOUND
  • POTENTIAL_MATCHES_REMOVED
Q. Can I configure multiple queues to the same destination?
A. Yes, you can configure multiple queues to the same destination. For more information, see topic Outbound Streaming Scenarios.
Q. Can I configure the same events to be streamed to multiple destinations?
A. Yes, you can configure the same events and filters on multiple queues pointing to different destinations. For more information, see topic Outbound Streaming Scenarios.
Q. How can I add static fields with Delta streaming?
A. Reltio doesn’t support selecting static fields while configuring delta streaming for objects. This is because the delta configuration sends only the changed attributes of the object and minimal static fields that can help you identify the object correctly in Reltio and in the source system. If you also want to use static fields to correctly link the changes to the source tables in your database, configure an extra queue using the Snapshot (Selected fields) option and select the required attributes.
Keep the same object and event filters for both the queues. This will ensure that the same events are transmitted in both the queues, and any differences can be seen in the JSON responses. The message in the delta queue will contain the changed attributes and the message in the snapshot queue will contain the static fields.
The external system can join the messages from the two queues using the following message attributes:
  • operationId in the message header - Several messages in the delta and snapshot queues will have the same operation ID if they’re generated from the same operation in the Reltio platform.
    For example, when two entities are merged, four events are generated:
    • Two Entity_Changed events
    • One Entity_Lost_Merge event
    • One Entities_Merged event
    All these events will have the same operationId value in the header of their message.
  • Object URI in the message body and objectVersion in the message header - The object URI and object version will be the same in both queues for messages that are generated from a single event.

The Delta payload snapshot includes only the object fields that have changed. These are presented in a before-after format similar to the Activity Log. For more information, see topic Activity Log).

This example shows the delta snapshot if only the value of the City attribute changes from InitialCity to ChangedCity in entity 00009ab.
{
	"type": "ENTITY_CHANGED",
	"uri": "entities/00009ab",
	"deltas": {
		"ovChanged": true,
		"delta": [{
			"type": "ATTRIBUTE_CHANGED",
			"attributeType": "configuration/entityTypes/Location/attributes/City",
			"newValue": {
				"value": "ChangedCity",
				"ov": true,
				"id": "8",
				"sources": [
					"Reltio"
				]
			},
			"oldValue": {
				"value": "InitialCity",
				"ov": true,
				"id": "8",
				"sources": [
					"Reltio"
				]
			}
		}]
	}
}