Message Streaming

The message streaming enables the Reltio platform to process events from an internal queue into an external queue or topic in JSON format.

When the Reltio Platform processes events from an internal queue, message Streaming sends Create/Read/Update/Delete (CRUD) and MATCH events into an external Message Bus Queue (Simple Queue Service(SQS))/PubSub/Azure or topics in JSON format.

Message Streaming includes the following tasks:
  • The messages are sent into different queues
  • The payload of messages are filtered
  • The messages are filtered by types of event
  • The messages are filtered by object values

Message Streaming

The Message Streaming image explains how the internal event queues are turned into external queues.

Configuration: Tenant Configuration

JSON Configuration Format

Note:
  • The skipPayload field for streaming configuration is added.
  • The original JMS attributes JMSEventsTimeToLive and JMSEventsFilteringFields are still used.
The Streaming Configuration example describes the new format of the streaming configuration.

Example: Streaming Configuration


"streamingConfig": {
  "streamingEnabled": "true",
  "streamingAPIEnabled": false,
  "analyzeOvChanges": false,
  "JMSEventsTimeToLive": 86400000,
  "largeObjectsSupport": false,
  "JMSEventsFilteringFields": [
    "uri",
    "crosswalks",
    "type",
    "VeevaSync",
    "createdBy",
    "createdTime",
    "updatedBy",
    "updatedTime",
    "startObject",
    "endObject",
    "attributes"
  ],
  "messaging": {
    "destinations": [
      {
        "provider": "default",
        "type": "queue",
        "name": "queue.MyTenant.allEvents"
      },
      {
        "provider": "default",
        "type": "queue",
        "name": "queue.MyTenant.dtssEvents",
        "dtssQueue": true
      }
    ]
  }
}
The Parameters: Streaming Configuration table describes various types of parameters in streaming configuration.
Table 1. Parameters: Streaming Configuration
Property name Required Description Type
streamingEnabled Yes This parameter enables or disables events publishing. boolean
streamingAPIEnabled No If the value of the parameter is false then events from a data load endpoint and a reindex endpoint are not sent to JMS.

The default value is true.

boolean
skipPayload No If the value is true, the platform sends the type of event and ovChanges property only, not the entire payload. The default value is false. boolean
analyzeOvChanges No If the value of the parameter is true, then all ENTITY_CHANGED events will include an additional boolean attribute:ovChanged.

This indicates whether ov=true attribute values were changed or not.

The default value is false. Also, you may specify analyzeOvChanges attribute for any REST request to the API. This attribute will override the option from streamingConfig.

boolean
JMSEventsTimeToLive Yes This parameter indicates the expiration time for events in milliseconds. long
JMSEventsFilteringFields No This parameter indicates the fields of objects to be included into the content of the events.

The parameter indicates the field names of any objects and the entities. The filtering cannot be done if the parameter is not there and all the content belonging to the object will be placed into the event.

array
RelationEventsFilteringFields No This parameter indicates that the fields of objects must be included into the content of the event.

Here the field names of the relations are listed. If it is not specified, filtering is not done. All the content belonging to the object will be placed into the event.

array
largeObjectsSupport No The support of large objects in External Streaming Service is implemented. When the compressed size of the event exceeds the following limit, the event will be sent again to include the URI of the object. The exceededQueueSizeLimit:true field instead of the entire message.

The default value is false.

boolean
JMSIncludeMergeTime No If the value of the parameter is true, the updatedTime are added to the ENTITIES_MERGED event.

The default value is false.

boolean
emptyStartEndRelationCrosswalks No If the value of the parameter is true then crosswalks collections ofstartObject and endObject in relation events are always empty.

The default value is false.

boolean
messaging No A list of Messaging Destination configurations.

At least one dest is required.

object array

Messaging Destinations

Tenants may have event messages published to multiple destinations.

The Parameter: Messaging Destination table explains each destination that is specified in the tenant configuration and consists of the following attributes.

Table 2. Parameter: Messaging Destination
Attribute Required Default Description
provider No default value Defines the alias of the provider, as configured for the Reltio environment.
type No queue Defines the type of destination. The supported types may be provider-specific.
name Yes   Defines the name of the destination.
dtssQueue No false Defines a true or false flag indicating that this queue is used by DTSS.
enabled No true Indicates whether this destination is used or ignored.
typeFilter No null Defines the collection of event type names to stream. Events of different types will be ignored.
awsAccountId No null Defines the AWS account identifier for SQS queue, must not be used. This provides a full queue for Amazon Resource Name (ARN) within the name field.
objectFilter No null Defines the filter expression for the event object. The event must not be sent to a destination if the object does not match with the filter. The expression structure is the same as for search filters.

For more information, see Filtering Entities.

format No JSON Defines the format of the events that are sent into the queue.

JSON - message will be sent in string JSON format. JSON_ZIP_BASE64 is the string JSON that was zipped by GZIP and converted to Base64.

payloadType No snapshot Specifies the content payload type, which can be either snapshot (the full object state when it is modified) or deltas (the changed object attributes).