Unify and manage your data

Message Streaming Capabilities

Learn about the message streaming capabilities.

Message streaming enables the Reltio platform to process events from an internal queue into an external queue or topic in JSON format. For more information, see topic Message streaming. Different formats of messages (JSON or zipped JSON) are supported.

Type Filter Property

Entity Type Filtering is supported using the typeFilter property. If typeFilter is defined, only events with type that are defined in this field will be sent to the external queue.

Object Filter Property

Entity Type Filtering is supported using the objectFilter property. The objectFilter attribute is a filter expression for an event object. An event is not sent to a destination if its object does not match the filter. The expression structure is the same as for search filters. This attribute is optional, and the default value is null.

For example, the following filter is valid for CRUD Events:

objectFilter=equals(type,'configuration/entityTypes/Individual') and
          startsWith(attributes.FirstName, 'An')

This type of filter does not work properly for MATCHING events, even if the Winnerentity has FirstName = 'Ann' or FirstName = 'Andy'. Such an event does not appear in the queue.

However, the following filter works with CRUD/MATCH Events properly:

objectFilter=equals(type,'configuration/entityTypes/Individual') and gt(updatedTime,
          1567595145000)

objectFilter syntax is checked before a modification. Validation errors are displayed in case of an invalid expression.

When filtering entities for streaming, you cannot search for entities based on their OV value. In other words, you cannot include the searchByOv option in streaming queue filters.

Various Formats of messages

By default, messages are sent into a queue in string JSON format, but sometimes long messages cannot be sent due to their size. In this case, you can configure Message Streaming to zip messages before sending. The format attribute allows to specify message format:

  • JSON- JSON string (default)
  • JSON_ZIP_BASE64- JSON string zipped with GZIP and Base64-enconded

Hidden attributes in SQS messages

When you send data to SQS, all attributes — including hidden attributes — are included in the payload. There are no configuration flags available that can prevent hidden attributes from being sent to the SQS.