Message Streaming Capabilities
Learn about the message streaming capabilities.
Message streaming enables the Reltio platform to process events from an internal queue into an external queue or topic in JSON format. For more information, see topic Message streaming. Different formats of messages (JSON or zipped JSON) are supported.
Type Filter Property
Entity Type Filtering is supported using the typeFilter
property. If typeFilter
is defined, only events with type that are defined in this field will be sent to the external queue.
Object Filter Property
Entity Type Filtering is supported using the objectFilter
property. The objectFilter
attribute is a filter expression for an event object. An event is not sent to a destination if its object does not match the filter. The expression structure is the same as for search filters. This attribute is optional, and the default value is null
.
For example, the following filter is valid for CRUD
Events:
objectFilter=equals(type,'configuration/entityTypes/Individual') and
startsWith(attributes.FirstName, 'An')
This type of filter does not work properly for MATCHING
events, even if the Winner
entity has FirstName
= 'Ann'
or FirstName
= 'Andy'. Such an event does not appear in the queue.
However, the following filter works with CRUD
/MATCH
Events properly:
objectFilter=equals(type,'configuration/entityTypes/Individual') and gt(updatedTime,
1567595145000)
objectFilter
syntax is checked before a modification. Validation errors are displayed in case of an invalid expression.
When filtering entities for streaming, you cannot search for entities based on their OV value. In other words, you cannot include the searchByOv
option in streaming queue filters.
Various Formats of messages
By default, messages are sent into a queue in string JSON format, but sometimes long messages cannot be sent due to their size. In this case, you can configure Message Streaming to zip messages before sending. The format
attribute allows to specify message format:
JSON
- JSON string (default)JSON_ZIP_BASE64
- JSON string zipped with GZIP and Base64-enconded
Hidden attributes in SQS messages
When you send data to SQS, all attributes — including hidden attributes — are included in the payload. There are no configuration flags available that can prevent hidden attributes from being sent to the SQS.