Message Queue

Learn about the message queue functionality in Reltio.

Your Reltio subscription includes a message queue as an AWS SQS queue, AWS SNS queue, or GCP PubSub topic for each tenant.

As needed, you can subscribe to this queue. You can use the subscription to be utilized to read events off of the queue to progate to the rest of your enterprise.

  • Messages are automatically placed in the queue when an event occurs in the platform
  • Messages in the queue indicate the event type that triggered the message
  • Messages can be configured to return the ID, metadata about the event (such as timestamps) and the full payload of the impacted Reltio Entity

Best Practices for Message Queue

Connection

For details about connecting to AWS SQS, consult the Amazon documentation.

For details about connecting to AWS SNS, consult the Amazon documentation.

To configure PubSub topic subscriptions or to learn how to read messages from PULL subscriptions, consult the Understanding subscriptions in PubSub.

Security

To access AWS SQS queue or AWS SNS topics, you need to provide valid Amazon credentials: AWS Access Key, AWS Secret Key.

To access PubSub topics, you need to provide valid Google credentials: GCP Client email, GCP Private Key.

Messaging Destinations

Each Tenant in Reltio API has its own event queues/topics.

Message Consumer

Tolerate Duplicate and Out-of-order Messages.

Most highly scalable cloud systems (Amazon SQS/SNS and Google PubSub) do not guarantee message order and may occasionally deliver duplicate messages. Make your processing logic tolerant of duplicate and out-of-order messages. These steps will also ensure you can scale your consumer using multi-threading.

Message Delivery Considerations

SQS/SNS and PubSub guarantee that messages will be delivered "at least" once. As a result, you might receive a duplicate copy of the message (at-least-once delivery guaranteed). When using SQS/SNS/PubSub, the design and code should be done in a manner that they are prepared to receive the same message more than once, but only process it one time. For more information about considerations for SNS, consult the Amazon documentation.

Improve Receive Throughput by Processing Concurrently

SQS queue semantics support multiple consumers on the same queue, with each message delivered to one and only one consumer. This means that you may have consumers processing messages in 'n' different threads, but each message will only be processed once. To create concurrent consumers using SQS, use multiple polling threads.

Understanding Topic Subscriptions

A topic subscription can be passive (PULL in terms of GCP) or active (PUSH in terms of GCP).
  • Passive subscriptions just aggregate messages sent to them and do nothing with the messages until explicitly asked.
    • Such subscriptions can be pulled by a consumer.
    • After processing the messages successfully, the consumer should delete them from subscription with another explicit call.
  • Active subscriptions take some action when they get a new message. Depending upon the platform, the actions taken could be similar to these examples:
    • Calling some HTTP endpoint with message content as a payload
    • Sending text message to a phone number (SMS)
    • Sending email with message content

Optimize Network IO and AWS Cost by Batching

SQS is based on Amazon Web Services (AWS), so it uses an HTTP request to retrieve messages and perform all other SQS tasks. Making fewer network round-trips can improve performance.

PubSub is based on Google Cloud Platform (GCP), so requesting messages in batches (instead of one message at a time) is more optimal.

Filter Out Events that are not needed

Before processing a Reltio Event, you can pre-parse the message body and discard event types that are never going to be used in your integrations.