Managing External Queues

The Tenant Management application allows you to add and manage the external queues for streaming data between Reltio and external systems.

The external queue is used for real-time integration with systems that are external to Reltio.

For more information on the required permissions, see Permissions for the Console Service.

Related Video:

You can specify the Provider as Amazon SNS/SQS, Google Pub/Sub, or Microsoft Azure.
Note: By default, Amazon SNS/SQS is selected.
.

Adding an Amazon SNS or SQS Queue Configuration

To add a queue to stream data to AWS, do the following steps:

  1. Go to the home page.
  2. In the Console, click Tenant Management.
  3. On the left navigation bar, click External Queue.

    The list of queues is displayed.

  4. Click ADD CONFIGURATION.

    The Add Configuration dialog box is displayed.

    You can select the Authenticate using role checkbox and specify the Role and External ID details. Alternatively, you can specify Amazon Key and Secret for authenticating yourself by using Amazon Key credentials.

  5. Select the Authenticate using role checkbox to use role-based authentication.
    1. Enter the Role.

      For more information, see Creating a role for an IAM user and How to use an external ID.

    2. Enter the External ID.

    OR

    Clear the Authenticate using role checkbox.

    1. Enter the AWS Key.
      Note: If the AWS Key contains characters that are not allowed in URI then these characters must be encoded. For more information about the common encodings, see the Encodings table in Message Streaming Provider.

      For more information, see AWS Account and Access Keys.

    2. Enter the Secret.
  6. Add Queue Configuration details.

    Queue details can be provided using ARN or by adding the queue details.

  7. Select the Use ARN checkbox.

    This will help you to use the Amazon Resource Name (ARN) of your AWS queue. In this case, the Queue name and Type fields are not required.

  8. If you deselect the Use ARN checkbox:
    1. Enter a name for the queue.
    2. Select the queue type as SQS or SNS.
    3. Enter the Region.

  9. Select the streaming format as JSON or JSON_ZIP_BASE64.
    Note: The default streaming format is JSON_ZIP_BASE64.
  10. Specify the object filter condition as another option.

    This helps you to filter out the object that must not be sent to this queue. For more information, see Events API.

  11. Select the Type filter.

    You can select the events that you want to stream using this queue. The unselected Event Types will be ignored during transmission to this external queue.

  12. Select Payload Type.

    Use Payload Type if you want the messages to be streamed to the external queue. For more information, see Message Streaming.

    There are three options:
    1. Snapshot (All fields)
    2. Snapshot (Selected fields)
    3. Delta

    1. By default, the Snapshot (All fields) option is selected.

      You can stream all the fields of the objects that have changed since the last snapshot.

      OR

    2. Select Snapshot (selected fields).

      On selecting this option, the Event Payload drop-down is displayed.

      By default the uri, type, createdTime,createdBy, and objectVersion are selected.

      1. Click the Event Payload drop-down.

        It helps you to select the required fields or JMS events that must be streamed to the message queue.

        Note: The uri, type, and objectVersion are mandatory event payload fields. You will not be able to save the queue details if the mandatory fields are deselected and an error will be displayed. Currently, the simple attributes are supported here.

      OR

    3. Select Delta.

      This will enable streaming for only the changed part of the object in a format similar to the activity log.

      For more information, see Message Streaming.

    Note: By default, the Enable Streaming checkbox is selected. The events will start publishing to the queue. If you do not want to enable streaming immediately, you can deselect this checkbox.

  13. Click SAVE.

    After you have saved the queue it is added to the External Queues page.

Adding a Google Pub/Sub Queue Configuration

To add a queue to stream data to Google Pub/Sub, do the following steps:

  1. Go to the home page.
  2. In the Console, click Tenant Management.
  3. On the left navigation bar, click External Queue.

    The list of queues are displayed.

  4. Click ADD CONFIGURATION.

    The Add Configuration dialog box is displayed.

  5. From the Provider drop-down list, select Google Pub/Sub.
  6. Enter the Queue Name.
  7. Click Upload Service Account Key File.

    You can upload the Service Account Key file in JSON format from your local system.

    The JSON file is generated by using the Google Developer Console. The file includes the project ID, private key, and the client email.

    Note: If the AWS Key contains characters that are not allowed in URI then these characters must be encoded. For more information about the common encodings, see the Encodings table in Message Streaming Provider.

    If you do not upload the JSON file, you must specify values for Project ID, Private key, and the Client email fields.

    Note: You need to change the Pub/Sub manually. You must remove the new line breaks including the first and last extra lines from the original Private key manually.
  8. Select the streaming format as JSON, or JSON_ZIP_BASE64.
    Note: The default streaming format is JSON_ZIP_BASE64.
  9. Specify the Object Filter condition as another option.

    This helps you to filter out the object that must not be sent to this queue. For more information, see Events API.

  10. Select the Type filter.

    You can select the events that you want to stream using this queue. The unselected Event Types will be ignored during transmission to this external queue.

  11. Select Payload Type.

    Use Payload Type if you want the messages to be streamed to the external queue. For more information, see Message Streaming.

    In Payload type, there are three options:
    1. Snapshot (All fields)
    2. Snapshot (Selected fields)
    3. Delta

    1. By default, the Snapshot (All fields) option is selected.

      You can stream all the fields of the objects that have changed since the last snapshot.

      OR

    2. Select Snapshot (selected fields)

      .

      On selecting this option, the Event Payload drop-down is displayed.

      By default, uri, type, createdTime, createdBy, and objectVersion are selected.

      1. Click the Event Payload drop-down.

        This will help you to select the required fields or JMS events that should be streamed to the message queue.

        Note: The uri, type, and objectVersion are mandatory event payload fields. You will not be able to save the queue details if the mandatory fields are deselected and an error will be displayed. Currently, the simple attributes are supported here.
    3. Select Delta.

      Select Delta to add only information about the changed part of the object in a format very similar to the activity log.

      For more information, see Message Streaming.

    Note: The Enable Streaming checkbox is selected, by default. The events will start publishing to the queue. If you do not want to enable streaming immediately, you can deselect this checkbox.
  12. Click SAVE.

    After you have saved the queue it is added to the External Queues page.

Adding a Microsoft Azure Queue

To add a queue to stream data to Microsoft Azure, do the following steps:

  1. In the Console, click the Tenant Management.
  2. Click External Queue on the left navigation bar.
  3. Click ADD CONFIGURATION.

    The Add configuration dialog box is displayed.

  4. From the Provider drop-down list, select Microsoft Azure.
  5. Select Type as Queue or Topic depending on where you want to stream messages in Microsoft Azure.

  6. Enter the Queue name.
    1. A Queue name can include letters, numbers, periods (.), hyphens (-), underscores (_), and slashes (/), up to 260 characters. The Queue name must start and end with a letter or number.
    2. The Queue name must start and end with a letter or number.
  7. Enter the Namespace name.
    1. The namespace must be between 6 and 50 characters long.
    2. The namespace can contain only letters, numbers, and hyphens.
    3. The namespace must start with a letter, and it must end with a letter or a number.
  8. Enter the Authorization policy rule name.
  9. Enter the access key.
    Note: If the AWS Key contains characters that are not allowed in URI then these characters must be encoded. For more information about the common encodings, see the Encodings table in Message Streaming Provider.
  10. Select the streaming format as JSON, or JSON_ZIP_BASE64.
    Note: The default streaming format is JSON_ZIP_BASE64.
  11. Specify the object filter condition as another option.

    This helps you to filter the object that must not be sent to this queue. For more information, see Events API.

  12. Select the Type filter.

    You can select the events that you want to stream using this queue. The unselected Event Types will be ignored during transmission to this external queue.

  13. Select Payload Type.

    Use Payload Type if you want the messages to be streamed to the external queue. For more information, see Message Streaming.

    In Payload type, there are three options:
    1. Snapshot (All fields)
    2. Snapshot (Selected fields)
    3. Delta
    1. By default, the Select Snapshot (All fields) option is selected.

      You can stream all the selected fields of the objects that have changed since the last snapshot.

      OR

    2. Select Snapshot (Selected fields).

      On selecting this option, the Event Payload drop-down is displayed.

      By default, the uri, type, createdTime, createdBy, and objectVersion are selected.
      1. Click the Event Payload drop-down.

        It helps you to select the required fields or JMS events that must be streamed to the message queue.

      Note: The uri, type, and objectVersion are mandatory event payload fields. You will not be able to save the queue details if the mandatory fields are deselected and an error will be displayed. Currently, the simple attributes are supported here.
    3. Select Delta.

      This will enable streaming for only the changed part of the object in a format similar to the activity log.

      For more information, see Message Streaming.

    Note: The Enable Streaming checkbox is selected, by default. The events will start publishing to the queue. If you do not want to enable streaming immediately, you can deselect this checkbox.
  14. Click SAVE.

    After you have saved the queue it is added to the External Queues page.

Editing or Deleting a Queue Configuration

  1. Hover over the queue on the right that you want to edit or delete.
  2. Click the Edit icon.

    You can edit a selected item in the queue .

    The Edit Configuration dialog box appears.

    In this example, we selected the option to Use ARN. While editing, you can also enable or disable streaming and delete the queue.

  3. After editing, click SAVE.
  4. To delete the queue configuration, click DELETE.

    Once you confirm the deletion, the queue configuration is successfully deleted.

Monitoring

On the External Queues page, click Monitor Queues to monitor the processing of the external queues that you created as shown below.

From this page, you can also enable or disable streaming and edit or delete a queue as shown in the image above.