Streaming RDM Events

Learn how the RDM service streams events that create, update, and delete lookup values.

The RDM service streams reference data changes in real time by publishing event messages whenever it creates, updates, or deletes a lookup value. Each change generates a separate event that contains complete details of the affected lookup value, allowing downstream systems to stay synchronized without polling. RDM streams these events to supported messaging platforms, such as Google Pub/Sub and AWS SQS, using tenant-level configuration of messaging providers and destinations to support event-driven integration with external systems.

Event Format

RDM service events use the following format:

{
  "type": <event_type>,
  "object": <changed_object>
}

The following table lists the fields included in an RDM service event message:

Table 1. Event Fields
Field	Description
`type`	Event type. The following types are supported: `LOOKUP_VALUE_CREATED` `LOOKUP_VALUE_UPDATED` `LOOKUP_VALUE_DELETED`
`object`	Contains details about the created, updated, or deleted value.

The following example shows an event generated when a lookup value is created:

{
  "type": "LOOKUP_VALUE_CREATED",
  "object":
    {
      "tenantId": "rdmabc",
      "type": "rdm/lookupTypes/Gender",
      "code": "M",
      "enabled": true,
      "sourceMappings": [
        {
          "source": "Reltio",
          "values": [
            {
              "code": "MALE",
              "value": "Male",
              "description": "",
              "enabled": true,
              "canonicalValue": true,
              "downStreamDefaultValue": true
            }
          ]
        }
      ],
      "localizations": [
        {
          "languageCode": "sv-se",
          "value": "Manlig"
        }
      ],
      "startDate": 0,
      "endDate": 0,
      "updatedBy": "test.user",
      "updateDate": 1573118722139,
      "version": 1
    }
}

Message headers

In addition to the event payload, RDM publishes message headers that provide metadata about lookup change events.

Standard message header

All lookup change events include the following message header.


Header	Description
`MESSAGE_ID`	Unique identifier of the published message.

Optional message headers

RDM supports optional message headers that provide more context about the lookup change.


Header	Description
`COMMIT_TIME`	Timestamp (milliseconds since epoch) when the lookup change was committed.
`OBJECT_VERSION`	Version of the lookup value after the change.

These headers are provided in addition to MESSAGE_ID and do not modify the event payload.

Suppressing optional message headers

RDM controls the inclusion of optional message headers by using a tenant-level configuration property named suppressMessageHeaders. By default, this property is set to true, and RDM publishes only the standard MESSAGE_ID header.

When you set suppressMessageHeaders to false, RDM publishes the standard MESSAGE_ID header along with all supported optional message headers, such as COMMIT_TIME and OBJECT_VERSION. These headers provide additional context about the lookup change.

This configuration allows you to include additional message metadata while maintaining compatibility with existing systems that use only the standard message header.

Messaging Provider

A messaging provider contains information that allows you to connect to a messaging destination. You must define the following properties for each messaging provider:

type
host
username
password

The last three properties contain different information depending on the type specified.

Note: RDM API does not support Azure providers for now.

Google Pub/Sub

The following table explains the values to be provided for the Google Pub/Sub messaging provider:


Field	Description
`type`	The message provider type. Supported type: `google`
`host`	The name of the default GCP project in which the topics are created.
`username`	The client email from GCP service account key JSON with Pub/Sub permissions.
`password`	The private key from GCP service account key JSON with Pub/Sub permissions. For example, GCP account must have the following permissions: pubsub.topics.create pubsub.topics.publish pubsub.topics.get Note: You must remove the new line breaks from the original key in addition to the first and the last extra lines.

AWS

The following table explains the values to be provided for the AWS messaging provider:


Property	Description
`type`	Represents the messaging provider type. Supported type is AWS
`host`	Represents the name of the Amazon region in which the queues will be created.
`username`	Represents the Amazon Access Key for an account with Simple Queue Service (SQS) permission or the Amazon Role Application reference number (ARN). For example, arn:aws:iam::CUSTOMER-AWS-ACCOUNT-ID:role/RoleToStreamToQueue.
`password`	Represents the Amazon Secret Key for an account with SQS permission, for corresponding Amazon Access Key or the Amazon Role External ID in GUID format. For example, 07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d.

For more information on delegating access using roles, see : IAM Tutorial: Delegate access across AWS accounts using IAM roles.

The policy with following permissions are attached with the roles provided:

sqs:SendMessage
sqs:GetQueueUrl
sqs:GetQueueAttributes

The following permissions are granted when the queue does not exist. The queues are created automatically before the first message is sent.

sqs:CreateQueue
sqs:SetQueueAttributes

Messaging Destination

A messaging destination is the topic or queue to which the RDM API sends events . You must define the following properties for each messaging destination:

provider
type
name

Note: The number of destinations to which RDM API can send events to is limited to three.

The following table explains the values to be provided for the messaging destination properties:

Table 2. Messaging Destination Properties
Field	Description
`provider`	The alias of the provider, as configured for the tenant or a special `messaging-provider-agnostic` URI to embed connection details.
`type`	The type of a destination. Supported types: topic, queue.
`name`	The name of a topic. It has the following formats: Google Pub/Sub `{name}` - topic name. Google project name is taken from the provider definition. `projects/{project}/topics/{name}` - where project is a Google project and name is a topic name. Note: Names must start with a letter, and contain only the following characters: letters, numbers, dashes (-), periods (.), underscores (_), tildes (~), percents (%) or plus signs (+). Cannot start with goog. AWS SQS Names can be a combination of Alphanumeric characters '-' (dash) '_' (underscore) Note: The '.' (dot) is not valid in SQS names! You can set topic name as the SQS ARN link in the following format: arn:aws:sqs:<region>:<account>:<name> Note: You can use SQS ARN link only on the already existing queue.

Messaging Destination Provider as URI

Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration. The format of the messaging-provider-agnostic URI is as follows:

<scheme>://<username>:<password>@<host>

The following table explains the fields of the messaging-provider-agnostic URI:

Table 3. Messaging Provider Fields
Field	Description
`scheme`	A valid scheme. Supported scheme: google, aws.
`username and password`	Google The client email and private key are escaped by URI rules and separated by a colon. AWS The access-key and secret key are separated by a colon. AWS Role ARN and Role External ID (format: GUID) are separated by a colon.
`host`	Google The default GCP project for the Pub/Sub topic. AWS The AWS region name for the SQS queue.

The following is an example of Google provider definition in URI style:

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

The following is an example of AWS provider definition in URI style:

aws://AccessKey:SecretKey@us-east-1 [Option 1]
aws://username:password@region [Option 2]

The following example defines the url_encoded role as username and External ID as password:

"aws://arn%3Aaws%3Aiam%3A%3ACUSTOMER-AWS-ACCOUNT-ID%3Arole%2FRoleToStreamToQueue:07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d@us-east-1"

Encoding

When using URI style configuration, it is important to encode characters that are not legal within URI fields. The common forms of encoding are as follows:

      !              %21
      "              %22
      #              %23
      $              %24
      %              %25
      &              %26
      '              %27
      (              %28
      )              %29
      *              %2A
      +              %2B
      ,              %2C
      -              %2D
      .              %2E
      /              %2F
      :              %3A
      ;              %3B
      <              %3C
      =              %3D
      >              %3E
      ?              %3F
      @              %40
      [              %5B
      \              %5C
      ]              %5D
      {              %7B
      |              %7C
      }              %7D

The following commands can be used for encoding:

Python 2

echo '789secretkey999' | python2 -c 'import urllib, sys; sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in sys.stdin)'

Python 3

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys; sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

Messaging Configuration in the RDM Tenant

To ensure that RDM events can be streamed to the specified destination, the following configuration must be performed on the RDM tenant. RDM API allows to configure streaming lookups changes in RDM tenant into supported messaging destinations by users of this tenant with appropriate permissions.

{
  "enabled": true,
  "providers":
  {
    "<messaging_provider_alias>":
    {
      "type": "<provider_type>",
      "host": "<provider_host>",
      "username": "<provider_username>",
      "password": "<provider_password>"
    }
  },
  "destinations": [
    {
      "provider": "<provider_alias_or_uri>",
      "type": "<destination_type>",
      "name": "<topic>"
    }
  ]
}

The following table explains the fields that are part of the messaging configuration:

Table 4. Messaging Configuration Fields
Field	Description
`enabled`	Enable or disable RDM events streaming.
`providers`	A map of messaging providers.
`destination`	A list of messaging destinations.

Messaging APIs

Get Messaging Configuration

This API allows to get the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

Permission: rdm:config.messaging
Privilege: READ

Request

GET https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Update Messaging Configuration

This API allows to update the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

Permission: rdm:config.messaging
Privilege: UPDATE