Streaming RDM Events

Learn how the RDM service streams events that create, update, and delete RDM values.

The RDM service streams individual event messages for each RDM value change (create, update, delete) to any topic.

Event Format

RDM service events have this format:

{
  "type": <event_type>,
  "object": <changed_object>
}

With these fields:

Table 1. Event Fields
Field	Description
`type`	Event type. The following types are supported: `LOOKUP_VALUE_CREATED` `LOOKUP_VALUE_UPDATED` `LOOKUP_VALUE_DELETED` `UNMAPPED_VALUE_CREATED` `UNMAPPED_VALUE_UPDATED` `UNMAPPED_VALUE_DELETED`
`object`	Contains details about the created, updated, or deleted value.

The following is an example of an event when a lookup value is created:

{
  "type": "LOOKUP_VALUE_CREATED",
  "object":
    {
      "tenantId": "rdmabc",
      "type": "rdm/lookupTypes/Gender",
      "code": "M",
      "enabled": true,
      "sourceMappings": [
        {
          "source": "Reltio",
          "values": [
            {
              "code": "MALE",
              "value": "Male",
              "description": "",
              "enabled": true,
              "canonicalValue": true,
              "downStreamDefaultValue": true
            }
          ]
        }
      ],
      "localizations": [
        {
          "languageCode": "sv-se",
          "value": "Manlig"
        }
      ],
      "startDate": 0,
      "endDate": 0,
      "updatedBy": "test.user",
      "updateDate": 1573118722139,
      "version": 1
    }
}

Messaging Provider

A messaging provider contains information that allows you to connect to a messaging destination. You must define the following properties for each messaging provider:

type
host
username
password

The last three properties contain different information depending on the type specified.

Note: RDM API does not support Azure providers now.

The following table explains the values to be provided for the Google Pub/Sub messaging provider:

Table 2. Google Pub/Sub
Field	Description
`type`	The message provider type. Supported type: `google`
`host`	The name of the default GCP project in which the topics are created.
`username`	The client email from GCP service account key JSON with Pub/Sub permissions.
`password`	The private key from GCP service account key JSON with Pub/Sub permissions. For example, GCP account must have the following permissions: pubsub.topics.create pubsub.topics.publish pubsub.topics.get Note: You must remove the new line breaks from the original key in addition to the first and the last extra lines.

AWS

The following table explains the values to be provided for the AWS messaging provider:

Table 3. AWS
Property	Description
type	Represents the messaging provider type. Only the aws type is supported.
host	Represents the name of the Amazon region in which the queues will be created.
username	Represents the Amazon Access Key for an account with Simple Queue Service (SQS) permission or or the Amazon Role Application reference number (ARN). For example, arn:aws:iam::CUSTOMER-AWS-ACCOUNT-ID:role/RoleToStreamToQueue.
password	Represents the Amazon Secret Key for an account with SQS permission, for corresponding Amazon Access Key or the Amazon Role External ID in GUID format. For example, 07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d.

For more information on delegating access using roles, see : IAM Tutorial: Delegate access across AWS accounts using IAM roles.

The policy with following permissions are attached with the roles provided:

sqs:SendMessage
sqs:GetQueueUrl
sqs:GetQueueAttributes

The following permissions are granted when the queue does not exist. The queues are created automatically before the first message is sent.

sqs:CreateQueue
sqs:SetQueueAttributes

Messaging Destination

A messaging destination is the topic where the RDM API will send the event to. You must define the following properties for each messaging destination:

provider
type
name

Note: The number of destinations to which RDM API can send events to is limited to three.

The following table explains the values to be provided for the messaging destination properties:

Table 4. Messaging Destination Properties
Field	Description
`provider`	The alias of the provider, as configured for the tenant or a special `messaging-provider-agnostic` URI to embed connection details.
`type`	The type of a destination. Supported types: topic, queue.
`name`	The name of a topic. It has the following formats: Google Pub/Sub `{name}` - topic name. Google project name is taken from the provider definition. `projects/{project}/topics/{name}` - where project is a Google project and name is a topic name. Note: Names must start with a letter, and contain only the following characters: letters, numbers, dashes (-), periods (.), underscores (_), tildes (~), percents (%) or plus signs (+). Cannot start with goog. AWS SQS Names can be a combination of Alphanumeric characters '-' (dash) '_' (underscore) Note: The '.' (dot) is not valid in SQS names! You can set topic name as the SQS ARN link in the following format: arn:aws:sqs:<region>:<account>:<name> Note: You can use SQS ARN link only on the already existing queue.

Messaging Destination Provider as URI

Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration. The format of the messaging-provider-agnostic URI is as follows:

<scheme>://<username>:<password>@<host>

The following table explains the fields of the messaging-provider-agnostic URI:

Table 5. Messaging Provider Fields
Field	Description
`scheme`	A valid scheme. Supported scheme: google, aws.
`username and password`	Google The client email and private key are escaped by URI rules and separated by a colon. AWS The access-key and secret key are separated by a colon. AWS Role ARN and Role External ID (format: GUID) are separated by a colon.
`host`	Google The default GCP project for the Pub/Sub topic. AWS The AWS region name for the SQS queue.

The following is an example of Google provider definition in URI style:

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

The following is an example of AWS provider definition in URI style:

aws://AccessKey:SecretKey@us-east-1 [Option 1]
aws://username:password@region [Option 2]

The following example defines the url_encoded role as username and External ID as password:

"aws://arn%3Aaws%3Aiam%3A%3ACUSTOMER-AWS-ACCOUNT-ID%3Arole%2FRoleToStreamToQueue:07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d@us-east-1"

Encoding

When using URI style configuration, it is important to encode characters that are not legal within URI fields. The common forms of encoding are as follows:

      !              %21
      "              %22
      #              %23
      $              %24
      %              %25
      &              %26
      '              %27
      (              %28
      )              %29
      *              %2A
      +              %2B
      ,              %2C
      -              %2D
      .              %2E
      /              %2F
      :              %3A
      ;              %3B
      <              %3C
      =              %3D
      >              %3E
      ?              %3F
      @              %40
      [              %5B
      \              %5C
      ]              %5D
      {              %7B
      |              %7C
      }              %7D

The following commands can be used for encoding:

Python 2

echo '789secretkey999' | python2 -c 'import urllib, sys; sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in sys.stdin)'

Python 3

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys; sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

Messaging Configuration in the RDM Tenant

To ensure that RDM events can be streamed to the specified destination, the following configuration must be performed on the RDM tenant. RDM API allows to configure streaming lookups changes in RDM tenant into supported messaging destinations by users of this tenant with appropriate permissions.

{
  "enabled": true,
  "providers":
  {
    "<messaging_provider_alias>":
    {
      "type": "<provider_type>",
      "host": "<provider_host>",
      "username": "<provider_username>",
      "password": "<provider_password>"
    }
  },
  "destinations": [
    {
      "provider": "<provider_alias_or_uri>",
      "type": "<destination_type>",
      "name": "<topic>"
    }
  ]
}

The following table explains the fields that are part of the messaging configuration:

Table 6. Messaging Configuration Fields
Field	Description
`enabled`	Enable or disable RDM events streaming.
`providers`	A map of messaging providers.
`destination`	A list of messaging destinations.

Messaging APIs

Get Messaging Configuration

This API allows to get the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

Permission: rdm:config.messaging
Privilege: READ

Request

GET https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Update Messaging Configuration

This API allows to update the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

Permission: rdm:config.messaging
Privilege: UPDATE