Accelerate the Value of Data

Streaming RDM Events

RDM API can stream events that contain changes to create, update and delete lookup values in Pub/Sub topics.

The changes to lookup values (create, update, delete) can be streamed to any Google Pub/Sub topics and is not limited to Reltio Google Project. Each change is streamed as a separate message to the Google Pub/Sub topics and AWS streaming.

Event Format

The event streamed from RDM has the following format:

{
  "type": <event_type>,
  "object": <changed_object>
}

The following table explains the fields in the event:

Table 1. Event Fields
Field Description
type Event type. The following types are supported:
  • LOOKUP_VALUE_CREATED
  • LOOKUP_VALUE_UPDATED
  • LOOKUP_VALUE_DELETED
object Contains details about the created, updated, or deleted lookup value.

The following is an example of an event when a lookup value is created:

{
  "type": "LOOKUP_VALUE_CREATED",
  "object":
    {
      "tenantId": "rdmabc",
      "type": "rdm/lookupTypes/Gender",
      "code": "M",
      "enabled": true,
      "sourceMappings": [
        {
          "source": "Reltio",
          "values": [
            {
              "code": "MALE",
              "value": "Male",
              "description": "",
              "enabled": true,
              "canonicalValue": true,
              "downStreamDefaultValue": true
            }
          ]
        }
      ],
      "localizations": [
        {
          "languageCode": "sv-se",
          "value": "Manlig"
        }
      ],
      "startDate": 0,
      "endDate": 0,
      "updatedBy": "test.user",
      "updateDate": 1573118722139,
      "version": 1
    }
}

Messaging Provider

A messaging provider contains information that allows you to connect to a messaging destination. You must define the following properties for each messaging provider:
  • type
  • host
  • username
  • password
The last three properties contain different information depending on the type specified.
Note: RDM API does not support Azure providers now.

The following table explains the values to be provided for the Google Pub/Sub messaging provider:

Table 2. Google Pub/Sub
Field Description
type The message provider type. Supported type: google
host The name of the default GCP project in which the topics are created.
username The client email from GCP service account key JSON with Pub/Sub permissions.
password The private key from GCP service account key JSON with Pub/Sub permissions.

For example, GCP account must have the following permissions:

  • pubsub.topics.create
  • pubsub.topics.publish
  • pubsub.topics.get
Note: You must remove the new line breaks from the original key in addition to the first and the last extra lines.

AWS

The following table explains the values to be provided for the AWS messaging provider:

Table 3. AWS
Property Description
type Represents the messaging provider type. Only the aws type is supported.
host Represents the name of the Amazon region in which the queues will be created.
username Represents the Amazon Access Key for an account with Simple Queue Service (SQS) permission or or the Amazon Role Application reference number (ARN).

For example, arn:aws:iam::CUSTOMER-AWS-ACCOUNT-ID:role/RoleToStreamToQueue.

password Represents the Amazon Secret Key for an account with SQS permission, for corresponding Amazon Access Key or the Amazon Role External ID in GUID format.

For example, 07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d.

For more information on delegating access using roles, see : IAM Tutorial: Delegate access across AWS accounts using IAM roles.

The policy with following permissions are attached with the roles provided:

  • sqs:SendMessage
  • sqs:GetQueueUrl
  • sqs:GetQueueAttributes

The following permissions are granted when the queue does not exist. The queues are created automatically before the first message is sent.

  • sqs:CreateQueue
  • sqs:SetQueueAttributes

Messaging Destination

A messaging destination is the Pub/Sub topic where the RDM API will send the event. You must define the following properties for each messaging destination:
  • provider
  • type
  • name
Note: The number of destinations to which RDM API can send events is limited to three.
The following table explains the values to be provided for the messaging destination properties:
Table 4. Messaging Destination Properties
Field Description
provider The alias of the provider, as configured for the tenant or a special messaging-provider-agnostic URI to embed connection details.
type The type of a destination. Supported types: topic, queue.
name The name of a topic. It has the following formats:

Google Pub/Sub

  • {name} - topic name. Google project name is taken from the provider definition.
  • projects/{project}/topics/{name} - where project is a Google project and name is a topic name.
Note: Names must start with a letter, and contain only the following characters: letters, numbers, dashes (-), periods (.), underscores (_), tildes (~), percents (%) or plus signs (+). Cannot start with goog.

AWS SQS

Names can be a combination of
  • Alphanumeric characters
  • '-' (dash)
  • '_' (underscore)
Note: The '.' (dot) is not valid in SQS names!
You can set topic name as the SQS ARN link in the following format: arn:aws:sqs:<region>:<account>:<name>
Note: You can use SQS ARN link only on the already existing queue.

Messaging Destination Provider as URI

Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration. The format of the messaging-provider-agnostic URI is as follows:

<scheme>://<username>:<password>@<host>

The following table explains the fields of the messaging-provider-agnostic URI:

Table 5. Messaging Provider Fields
Field Description
scheme A valid scheme. Supported scheme: google, aws.
username and password

Google

The client email and private key are escaped by URI rules and separated by a colon.
AWS
  • The access-key and secret key are separated by a colon.
  • AWS Role ARN and Role External ID (format: GUID) are separated by a colon.
host Google

The default GCP project for the Pub/Sub topic.

AWS

The AWS region name for the SQS queue.

The following is an example of Google provider definition in URI style:

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project
The following is an example of AWS provider definition in URI style:
aws://AccessKey:SecretKey@us-east-1 [Option 1]
aws://username:password@region [Option 2]
The following example defines the url_encoded role as username and External ID as password:
"aws://arn%3Aaws%3Aiam%3A%3ACUSTOMER-AWS-ACCOUNT-ID%3Arole%2FRoleToStreamToQueue:07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d@us-east-1"

Encoding

When using URI style configuration, it is important to encode characters that are not legal within URI fields. The common forms of encoding are as follows:
      !              %21
      "              %22
      #              %23
      $              %24
      %              %25
      &              %26
      '              %27
      (              %28
      )              %29
      *              %2A
      +              %2B
      ,              %2C
      -              %2D
      .              %2E
      /              %2F
      :              %3A
      ;              %3B
      <              %3C
      =              %3D
      >              %3E
      ?              %3F
      @              %40
      [              %5B
      \              %5C
      ]              %5D
      {              %7B
      |              %7C
      }              %7D

The following commands can be used for encoding:

Python 2

echo '789secretkey999' | python2 -c 'import urllib, sys; sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in sys.stdin)'

Python 3

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys; sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

Messaging Configuration in the RDM Tenant

To ensure that RDM events can be streamed to the specified destination, the following configuration must be performed on the RDM tenant. RDM API allows to configure streaming lookups changes in RDM tenant into supported messaging destinations by users of this tenant with appropriate permissions.

{
  "enabled": true,
  "providers":
  {
    "<messaging_provider_alias>":
    {
      "type": "<provider_type>",
      "host": "<provider_host>",
      "username": "<provider_username>",
      "password": "<provider_password>"
    }
  },
  "destinations": [
    {
      "provider": "<provider_alias_or_uri>",
      "type": "<destination_type>",
      "name": "<topic>"
    }
  ]
}

The following table explains the fields that are part of the messaging configuration:

Table 6. Messaging Configuration Fields
Field Description
enabled Enable or disable RDM events streaming.
providers A map of messaging providers.
destination A list of messaging destinations.

Messaging APIs

Get Messaging Configuration

This API allows to get the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

  • Permission: rdm:config.messaging
  • Privilege: READ

Request

GET https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Update Messaging Configuration

This API allows to update the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

  • Permission: rdm:config.messaging
  • Privilege: UPDATE

Request

PUT https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging
Body
{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}