Streaming RDM Events to Google Pub/Sub

RDM API can stream events that contain changes to lookup values to Google Pub/Sub topics.

The changes to lookup values (create, update, delete) can be streamed to any Google Pub/Sub topics and is not limited to Reltio Google Project. Each change is streamed as a separate message to the Google Pub/Sub topic.

Event Format

The event streamed from RDM has the following format:

{
  "type": <event_type>,
  "object": <changed_object>
}

The following table explains the fields in the event:

Table 1. Event Fields
Field Description
type Event type. The following types are supported:
  • LOOKUP_VALUE_CREATED
  • LOOKUP_VALUE_UPDATED
  • LOOKUP_VALUE_DELETED
object Contains details about the created, updated, or deleted lookup value.

The following is an example of an event when a lookup value is created:

{
  "type": "LOOKUP_VALUE_CREATED",
  "object":
    {
      "tenantId": "rdmabc",
      "type": "rdm/lookupTypes/Gender",
      "code": "M",
      "enabled": true,
      "sourceMappings": [
        {
          "source": "Reltio",
          "values": [
            {
              "code": "MALE",
              "value": "Male",
              "description": "",
              "enabled": true,
              "canonicalValue": true,
              "downStreamDefaultValue": true
            }
          ]
        }
      ],
      "localizations": [
        {
          "languageCode": "sv-se",
          "value": "Manlig"
        }
      ],
      "startDate": 0,
      "endDate": 0,
      "updatedBy": "test.user",
      "updateDate": 1573118722139,
      "version": 1
    }
}

Messaging Provider

A messaging provider contains information that allows you to connect to a messaging destination. You must define the following properties for each messaging provider:
  • type
  • host
  • username
  • password
The last three properties contain different information depending on the type specified.

The following table explains the values to be provided for the messaging provider:

Table 2. Messaging Provider
Field Description
type Supported type: google
Note: AWS and MS Azure are not currently supported.
host The name of the default GCP project in which the topics are created.
username The client email from GCP service account key JSON with Pub/Sub permissions.
password The private key from GCP service account key JSON with minimul Pub/Sub permissions, such as pubsub.topics.create, pubsub.topics.publish, and pubsub.topics.get.
Note: You must remove the new line breaks from the original key in addition to the first and the last extra lines.

Messaging Destination

A messaging destination is the Pub/Sub topic where the RDM API will send the event. You must define the following properties for each messaging destination:
  • provider
  • type
  • name
Note: The number of destinations to which RDM API can send events is limited to three.
The following table explains the values to be provided for the messaging destination properties:
Table 3. Messaging Destination Properties
Field Description
provider The alias of the provider, as configured for the tenant or a special messaging-provider-agnostic URI to embed connection details.
type The type of a destination. Supported types: topic
name The name of a topic. It has the following formats:
  • {name} - topic name. Google project name is taken from the provider definition.
  • projects/{project}/topics/{name} - where project is a Google project and name is a topic name.

Messaging Destination Provider as URI

Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration. The format of the messaging-provider-agnostic URI is as follows:

<scheme>://<username>:<password>@<host>

The following table explains the fields of the messaging-provider-agnostic URI:

Table 4. Messaging Provider Fields
Field Description
scheme A valid scheme. Supported scheme: google
username and password The client email and private key are escaped by URI rules and separated by a colon.
host The default GCP project for the Pub/Sub topic.

The following is an example of Google provider definition in URI style:

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

Encoding

When using URI style configuration, it is important to encode characters that are not legal within URI fields. The common forms of encoding are as follows:

      !              %21
      "              %22
      #              %23
      $              %24
      %              %25
      &              %26
      '              %27
      (              %28
      )              %29
      *              %2A
      +              %2B
      ,              %2C
      -              %2D
      .              %2E
      /              %2F
      :              %3A
      ;              %3B
      <              %3C
      =              %3D
      >              %3E
      ?              %3F
      @              %40
      [              %5B
      \              %5C
      ]              %5D
      {              %7B
      |              %7C
      }              %7D

The following commands can be used for encoding:

Python 2

echo '789secretkey999' | python2 -c 'import urllib, sys; sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in sys.stdin)'

Python 3

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys; sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

Messaging Configuration in the RDM Tenant

To ensure that RDM events can be streamed to the specified destination, the following configuration must be performed on the RDM tenant.

{
  "enabled": true,
  "providers":
  {
    "<messaging_provider_alias>":
    {
      "type": "<provider_type>",
      "host": "<provider_host>",
      "username": "<provider_username>",
      "password": "<provider_password>"
    }
  },
  "destinations": [
    {
      "provider": "<provider_alias_or_uri>",
      "type": "<destination_type>",
      "name": "<topic>"
    }
  ]
}

The following table explains the fields that are part of the messaging configuration:

Table 5. Messaging Configuration Fields
Field Description
enabled Enable or disable RDM events streaming.
providers A map of messaging providers.
destination A list of messaging destinations.

Messaging APIs

Get Messaging Configuration

This API allows to get the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

  • Permission: rdm:config.messaging
  • Privilege: READ

Request

GET https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Update Messaging Configuration

This API allows to update the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:

  • Permission: rdm:config.messaging
  • Privilege: UPDATE

Request

PUT https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging
Body
{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}

Response

{
  "enabled": true,
  "providers":
  {
    "some-provider":
    {
      "type": "google",
      "host": "some-google-project",
      "username": "service-account@email",
      "password": "private_key"
    }
  },
  "destinations": [
    {
      "provider": "some-provider",
      "type": "topic",
      "name": "topicId"
    },
    {
      "provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
      "type": "topic",
      "name": "topicIdInMyProject"
    }
  ]
}