Streaming RDM Events
RDM API can stream events that contain changes to create, update and delete lookup values in Pub/Sub topics.
The changes to lookup values (create, update, delete) can be streamed to any Google Pub/Sub topics and is not limited to Reltio Google Project. Each change is streamed as a separate message to the Google Pub/Sub topics and AWS streaming.
Event Format
The event streamed from RDM has the following format:
{
"type": <event_type>,
"object": <changed_object>
}
The following table explains the fields in the event:
Field | Description |
---|---|
type |
Event type. The following types are supported:
|
object |
Contains details about the created, updated, or deleted lookup value. |
The following is an example of an event when a lookup value is created:
{
"type": "LOOKUP_VALUE_CREATED",
"object":
{
"tenantId": "rdmabc",
"type": "rdm/lookupTypes/Gender",
"code": "M",
"enabled": true,
"sourceMappings": [
{
"source": "Reltio",
"values": [
{
"code": "MALE",
"value": "Male",
"description": "",
"enabled": true,
"canonicalValue": true,
"downStreamDefaultValue": true
}
]
}
],
"localizations": [
{
"languageCode": "sv-se",
"value": "Manlig"
}
],
"startDate": 0,
"endDate": 0,
"updatedBy": "test.user",
"updateDate": 1573118722139,
"version": 1
}
}
Messaging Provider
type
host
username
password
type
specified.The following table explains the values to be provided for the Google Pub/Sub messaging provider:
Field | Description |
---|---|
type |
The message provider type. Supported type:
google
|
host |
The name of the default GCP project in which the topics are created. |
username |
The client email from GCP service account key JSON with Pub/Sub permissions. |
password |
The private key from GCP service account key JSON with Pub/Sub
permissions. For example, GCP account must have the following permissions:
Note: You must remove the new line breaks from the original key
in addition to the first and the last extra
lines. |
AWS
The following table explains the values to be provided for the AWS messaging provider:
Property | Description |
---|---|
type | Represents the messaging provider type. Only the aws type is supported. |
host | Represents the name of the Amazon region in which the queues will be created. |
username | Represents the Amazon Access Key for an account with
Simple Queue Service (SQS) permission or or the Amazon Role
Application reference number (ARN). For example, arn:aws:iam::CUSTOMER-AWS-ACCOUNT-ID:role/RoleToStreamToQueue. |
password | Represents the Amazon Secret Key for an account with
SQS permission, for corresponding Amazon Access
Key or the Amazon Role External ID in GUID format. For example, 07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d. |
For more information on delegating access using roles, see : IAM Tutorial: Delegate access across AWS accounts using IAM roles.
The policy with following permissions are attached with the roles provided:
- sqs:SendMessage
- sqs:GetQueueUrl
- sqs:GetQueueAttributes
The following permissions are granted when the queue does not exist. The queues are created automatically before the first message is sent.
- sqs:CreateQueue
- sqs:SetQueueAttributes
Messaging Destination
provider
type
name
Field | Description |
---|---|
provider |
The alias of the provider, as configured for the tenant or a
special messaging-provider-agnostic URI to embed
connection details. |
type |
The type of a destination. Supported types: topic, queue. |
name |
The name of a topic. It has the following formats: Google Pub/Sub
Note: Names must start with a letter, and contain only the following
characters: letters, numbers, dashes (-), periods (.),
underscores (_), tildes (~), percents (%) or plus signs (+).
Cannot start with goog. AWS SQS Names can be a combination of
Note: The '.' (dot) is not valid in SQS
names!
You can set topic name as the SQS ARN link in the
following format:
arn:aws:sqs:<region>:<account>:<name> Note: You
can use SQS ARN link only on the already existing
queue. |
Messaging Destination Provider as URI
Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration. The format of the messaging-provider-agnostic URI is as follows:
<scheme>://<username>:<password>@<host>
The following table explains the fields of the
messaging-provider-agnostic
URI:
Field | Description |
---|---|
scheme |
A valid scheme. Supported scheme: google, aws. |
username and password |
AWS
|
host |
Google The default GCP project for the Pub/Sub topic. AWS The AWS region name for the SQS queue. |
The following is an example of Google provider definition in URI style:
google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project
The following is an example of AWS provider definition in URI
style:aws://AccessKey:SecretKey@us-east-1 [Option 1]
aws://username:password@region [Option 2]
"aws://arn%3Aaws%3Aiam%3A%3ACUSTOMER-AWS-ACCOUNT-ID%3Arole%2FRoleToStreamToQueue:07515ab2-f3f0-4ac2-a7cf-3fe58e3b2b4d@us-east-1"
Encoding
When using URI style configuration, it is important to encode characters that are not legal within URI fields. The common forms of encoding are as follows: ! %21
" %22
# %23
$ %24
% %25
& %26
' %27
( %28
) %29
* %2A
+ %2B
, %2C
- %2D
. %2E
/ %2F
: %3A
; %3B
< %3C
= %3D
> %3E
? %3F
@ %40
[ %5B
\ %5C
] %5D
{ %7B
| %7C
} %7D
The following commands can be used for encoding:
Python 2
echo '789secretkey999' | python2 -c 'import urllib, sys; sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in sys.stdin)'
Python 3
echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys; sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'
Messaging Configuration in the RDM Tenant
To ensure that RDM events can be streamed to the specified destination, the following configuration must be performed on the RDM tenant. RDM API allows to configure streaming lookups changes in RDM tenant into supported messaging destinations by users of this tenant with appropriate permissions.
{
"enabled": true,
"providers":
{
"<messaging_provider_alias>":
{
"type": "<provider_type>",
"host": "<provider_host>",
"username": "<provider_username>",
"password": "<provider_password>"
}
},
"destinations": [
{
"provider": "<provider_alias_or_uri>",
"type": "<destination_type>",
"name": "<topic>"
}
]
}
The following table explains the fields that are part of the messaging configuration:
Field | Description |
---|---|
enabled |
Enable or disable RDM events streaming. |
providers |
A map of messaging providers. |
destination |
A list of messaging destinations. |
Messaging APIs
Get Messaging Configuration
This API allows to get the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:
- Permission:
rdm:config.messaging
- Privilege:
READ
Request
GET https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging
Response
{
"enabled": true,
"providers":
{
"some-provider":
{
"type": "google",
"host": "some-google-project",
"username": "service-account@email",
"password": "private_key"
}
},
"destinations": [
{
"provider": "some-provider",
"type": "topic",
"name": "topicId"
},
{
"provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
"type": "topic",
"name": "topicIdInMyProject"
}
]
}
Update Messaging Configuration
This API allows to update the messaging configuration of the RDM tenant. The user running this API must have the following permission and privilege:
- Permission:
rdm:config.messaging
- Privilege:
UPDATE
Request
PUT https://{{rdm-service}}/configuration/{{rdm_tenant_name}}/messaging
Body
{
"enabled": true,
"providers":
{
"some-provider":
{
"type": "google",
"host": "some-google-project",
"username": "service-account@email",
"password": "private_key"
}
},
"destinations": [
{
"provider": "some-provider",
"type": "topic",
"name": "topicId"
},
{
"provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
"type": "topic",
"name": "topicIdInMyProject"
}
]
}
Response
{
"enabled": true,
"providers":
{
"some-provider":
{
"type": "google",
"host": "some-google-project",
"username": "service-account@email",
"password": "private_key"
}
},
"destinations": [
{
"provider": "some-provider",
"type": "topic",
"name": "topicId"
},
{
"provider": "google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project",
"type": "topic",
"name": "topicIdInMyProject"
}
]
}