Message Streaming Provider

In most cases, the provider is a simple string alias that matches a configuration section from the MESSAGING_CONFIG_NAME variable.

Assuming that an environment's default provider has been configured, the provider alias may be omitted from the configuration.

If the list of providers must be overridden for some reason, the following feature may be leveraged. Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration.

The form of the URI is:


activemq:jms://user:pass@test.reltio.com:61616

aws://AccessKey:SecretKey@us-east-1

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

A valid URI must include the following sections:

The Table1: Parameter explains the URI fields and relevant description of the parameters.

Table 1. Parameters
URI field	Description	Example
`scheme`	This field defines a valid broker type and protocol. Currently, `activemq:jms`,`AWS`, `Google` and `Azure` are supported.	`activemq:jms`
`username` and `password`	ActiveMQ: The username and password (separated by a colon) configured for queue or topic access within the broker. AWS: The access-key and secret key, separated by a colon. The Google Cloud Platform (GCP): The client email and private key (in a plain form with first or last lines with line breaks removed. It is escaped by URI rules), separated by a colon. Azure: The Shared Access Signature (SAS).	`user:pass`
`host` and `port`	ActiveMQ: The host and port of the message broker (separated by a colon). AWS: The AWS region name for the Simple Notification Service (SNS) topic/ Simple Queue Service (SQS). GCP: The GCP default project for the PubSub topic. Azure	`test.reltio.com:61616`

Use this simple Python2 command shown as following:

echo '789secretkey999' | python2 -c 'import urllib, sys;
        sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in
    sys.stdin)'

The following example displays how the Python3 works:

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys;
        sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

When you are using URI style configuration, it is important to encode characters that are not legal within URI fields. The most common case is the occurrence of a slash / character in Amazon Secret-Keys. Common encodings are listed below.

Use the following list of special characters that must be encoded with the corresponding ASCII format as displayed in the Table2: Encodings.

Table 2. Encodings
Special Characters	Encodings
!	%21
"	%22
#	%23
$	%24
%	%25
&	%26
'	%27
(	%28
)	%29
*	%2A
+	%2B
,	%2C
-	%2D
.	%2E
/	%2F
:	%3A
;	%3B
<	%3C
=	%3D
>	%3E
?	%3F
@	%40
[	%5B
\	%5C
]	%5D
{	%7B
\|	%7C
}	%7D

Note: If the provider URI is not parsed, due to syntax errors or unencoded special characters, an error will be logged. The messages will not be sent to that provider's destinations.

Type

The type of destination may be provider specific. If this is omitted, the default type is queue. Supported types are:

queue- supported by both ActiveMQ and Amazon SQS
topic- supported by all: ActiveMQ, Amazon SNS, and Google PubSub

Name

The name of the destination can be any arbitrary string.

Character restrictions may be provider specific.

The Provider Restriction table displays the list of providers along with the character restrictions.

Table 3. Provider Restriction
`Provider RestrictionJMS`	The names can be a combination of alphanumeric characters. For example, the '.' (dot), '-'(dash) or '_' (underscore) characters.
`AWS`	The names can be a combination of alphanumeric characters and the '-' (dash) or '_' (underscore) characters. Note: The '.' (dot) is invalid in `SNS`/`SQS` names.
`GCP`	Names must start with a letter, and includes only the following characters: letters numbers dashes (-) periods (.) underscores (_) tildes (~) percents (%) plus signs (+) Cannot start with goog.
`Azure`	Names can include letters, numbers, periods (.), hyphens (-), underscores (_), and slashes (/), up to 260 characters. Must start and end with a letter or number. Queue names are also case-insensitive.

Note: It is recommended to use valid name identifiers instead oif simple names:

The Restriction for Queue Name table displays the naming convention for queue names.

Table 4. Restriction for Queue Name
`JMS`	Just a simple string name, there is no special ID format.
`AWS`	SNS/SQS ARN in the following format: `arn:aws:sns:<region>:<account>:<name> / arn:aws:sqs:<region>:<account>:<name>`
`GCP`	GCP resource identifier for PubSub topics: `projects/{project}/topics/{name}`
`Azure`	Azure resource identifier for queues or topics: `<namespace>:{name}`

The naming convention is observed when assigning destination names to tenants. It is especially important to maintain uniqueness, as multiple tenants with the same destination name will have their data mixed within the queue or topic. A suggested convention is to use a character-separated sequence which includes the destination-type and tenant-id. It is followed by a descriptive queue-name.

The example for a JMSis: queue.TestTenant.allEvents. The example for an SQS is: queue_TestTenant_allEvents.

Accelerate the Value of Data

Message Streaming Provider

Footer