Message Streaming Provider

In most cases, the provider is a simple string alias that matches a configuration section from the MESSAGING_CONFIG_NAME variable.

Assuming that an environment's default provider has been configured, the provider alias may be omitted from the configuration.

If the list of providers must be overridden for some reason, the following feature may be leveraged. Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration.

The form of the URI is:

aws://AccessKey:SecretKey@us-east-1

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

azure://username:password@my-namespace

kafka-plain://username:password@host:9092

A valid URI must include the following sections:

The Table1: Parameter explains the URI fields and relevant description of the parameters.

Table 1. Parameters
URI field	Description	Example
`scheme`	This field defines a valid broker type and protocol. Currently, `AWS`, `Google`, `Azure` and `Confluent Kafka` are supported.	`aws:`
`username` and `password`	AWS: The access key and secret key, separated by a colon. Google Cloud Platform (GCP): The client email and private key, separated by a colon. Azure: The Shared Access Signature (SAS). Kafka: The API key (username) and API secret (password), separated by a colon.	`AccessKey:SecretKey`
`host` and `port`	AWS: The AWS region name for the Simple Notification Service (SNS) topic/ Simple Queue Service (SQS). GCP: The GCP default project for the PubSub topic. Azure: The namespace name. Kafka: The Kafka server in `host:port` format.	`us-east-1`

Use this simple Python2 command shown as following:

echo '789secretkey999' | python2 -c 'import urllib, sys;
        sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in
    sys.stdin)'

The following example displays how the Python3 works:

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys;
        sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'

When you are using URI style configuration, it is important to encode characters that are not legal within URI fields. The most common case is the occurrence of a slash / character in Amazon Secret-Keys. Common encodings are listed below.

You can use an external tool that automatically encodes your URI. These types of tools provide you with an encoded URI, ready to be copy and pasted.

Alternatively, use the following list of special characters that must be encoded with the corresponding ASCII format as displayed in the Table2: Encodings.

Table 2. Encodings
Special Characters	Encodings
!	%21
"	%22
#	%23
$	%24
%	%25
&	%26
'	%27
(	%28
)	%29
*	%2A
+	%2B
,	%2C
-	%2D
.	%2E
/	%2F
:	%3A
;	%3B
<	%3C
=	%3D
>	%3E
?	%3F
@	%40
[	%5B
\	%5C
]	%5D
{	%7B
\|	%7C
}	%7D

Note: If the provider URI is not parsed, due to syntax errors or not having special characters encoded, an error will be logged. The messages will not be sent to that provider's destinations.

Note: Confluent Kafka providers use topic-based messaging instead of queue-based messaging. However, they follow the same destination configuration model used for other streaming providers.

Type

The type of destination may be provider specific. If this is omitted, the default type is queue. Supported types are:

queue- supported by Amazon SQS
topic- supported by all: Amazon SNS, and Google PubSub

Name

The name of the destination can be any arbitrary string.

Character restrictions may be provider specific.

The Provider Restriction table displays the list of providers along with the character restrictions.

Table 3. Provider Restriction
`Provider RestrictionJMS`	The names can be a combination of alphanumeric characters. For example, the '.' (dot), '-'(dash) or '_' (underscore) characters.
`AWS`	The names can be a combination of alphanumeric characters and the '-' (dash) or '_' (underscore) characters. Note: The '.' (dot) is invalid in `SNS`/`SQS` names.
`GCP`	Names must start with a letter, and includes only the following characters: letters numbers dashes (-) periods (.) underscores (_) tildes (~) percents (%) plus signs (+) Cannot start with goog.
`Azure`	Names can include letters, numbers, periods (.), hyphens (-), underscores (_), and slashes (/), up to 260 characters. Must start and end with a letter or number. Queue names are also case-insensitive.

Note: It is recommended to use valid name identifiers instead oif simple names:

The Restriction for Queue Name table displays the naming convention for queue names.

Table 4. Restriction for Queue Name
`JMS`	Just a simple string name, there is no special ID format.
`AWS`	SNS/SQS ARN in the following format: `arn:aws:sns:<region>:<account>:<name> / arn:aws:sqs:<region>:<account>:<name>`
`GCP`	GCP resource identifier for PubSub topics: `projects/{project}/topics/{name}`
`Azure`	Azure resource identifier for queues or topics: `<namespace>:{name}`

The naming convention is observed when assigning destination names to tenants. It is especially important to maintain uniqueness, as multiple tenants with the same destination name will have their data mixed within the queue or topic. A suggested convention is to use a character-separated sequence which includes the destination-type and tenant-id. It is followed by a descriptive queue-name.

The example for a JMSis: queue.TestTenant.allEvents. The example for an SQS is: queue_TestTenant_allEvents.

Unify and manage your data

Message Streaming Provider

Footer