Set up automated pipelines

Learn how to create a document pipeline so Unstructured augmentation can process documents from a configured source by using a published extraction template.

A pipeline is configured to use a selected source and a published template to process documents. It also lets you define the following settings:

Crosswalk source, which defaults to the document source
Whether to create a data change request before publishing extracted data to Reltio
An optional schedule for pipeline runs

The following roles use automated pipelines:

Data Steward

For role definitions, see About roles.

Prerequisites

Before you begin, ensure you meet the following requirements:

You must have an active Reltio tenant with Unstructured augmentation enabled.
You must have a published extraction template and a configured document source for the pipeline.
For Google Drive, you must have the ID of the folder that contains the documents to process.

To set up a pipeline

In Unstructured augmentation, select Unstructured extraction.
Under Automated Bulk Extraction, in the Setup automated pipelines section, select Let's get started. The Create New Pipeline page is displayed.
In Select Source, select the source type you want to use: AWS S3, GCS or Google Drive.
Search and select the source from the list of configured sources.
In the Folder path to be included field, enter the folder path for the pipeline. You can leave this field empty if the documents are stored directly in the bucket. For Google Drive, enter the Google Drive folder ID in this field.
Select Continue to open the Select template page.
1. In the Template field, select the published template you want to use for the pipeline.
Select Continue to open the Save & Schedule page.
1. In the Pipeline Name field, enter a name that identifies the pipeline.
2. In the Description field, enter a short summary of what the pipeline does.
3. In the Crosswalk field, select the Reltio crosswalk source. The pipeline uses this crosswalk source when it publishes the extracted data to Reltio. By default, the field is set to the document source, such as AWS S3, GCS, or Google Drive. You can select a different crosswalk source.
4. In Do you want to make this a DCR?, select Yes to create a data change request (DCR) before publishing the extracted data to Reltio, or select No to publish without a data change request.
  - If you select Yes, set Enable Reltio Workflow for this DCR? to Disabled or Enabled. If you enable Reltio Workflow for the DCR, use the default reviewer or assign a specific reviewer.
    If you enable DCR, select Default to assign the review task to the default candidate group. Select Specific user to assign the task to an individual review and enter the reviewer's email below.
5. In Do you want to schedule this workflow?, select Yes to run the pipeline on a schedule, or select No to run it manually.
  - If you select Yes, enter the schedule in the Cron expression field.
Select Save Pipeline.

Result

The pipeline is saved and available to process of documents. For more information, see Review pipeline executions.

Unify and manage your data

Set up automated pipelines

Footer