Workflows with a Dynamic Destination Path
Last updated
Last updated
Workflows are designed to orchestrate an automated and iterative execution of ordered tasks. When designing a workflow to orchestrate a process, a source object is required, that can provide access to a directory containing bulk load of source files of the same layout and format. A workflow is designed, such that each source file in that particular directory goes through the same data processing.
This requires a source file to be replaced in the run-time until all source files (of the same layout and format) within a directory are processed and loaded to respective destination files. Now, this further involves the associated destination files to be replaced in the run-time as well, hence making it essential to provide a dynamic destination path. For example, a source directory containing 100 excel source files in this workflow will create 100 destination files after going through the process.
In this document, we will learn to create workflows with a dynamic destination path that directs Astera to create a unique destination file for each source file.
In Astera, a dynamic destination path is achieved in two steps:
Parameterizing the dataflow through a Variables object.
Providing a run-time destination path through an Expression Transformation object in the workflow.
In this case, we are orchestrating a simple dataflow that contains customer account details coming in through an Excel Workbook Source. The source data is passed through an Aggregate Transformation and loaded to an Excel Workbook Destination.
Now, there are 100 Excel source files in a directory containing customer account details of 100 customers. Each of these Customer_Accounts files must go through the same aggregate transformation and loaded to a distinctive Excel Workbook Destination file.
For this purpose, we will add a Variables object and parameterize the source file path and destination file path in the dataflow.
Go to Toolbox > Resources > Variables, and drag-and drop the Variables object onto the dataflow designer.
The Variables object is currently empty and needs to be configured.
Right-click on the object’s header and select Properties from the context menu. A properties window will open.
On the Variable Properties window, create two new fields:
Excel_Source
Excel_Destination
Set both fields as Input. For their Default Value, paste the path for customer account file source and customer account file destination in each field, respectively.
Click OK.
Double-click on the Customer_Account source object header. An Excel Source Properties window will open. Click Next, and the Layout Builder will open. Click Next again, and the Config Parameters screen will open.
On the Config Parameters window, under the Value column for File Path, provide the name of the field enclosed in {….} that points to the source file path in the Variables object in step 3.
Now, Repeat steps 4 and 5 for the ExcelDestination object.
Our dataflow has been parameterized. The next step is to create a workflow with a dynamic destination path.
Open a new workflow by going to File > New > Workflow.
To provide access to the local source directory containing 100 customer account files, go to Toolbox > Sources > File System Item Source and drag-and-drop the File System Item Source object onto the workflow designer.
Right-click on the object’s header and select Properties from the context menu. A configuration window will open for the FileSystem object.
Provide the local directory path in File Location. Apply a Filter, *.xls and click OK.
Right-click on the FileSystem source object’s header and select Loop from the context menu. This will direct Astera to run the task in a loop, replacing the source file paths in real-time, until all source files have been processed.
Go to Toolbox > Workflow Tasks > Run Dataflow and drag-and-drop the Run Dataflow object onto the workflow designer. Map the header output port of the FileSystem source object onto the header input port of the Run Dataflow object.
Right-click on the Run Dataflow object’s header and select Properties from the context menu. Here, provide the Job Info Path for the dataflow that is being orchestrated and click OK.
Go to Toolbox > Transformation > Constant Value, drag-and-drop the Constant Value transformation object onto the workflow designer. Double-click on its header, a Constant Value Map Properties window will open.
Paste the path of the folder where Astera will load the destination files, in the Constant Value box as shown below. Click OK.
Go to Toolbox > Transformation > Expression, drag-and-drop the Expression transformation object onto the workflow designer. Double-click on its header, and a Layout Builder screen will open.
On this Layout Builder screen, create four new fields and set the data type accordingly.
FullPath
DestinationDirectory
FileName
DestinationFilePath
Click OK.
Define field mappings as follows:
FullPath field under FileSystem source object onto the FullPath field under Expression transformation object.
FullPath field under FileSystem source object onto the Excel_Source field under RunDataflow task object.
Value field under Constant Value transformation object onto the DestinationDirectory field under Expression transformation object.
DestinationFilePath field under Expression transformation object onto the Excel_Destination field under RunDataflow task object.
Double-click on the Expression transformation object’s header, to open its Layout Builder screen. Write the following expressions for these fields:
FileName : LSplit(RSplit(FullPath,2,”\”,0),2,”.”,0)
DestinationFilePath : DestinationDirectory+FileName+”.xls”
Click OK. A dynamic destination file path has been created.
Click on the Start Workflow icon located in the toolbar at the top, and execute this workflow.
This will create a unique destination file for each source file.