Creating Workflows in Astera
Last updated
Last updated
A workflow enables automated and repeated execution of a sequence of tasks, such as running programs, sending emails, uploading files, running a transfer setting or batch, executing SQL code, and many others.
The tasks run according to some predefined path and follow the custom logic that controls under what circumstances specific paths should be activated.
You can add any number of tasks on a single visual workflow diagram, as well as specify what should happen at each step depending on the successful or erroneous completion of a task.
You can also route workflows one way or another by using Boolean conditions that are suitable for your scenario. You can even call other workflows or dataflows to run within your main workflow.
A workflow can be created quickly and easily in Astera's intuitive drag-and-drop environment. The Astera Data Stack workflow editor allows you to copy or move workflow tasks, source objects, or workflow parameters, change their properties, and perform a number of other actions with the capability of unlimited undo-redo of any previous actions.
A workflow makes it easy to visualize and implement complex sequences of tasks because pieces of a workflow can be used like ‘building blocks’. These building blocks can be pasted into different workflows, or even into the same workflow, which allows you to quickly replicate tasks, or even sequences of tasks, with similar properties.
Objects can be added to a workflow in several ways, such as direct drag and drop of files from any Explorer window, drag-and-drop of tables or views from the built-in Data Source Browser, or by adding an object from the Toolbox.
A workflow that you create in Astera can run on local or remote servers. The workflow can also be ported to run in any number of target environments. This is achieved by using run-time parameters which change values depending on the specific context in which the workflow will run.
Finally, to make sure that your workflow is continuously up to date with your changing environment, you can easily update data connections, such as server names, login credentials, etc. throughout the entire workflow, using a single easy-to-use interface.
This means that your workflow will stay in sync with the current requirements, even if the data connections have changed since the workflow was created.
To create a new workflow, go to File > New > Workflow on the main menu. Alternatively, you can expand the Create a New Dataflow dropdown icon on the main toolbar and select Workflow as shown below.
A workflow always has at least one workflow task and may have none, one, or multiple sources. Sources, workflow tasks, and resources are represented as objects in the workflow.
Depending on the type of object, an object can be added to the workflow in one of the following ways:
Using drag-and-drop: You can drag a flat file type (Excel, Delimited, Fixed-Length File) listed below from an Explorer window and drop it onto an open workflow tab in Astera.
The advantage of using the drag-and-drop method, compared to other methods, is that many of the object’s properties are pre-populated for you based on the file’s content. For example, the field layout is automatically filled out so that there is no need to create it manually.
Note: XML files cannot be added to a dataflow using the drag-and-drop method since they require a schema to be defined first for the layout.
Using the Toolbox: You can add a source object by selecting it from the appropriate category in the Toolbox.
For example, to add a source comma-delimited file object, expand the Sources group in the workflow Toolbox, and drag-and-drop the Delimited File Source object onto the workflow.
Note that an object added this way initially does not have any properties defined. To define its properties, double-click on the object’s header or right-click on its header and select Properties from the context menu.
In the Properties screen that opens, select the File Path of the file that will be associated with the object. Field layout and other properties can then be populated based on the file’s content.
Copying and pasting an existing object from the same or different workflow or dataflow: If your source is already defined in the same or different workflow (or dataflow), you can copy the existing object and paste it into your workflow.
The object being copied retains the properties of the original object and is assigned a unique new name to distinguish it from the original object.
Using the Toolbox: To add an XML source to the workflow, use the XML/JSON File Source object in the Sources section of Toolbox.
Note that the XML File object initially will not have any properties defined. To define its properties, double-click on the object’s header or right-click, and select Properties from the context menu.
On the Properties window that opens, select the File Path of the XML file that will be associated with the object. Additionally, you need to provide the path to the XSD schema that defines the layout of your XML file.
As with flat files, you can copy and paste an existing XML object from the same or different workflow or dataflow. The object being copied retains the properties of the original object and is assigned a unique new name to distinguish it from the original object.
Using drag-and-drop: You can drag a database table or view from the Data Source Browser and drop it on an open workflow tab.
To open the Data Source Browser, go to View > Data Source Browser. Connect to the server, then expand the Database tree and expand Tables (or Views) tree to select your table (or view). Drag-and-drop the selected table or view to the workflow.
By default, the database table is added as a Database Table Source object.
To add a data model source, press and hold the ‘Ctrl’ key while dragging and dropping a table (or view) from the Data Source Browser.
As with files, you can copy and paste an existing database table object from the same or different workflow or dataflow. The object being copied retains the properties of the original object and is assigned a unique new name to distinguish it from the original object.
Using the Toolbox: You can add an object by selecting it from the appropriate category in the Toolbox.
Note that an object added this way initially does not have any properties defined. To define its properties, double-click on the object’s header or right-click, and select Properties from the context menu.
Copying and pasting an existing object from the same or different workflow or dataflow: If your object is already defined in the same or different workflow (or dataflow), you can copy the existing object and paste it into your workflow.
The object being copied retains the properties of the original object and is assigned a unique new name to distinguish it from the original object.
The workflow editor supports unlimited undo and redo capability. You can quickly undo or redo the last action done, or undo/redo several actions at once.
To undo the last action, open the Edit menu and select Undo. You can also click the Undo icon on the Workflow toolbar. You can also use the CTRL+Z shortcut. To undo several actions at once, select the first action you wish to undo from the Undo dropdown. The actions following the selected action will also be undone.
To redo the last action, open the Edit menu and select Redo. You can also click the Redo icon on the Workflow toolbar. You can also use the CTRL+Y shortcut. To redo several actions at once, select the first action you wish to redo from the Redo dropdown. The actions following the selected action will also be redone.
Using the copy and paste feature, you can replicate an object on your workflow by copying it into a new object with a different name to distinguish it from the original object.
You can paste this object into the same workflow, or a different workflow.
You can also copy several objects at once. To do so, click the objects you wish to copy while pressing the Ctrl key. Or, you can draw a rectangle with your mouse while holding down the LEFT mouse button. The objects inside the rectangle will be selected. Right-click on a selected object and select Copy from the context menu. Then right-click on white space in the same or different workflow, and select Paste from the context menu.
To move an object or a set of objects, use the cut-and-paste sequence similar to the one described above.
Note: When you move objects, they keep their original names.
Note: You can use CTRL+C shortcut to copy the selected object into the clipboard. CTRL+V will paste it from the clipboard. CTRL+X will cut it onto the clipboard.
The Auto Layout feature allows you to arrange objects on the workflow improving its visual representation.
To invoke the Auto Layout feature, click the Auto Layout Diagram icon on the Workflow toolbar. Or you can open the Workflow menu and select Auto Layout.
Note: You can manually move the object around the workflow by holding the LEFT mouse button down over the object title and moving it to a new location.
The following tools are available in the Workflow menu to help you adjust the display size of the dataflow:
Zoom In
Zoom Out
Fit To Screen
Additionally, you can select a custom zoom percentage using the Zoom % input on the Workflow toolbar.
Any workflow task objects you add to the workflow will show Input and Output ports. Ports make it possible to connect an object to another object via links, creating the required workflow path.
The Input port on an object allows you to connect it to an upstream object on the workflow. Conversely, the Output port allows you to connect your object to a downstream object on the workflow. The downstream object representing a task will be processed after the upstream object it is connected to has finished running (with some exceptions, such as the Run Program task, which may not require waiting for the program to exit).
Source and resource objects have no ports and cannot be linked to other objects on the workflow. These types of objects cannot be linked to any specific object as they provide values that can be used by any task throughout the workflow.
All source objects added to the workflow are used in Singleton mode only. A Singleton mode object only returns the first record of data (excluding the header) from the object. This makes it possible for you to get the values from the first record anywhere on the workflow, using the following notation: ObjectName.FieldName.
Similar to Source objects, Resource Objects, such as Parameters or Context Info, provide parameters used throughout the workflow. Parameters have a similar notation of ObjectName.ParameterName.
To create links between two objects, drag the Output port of the upstream object and drop it on the Input port of the downstream object.
To remove a link between two objects, right-click on the link and select Delete, or left-click on the link and press the DEL key on the keyboard.
Note: By default, the link of Normal type is created between two objects. Normal link means that the downstream object will be processed in the event of a successful completion of the upstream object to which it is connected. Normal links are displayed in green.
In contrast, an Error link can be created between two objects, meaning that the downstream object will be processed only in the event of a failed (error) status of the upstream object to which it is connected. Error links are displayed in orange.
To change the link type, right-click on the link and select Change Link Type.
An example of linked workflow tasks is shown below.
To open an object’s properties, double-click on the object’s header. Alternatively, you can right-click on the object’s header and select Properties from the context menu.
Note: On the Properties screen, you can switch to the Properties of another object on the workflow. To do so, select an object in the Editing dropdown.
The following functions are common for many types of objects on the workflow. The following functions are available in the context menu:
Rename – Renames the object.
Edit File – Only applies to files. It opens the file for editing in a new tab.
Edit Schema – Only applies to XML files. It opens the XSD file for editing in a new tab.
View Table Data – Only applies to database tables. It opens the table for viewing in a new tab.
View Table Schema – Only applies to database tables. It opens the database schema for viewing in a new tab.
Delete – Deletes the object from the workflow. This will also remove any links to and from the object.
Cut – Removes the object from the workflow and places it into the clipboard. The object can then be ‘pasted’.
Copy – Copies the object into the clipboard leaving the original object as it is. A copy of the object can then be ‘pasted’.
Paste - Pastes the object from the clipboard.
Verifying a workflow will list any errors or warnings present in the workflow design. Correct any such errors or warnings, and verify your workflow again to ensure there are no errors.
To verify a workflow, click the Start Verification icon on the main toolbar. Verification results will be displayed in the Verify window.
To run your workflow, click the Start Workflow icon on the main toolbar.
To stop a workflow that is currently running, click the Stop Job icon in the Job Progress window toolbar.
Each source on the workflow is represented as a source object. You can have any number of sources in a workflow, but they can only be used in Singleton mode. A Singleton mode source only returns the first record of data (excluding the header) from the source.
This makes it possible for you to get the values from the first record anywhere on the workflow, using the following notation: ObjectName.FieldName.
The following source types are supported by the workflow engine:
All sources can be added to the workflow by picking a source type from Toolbox and dropping it on the workflow. File sources can also be added by dragging and dropping a file from the Explorer window. Database sources can be drag-and-dropped from the Data Source Browser.
