Filter Transformation

Filter transformation is similar to data quality rules. But unlike data quality rules, which return an error or warning when the rule condition fails while still passing the record downstream, the Filter transformation will completely filter out any such records. The filtered records, as well as their status, will not be accessible to any downstream object on the dataflow, including any type of log.

Steps

To add a Filter transformation, drag the Filter object from the Transformations group in the Flow toolbox and drop it on the dataflow.

An example of what a Filter object might look like is shown below:

To configure the properties of a Filter object after it was added to the dataflow, right-click on it and select Properties from the context menu. The following properties are available:

• Meta Object Builder screen: It is used to add or remove fields in the field layout, as well as select their data type.

Note: To quickly add fields to the layout, drag and drop the node Output port of the object whose layout you wish to replicate into the node Input port of the Filter object. The fields added this way show in the list of fields inside the node and as well as in the Meta Object Builder.

• Filter Transformation Properties screen: Provides the interface to create your filter rule. In the Expression input, enter the expression making the filter rule. For example, LTV > 60 and LTV <= 80.

You can select from the list of available functions arranged by Category (left-hand pane), and you may also double-click on an object in the Object tree (right-hand pane) to paste the selected object into the expression.

Click Compile to check for any syntax errors in your rule. The Status should read “Successful” for a successful compilation.

Your filter expression is complete.

• General Options screen: This screen shares the options common to most objects on the dataflow:

  • Clear Incoming Record Messages: When this option is on, any messages coming in from objects preceding the current object will be cleared. This is useful when you need to capture record messages in the log generated by the current object and filter out any record messages generated earlier in the dataflow.

  • Do Not Process Records with Errors: When this option is on, records with errors will not be output by the object. When this option is off, records with errors will be output by the object, and a record message will be attached to the record. This record message can then feed into downstream objects on the dataflow, for example, a destination file that will capture record messages or a log that will capture the messages and as well as collect their statistics.

  • The Comments input allows you to enter comments associated with this object.

Usage

An example of a dataflow using a Filter to filter out column sizes of greater than 5 characters is shown below: