Data Quality Rules Transformation
Last updated
Last updated
© Copyright 2023, Astera Software
A Data Quality Rules transformation can be used to apply one or more validation checks to incoming records. If a data quality rule fails for a record, the corresponding error message is attached to the data record. This error message as well as the record status can be mapped in downstream transformations. You can also use this information to route and filter records as well as create custom reject files and logs.
To add a Data Quality Rules transformation, drag the Data Quality Rules object from the Data Profiling group in the Flow toolbox and drop it on the dataflow.
An example of what a Data Quality Rules object might look like is shown below.
To configure the properties of a Data Quality Rules object after it was added to the dataflow, right-click on it and select Properties from the context menu. The following properties are available:
Meta Object Builder screen: Meta Object Builder screen allows you to add or remove fields in the field layout, as well as select their data type.
Note: To quickly add fields to the layout, drag and drop the node Output port of the object whose layout you wish to replicate into the node Input port of the Data Quality Rules object. The fields added this way show in the list of fields inside the node and as well as in the Meta Object Builder.
The Data Quality Rules screen provides the interface to manage your data quality rules.
Type a descriptive name for the rule in the Description field.
Using the Attach Rule to the Field dropdown, select which field will be returned in the transfer Profile, should the rule fail.
In the Expression input, enter the expression making the rule. For example, LTV > 60 and LTV <= 80. Or click … to open Expression Builder, a tool that allows you to visually build your rule using Record tree and Intellisense.
Click Compile to check for any syntax errors in your rule. The Status should read Successful for a successful compilation.
If your data quality rule has compiled successfully, enter a short message in the Show Message input. This custom message is displayed in the transfer Profile when a record fails the expression criteria. (If your data quality rule has not compiled successfully, correct the errors and click Compile again).
To activate the rule, check the Active switch.
If the Do not process record if rule fails switch is checked, a record that fails the data quality rule will be filtered out by this object and will not make it to downstream objects on the dataflow.
If the Do not process record if rule fails switch is cleared, a record that fails the data quality rule will be assigned the Warning status, but will not be filtered out by this object.
Click the icon to create a new data quality rule.
Add other data quality rules if necessary. To delete an existing data quality rule, select it and click the icon.
