Using Data Quality Mode
Data Quality Mode
In addition to standard logging functionality, Astera provides a special data quality mode which is useful for advanced profiling or debugging. When a dataflow is used in data quality mode, most objects on the dataflow show the Messages node with output ports. An example of an object with the Messages node is shown below.
Connecting the Messages node’s outputs to another object’s input ports on the dataflow makes it possible to get both summary and record-level statistics for the object’s record set, which is useful for analysis or debugging.
The Messages node captures the following statistical data:
TotalCount
ErrorCount
WarningCount
InfoCount
MessagesText
The metrics listed above aggregate data for the entire record set.
In addition, FirstItem, LastItem, and Items sub-nodes provide a way to collect quality control data on a record-by-record basis. The quality control data, such as ElementName, MessageType, or Action, can be written into a dedicated destination object for record-keeping purposes.
Usage
An example of a dataflow using data quality mode is shown below. Notice that all dataflow objects have Messages node enabled.
Last updated
