Using Data Quality Mode

In addition to standard logging functionality, Astera provides a special data quality mode which is useful for advanced profiling or debugging. When a dataflow is used in data quality mode, most objects on the dataflow show the Messages node with output ports. An example of an object with the Messages node is shown below:

Connecting the Messages node’s outputs to another object’s input ports on the dataflow makes it possible to get both summary and record-level statistics for the object’s record set, which is useful for analysis or debugging.

The Messages node captures the following statistical data:

• TotalCount

• ErrorCount

• WarningCount

• InfoCount

• MessagesText

The metrics listed above aggregate data for the entire record set.

In addition, FirstItem, LastItem, and Items sub-nodes provide a way to collect quality control data on a record-by-record basis. The quality control data, such as ElementName, MessageType, or Action, can be written into a dedicated destination object for record-keeping purposes.

To turn on data quality mode, click the icon a second time turns data quality mode off.

Usage

An example of a dataflow using data quality mode is shown below. Notice that all dataflow objects have Messages node enabled.