# Data Quality Mode

In addition to the standard logging functionality, Astera provides a special *Data Quality Mode* option, useful for advanced profiling and debugging. When a dataflow is created/opened in *Data Quality Mode*, most objects on the dataflow show the *Messages* node with output ports.

In this document, we will learn how to use the *Data Quality Mode* in Astera.

### Sample Use-Case

In this case, we have a simple dataflow designed to perform a data quality check. It contains customers’ data coming in from an [*Excel Workbook Source*](https://documentation.astera.com/dataflows/sources/excel-workbook-source). A [*Data Quality Rule*](https://documentation.astera.com/dataflows/data-logging-and-profiling/using-data-quality-rules-in-astera) object is added to validate data for null values and perform warning checks.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/TCiuUYsdVRiIlb4lXzwW/1_use_case.PNG)

If you preview the *Customers* dataset output at this stage, you will see that some of the records have missing values in the *Region* and *Fax* fields.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/xjS5gkNdy9m1depxOGx3/2_data.png)

Data quality rules are set so that records with empty *Region* values are marked as errors and records with empty *Fax* values are marked as warnings. A red exclamation sign in the *Data Preview* window identifies the records that have failed to match the rule and returned an error or a warning as a result.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/YytqP3uGUVdn7hNgmALC/3.PNG)

Now, for instance, we want to collect information regarding the number of errors/warnings in a single record, the error/warning messages attached to these records, and write this information to a destination. For this purpose, we will use *Data Quality Mode*.

{% hint style="info" %}
**Note:** The *Record Level Log* feature also collects and records this information but we can not further process it in the dataflow.
{% endhint %}

#### Activating Data Quality Mode

1. To activate this feature, click on the *Data Quality Mode* <img src="https://docs.astera.com/projects/centerprise/en/10/_images/10icon.PNG" alt="10icon" data-size="line"> icon located at the top of the dataflow designer.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/ZICxrlWPNRdDmzdztyN8/4_dq_mode.png)

Once *Data Quality Mode* is activated, a *Messages* node will be added to all the objects in the dataflow.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/33tGJRWAJdyI4e4Jssv3/5_messages_node.png)

The *Messages* node captures the following statistical information:

* *TotalCount*
* *ErrorCount*
* *WarningCount*
* *InfoCount*
* *MessagesText*
* *DbAction*
* *Custom*

In addition, *FirstItem*, *LastItem*, and *Items* sub-nodes provide a way to collect quality control data for each of the records. The quality control data includes *ElementName*, *MessageType*, or *Action* etc. and can be written to a destination object for record-keeping purposes.

#### Writing to a Destination

Connecting the *Messages* node’s output ports to another object’s input ports on the dataflow makes it possible to get both - summary statistics and record-level statistics for the dataset, which are useful for analysis and debugging. To do this:

1. Right-click on the *Messages* node inside the *NoNullValues\_Rule* object and go to *Write to > Delimited File Destination*.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/Ur47P6X7PVVpHNkBGRsU/6_destination.png)

A *Delimited Destination* object is added to the designer with mapped fields.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/jZUd69kMdjXK0x6CM2YS/7_mappings.PNG)

2. Configure settings for the [*Delimited File Destination*](https://documentation.astera.com/dataflows/destinations/delimited-file-destination) to save this data.
3. Right-click on the header of the destination object and select *Preview Output* from the context menu.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/OOhtjpDGfHFOk9znwDmM/8_properties.png)

A *Data Preview* window will open, showing error and warning information.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/arCqDcYth3ScbFJdjUtS/9_preview.PNG)