# Data Quality Mode

In addition to the standard logging functionality, Astera provides a special *Data Quality Mode* option, useful for advanced profiling and debugging.

When a dataflow is created/opened in the *Data Quality Mode*, most objects on the dataflow show the *Messages* node with output ports..

## **Sample Use-Case**

In this case, we have a simple dataflow designed to perform a data quality check. It contains customers’ data coming in from an [*Excel Workbook Source*](https://documentation.astera.com/v/astera-data-stack-v7/dataflows/sources/excel-workbook-source). A [*Data Quality Rule*](https://documentation.astera.com/astera-data-stack-v7/dataflows/data-logging-and-profiling/using-data-quality-rules-in-astera) object is added to validate data for null values and perform warning checks.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FvvHoBPp2dTszbdkZpcuD%2F0.png?alt=media)

If you preview the *Customers* dataset output at this stage, you will see that some of the records have missing values in the *Region* and *Fax* fields.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2Fstynq072FSqY24of09I7%2F1.png?alt=media)

Data quality rules are set so that records with empty *Region* values are marked as errors and records with empty *Fax* values are marked as warnings.

A red exclamation sign in the *Data Preview* window identifies the records that have failed to match the rule and returned an error or a warning as a result.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FWDj0HI3LuowWYGADBb60%2F2.png?alt=media)

Now, for instance, we want to collect information regarding the number of errors/warnings in a single record along with the error/warning messages attached to these records, and write this information to a destination. For this purpose, we will use *Data Quality Mode*.

{% hint style="info" %}
**Note:** The *Record Level Log* feature also collects and records this information but we can not further process it in the dataflow.
{% endhint %}

## **Activating Data Quality Mode**

1. To activate this feature, click on the *Data Quality Mode* ![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FZgqLcuV2pRKF80StaK3v%2F3.png?alt=media)icon located at the top of the dataflow designer.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FrKE2vAzUIlP4wnTTXn5O%2F4.png?alt=media)

Once the *Data Quality Mode* is activated, a *Messages* node will be added to all the objects in the dataflow.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FrEjUh2BOrR1qrSyEWNsZ%2F5.png?alt=media)

The *Messages* node captures the following statistical information:

* *TotalCount*
* *ErrorCount*
* *WarningCount*
* *InfoCount*
* *MessagesText*
* *DbAction*
* *Custom*

In addition, *FirstItem*, *LastItem*, and *Items* sub-nodes provide a way to collect quality control data for each of the records. The quality control data includes *ElementName*, *MessageType*, *Action,* etc., and can be written to a destination object for record-keeping purposes.

## **Writing to a Destination**

Connecting the *Messages* node’s output ports to another object’s input ports on the dataflow makes it possible to get both - summary statistics and record-level statistics for the dataset, which are useful for analysis and debugging. To do this:

1. Drag-and-drop *Delimited File Destination* onto the dataflow designer by going to *Toolbox > Destinations > Delimited File Destination*.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FZCxEnCqYdYfFeJ1MjWaA%2F6.png?alt=media)

2. Auto-map fields under the *Messages* node onto the destination object.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FKrupUdpOwaHf9aP7D1Qg%2F7.png?alt=media)

3. Configure settings for the [*Delimited File Destination*](https://documentation.astera.com/v/astera-data-stack-v7/dataflows/destinations/delimited-file-destination) to save this data.
4. Right-click on the header of the destination object and select *Preview Output* from the context menu.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2Fw5HfSGIVdWpzhREGP2hZ%2F8.png?alt=media)

A *Data Preview* window will open, showing error and warning information.

![](https://627607815-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6xzBT0roYJkfVS5klkLl%2Fuploads%2FC7Ory614SzfTJ1tvhyhA%2F9.png?alt=media)