# Data Quality Mode

In addition to the standard logging functionality, Astera provides a special *Data Quality Mode* option, useful for advanced profiling and debugging. When a dataflow is created/opened in *Data Quality Mode*, most objects on the dataflow show the *Messages* node with output ports.

In this document, we will learn how to use the *Data Quality Mode* in Astera.

### Sample Use-Case

In this case, we have a simple dataflow designed to perform a data quality check. It contains customers’ data coming in from an [*Excel Workbook Source*](https://documentation.astera.com/astera-data-stack-v10/dataflows/sources/excel-workbook-source). A [*Data Quality Rule*](https://documentation.astera.com/astera-data-stack-v10/dataflows/data-logging-and-profiling/using-data-quality-rules-in-astera) object is added to validate data for null values and perform warning checks.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FwXpRigXq97wDOhgtkrkS%2F1_use_case.PNG?alt=media\&token=4d7700c3-7981-4908-bea4-71d8fc2fd5a1)

If you preview the *Customers* dataset output at this stage, you will see that some of the records have missing values in the *Region* and *Fax* fields.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FBU4fLqu5sfyY5vlIoOc9%2F2_data.png?alt=media\&token=d75f2efe-0134-4aea-8044-246f1b0c58e8)

Data quality rules are set so that records with empty *Region* values are marked as errors and records with empty *Fax* values are marked as warnings. A red exclamation sign in the *Data Preview* window identifies the records that have failed to match the rule and returned an error or a warning as a result.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FjUTJjwoniJSsuEMlImv5%2F3.PNG?alt=media\&token=de65b9db-9d9f-42d5-b69c-d931c876f3b5)

Now, for instance, we want to collect information regarding the number of errors/warnings in a single record, the error/warning messages attached to these records, and write this information to a destination. For this purpose, we will use *Data Quality Mode*.

{% hint style="info" %}
**Note:** The *Record Level Log* feature also collects and records this information but we can not further process it in the dataflow.
{% endhint %}

#### Activating Data Quality Mode

1. To activate this feature, click on the *Data Quality Mode* <img src="https://docs.astera.com/projects/centerprise/en/10/_images/10icon.PNG" alt="10icon" data-size="line"> icon located at the top of the dataflow designer.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2Ff3a988xKAtGxk16qvjzr%2F4_dq_mode.png?alt=media\&token=f7a8cc7f-a563-4ca1-9566-155aef5ab9f5)

Once *Data Quality Mode* is activated, a *Messages* node will be added to all the objects in the dataflow.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FsJD3Tx320N2yzOi3A2PT%2F5_messages_node.png?alt=media\&token=3e74ad18-83b8-4fed-b052-a1bb4c5e5a8b)

The *Messages* node captures the following statistical information:

* *TotalCount*
* *ErrorCount*
* *WarningCount*
* *InfoCount*
* *MessagesText*
* *DbAction*
* *Custom*

In addition, *FirstItem*, *LastItem*, and *Items* sub-nodes provide a way to collect quality control data for each of the records. The quality control data includes *ElementName*, *MessageType*, or *Action* etc. and can be written to a destination object for record-keeping purposes.

#### Writing to a Destination

Connecting the *Messages* node’s output ports to another object’s input ports on the dataflow makes it possible to get both - summary statistics and record-level statistics for the dataset, which are useful for analysis and debugging. To do this:

1. Right-click on the *Messages* node inside the *NoNullValues\_Rule* object and go to *Write to > Delimited File Destination*.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FEOKXUnnPupSsZXKZtY1Q%2F6_destination.png?alt=media\&token=4ffd7285-ecdf-427e-9216-b7f6eeb41096)

A *Delimited Destination* object is added to the designer with mapped fields.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FTa5he5rJzNyRCVBViNCr%2F7_mappings.PNG?alt=media\&token=94dc5eb3-f660-4830-a18d-3f163a9d12ff)

2. Configure settings for the [*Delimited File Destination*](https://documentation.astera.com/astera-data-stack-v10/dataflows/destinations/delimited-file-destination) to save this data.
3. Right-click on the header of the destination object and select *Preview Output* from the context menu.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2Fa01Dz2BH8QO4ZUO8lx7r%2F8_properties.png?alt=media\&token=9ada1252-3514-41b6-ac03-3dd55e0a961a)

A *Data Preview* window will open, showing error and warning information.

![](https://3083465318-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FsR50Wa7EwZGlmPSAMkkf%2Fuploads%2FOXPr0IoQ2adjy3Z1jmB3%2F9_preview.PNG?alt=media\&token=d76bc0b0-19e4-4aa9-9041-ec7f086f87e2)