# Data Quality Mode

In addition to the standard logging functionality, Astera provides a special *Data Quality Mode* option, useful for advanced profiling and debugging.

When a dataflow is created/opened in the *Data Quality Mode*, most objects on the dataflow show the *Messages* node with output ports.

**Sample Use-Case**

In this case, we have a simple dataflow designed to perform a data quality check. It contains customers’ data coming in from an [*Excel Workbook Source*](https://documentation.astera.com/astera-data-stack-v9/dataflows/sources/excel-workbook-source). A [*Data Quality Rule*](https://documentation.astera.com/astera-data-stack-v9/dataflows/data-logging-and-profiling/using-data-quality-rules-in-astera) object is added to validate data for null values and perform warning checks.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2F4a1kLUGi8UxN88H9983o%2F0.png?alt=media)

If you preview the *Customers* dataset output at this stage, you will see that some of the records have missing values in the *Region* and *Fax* fields.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FFB4DBlYFryNpQSLc9Gfs%2F1.png?alt=media)

Data quality rules are set so that records with empty *Region* values are marked as errors and records with empty *Fax* values are marked as warnings.

A red exclamation sign in the *Data Preview* window identifies the records that have failed to match the rule and returned an error or a warning as a result.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FSpkshyCnQSt6goqBHcY4%2F2.png?alt=media)

Now, for instance, we want to collect information regarding the number of errors/warnings in a single record along with the error/warning messages attached to these records, and write this information to a destination. For this purpose, we will use *Data Quality Mode*.

**Note:** The *Record Level Log* feature also collects and records this information but we can not further process it in the dataflow.

**Activating Data Quality Mode**

1. To activate this feature, click on the *Data Quality Mode* ![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2F03Spwy5vEjGZi00OBUfm%2F3.png?alt=media)icon located at the top of the dataflow designer.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FPlh8PV6PtdaUtbjwWDwE%2F4.png?alt=media)

Once the *Data Quality Mode* is activated, a *Messages* node will be added to all the objects in the dataflow.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FTuqlOcl8ggHRfudBtUs1%2F5.png?alt=media)

The *Messages* node captures the following statistical information:

* *TotalCount*
* *ErrorCount*
* *WarningCount*
* *InfoCount*
* *MessagesText*
* *DbAction*
* *Custom*

In addition, *FirstItem*, *LastItem*, and *Items* sub-nodes provide a way to collect quality control data for each of the records. The quality control data includes *ElementName*, *MessageType*, *Action,* etc., and can be written to a destination object for record-keeping purposes.

**Writing to a Destination**

Connecting the *Messages* node’s output ports to another object’s input ports on the dataflow makes it possible to get both - summary statistics and record-level statistics for the dataset, which are useful for analysis and debugging. To do this:

1. Right-click on the *Messages* node inside the *NoNullValues\_Rule* object and go to *Write to > Delimited File Destination*.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FWhNEbLwuqbFJvaRImdUF%2F6.png?alt=media)

A *Delimited Destination* object is added to the designer with mapped fields.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2Fzc6a4ypkclzyZjkp0etv%2F7.png?alt=media)

2. Configure settings for the [*Delimited File Destination*](https://documentation.astera.com/astera-data-stack-v9/dataflows/destinations/delimited-file-destination) to save this data.
3. Right-click on the header of the destination object and select *Preview Output* from the context menu.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FAMQeQzl497NQZJKocW2r%2F8.png?alt=media)

A *Data Preview* window will open, showing error and warning information.

![](https://3891069449-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F28dJIkGHacXhBlux3efx%2Fuploads%2FVkidFXXHj5GJSqcLQ1pB%2F9.png?alt=media)