# Data Quality Mode

In addition to the standard logging functionality, Astera provides a special *Data Quality Mode* option, useful for advanced profiling and debugging. When a dataflow is created/opened in *Data Quality Mode*, most objects on the dataflow show the *Messages* node with output ports.

In this document, we will learn how to use the *Data Quality Mode* in Astera.

### Sample Use-Case

In this case, we have a simple dataflow designed to perform a data quality check. It contains customers’ data coming in from an [*Excel Workbook Source*](https://documentation.astera.com/dataflows/sources/excel-workbook-source). A [*Data Quality Rule*](https://documentation.astera.com/dataflows/data-logging-and-profiling/using-data-quality-rules-in-astera) object is added to validate data for null values and perform warning checks.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/TCiuUYsdVRiIlb4lXzwW/1_use_case.PNG)

If you preview the *Customers* dataset output at this stage, you will see that some of the records have missing values in the *Region* and *Fax* fields.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/xjS5gkNdy9m1depxOGx3/2_data.png)

Data quality rules are set so that records with empty *Region* values are marked as errors and records with empty *Fax* values are marked as warnings. A red exclamation sign in the *Data Preview* window identifies the records that have failed to match the rule and returned an error or a warning as a result.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/YytqP3uGUVdn7hNgmALC/3.PNG)

Now, for instance, we want to collect information regarding the number of errors/warnings in a single record, the error/warning messages attached to these records, and write this information to a destination. For this purpose, we will use *Data Quality Mode*.

{% hint style="info" %}
**Note:** The *Record Level Log* feature also collects and records this information but we can not further process it in the dataflow.
{% endhint %}

#### Activating Data Quality Mode

1. To activate this feature, click on the *Data Quality Mode* <img src="https://docs.astera.com/projects/centerprise/en/10/_images/10icon.PNG" alt="10icon" data-size="line"> icon located at the top of the dataflow designer.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/ZICxrlWPNRdDmzdztyN8/4_dq_mode.png)

Once *Data Quality Mode* is activated, a *Messages* node will be added to all the objects in the dataflow.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/33tGJRWAJdyI4e4Jssv3/5_messages_node.png)

The *Messages* node captures the following statistical information:

* *TotalCount*
* *ErrorCount*
* *WarningCount*
* *InfoCount*
* *MessagesText*
* *DbAction*
* *Custom*

In addition, *FirstItem*, *LastItem*, and *Items* sub-nodes provide a way to collect quality control data for each of the records. The quality control data includes *ElementName*, *MessageType*, or *Action* etc. and can be written to a destination object for record-keeping purposes.

#### Writing to a Destination

Connecting the *Messages* node’s output ports to another object’s input ports on the dataflow makes it possible to get both - summary statistics and record-level statistics for the dataset, which are useful for analysis and debugging. To do this:

1. Right-click on the *Messages* node inside the *NoNullValues\_Rule* object and go to *Write to > Delimited File Destination*.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/Ur47P6X7PVVpHNkBGRsU/6_destination.png)

A *Delimited Destination* object is added to the designer with mapped fields.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/jZUd69kMdjXK0x6CM2YS/7_mappings.PNG)

2. Configure settings for the [*Delimited File Destination*](https://documentation.astera.com/dataflows/destinations/delimited-file-destination) to save this data.
3. Right-click on the header of the destination object and select *Preview Output* from the context menu.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/OOhtjpDGfHFOk9znwDmM/8_properties.png)

A *Data Preview* window will open, showing error and warning information.

![](https://content.gitbook.com/content/zEifS4h8yurLAAwiGNX2/blobs/arCqDcYth3ScbFJdjUtS/9_preview.PNG)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://documentation.astera.com/dataflows/data-logging-and-profiling/data-quality-mode.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
