Data Quality

What is Data Quality?

Data Quality applies one or more conditions or rules, called Data Quality Rules, on different file columns. It measures data conditions based on factors such as accuracy, completeness, consistency, reliability, and whether it's up to date.

Previously, we generated the data profile for a specific data asset. Now, we will be adding some validation checks and running them.

1. Navigate to the Data Quality page from the tab above.

You will see an empty grid since no validation rules have been added here. Furthermore, the timestamp has already been updated since we generated the data profile. The Data Profile and Data Quality grids are controlled collectively.

1. Click on Add Rule to create Data Quality Rule(s).

A panel to add the rule will appear at the bottom of the page.

1. Enter the Name, Select Field from 'Attach Rule To Field' dropdown, description (optional) and then click Define Expression.

A window will appear where you can search for expressions through the Search text box, select a specific Category from the 'Categories' dropdown, or manually write it in the expression box.

1. Define the expression and verify the Compile status.

1. Next, click on Confirm.

The provided expression has been added to the Add Rule panel.

1. Then, click Add.

You will receive a message stating that the rule has been added, and it will get populated in the Data Quality grid.

1. Now, we can run the rule we have just added. Click on the Run icon.

You will receive a message stating that the job has been scheduled successfully. At this point, you cannot add any more rules until the run is complete.

1. After completion, the validity will be populated, along with the date timestamp of when the Data Quality was completed.

This validity percentage signifies the percentage of records that satisfy the defined rule.

Editing and Deleting Rules

You can edit or delete this rule through the commands in the Action column.

1. Click on the edit icon.

The edit panel will appear populated with the current configurations of the rule.

1. Edit the rule details to calculate the percentage of orders received in 2020.

1. The update button will now be enabled. Click Update.

The rule has now been updated. Note that the old validity is not populated anymore since an edit was made.

1. Run this rule to populate the updated percentage.

1. To delete the rule, click the Delete icon.

1. Then click Yes on the confirmation prompt.

The rule will be deleted.

Now that we have seen how to run both, Data Profile and Data Quality rules, let's explore how we can schedule these to run automatically in the following article.