Variables

The Variable object in Dataprep allows you to parameterize values such as file paths, dataset names, or filter conditions, that can be reused throughout your recipe without reconfiguration. By using variables, you can make your recipes dynamic, flexible, and easier to manage when integrated into dataflows or workflows.

Example Use Case

Imagine you receive monthly sales files that all need the same cleansing and transformation. Without variables, you’d have to create a separate recipe for each file, which is time-consuming and hard to maintain.

This is where variables help. Instead of hardcoding the file path in your recipe, you can define a variable and use it as the input source. When the recipe is run in a dataflow, the variable can be mapped to different files or even an entire directory.

Creating a Variable

1. Open your recipe in Dataprep.

1. Navigate to Define > Variable in the toolbar.

1. In the Recipe Configuration – Variable window:

• Name: Give the variable a descriptive name.

• Type: Choose a data type (e.g., String, Integer, Boolean).

• Value: Enter the value of this variable. For this use case, we can paste the file path of one of the sources here.

1. Click Apply to save the variable.

Using Variables in Recipes

1. Add a File Source object.

1. Set File Location to Path from Variable.

2. Select the variable you defined.

3. Provide a dataset name and click Apply.

1. Perform the required cleansing and transformation steps on this dataset. This recipe will later be reused for other files through the variable.

Using Variables in Dataflows

Once the Dataprep recipe has been configured, create a dataflow to process all datasets using this recipe. To do this:

1. Add a Dataflow to your project.

2. Drag-and-drop a Dataprep Source object onto the designer, right-click on the header and select Properties from the context menu.

1. Provide the file path to the Dataprep recipe and click OK.

1. Right-click on the Dataprep source object and select Transformation. This will allow you to map any inputs to the source object.

1. Drag-and-drop a File System Items source object from the toolbox onto the designer and configure it to the directory where the datasets are stored.

1. Map the Full Path to the File Path in the Dataprep Source object’s input node.

1. Your Dataprep Source has now been configured successfully, and you can now write the output to a destination.

Now, your dataflow dynamically processes all datasets in the folder using the same recipe.