Azure Test Plans example for Azure Data Factory

Reading Time: 4 minutes

In this post I want to cover an Azure Test Plans example for Azure Data Factory as part of a series of posts. For a few reasons.

Firstly, because I want to do a series of posts about how Azure Test Plans can be used for deployments to various Data Platform services. So that others gain ideas of how to use Azure Test Plans.

Secondly, I want to introduce some Azure Data services to people because I know some people have not seen them before. Finally, because I promised a Data Platform MVP that I would do a Power BI example, which I will cover in a later post.

In reality, I have already done a couple of SQL Server examples. Including one in my post about using Azure Test Plans for Data Platform deployments.

With this in mind, I am going to cover examples based on other Data Platform services. Which are Azure Data Factory, Azure Synapse Analytics and Power BI.

You can find links to the other two posts in this series below:

• Azure Test Plans example for Azure Synapse Analytics
• Azure Test Plans example for Power BI

In addition, each example in this series will use a different type of test suite. For more about the different test suites you can read my post about Azure Test Plans jargon for Data Platform professionals.

Azure Test Plans example for Azure Data Factory

In this example I use a static test suite with Azure Data Factory. Which is a cloud-based service which allows you to move data around. Those of you with a SQL Server background can think of it along the lines of a cloud-based version of SSIS. However, it is different in both functionality and use.

For this basic example I created a mapping data flow, which allows me to get data from two datasets and put them into an Azure SQL database.

In the example below, the two sources are two tables in the AdventureWorksLt sample database.

If you go from left to right below you can see that I first filter the Product table for only road bikes. Afterwards, I perform a lookup to the Model table. I then select which columns I actually want to keep with a Select modifier. Once done it goes to a new table in the RoadBike database.

In reality, the two sources above can be from a number of other places instead of Azure SQL Databases. For example, flat files and Snowflake. You can view the list for which sources you can use in the Azure Data Factory and Azure Synapse Analytics connector overview.

Azure Test Plans

In Azure Test Plans within Azure Devops we can create a static test suite to check that this data flow works after creating it in various ways.

For example, we can test that Linked services exist and that the connections are setup correctly. Which can be useful if your company policy states they have to be setup using key vault.

From there you can build up your test case to test various stages of your data flow as below.

Once you are happy with your data flow you can add a step to test it like I have done as the final step above. Which you can do by adding it as a data flow activity in a pipeline and clicking ‘Debug’.

Of course, this is just a basic example. You can expand on this by having multiple test cases inside your static test suite for various pipelines. Because it’s a static test suite you can add as many as different ones as needed.

More about Azure Data Factory

If you want to learn more about Azure Data Factory, Microsoft Learn has a learning path for you. It’s called Data integration at scale with Azure Data Factory or Azure Synapse Pipeline.

In addition, there are various books available. Including a recent one by Richard Swinbank called Azure Data Factory by Example. Which I have been recommending to others who are learning Azure Data Factory. It contains hands-on examples.

Final words

I hope this Azure Test Plans example for Azure Data Factory has been an interesting insight.

Because, I had two aims for this post. To give others’ ideas about how Azure Test Plans can be used with Azure Data Factory and to introduce some to Azure Data Factory.

Of course, if you have any comments or queries about this post feel free to reach out to me.