Using Azure Test Plans for Data Platform deployments

Reading Time: 7 minutes

In this post I want to cover using Azure Test Plans for Data Platform deployments. Because using it to manage test plans can be very useful.

By the end of this post, you will know what Azure Test Plans are and how they can be useful for data Platform deployments.

In reality, you can use a lot of this post to use Azure Test Plans with other types of deployments as well.

To clarify, Azure Test Plans is a service which is available within Azure DevOps. You can use it to manage your manual tests.

To use some of the features mentioned in this post you do need either a Visual Studio subscription, MSDN Platforms subscription or a Basic + Test Plans Azure DevOps license.

One way you can check your access level is within the ‘Users’ section in Organization Settings as below. You can read more about this in the Microsoft guide about manual test permissions and access.

In addition, when I say Data Platform in this post, I mean the Microsoft Data Platform. Because I refer to services like SQL Server and Azure Synapse Analytics.

Testing Data Platform deployments

I want to cover why you should test Data Platform deployments briefly here. Because there are some very good reasons for doing so.

Over the years people have tended to shy away from doing various tests for Data Platform deployments. However, due to a higher adoption of CI/CD pipelines more people are doing it in some form. More people are doing automated unit tests and user acceptance tests.

For example, the below highlights a unit test taking place within Azure Pipelines. In this example, unit tests are performed against a SQL Server database using tSQLt.

Testing is an important part of building up confidence with your stakeholders and colleagues.

Building confidence example

For example, say you are looking to work in a DevOps related way and intend to have fully (or partially) automated deployments. So that you automatically deploy to servers in various environments one environment at a time. For instance, a Development, User Acceptance Test (UAT) and Production environments.

However, one or more team members have concerns. In the short-term team members can check the work that has been completed in each environment. By doing something along the lines of a user acceptance test.

You can also configure your pipeline with approvals. Giving them a chance to do these tests before they approve the pipeline so that it can continue. For instance, setting up the pipeline so that they must approve releases to the Production environment after they have done tests in the UAT environment.

Doing this builds up the trust until it gets to a stage where they are confident enough that manual approvals are no longer required. Or at least a stage where manual approvals are only required for Production.

In addition, exploratory testing can be useful when doing Data Platform deployments. Because you can get feedback from stakeholders.

Using spreadsheets for test plans

In reality, one option is to manage these kinds of tests using spreadsheets. However, that approach can introduce issues of its own. Especially with the methods people have used over the years to manage them.

To get a better idea of the sprawl that can happen you can read a list of methods to manage scripts I put together. It’s in my post about how SQL Server Professionals can benefit from using version control.

Why Azure Test Plans for Data Platform deployments?

Azure Test Plans is great for the above scenarios. It’s even better of you are using the other services in Azure DevOps already.

Especially Azure Boards. Because the two services are tightly integrated.

In fact, as you can see below you can only use Azure Test Plans inside an Azure DevOps project if Azure Boards is enabled.

You can link test plans to work items a few ways in Azure DevOps. For example, say you are using the Boards feature in Azure Boards to work in a Kanban or Scrumban kind of way. You can quickly add a Test case by clicking on the ellipsis in the work item and select ‘Add Test’.

However, that test case only applies for that particular work item. For a more complex tests, I recommend creating test plans in Azure Test Plans.

Create test plan in Azure Test Plans

When you create a new test plan you have to define it before you can execute it. Easiest way to start is to click the ‘New Test Case’ button.

Once done you can create a test case. Like the one I have done below. Note that I have added an attachment to one of the steps.

After setting up the test case I can go to the ‘Execute’ section and click on ‘Run for web application’ to run it.

When I run this the picture that I attached is now visible in the web application. Some of you might recognize this image from my post about deploying to multiple SQL Server database types using Azure DevOps. Where I deployed to various SQL Server related services like Azure SQL Database and Azure Synapse Analytics.

In reality, it can be an image of anything. Including a GitHub Action deployment or a screenshot of an Azure Data Factory pipeline. To save downloading the image I can hover over it in the application to zoom in instead.

You can do a lot within the Runner application. It’s fairly easy to understand as well. Click on a tick if everything is OK, otherwise click on the cross. You can also add text about a check as well.

You can also create a bug in Azure Boards as well. If you click on ‘Create bug’ a new bug window appears. It automatically includes progress within your test case as well, as you can see below.

You can add the usual work item links to this bug as well. Including links to any potential fixes.

Data Platform example

Using Azure Test Plans with other services in Azure DevOps can help you complete part of your application lifecycle management for Data Platform deployments. Below is a very basic flow of work to give you an idea of how this can work with Azure DevOps.

Below is an example of how you can use the above diagram for a SQL Server database update.

1. First, a work item is created in Azure Boards to update a table in a SQL Server database.
   Somebody updates the code for the database using their clone (copy) of the repository in Visual Studio.
   Once finished they commit (save) their work locally and synchronize their clone (copy) with the repository in Azure Repos.
   Once done, a link to the commit is added to the work item within Azure Boards. Which is why the arrow goes both ways.
2. Meanwhile, the commit triggers a CI/CD pipeline to start in Azure Pipelines. Deploying the update(s) to multiple SQL Server databases.
3. After the updates are done, somebody runs a test plan created in Azure Test Plans.
   Checking that the pipeline has completed properly and that the databases are all OK.
4. If any bugs are found in the test case, they are reported back to Azure Boards as a work item. Starting a new flow of work.

Now I must state the above is a very simplified version. In real-life it tends to be more complex. However, this example helps you visualize how Azure Test Plans can be used for Data Platform deployments.

Using GitHub with Azure Test Plans

In reality, you can use a modified version of the flow above if your repository is in GitHub and you work with GitHub Actions. It’s even easier to do this if you have your GitHub account connected to your Azure DevOps project. You can set this up using the GitHub connections setting within Project settings.

Setting this up makes it a lot easier to implement the below flow of work. Because it means you can do some additional things. Like adding links to GitHub commits directly in work items that are in Azure Boards.

Azure Test Plans jargon

I do apologize for not fully explaining some jargon in this post. However, I wanted to focus on how you can use Azure Test Plans for Data Platform deployments.

In reality, explaining all the jargon will make this post longer. So, I will explain some jargon in a future post.

Final words about using Azure Test Plans for Data Platform deployments

I hope this post about using Azure Test Plans for Data Platform deployments has inspired some of you. Because I think it’s a service that can help many Data Platform professionals.

In fact, I’m very interested to hear from people who currently use Azure Test Plans. So, if this includes you feel free to comment.

Of course, if you have any other comments or queries about this post feel free to reach out to me.