Increase in demand for Data Platform automation

Reading Time: 5 minutes

Recently I have taken part in various discussions about why there is an increase in demand for Data Platform automation. Mostly linked to CI/CD deployments and working in a DevOps related way.

In this post I want to share my personal thoughts about this, as well as some advice. Towards the end I mention a couple of conferences happening in November as well. Due to my involvement with them.

I know this will not apply to some of you reading this. Because of your current skillsets.

However, this will apply to a large amount of Data Platform professionals. Because as much as you might want to keep just working with data your employers will expect this. Either now or in the future.

With this in mind, I have a clear aim for this post. Which is to get the message across that the demand for this is increasing and to encourage people to learn more about it.

It is a known fact that more companies are moving away from doing manual deployments. Instead, they are expecting Data Platform professionals to do automated deployments. For all kinds of Microsoft Data Platform offerings. Including SQL Server, Azure Data Factory and Azure Synapse Analytics.

A lot of companies are taking steps to ensure that Infrastructure as Code is becoming the standard. I got reminded about this fact whilst writing my last post. Where I introduced some Azure DevOps templates for Data Platform deployments that I now share on GitHub.

Examples

For example, say that a company wants to deploy Azure services based on the Microsoft solution idea online for an advanced analytics architecture. Nowadays the expectation is that the suggested services are deployed together to different environments. Like in the below diagram.

Usually this is done within a service or feature that supports CI/CD deployments using Infrastructure as Code. Like Azure Pipelines or GitHub Actions.

However, a lot of companies want more than just the services themselves deployed automatically. Now there is an increase in demand for updates to objects within these services to be deployed using some form of CI/CD as well.

For example, changes made to the schema of an Azure SQL Database are expected to be deployed across the different environments.

If a new table was created in the Azure SQL Database in the development resource group, then that update would be deployed to the other Azure SQL Databases in the other environments. Like in the below diagram.

I want to make one key point about the above diagrams. Even though they both state that approval is required to production it does not have to be a manual approval. Companies can put in automated approvals based on various things.

For instance, in Azure DevOps they can create an automated approval that queries Azure Monitor. So that it can check there are no performance issues in the other environments before deployments happen.

On a side note, I wrote a post about relating to this a while back. It is about how to deploy to Azure SQL Database using GitHub Actions.

Manual Data Platform builds

Like others, in the past I use to manually build Data Platform servers for use. For example, servers running Hadoop or SQL Server. I spent a lot of time configuring servers to perfection, as if they were a work of art.

However, more companies are now seeing the benefits of automating these deployments. In fact, some companies only allow deployments that are securely automated.

It is easier to do with modern technology. Especially with more companies moving to the cloud.

I noticed this some time ago. Which is why I put in a reasonable amount of effort in gaining the MCSD Azure Architect certification.

In addition, I put the effort into making the most out of some of the modern toolsets to manage your application lifecycle management. To keep up with the increase in demand. For example, Azure DevOps and GitHub. To the extent that I found out how to use the below services in Azure DevOps together.

It has allowed me to deliver some great things for companies. Due to the increase in demand for these skills. In addition, it has opened some very interesting doors along the way.

For example, I have provided customized Azure DevOps training days for a client. In addition, I have written a whitepaper for SogetiLabs which is a quick introduction to Azure DevOps.

I see it as a healthy addition to the work I do with the Microsoft Data Platform.

My advice about the increase in demand

My advice is to at least gain a basic understanding of how these technologies work if you think there is even a slight chance this will ever apply to you. In case you need to build on it.

So that if you discover your company is making this a standard you can build on that knowledge. To set realistic expectations, this is more likely to be “when” and not an “if” for certain roles.

Because demand for using CI/CD and working in a DevOps related way is increasing. Feel free to check this yourself by looking online or asking your peers. Because I am finding that I am talking to more members of the Data Platform community about it these days. Especially those who have a SQL Server background.

I suspect some of you reading this and are thinking that if this ever happens at your workplace you will simply change companies.

However, here is the thing. Odds are that if you change companies and accept a similar role elsewhere that does not do automated deployments yet, you might get a future surprise. Because the chances are you will also be expected to do it there at some stage in the future as well.

November Data Platform events

In reality, I would love to cover more about the topics in this post in November at the next DataWeekender. However, I cannot because I am an organizer. It did cross my mind that I could be an anonymous co-presenter. However, we need to give other speakers opportunities.

So instead, I will mention that the call for speakers has just finished and you can expect to see the schedule soon.

You can register to attend using this Meetup link or the logo below.

I should point out that I am presenting about a topic relating to the increase in demand for Data Platform automation at the PASS Data Community Summit 2021. Which takes place the week after DataWeekender. I am co-presenting with Sander Stad.

We will be covering using parts of GitHub and Azure DevOps together for CI/CD deployments. We cover various topics in this session including work items and of course pipelines. Based on real-life experience.

You can read the session details on the PASS Data Community Summit website. Feel free to come along and watch. Because I will be showing an interesting alternative to the above diagram. In addition, you can ask me questions along the way as well.

Final words about increase in demand

I hope that my personal thoughts about the increase in demand for Data Platform automation has given some of you food for thought. Because it has been on my mind more since my last post and I wanted to express my thoughts about it.

I mentioned DataWeekender because I would like to see some more sessions relating to this.

Of course, if you have any questions or comments about this post feel free to reach out to me.