CI/CD for Microsoft Fabric Data Warehouses using GitHub Actions

Reading Time: 6 minutes

In this post I want to cover CI/CD for Microsoft Fabric Data Warehouses using GitHub Actions. I want to do this post for various reasons.

Including the fact that when I was presenting at the Microsoft office in the Netherlands earlier this year I mentioned to somebody that this can be done in GitHub. I feel now is the right time to cover it. Since you can now select GitHub as a provider for Git integration.

Plus, you may be aware that Microsoft Fabric Git integration now supports Microsoft Fabric Data Warehouses.

When the Data Warehouse is synchronized what Git integration actually does is save a version of a database project for the Data Warehouse. Which it saves to a subfolder within the Git repository.

With the above in mind, and in order to make this post stand out compared to previous posts about performing CI/CD using GitHub Actions I intend to show the following:

1. How to work with a database project created in a subfolder within a Git repository when you configure Microsoft Fabric Git integration.
2. How to perform CI/CD for a Microsoft Fabric Data Warehouse using GitHub Actions.
3. How to use Azure SQL Deploy GitHub Action with a service principal.

So, the entire flow will look like the below example with two Data Warehouses in separate development and production workspaces.

At times I reference other posts to save repetition.

If you need help with any jargon used in this post, then I recommend that you read one my other posts. Which is a Microsoft Fabric Git integration jargon guide for Fabricators.

GitHub Actions template

I have made a template for the contents of this post publicly available in GitHub. Which you are free to download and work with. It is called GitHub-FabricDWDBProject.

However, I do have one request. If you do find the repository useful, please click on the star in the right-hand corner of the repository GitHub.

Preparing CI/CD for Microsoft Fabric Data Warehouses using GitHub Actions

First of all, I created a new workspace. I then created a sample Data Warehouse in my workspace. Ensuring that I closed the Data Warehouse once it was populated. Before I configured Microsoft Fabric Git integration.

I then connect the workspace to a new GitHub repository. For the benefit of good working practices, I configured it to work with a new dev branch.

I covered how to do this in a previous post where I show my initial tests of GitHub support for Microsoft Fabric Git integration.

After connecting the workspace to a Git repository, I went into GitHub and explored the dev branch of the Git repository. When I expanded the file tree view in GitHub, I could see a populated database project.

For full disclosure, I created the Data Warehouse first due to the fact I get an error when I try to configure Git integration first. However, configuring the workspace after a Data Warehouse had been deployed in the workspace works fine.

Anyway, once done I created a new workspace to represent a production workspace. I then added my service principal account as a contributor to the workspace and created an empty Data Warehouse.

My final step was to go into the repository settings and add a secret to the repository called FabricDW_Connection_String. Which contained the credentials to connect to the target Data Warehouse in the below format:

Server=<Fabric DW connection string>;Initial Catalog=<Fabric DW name>;Authentication=Microsoft Entra Service Principal; User ID=<clientId>; Password=<clientSecret>; Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

I was then ready to configure GitHub Actions.

Building the dacpac for Microsoft Fabric Data Warehouses using GitHub Actions (CI)

I returned to my repository GitHub, clicked on the actions tab and selected the option to setup a workflow myself. That way I had an empty workflow.

I renamed the YAML file. In order to reflect the fact that the dacpac is created from a database project. First, I added the below code. Which sets the trigger to be a pull request and specifies the workflow variables.

name: Update Fabric Data Warehouse

#Sets the trigger to update when pull request is done to the main branch
on:
  pull_request:
    branches: 
      - main
# Specify source and destination Data Warehouses
env:
  SOURCE_DW: CICDDWGH
  TARGET_DW: CICDDWGH

Afterwards, I added my first job to create the dacpac file.

jobs:

  # Job to build and publish the dacpac
  BuildDacpac:
    # I use a GitHub-Hosted runner here, you can opt to use a self-hosted one instead
    runs-on: windows-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v4.1.7

      # Install latest version of SqlPackage on the runner
      - name: Build Database project
        run: |
          dotnet tool update -g microsoft.sqlpackage

      # Build dacpac using dotnet build
      - name: Build Database project based on the contents of the Data Warehouse subfolder
        run: |

          dotnet build "${{env.SOURCE_DW}}.Warehouse"
          
      # Publish artifact
      - uses: actions/upload-artifact@v4.3.4
        with:
          name: FabricDW
          path: ${{ github.workspace }}/${{env.SOURCE_DW}}.Warehouse/bin/Debug/ 

I want to highlight a couple of key points about the above code:

• To create the database project, I specify the folder that contains the Data Warehouse database project.
• It only creates the dacpac file on the GitHub Runner, it does not bloat the original repository.

I then performed an initial pull request in GitHub from the dev to the main branch. Afterwards, I confirmed that an artifact containing the dacpac had been created.

Deploying the dacpac to a Microsoft Fabric Data Warehouse using GitHub Actions (CD)

After confirming that the dacpac had been created I added the below code to the workflow. In order to deploy the contents of the dacpac to the new Microsoft Fabric Data Warehouse.

# Deploy dacpac job
  DeployFabricDWUpdates:
    # Set the dependency for the build job
    needs: BuildDacpac
    # Easier to use Github-hosted runner if updating in GitHub
    runs-on: windows-latest

    # Steps to deploy the updates to a Microsoft Fabric Data Warehouse
    steps:
      - name: download artifact containing dacpac
        # Downloads Data artifact to dacpac folder
        uses: actions/download-artifact@v4.1.8
        with:
          name: FabricDW

      # Installs dacpac
      # For this to work you MUST have a secret specified, default in my repo is blank
      # Requires a secret SQLDB_CONNECTION_STRING which contains connection string to your Fabric Data Warehouse
      - name: Fabric Data Warehouse  Deploy
        uses: Azure/sql-action@v2.3
        with:
          # The connection string, including authentication information, for the Azure SQL Database.
          connection-string: '${{ secrets.FabricDW_Connection_String }}'
          # Path to the dacpac file in the artifact
          path: './${{env.SOURCE_DW}}.dacpac'
          # Action we want it to do, in this case 'Publish' the contents of the dacpac to the database
          action: 'publish'

I want to highlight a few key points about the above code:

• This code contains the latest versions of GitHub Actions.
• It is very similar to the code I showed in a previous post about deploying a dacpac to a serverless SQL Pool.
• As you can see, code to deploy with a service principal is the same as connecting with a regular user. Only difference is the details within the connection string that I added as a secret earlier.

Afterwards, I did a pull request from the dev to the main branch to start the workflow and confirmed it had completed successfully.

I then went into the destination Microsoft Fabric Data Warehouse and confirmed that the empty tables had been created.

Further steps

Of course, this is just one example of how it can be done. You can look to take this further by adding additional Data Warehouses or working with GitHub environments.

To test this yourself you can setup a couple of Microsoft Fabric Data Warehouses and then test with either a clone or a fork of my sample GitHub repository. Changing the variables and adding your own secret where required.

Alternatively, you can work with your own your own database project. Ensuring that you change the location of the project in the code. I covered three ways to create a Microsoft Fabric Data Warehouse Database Project in a previous post.

For additional security, I recommend you read my post about security considerations when using GitHub with Microsoft Fabric Git integration first.

Final words about CI/CD for Microsoft Fabric Data Warehouses using GitHub Actions

I hope this post about how to perform CI/CD for Microsoft Fabric Data Warehouses using GitHub Actions helps some of you.

Because I wanted to raise awareness that this was possible. Especially now that GitHub can be a provider when working with Microsoft Fabric Git integration.

Of course, if you have any comments or queries about this post feel free to reach out to me.