Operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps

Reading Time: 6 minutes

In this post I want to show how you can operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps. Which I exclusively revealed at Power BI Gebruikersdag over the weekend.

Just so that everybody is aware, fabric-cicd is a Python library that allows you to perform CI/CD of various Microsoft Fabric items into Microsoft Fabric workspaces. At this moment in time there is a limited number of supported item types. However, that list is increasing.

I recently shared the results of my initial tests of fabric-cicd. Now I want to show how you can implement fabric-cicd in an efficient manner. In order to deploy to multiple Microsoft Fabric workspaces with Azure DevOps in a clean and more modular way.

Like in the below diagram. Where I assume you have development branch in a Git repository. preferably updated via the recommended development process by Microsoft.

To manage expectations, this post shows how to do it with the GUI-based Releases feature for deployments. Along the way I share plenty of links.

If you need help with any jargon used in this post, then I recommend that you read one my other posts. Which is a Microsoft Fabric Git integration jargon guide for Fabricators.

Operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps

To make fabric-cicd operate efficiently with Azure Pipelines I first copied some of the sample items into a Git repository in Azure DevOps.

I then created a new release pipeline. Within the pipeline, I added the existing repository as the artifact. Selecting the dev branch as the source for deployments.

Once I added my artifact I created two stages. One for Test and one for Production. I then had to add the tasks to both stages.

Both stages had the below tasks. Only slight difference in the task names was to reflect different destinations.

Below is a breakdown of each task.

Use Python 3.11

First task to run is to select the version of Python to work with via the Python version task. If you do not select a version of Python for this particular method, you will get an error about invalid syntax.

Install necessary libraries

Afterwards I had to install the necessary libraries in a PowerShell task. In this task I ran the below code.

python -m pip install --upgrade pip
pip install fabric-cicd
pip install azure-identity

I needed fabric-cicd since that was the main library required in order to perform this. In addition, I needed to install azure-identity in order for my Python script to utilize the authenticated service principal. Which occurs in the task that follows.

Authenticate as a service principal

Once the libraries are installed the next task authenticates as a service principal with the below PowerShell code.

Install-Module -Name Az.Accounts -AllowClobber -Force

$SecureStringPwd = ConvertTo-SecureString $(AZURE_CLIENT_SECRET) -AsPlainText -Force
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(AZURE_CLIENT_ID), $SecureStringPwd
                  
Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(AZURE_TENANT_ID)

$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

As you can see you need three Azure Pipeline variables added for this task:

• AZURE_CLIENT_ID – Your service principal client ID.
• AZURE_CLIENT_SECRET – Your service principal secret. Note this is the secret value.
• AZURE_TENANT_ID – Your Microsoft Entra tenant ID.

One key point I must stress here is that your service principal must have permissions to change the Microsoft Fabric workspace in question.

Anyway, I chose to do this with PowerShell since this also method allows you to authenticate with trial E5 subscriptions. However, those operating with a regular tenant can also configure an Azure DevOps service connection as an alternative to this step.

In addition, these variables are capitalized due to my testing methods. Feel free to change them as required.

Run script to deploy with fabric-cicd

In the final stage I call the Python script that orchestrates the deployment with fabric-cicd with a Python script task. With the script path configured to “$(System.DefaultWorkingDirectory)/_fabric-cicd-sample/auth_spn_secret_AzDo.py”.

I decided on this naming convention since it is a modified version of the sample ‘auth_spn_secret.py’ provided with the fabric-cicd GitHub repository. Which I changed to work with Azure DevOps in a more efficient manner.

For each environment I pass through the below arguments. In order to pass parameters through to the Python script.

--WorkspaceId $(TestWorkspace) --Environment "Test" --RepositoryDirectory "$(System.DefaultWorkingDirectory)\_fabric-cicd-sample\workspace" --ItemsInScope  $(ItemsInScope)

Note that you need the below variables configured for this to work:

• TestWorkspace – GUID value of the workspace destination. In this case this variable is for my Test workspace. However, it can be the variable of your choice.
• ItemsInScope- All of the Fabric items that you want to be deployed. For this example, I opted for the below items in the below format:
  “Notebook,Environment,Report,SemanticModel”

Once the tasks were in place it was time to configure my modified Python script.

Modified Python script to operationalize fabric-cicd

Once everything else is configured the final piece of the puzzle is to modify the Python deployment script.

First of all, I ran the below code at the top of my Python script.

from azure.identity import DefaultAzureCredential
from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items
import argparse

Which does the below actions (in line order):

1. Imports the DefaultAzureCredential class from the Azure SDK for Python. Allowing me to pass through the authenticate service principal properly.
2. Imports items from fabric-cicd library into my script.
3. Allows me to work with the argparse module. So that I can work with the pipeline variables that I specified.

Afterwards, I pass through the variables that I specified in the arguments section to run the Python script. As per the below code.

parser = argparse.ArgumentParser(description='Process some variables.')
parser.add_argument('--WorkspaceId', type=str)
parser.add_argument('--Environment', type=str)
parser.add_argument('--RepositoryDirectory', type=str)
parser.add_argument('--ItemsInScope', type=str)
args = parser.parse_args()

I then convert the items I want to be deployed as a list with the split functionality. In order for it to be properly recognized by the item_type_in_scope parameter.

# Convert item_type_in_scope into a list
allitems = args.ItemsInScope
item_type_in_scope=allitems.split(",")
print(item_type_in_scope)

Once done I authenticate the currently logged in service principal with DefaultAzureCredential.

# Authenticate with DefaultAzureCredential authenticated by PowerShell
credential = DefaultAzureCredential()

From there, I initialize the FabricWorkspace object with the passed through variables and the converted list.

target_workspace = FabricWorkspace(
    workspace_id= args.WorkspaceId,
    environment=args.Environment,
    repository_directory=args.RepositoryDirectory,
    item_type_in_scope=item_type_in_scope,    
)

I can then publish and unpublish items as per the original sample code.

# # # Publish all items defined in item_type_in_scope
publish_all_items(target_workspace)

# # # Unpublish all items defined in item_type_in_scope not found in repository
unpublish_all_orphan_items(target_workspace)

Once done, I was ready to run my pipeline.

Test results to operationalize fabric-cicd to work with Microsoft Fabric

When I run the pipeline the two stages complete like in the below example.

When I checked the two Fabric workspaces, I saw that all the items had deployed to it.

However, those of you who attended my session at Power BI Gebruikersdag know that I took my testing one step further. By subtly changing the sample Power BI report as below.

I then changed my “parameter.yml” file to include the below statements. In order to utilize the find_replace functionality.

find_replace:
    # Change report text
    "This is a Report in Dev!!!":
        Test: "This is a Report in Test!!!"
        Prod: "This is a Report in Prod!!!"

Which meant that after running my pipeline again my report in my Test workspace was subtly different.

Of course, this was just a basic example to highlight the possibilities of the find_replace functionality.

Conclusion of test results to operationalize fabric-cicd

By following the above, you can operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps. Bear in mind that not all Fabric items are supported yet. However, this is ideal for the supported item types specified on the fabric-cicd page.

Of course, what I share can also be customized to suit your needs. Some things you can consider doing is as follows:

• Keep an eye on fabric-cicd updates.
• YAML pipeline instead of Release pipeline.
• Create a service connection in Azure DevOps that authenticates as a service principal.
• Test with your own development items configured with Microsoft Fabric Git integration.
• Alternative variables to suit your own strategy. Plus, alternative methods to pass through the parameters.

Final words

I hope that showing this way to operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps helps some of you get started. Plus, inspire some of you to experiment in your own environments.

Because I do believe that this Python library has a lot of potential.

Of course, if you have any comments or queries about this post feel free to reach out to me.