Operationalize fabric-cicd to work with Microsoft Fabric and YAML Pipelines

Reading Time: 8 minutes

In this post I want to show how you can operationalize fabric-cicd to work with Microsoft Fabric and YAML pipelines in Azure DevOps. As a follow up to my previous post on how to work with fabric-cicd and Classic pipelines.

I want to highlight some subtle differences in this post compared to my last post. Which are for the better. Including the new mirrored databases sample that is shared in the fabric-cicd GitHub repository.

Just so that everybody is aware, fabric-cicd is a Python library that allows you to perform CI/CD of various Microsoft Fabric items into Microsoft Fabric workspaces. At this moment in time there is a limited number of supported item types. However, that list is increasing.

Below is a diagram to help visualize this. Based on a development branch in a Git repository. Preferably updated via the recommended development process by Microsoft.

I recently shared the results of my initial tests of fabric-cicd. Now I want to show how you can implement fabric-cicd in an efficient manner.

In order to deploy to multiple Microsoft Fabric workspaces with YAML Pipelines in Azure DevOps for a cleaner and more modular approach. Working with Azure DevOps features such as variables and environments.

I share plenty of links in this post. However if you need help with any jargon, then I recommend that you read one my other posts. Which is a Microsoft Fabric Git integration jargon guide for Fabricators.

Sample Git repository

I created a Git repository to accompany this post called AzureDevOps-fabric-cicd-sample. Within it contains three different YAML files which can be used to create YAML pipelines for the below scenarios.

1. fabric-cicd-demo-variables.yml – Pipeline that contains references to Azure Pipeline variables. For scenarios where all the values are constant.
2. fabric-cicd-demo-wsparameters.yml – Pipeline that contains parameters that affect workspace values. Including workspace ID and items to deploy.
3. fabric-cicd-demo-gblparameters.yml – Pipeline that contains global parameters. Suited for Option four in the recommended CI/CD options article by Microsoft.

One quick way to get going is to import the repository into Azure DevOps. From there, create a pipeline from an existing YAML file.

When you start any of YAML pipelines in Azure DevOps you may need to permit them to access resources when you first run the pipeline. Just permit them and let the pipeline resume.

All of the sample fabric items in this repository are from the original fabric-cicd GitHub repository. I changed the Power BI report to test parameterization.

My only ask is that you give this repository a star in GitHub if it proves to be helpful.

Azure DevOps variable groups and environment

Before working on the below example, I first created two variable groups in Azure DevOps. One that contained sensitive values and one that contained non-sensitive values.

Ideally the variable group that contains sensitive values should link to secrets in Azure Key Vault. Alternatively, you can look to work with the Azure Key Vault task in your YAML pipeline instead.

In the variable group for sensitive values the below variables are in-place:

• AZURE_CLIENT_ID – Your service principal client ID.
• AZURE_CLIENT_SECRET – Your service principal secret. Note this is the secret value.
• AZURE_TENANT_ID – Your Microsoft Entra tenant ID.

Whereas the variable group for non-sensitive values contains the below:

• ItemsInScope – List of all items you want deployed. For this example, I opted for the below items in the below format:
  “Notebook,Environment,Report,SemanticModel”
• ProdEnv – Name of production environment in the context of fabric-cicd parameters. As opposed to the Azure DevOps environment.
• ProdWorkspace – GUID value of the production workspace.
• resourceUrl – URL value to get a bearer token for the specific API. In this case for “https://api.fabric.microsoft.com”.
• TestEnv – Name of production environment in the context of fabric-cicd parameters.
• TestWorkspace – GUID value of the test workspace.

I already had a Production environment configured in Azure DevOps. Along with an approval configured. One key point I must add is that the approvals and checks for deployment to an environment can be in many forms. As you can see below.

Operationalize fabric-cicd to work with Microsoft Fabric and YAML Pipelines

To make fabric-cicd operate efficiently with Azure Pipelines I first updated the sample items that I had stored in a Git repository in Azure DevOps. So that they included an additional sample for a mirrored database.

I then created a new YAML pipeline. I selected a starter pipeline. However, if you download my sample GitHub repository you can create a pipeline from an existing YAML file instead.

Once in my YAML pipeline I specified the two variable groups that I had created. Plus, the fact that I did not want a trigger at this moment in time so that I can manually run the pipeline. In addition, I specified I wanted an Azure Pipelines Agent with the latest windows image.

variables:
- group: fabric-cicd-ns
- group: fabric-cicd-s

trigger: none
# In this pipeline I use a Microsoft-hosted agent
pool: 
  vmImage: 'windows-latest'

Afterwards, I configure two stages in my YAML pipeline. One to deploy to a test workspace and another to deploy to the production workspace that is connected to an Azure DevOps environment.

Stage to deploy to a test workspace in Microsoft Fabric with fabric-cicd

I first created a stage to deploy to the test workspace. Which contained a single job.

- stage: Test
  displayName: 'Deploy to Test'
  
  jobs:
  - job: 'DeployTest'
    displayName: 'Deploy To Test'

My first task in this job was to specify the Python version. Which is essential when working with fabric-cicd to avoid any errors about invalid syntax.

- task: UsePythonVersion@0
  displayName: 'Use Python 3.11'
  inputs:
    versionSpec: 3.11

Afterwards I installed the fabric-cicd library.

- task: PowerShell@2
  displayName: 'Install necessary libraries'
  inputs:
    targetType: 'inline'
    script: |
      python -m pip install --upgrade pip
      pip install fabric-cicd
      # pip install azure-identity
    pwsh: true

Unlike a previous example that I showed, the azure-identity library is not required. If I was to encounter an issue with the service principal credentials not passing through properly, I can uncomment it and specify DefaultAzureCredential() in the Python script.

Afterwards, I added the below PowerShell task to authenticate a service principal.

- task: PowerShell@2
  displayName: 'Authenticate as Service Principal'
  inputs:
    targetType: 'inline'
    script: |
      Install-Module -Name Az.Accounts -AllowClobber -Force

      $SecureStringPwd = ConvertTo-SecureString $(AZURE_CLIENT_SECRET) -AsPlainText -Force
      $pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(AZURE_CLIENT_ID), $SecureStringPwd
                        
      Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(AZURE_TENANT_ID)

      $fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token
    pwsh: true

I decided to authenticate this way because it also works with those working with trial Microsoft 365 E5 tenants.

Another way is to create a service connection in your Azure DevOps project. As shown in this post by Richard Mintz. However, you require a subscription ID to create a service connection. Which is not available with trial tenants.

Final task created in the stage to deploy to test is a Python script task to call the Python script that orchestrates the fabric-cicd deployments. Which passes through various variables as arguments. So that they can be utilized in the Python script.

      - task: PythonScript@0
        displayName: 'Run script to deploy with fabric-cicd to Test'
        inputs:
          scriptPath: 'auth_spn_secret_AzDo.py'
          arguments: '--WorkspaceId $(TestWorkspace) --Environment "Test" --RepositoryDirectory "$(Build.SourcesDirectory)\workspace" --ItemsInScope  $(ItemsInScope)'

I cover the Python script later in this post after I have shown the final part of the YAML pipeline.

Stage to deploy to a Production workspace in Microsoft Fabric with fabric-cicd

I then added an additional stage to deploy to the production workspace. Because it utilizes an Azure DevOps environment, I specified the below deployment job.

jobs:
  - deployment: 'DeployProd'
    displayName: 'Deploy to Prod'
    environment: Production

    # Just to raise awareness that you can select pools at different stages
    # pool: 
    #   vmImage: 'windows-latest'

    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            path: 'self'

I then added the same tasks to this stage as I did with the stage for test deployments. Only difference was that I changed the arguments in my Python script task.

- task: PythonScript@0
  displayName: 'Run script to deploy with fabric-cicd to Test'
  inputs:
    scriptPath: 'auth_spn_secret_AzDo.py'
    arguments: '--WorkspaceId $(ProdWorkspace) --Environment "Prod" --RepositoryDirectory "$(Build.SourcesDirectory)\workspace" --ItemsInScope  $(ItemsInScope)'

Once all the tasks were created it was time to configure my modified Python script.

Modified Python script to operationalize fabric-cicd

In reality, the modified Python script is similar to the one in my previous post with two main differences. Which are that I no longer need to import the DefaultAzureCredential class or specify DefaultAzureCredential.

I start my updated Python script with the below code.

from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items
import argparse

Which does the below actions (in line order):

1. Imports items from fabric-cicd library into my script.
2. Allows me to work with the argparse module. So that I can work with the pipeline variables that I specified.

Afterwards, the script passes through the variables that I specified in the arguments section in my Python script task in Azure DevOps. As per the below code.

parser = argparse.ArgumentParser(description='Process some variables.')
parser.add_argument('--WorkspaceId', type=str)
parser.add_argument('--Environment', type=str)
parser.add_argument('--RepositoryDirectory', type=str)
parser.add_argument('--ItemsInScope', type=str)
args = parser.parse_args()

Afterwards, the script converts the items that are specified to be deployed as a list with the split method. In order for it to be properly recognized by the item_type_in_scope parameter.

# Convert item_type_in_scope into a list
allitems = args.ItemsInScope
item_type_in_scope=allitems.split(",")
print(item_type_in_scope)

From there, the FabricWorkspace object is initialized with the passed through variables and the converted list.

target_workspace = FabricWorkspace(
    workspace_id= args.WorkspaceId,
    environment=args.Environment,
    repository_directory=args.RepositoryDirectory,
    item_type_in_scope=item_type_in_scope,    
)

Finally, it specifies to publish and unpublish items as per the original sample code.

# # # Publish all items defined in item_type_in_scope
publish_all_items(target_workspace)

# # # Unpublish all items defined in item_type_in_scope not found in repository
unpublish_all_orphan_items(target_workspace)

Once this Python script was modified, I was ready to run my pipeline.

Test results to operationalize fabric-cicd to work with Microsoft Fabric

When I ran the pipeline the test deployment completed. I was then asked to approve the deployment to the production stage. Once I approved the deployment the two stages complete like in the below example. Which shows the completed check for the approval.

When I checked my production workspace all the items I could see all the deployed items. Including the mirrored database.

Just like in my previous post my “parameter.yml” file included the below statements. In order to utilize the find_replace functionality.

"parameter.yml" file to include the below statements. In order to utilize the find_replace functionality.

find_replace:
    # Change report text
    "This is a Report in Dev!!!":
        Test: "This is a Report in Test!!!"
        Prod: "This is a Report in Prod!!!"

I checked, and both the test and production reports had been updated accordingly.

Conclusion of test results to operationalize fabric-cicd

By following the above, you can operationalize fabric-cicd to work with Microsoft Fabric and Azure DevOps. Like I mentioned earlier in this post, fabric-cicd is ideal for the supported item types specified on the fabric-cicd page. Plus, the list is growing.

At this moment in time you can experience errors when attempting to unpublish mirrored databases. You can publish them fine though.

Of course, everything I shared both in this post and the Git repository can be customized to suit your needs.

Plus, I recommend commenting out the production stage when testing for first-time. In order to check that everything works before deploying to a stage that requires an approval.

Other things you can consider doing are as follows:

• Keep an eye on fabric-cicd updates.
• Create a service connection in Azure DevOps that authenticates as a service principal.
• Test with your own development items configured with Microsoft Fabric Git integration.
• Work with alternative variables to suit your own strategy. Plus, alternative methods to pass through the parameters.

Final words

I hope that showing this way to operationalize fabric-cicd to work with Microsoft Fabric and YAML Pipelines in Azure DevOps helps some of you get started. Plus, inspire some of you to experiment in your own environments.

Because I do believe that fabric-cicd has a lot of potential.

Of course, if you have any comments or queries about this post feel free to reach out to me.