Photo Credits: Unsplash and Microsoft

Contribution Link: PR #5629

Introduction

Microsoft365DSC is a PowerShell module for applying the Desired State Configuration framework to Microsoft 365 configurations - in short, it allows you to backup and idempotently apply configuration items like Entra ID users and groups, Purview Data Loss Prevention Policies, Exchange mailbox settings, or any combination of more than 400 supported resource types.

I frequently develop custom tools and reporting platforms integrated with Microsoft 365, and I find the ability to export configurations from a short-lived tenant very useful.

The only problem? When I first attempted to perform an export of ~70 resource types for a small tenant, it took over 2 days.

My Contribution

Note: this section is largely duplicated from the issue (#5615) I opened prior to making my contribution.

The module is structured such that each exportable resource implements an Export-TargetResource and Get-TargetResource function (among others that are not relevant here). As I originally found it, most resources followed this pattern:

function Export-TargetResource
{
    Connect-ToWorkload
    Send-Telemetry
    $instances = Get-AllResources
    foreach ($instance in $instances)
    {
        Get-TargetResource -Instance $instance
    }
}

function Get-TargetResource
{
    param($Instance)
    Connect-ToWorkload
    Send-Telemetry
    $instance = Fetch-Resource -Instance $Instance
    Format-Resource # AndFetchAdditionalProperties
    return $instance
}

There are three significant performance issues:

1. Calling the workload connection function for every iteration of Get-TargetResource not only resulted in thousands of unnecessary function invocations - which should clearly be avoided - but the connection function itself also triggered API calls on every invocation (which I verified with several Fiddler traces), causing a great deal of superfluous I/O.
2. Telemetry was being sent on every iteration of Get-TargetResource within the Export-TargetResource foreach loop. It was enormously impactful; Without any other changes, I determined that the same export took 8,925 seconds with telemetry enabled and 1,400 seconds without. In other words, telemetry - just telemetry - accounted for ~85% of the export's runtime. Further, it is highly redundant to log the invocation of Export-TargetResource and also each invocation of Get-TargetResource from within the export. If there is a desire to track the number of Get-TargetResource calls or the average function time, it should be tracked within the export function and sent at the end of the function via a single log.
3. For many resources, the Get-TargetResource function actually re-fetched the same instance information from the same API as Export-TargetResource. This should clearly be avoided, and had been for many modules: Export-TargetResource would declare $Script:exportedInstances and Get-TargetResources would check whether such a variable existed, then filter the array to find the right instance. That was better but could be further improved. Instead, we now declare $Script:exportedInstance inside of the export's foreach loop and avoid the filtering inside of Get-TargetResource. For resources with large numbers of instances, this makes a meaningful impact.

Altogether, my code changes the relevant section of a typical Get-TargetResource from this:

New-M365DSCConnection -Workload 'MicrosoftGraph' `
        -InboundParameters $PSBoundParameters

Confirm-M365DSCDependencies

#region Telemetry
$ResourceName = $MyInvocation.MyCommand.ModuleName -replace 'MSFT_', ''
$CommandName = $MyInvocation.MyCommand
$data = Format-M365DSCTelemetryParameters -ResourceName $ResourceName `
    -CommandName $CommandName `
    -Parameters $PSBoundParameters
Add-M365DSCTelemetryEvent -Data $data
#endregion

$nullReturn = $PSBoundParameters
$nullReturn.Ensure = 'Absent'
try
{
    # fetch resource
    if ($null -eq $resource) { return $nullReturn }
    # format and enrich
    return
}
catch
{
    # log
}

to this:

try
{
    if (-not $Script:exportedInstance)
    {
        New-M365DSCConnection -Workload 'MicrosoftGraph' `
                -InboundParameters $PSBoundParameters

        Confirm-M365DSCDependencies

        #region Telemetry
        $ResourceName = $MyInvocation.MyCommand.ModuleName -replace 'MSFT_', ''
        $CommandName = $MyInvocation.MyCommand
        $data = Format-M365DSCTelemetryParameters -ResourceName $ResourceName `
            -CommandName $CommandName `
            -Parameters $PSBoundParameters
        Add-M365DSCTelemetryEvent -Data $data
        #endregion

        $nullReturn = $PSBoundParameters
        $nullReturn.Ensure = 'Absent'

        # fetch resource
        if ($null -eq $resource) { return $nullReturn }
    }
    else
    {
        $resource = $Script:exportedInstance
    }
    # format and enrich
    return
}
catch
{
    # log
}

In total, these simple modifications resulted in ~9,000 changed lines of code across ~60 supported resource types - as well as a number of broken unit tests I fixed and contributed in the process of verifying my contribution.

Note: There are better and more elegant solutions to the problems I identified than the fixes I applied. However, these fixes were (relatively) quick to implement and simple enough to minimize the back-and-forth approval discussions that come with open-source development. This felt like the right approach once pragmatic concerns were factored in.

Impact

My performance test results with identical configurations (runtime, measured in seconds):