search-unifiedauditlog highcompletenes parameter

[Passimist]

Hello everyone,

first of thank you for providing this cool project. I stumbled upon a parameter in the documentation of the search-unifiedauditlog cmdlet that got me worried: https://learn.microsoft.com/en-us/powershell/module/exchange/search-unifiedauditlog?view=exchange-ps#-highcompleteness

It says:

The HighCompleteness switch specifies completeness instead performance in the results. You don't need to specify a value with this switch.

When you use this switch, the query returns more complete search results but might take significantly longer to run. If you don't use this switch, the query runs faster but might have missing search results.

Did you know about this parameter? Did you actually experience any incomplete logs when comparing output with and without it? Would it make sense to include it in the cmdlet calls in this project?

[invictus-korstiaan]

Hi Passimist, thanks for bringing this to our attention. We were not aware of this parameter, we look through GitHub and found it was only recently added in the last two weeks. We will most likely add this to our script as we want to make sure we acquire everything.

[JoeyInvictus]

Hi @Passimist,

Small update. We've been conducting some tests, but the flag appears to be very unstable, resulting in errors most of the time. We attempted to integrate it into the Extractor for testing. Additionally, simply running the Search-UnifiedAuditLog cmdlet with this parameter results in errors over 50% of the time. There seems to be little information available online about this new parameter, and I was wondering if you had already conducted any testing?

[Passimist]

Hey @JoeyInvictus,

I sadly do not have more experience with this than you. I was thinking about contacting Microsoft support to get more information since the documentation is rather disappointing. But as you probably know it is uncertain if this will lead to anything. 😆
But since I was trying out your Microsoft Extractor Suite the same day that I stumbled over the "highcompleteness" parameter I figured I might as well ask here if you took this parameter into account.

Honestly I am hoping that this parameter will disappear again. Its description suggesting that log acquisition is incomplete by default is super concerning...

But maybe I interpret the parameter description wrong and it is actually used to acquire more logs than the limits that exist for some sources usually allow, which would be pretty cool.

[Calvindd2f]

A try-catch would be needed for -HighCompletness because they are records that are still processing. Something like this might do the trick but I don't have UAL enabled on test tenant.

$pageSize = 5000
$retryCount = 5
$retryDelaySeconds = 60
$OutputDir = "d:\AuditLogSearch"
$Output = "CSV"
$Encoding = [System.Text.Encoding]::UTF8

while ($currentStart -lt $end) {
    $currentEnd = $currentStart.AddMinutes($intervalMinutes)
    $sessionID = $currentStart.ToString("yyyyMMddHHmmss")
    $currentTries = 0

    while ($true) {
        try {
            $results = Search-UnifiedAuditLog -UserIds $UserIds -StartDate $currentStart -EndDate $currentEnd -HighCompleteness -ResultSize $pageSize -SessionCommand ReturnLargeSet
            $totalResults = $results[0].ResultCount
            $currentCount = $results.Count

            Write-LogFile -Message "[INFO] Found $totalResults audit logs between $($currentStart.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssK")) and $($currentEnd.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssK"))"

            if ($Output -eq "JSON") {
                $results = $results | Select-Object AuditData -ExpandProperty AuditData
                $results | Out-File -Append "$OutputDir/UAL-$sessionID.json" -Encoding $Encoding
                Write-LogFile -Message "[INFO] Successfully retrieved $currentCount records out of total $totalResults for the current time range. Moving on!" -Color "Green"
            }
            elseif ($Output -eq "CSV") {
                $results | Export-Csv "$OutputDir/UAL-$sessionID.csv" -NoTypeInformation -Append -Encoding $Encoding
                Write-LogFile -Message "[INFO] Successfully retrieved $currentCount records out of total $totalResults for the current time range. Moving on!" -Color "Green"
            }

            if ($totalResults -eq $currentCount) {
                break
            }
        }
        catch { # This part here handles -HighCompletness
            if ($currentTries -lt $retryCount) {
                Write-LogFile -Message "[WARNING] There are records still processing [-HighCompleteness] retrying so there is no incomplete data" -Color "Red"
                Write-LogFile -Message "[INFO] Sleeping $retryDelaySeconds seconds before we try again" -Color "Red"
                Start-Sleep -Seconds $retryDelaySeconds
                $currentTries++
                continue
            }
            else {
                Write-LogFile -Message "[WARNING] Empty data set returned between $($currentStart.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssK")) and $($currentEnd.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssK")). Retry count reached. Moving forward!" -Color "Red"
                break
            }
        }

        $currentStart = $currentEnd
        $currentEnd = $currentStart.AddMinutes($intervalMinutes)
    }}

[JoeyInvictus]

Hi @Calvindd2f,

Thank you for your response! What do you mean by "the results are still processing"? I agree with using the try-catch if we add this to the tool. However, I'd like to do some more testing because the cmdlet for me seems pretty unstable at the moment and takes significantly longer than running it without the flag. I would also like to see the difference in records that are acquired with and without the flag.

[Calvindd2f]

Hi @JoeyInvictus,

I am sorry about the delayed response. Thanks for your feedback. By "the results are still processing," I'm referring to the latency that occur when querying a large set of records, especially with the -HighCompleteness`` flag enabled in the Search-UnifiedAuditLog` cmdlet. This flag is intended to ensure a more comprehensive search, but it can cause delays because it waits for all logs to be indexed and available, which might not be instantaneous.

When accuracy is mission critical try-catch block allows graceful handle where the data isn't immediately available because of the processing kag. $retryCount is used to reattempt & eventually move on after reasonable effort has been made to retrieve all available logs.

Regarding instability and performance concerns you mentioned ; -HighCompleteness flag does guarantee queries will take longer, which is a trade-off between speed and data comprehensiveness.

I will generate some logs for testing purposes then comparse results obtained with and without this flag to guage data retrieved completeness and overall query execution time with Measure-Command cmdlet and I'll get back to you before Monday.

the comparison might be able to help decide if the additional wait time is justified depending on the difference in data completeness .

If there's minimal difference, especially with no significant addition to logs (especially logs that are genuinely useful for investigations), I'd consider the parameter to offer diminished returns. It might only be applicable in highly stakes envioments , like DoD or something.

[Calvindd2f]

Significant amount more data returned traded off for v. significant amount more query time. I'll review the difference in the files soon to see if there is anything worthy in the extra 25 MB data or if it is just junk.

Query time test.

{
    $withHCparam={Search-UnifiedAuditLog -EndDate (get-date) -HighCompleteness -StartDate (get-date).AddDays(-7) -UserIds '#calvindd2f'} ; $time1=Measure-Command -Expression $withHCparam;Write-Output "Time taken for Command 1: $($Time1.TotalMilliseconds) ms"

    $woHCparam={Search-UnifiedAuditLog -EndDate (get-date) -StartDate (get-date).AddDays(-7) -UserIds '#calvindd2f'} ;$time2=Measure-Command -Expression $woHCparam ;Write-Output "Time taken for Command 1: $($time2.TotalMilliseconds) ms"
}

Time taken for with -HighCompleteness : 152683.7185 ms or 152.6837185 Seconds
Time taken for without -HighCompleteness: 7707.614 ms or 7.707614 Seconds

153 is a 1812.5% increase of 8.

Cannot see this scaling well - I only enabled UAL 2 weeks ago on dev tenant with fairly little data generated (compared to business level end users inbox or activity).

---

Difference test / File sizes [Pending review of actual logs]

{
    Search-UnifiedAuditLog -EndDate (get-date) -HighCompleteness -StartDate (get-date).AddDays(-7) -UserIds 'calvindd2f'|ConvertTo-json > HCdataDiff.json # Includes Parameter

    Search-UnifiedAuditLog -EndDate (get-date) -StartDate (get-date).AddDays(-7) -UserIds 'calvindd2f' |ConvertTo-Json > woHCdataDiff.json # Excludes Parameter - this took a near comedic long time to finish.
}

It's 01:21 AM and I need to review the diffs to see if they are anything worth running parameter.
I will slap this over into the TODO: and just show the file size diff for the moment

Reference for getting file size [lazy i know]

$file=${$file};$size=(get-item $file).Length
Write-host "KB":($size/1KB);Write-host "MB":($size/1MB);Write-host "GB":($size/1GB)

OUTPUT

• KB : int
• MB : int
• GB : int

Without -HighCompleteness Parameter - woHCdataDiff.json

• KB : 560.36328125
• MB : 0.547229766845703
• GB : 0.000534404069185257

With -HighCompleteness Parameter - HCdataDiff.json

• KB : 30766.16015625
• MB : 30.0450782775879
• GB : 0.0293408967554569

[JoeyInvictus]

Hi @Calvindd2f,

Thanks for the detailed explanation and the tests you performed. I believe that from a forensic perspective, the most important aspect is to obtain all the data, even if it means waiting longer. However, currently, the parameter feels quite unstable as it keeps throwing errors in our test environment. I can't imagine how it would be if dealing with a significantly larger dataset.

As shown in the screenshot below, I attempted to execute the command you provided, but it failed on all three occasions when I requested 180 days of history. Since this is a test tenant with limited events (11229 total), it's concerning that the command failed even in this scenario. We've encountered tenants with event volumes exceeding 5000 per minute. If the command is already failing on our small environment, it raises significant concerns about its scalability and reliability.

[JoeyInvictus]

Hi, I will close this issue for now. We will still track changes to the highCompleteness parameter and implement it when it's more stable. However, we just published a way to extract the unified audit log via the Graph API. We did some testing, and the Graph API is way more stable in returning the same amount of data compared to the search-unifiedauditlog cmdlet. Since Microsoft is moving more and more towards doing everything with Graph, we expect that this method will work better and way less time consuming.