SCRT-HQ/PSGSuite

Implement retries with exponential backoff for long running commands like Get-GSDriveFileList

Celestica-Edward opened this issue · 0 comments

Describe the bug
Commands such as Get-GSDriveFileList calls Google APIs numerous times if the user has many files. Any of these API calls can transiently fail with 500 internal server error (http://disq.us/t/3ib2tne) or any other error. Retrying from the parent script would mean the file list needs to be from the beginning again, wasting lots of time.

To Reproduce
Steps to reproduce the behavior:

  1. Call Get-GSDriveFileList against a user with many thousands of files and if you're unlucky, an error is encountered.
    Capture
    Expected behavior
    The loop at https://github.com/SCRT-HQ/PSGSuite/blob/main/PSGSuite/Public/Drive/Get-GSDriveFileList.ps1#L170 can be modified to include retries, especially for 5xx errors. Other error codes such as 4xx should not be retried.

A possible example can be:

				$MaxRetries = 5
				$RetryDelaySeconds = 2
				$retryCount = 0
				while ($retryCount -lt $MaxRetries) {
					try {
						$result = $request.Execute()
						break
					}
					catch {
						Write-Verbose "Failed, retrying..."
						Write-Verbose $_
						$retryCount++
						Write-Verbose $("Sleeping " + ([math]::Pow($RetryDelaySeconds, $retryCount)) + " seconds")
						Start-Sleep -Seconds ([math]::Pow($RetryDelaySeconds, $retryCount))
					}
				}
				if ($retryCount -eq $MaxRetries) {
					Write-Verbose "Failed after $retryCount attempts"
					throw $_
				}

Capture2