Running Invoke-Parallel under VSO build agent stuck

[avishnyakov]

Hello team,

Invoke-Parallel looks really good working well under ISE / powershell console.
However using Invoke-Parallel under VSO build agent just stuck. It might be due to progress report (trying to figure out) or other things related to non-interactive session.

Anything you might suggest to look at?

[vors]

Could be a lot of different things. Invoke-Parallel code is relatively small. If you have time to investigate, to add debug output to different places and gather this data, it could be a great help to nail down the issue.

I seen you opened powercode/PSParallel#4
Are you looking for a good framework for the parallel execution in PS? :)

[avishnyakov]

Hey mate, thank you for getting back so fast.

I did open similar issue as the same story is happening with PSParallell. Furthermore, PSParallel stuck even with the PS console. Works only with ISE. Going to explore and run under VSO build agents.

The story around that has with Azure provision. We tend to parallel tons of operations getting the best of Azure PowerShell + either PSParallel or this Invoke-Parallel project. A wrapper on top of "parallel" execution helps us to switch between actual parallel run implementation, so that with one setting we either run Invoke-Parallel or PSParallel.

I stumbled upon that stuck behavior with TFS Online build agents. Troubleshooting today and tomorrow. Could be agent settings, could be something related to non-unteractive mode, could be silent switch or just something else. Could even be our Azure or PS mess as we do tons of weird stuff witin every parallel session. Need to try on bare array processing :)

And yes, there is almost NOTHING on really awesome PS parallel execution framework. PSParallel and your stuff really rock, guys. I really don't want to go with jobs. Been there, kinda hard to implement + pass params + get the feedback + return the values. I can invest time and make it right yet your scripts do a better job.

[RamblingCookieMonster]

@avishnyakov - It seems unlikely, but give the -Quiet parameter a try, it will suppress progress output. You might also take a look at PoshRSJob from @proxb, it's a fantastic, more fully featured framework for parallelization over runspaces.

@vors, @TravisEz13 - any plans to open source the jobs-based-tool that you mentioned? Seems like another path that might help here.

Cheers!

[avishnyakov]

Has something to do with calling Azure PowerShell command within parallel loop..

PSParallel works well on simple array, stuck on Azure PowerShell calls.
Checking Invoke-Parallel, gonna be interesting day.

[RamblingCookieMonster]

Oh! What Azure PowerShell commands are hanging up? I've been using invoke-parallel at $work recently. Occasionally (once or twice) get issues auto-loading the module in the first item, but works without issue the majority of the time. Is there a specific command/module you're running into trouble with?

Cheers!

[avishnyakov]

Oh, Invoke-Parallel does not seem to import functions, only vars and modules. That's why we switched to PSParallel and PSParallel now hangs up on Azure PowerShell calls. Such a mess,

[avishnyakov]

Okay, added custom function import into the initial session state:

 #If specified, add variables and modules/snapins to session state
            $sessionstate = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()

            $excludedFunctionNames = @(
                "cd",
                "cd\",
                "cd..",
                "Pause",
                "oos",
                "mkdir",
                "help"
            )

            $customFunctions = get-command | Where-Object  {
                                     $_.Module -eq $null `
                                     -and $_.Name.Contains(":") -eq $false `
                                     -and $excludedFunctionNames.Contains($_.Name) -eq $false `
                                     -and $_.CommandType -eq "Function"
            }

            foreach($customFunction in $customFunctions) {

                if($customFunction.Name -eq "Invoke-Parallel"){
                    continue;
                }

                Write-Verbose "Importing function:[$($customFunction.Name)]"

                $functionImpl = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $customFunction.Name, $customFunction.Definition
                $sessionstate.Commands.Add($functionImpl)
            }

That thing would import custom written functions from the current session into the new runspace. That's was kinda irritating issue initially that pushed me to PSParallel.

That's a temporary fix only, as I see that a proper option and built-in stuff needs to be added. I see that happening as an additional parameter + $StandardUserEnv variable needs to be enhanced with 'custom function bit'.

It would be really nice to see custom function import. My ad-hoc fix is only adding functions with a particular name convention (suits us), skips some built-in functions (just in case) and assumes the rest comes from the modules.

Testing under ISE, console and TFS agent stuff. Will keep you posted with the outcome. Hope that helps.

[proxb]

@avishnyakov Have you looked at PoshRSJob (https://github.com/proxb/PoshRSJob) to see if it does what you are looking to do? I am curious to see how it performs with your use case.

[avishnyakov]

No, I haven't yet. Pushing and testing Invoke-Parallel at this very moment.
PoshRSJob is the next in the list, thanks to you.

[avishnyakov]

Console / ISE works well.
TFS Build agent works well but somehow gives 'false exception' all the time in the $runspace.powershell.Streams.Error

VERBOSE: Exception calling "Invoke" with "1" argument(s): "Cannot index into a null array."
VERBOSE: System.Management.Automation.MethodInvocationException: Exception calling "Invoke" with "1" argument(s): 
"Cannot index into a null array." ---> System.Management.Automation.RuntimeException: Cannot index into a null array.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception 
exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
   at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockClauseToInvoke clauseToInvoke, Boolean 
createLocalScope, Dictionary`2 functionsToDefine, List`1 variablesToDefine, ErrorHandlingBehavior 
errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo 
invocationInfo, Object[] args)
   at System.Management.Automation.ScriptBlock.<>c__DisplayClassa.<InvokeWithPipe>b__8()
   at System.Management.Automation.Runspaces.RunspaceBase.RunActionIfNoRunningPipelinesWithThreadCheck(Action action)
   at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior 
errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo 
invocationInfo, Boolean propagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDefine, Object[]
 args)
   at System.Management.Automation.ScriptBlock.DoInvoke(Object dollarUnder, Object input, Object[] args)
   at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type 
typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
   at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
VERBOSE:    at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception 
exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
   at CallSite.Target(Closure , CallSite , ScriptBlock , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at System.Management.Automation.Interpreter.DynamicInstruction`3.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
VERBOSE: 

[avishnyakov]

Confirming that with custom function import all works as expected across the board.
The exception early seems to be our scripts, errors out only on the build controller and it's extremely hard to find the exact location due to mixture or script block invoking and Invoke-Parallel. Commenting out help.

Closing, thanks all for patience.

[avishnyakov]

Ans our build controller happen to have the old Azure PowerShell module, like 1.3.x something. That steals everything. All sorted now, Invoke-Parallel rocks!