DaedalicEntertainment/ue4-test-automation

Gauntlet: Array dimensions exceeded supported range

janousch opened this issue · 2 comments

janousch commented

After testing >300 tests we get the following error from Gauntlet:

Error: Array dimensions exceeded supported range..

   at System.String.SplitInternal(Char[] separator, Int32 count, StringSplitOptions options)
   at Gauntlet.UnrealLogParser.CreateSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 205
   at Gauntlet.UnrealLogParser.GetSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Utils\Gauntlet.UnrealLogParser.cs:line 189
   at Gauntlet.UnrealRoleArtifacts.get_LogSummary() in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealSession.cs:line 424
   at Gauntlet.UnrealTestNode`1.<>c.<GetWarnings>b__8_0(UnrealRoleArtifacts A) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs:line 116
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.<>c.<ReportMasterSummary>b__22_1(TestExecutionInfo T) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at System.Linq.Enumerable.WhereListIterator`1.MoveNext()
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source)
   at Gauntlet.TextExecutor.ReportMasterSummary(Int32 CurrentPass, Int32 NumPasses, TimeSpan Duration, IEnumerable`1 AllInfo) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 566
   at Gauntlet.TextExecutor.ExecuteTests(TestExecutorOptions InOptions, IEnumerable`1 RequiredTests) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs:line 440
   at Gauntlet.RunUnreal.ExecuteTests(UnrealTestOptions Options, IEnumerable`1 TestList) in D:\Workspace\UnrealEngine\Nightly\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs:line 307
AutomationTool exiting with ExitCode=152 (Error_TestFailure)
janousch commented

The problem is that we have an AnimationModifier class defined in C++ inside of an editor module. When we run the automated tests with Gauntlet we don't have access to that module. When the test runs, Unreal uses/opens the editor animation sequences, inside of the AnimSequence we got the blueprint version of that AnimationModifier and then we get the log message that the parent is missing. The logs are filling up the context string of the test artifact. When the UnrealLogParser tries to split that string he fails.

janousch commented

Fixed in https://github.com/janousch/ue4-test-automation/tree/develop
I added the option to execute console commands before starting the tests. By suppressing the logs about missing references to editor only objects the size of the log file decreased by several GB and the Error disappeared.
Currently, there are doubts whether these changes should be merged.