This Test Tool
simplifies testing of Unity
games and allows to run tests on mobile devices. It's based on methods, attributes, and logs, similar to those in NUnit.
License: Apache License 2.0
Just copy UITestTools
folder and paste it wherever you want inside Assets
folder of your project.
Flow Recorder
is an Unity Editor extension that allows to record user actions and generate test's source code from them.
To use Test Recorder
navigate to Window => Ui test tools => Flow recorder
, or press Ctrl+T
for Windows and Cmd+T
for Mac.
Start Record
button - enables recording mode. Once enabled, user's mouse input (LMB clicks) is recorded into test actions and appended to test's action list.Stop Record
- disables recording mode. All previously recorded actions are serialized automatically.Check
- toggles mode, in which clicks on UI buttons don't trigger them. The window will be colored green when check mode is enabled.Pause mode
- sets current timescale to 0 in play mode. Used in situations where user needs to emulate game pause yet still be able to record new actions.Clean
- deletes all previously recorded actions.+
button - manually adds new "empty" user action. Provides access to actions that don't requireGameObject
to be executed.Generate Code and Copy
- generate source code for test based on recorded user actions.
Recorded users actions are visually displayed as a list. User can edit recorded actions by selecting necessary assertation type, editing arguments, drag-and-dropping actions inside the list to change execution order, adding or removing actions manually.
Each action contains the following information:
6
- action's index in the list.Assertation Type
- a drop-down menu with a list of assertation types, available for currentGameObject
. Each assertation has its own list of required arguments.Description
- a brief description to selectedassertation method
. Results in a commentary above assertation method call in the generated source codePath
- path in the scene hierarchy to theGameObject
user interacted with.Select
button allows to select the specifiedGameObject
TimeOut
,IgnoreTimeScale
- selected assertation's arguments list. Argument values can be edited by user. Available arguments types areenum
,int
,float
,bool
,string
.►
- taplies assertation in runtime.Copy
- duplicates selected asertation and places the copy next to it. Hotkeys areCtrl+D
for Windows andCmd+D
for Mac.x
- removes selected action from list.select
- displays the location of the objects in hierarchy order.
When you perform a LMB click in the Game Window
during Unity
play mode, Flow Recorder
obtains click coordinates and uses UnityEngine.EventSystems.EventSystem
class to raycast by these coordinates to find the GameObject
user clicked. Then Flow Recorder
looks through a list of all existing assertations
and checks which assertation
could be applied to the GameObject
.
For example, if user clicked on UI text - all assertations
that compare text labels become available. Common assertations
, like Is Enable
or Is Exist
are available for any GameObject
. By default, Flow Recorder
supports only Unity UI
raycast targets. If a support for GameObject
s that don't have any UI components attached or use of custom assertations
is required you have to implement custom assertation method
. Please read below how to extend UI Test Tool
.
Launch the game from the editor, press Start Record
button to start recording actions, perform any desired actions in the game window, then press Stop Record
button to stop recording. You can change assertation types or parameters and change action order during the test recording or any time after it. Press Generate Code and Copy
to generate and obtain the source code for the test. Then you can paste the generated code to your own test class.
Also, you can create a new assertation with the object by making a Right Click
on the object in the hierarchy view and then selecting Create Assertation
, or you can select Copy Path
if you want to copy the path to the object.
To open Test Runner Window
navigate to Window => Runtime Test Runner
.
Test Runner
searches the project for all methods with the attribute [UnityTest]
and represents them as a tree structure with namespaces and classes as branches and methods as leaves. Press Run all
to run all tests.
You can select the number of times each test is run by dragging the Repeat Tests N Times
slider.
Also, you can change the TimeScale
value applied to game environment before the test run by dragging the Default Timescale
slider.
One of the greatest features of the Test Runner
tool is that it allows to run tests directly on target devices. See command line arguments for it in Command line section.
You can select specific tests that you want to run and press the Run
button. The set of selected tests is saved to UITestTools/PlayModeTestRunner/Resources/SelectedTests.asset
scriptable object.
We have a possibility to use a Filter
to search for tests. Just write the name or id of the test in the line.
You can run a single test by double clicking on test's name.
When you run tests on mobile device Play Mode Test Runner
tool loads SelectedTests
asset and performs only selected tests. If SelectedTests
asset does not exist - all tests will be performed.
You can mark any test with [SmokeTest]
attribute. Clicking the Rum Smoke
button allows to perform only tests marked with this attribute.
For example:
[SmokeTest]
[UnityTest]
public IEnumerator SomeTest()
{
// some code
}
Sometimes, tests may succeed or fail depending on the different screen resolution. For example, the test checks whether a specific GameObject
is located at the certain screen pixel coordinates. To ensure that the test is run only at the proper resolution you have to add [EditorResolutionAttribute]
attribute above test method declaration and specify target resolution for Editor. It doesn't affect tests that are performed on devices.
Also, you can add any amount of TargetResolutionAttribute
attributes to test. If device resolution matches not any of them, the test is ignored. If EditorResolutionAttribute
is not set, TargetResolutionAttribute
values are used for Editor Resolution.
For example:
[TargetResolution(1920, 1080)]
[TargetResolution(1334, 750)]
[TargetResolution(1136, 640)]
[EditorResolution(1920, 1080)]
[UnityTest]
public IEnumerator SomeTest()
{....}
You add link to test description in TestRails with attribute TestRailAttribute
. TestRailAttribute
constructor takes 2 arguments:
string subdomen
and int testId
.
For example if link to your test looks like this:
https://TESTRAILS_PROJECT_NAME.testrail.net/index.php?/cases/view/12345
where 12345
is some test id. Then you have to declare test method like that:
[TestRail("TESTRAILS_PROJECT_NAME", 12345)]
[UnityTest]
public IEnumerator SomeTest()
{....}
To use that link you have to right click on test name in TestRunner window and choose Open TestRail
in context menu.
You can open test context menu by right click on it's name in TestRunner window.
Open TestRails
- opens Testrail webpage with description of selected test. This option is shown only if you've addedTestRail
attribute to test method.Open Source
- opens .cs file with name equals to method's class name. If file contains test method name it will open source on corresponding line. This option is shown only if such file exists.Copy Name
- copies full name of test method with namespace.Run
- simply runs specific test method.
Test Helper
navigate to Window
=> Helper
that shows a list of possible assertations for the selected GameObject
in Hierarchy
and corresponding code lines next to them. In play mode Test Helper
allows user to obtain the list of sound clips that are currently playing.
Copy
- copies assertation's generated code.Find playing sounds
- searches for currently playing sound clips.Select
- used for sounds only, selects theGameObject
in hierarchy which hasAudio Source
component and it'sSound Clip
is playing a sound.
To run tests in Editor Mode via command line use the following command:
Unity.exe
-projectPath project_path
-executeMethod TestToolBuildScript.RunPlayModeTests
-runOnlySelectedTests
-runOnlySmokeTests
-timeScale time_scale
`-buildNumber` - build_numer
-projectPath
- absolute path to project folder (Unity attribute);-runOnlySelectedTests
- optional, runs only tests selected inPlay Mode Test Runner
window;-runOnlySmokeTests
- optional, runs test withSmokeTest
attribute only;-timeScale
- optional, timescale for tests;-buildNumber
- optional, used as postfix for file with test metrics.-testNamespaces
- optional, list of namespaces separated with comma, only tests which full name contains any of namespaces are run.
To create test build to run tests on the target device use TestToolBuildScript.TestBuild
method instead of TestToolBuildScript.RunPlayModeTests
:
Unity.exe
-projectPath project_path
-testBuildPath result_path
-executeMethod TestToolBuildScript.TestBuild
-runOnlySelectedTests
-runOnlySmokeTests
-buildNumber build_numer
-timeScale time_scale
Arguments are equal in both methods.
Test Tool
common actions are split to 4 static partial classes: Check
, Wait
, Interact
and AsyncWait
. Each of these classes contain a list of methods which perform a corresponding action like checking the state of the given object, waiting for specific condition or interacting with specific object.
For example:
yield return Wait.ObjectEnabled("SpeechBubble/avatar_king", 20f);
Check.CheckEnabled("content/overlay_square");
Check.CheckEnabled("content/overlay_square_5");
Check.CheckEnabled("content/speech bubble/SpeechBubble/speech bubble");
Check.TextMeshProEquals("content/CCLabelBMFontv2", "Match 4 to enchant the feather charms you need in the row.");
Check.CheckEnabled("content/pointer finger/SwipeAnim/pointer_finger");
yield return Interact.SwipeCell(4, 7, Interact.SwipeCellClass.SwipeDirection.Down, true, 1f);
Async
actions allow user to delay their finish conditions check until the following actions are finished.
For example:
var logWaiter = AsyncWait.StartWaitingForLog("some log");
yield return Interact.WaitDelayAndClick("LayoutComponent/ButtonCherryTree", 0, 20f);
yield return logWaiter;
You can check game state by making screenshot during test running and comparing it with reference image. This functionality works both in Editor and Mobile Build.
There are several methods to work with screenshots in Ineract
class:
IEnumerator MakeScreenShot(string name)
Method makes screenshot and store it in persistant data folder by screenshot path:
Screenshots/{resolution_WITH_HEIGHT}/{name_of_your_test}/{name_currentTime}
String Parameter name
is a name of screenshot. Current date and time and image extention (.png
) will be automatically added to screenshot name.
IEnumerator MakeScreenShotReference(string name)
Works only in editor!
Method makes screenshot and store it in resources folder by reference path:
Assets/.../PlayModeTestRunner/Editor/Resources/ReferenceScreenshots/{resolution_WITH_HEIGHT}/{name_of_your_test}/{name}
String Parameter name
is a name of screenshot. An image extention (.png
) will be automatically added.
IEnumerator MakeScreenshotAndCompare(string screenShotName, string referenceName, float percentOfCorrectPixels = 0.9f, bool dontFail = false)
Behaviour of this method depends on checkbox Make reference screenshots instead of comparing
in Advanced Options
foldout in Test Runner Window.
If you run this method from Build it will work as Make reference screenshots instead of comparing
paramert is False
.
Make reference screenshots instead of comparing
isFALSE
:
Method makes screenshot, stores it at screenshot path and compares with given reference screenshot.
Comparison is performed as follows: pixel arrays are extracted from current and reference screenshots, lengths of both arrays has to be equal, then method iterates through both arrays and compare corresponding pixels. Red, green and blue channels of current and reference pixels are comparing by formula abs(current - reference) < epsilon
where epsilon is empirically derived constant with value 20f
. Channels values are going from 0 to 255. If formula value is true - pixels are considered equal. If percent of matched pixels is greater or equals than percentOfCorrectPixels
value then method passes, else throw an Exception to fail test.
Sometimes you have several screenshot comparison in a one test. If first comparison fails test would failed, and further comparison would not happend. If you want to perform all comparisons in test disregard if comparisons succeed of failed you have to set bool dontFail
to true
. In this case if comparison fails error message would stored internally without raising error log preventing test from failing.
Examples:
[Test]
public IEnumerator SomeTest1()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: false); //this line fails
//test failed, further line will never performed
yield return MakeScreenshotAndCompare("screen2", "reference2", 0.9f, dontFail: true); //this line succeed
}
[Test]
public IEnumerator SomeTest2()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: true); //this line fails, error log stored internally
//further line will performed
yield return MakeScreenshotAndCompare("screen2", "reference2", 0.9f, dontFail: true); //this line succeed
//test succeed
}
If some screenshot comparing with flag dontFail
set to true
are failed, test will pass and you will not know that something failed. To fail test and log all stored internally errors you have to call next method:
void FailIfScreenShotsNotEquals()
Examples:
[Test]
public IEnumerator SomeTest1()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: true); //this line fails, error log "error 1" is stored
yield return MakeScreenshotAndCompare("screen2", "reference2", 0.9f, dontFail: true); //this line succeed
FailIfScreenShotsNotEquals(); //test failed, error log "error 1" is logged
}
[Test]
public IEnumerator SomeTest2()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: true); //this line succeed
yield return MakeScreenshotAndCompare("screen2", "reference2", 0.9f, dontFail: true); //this line succeed
FailIfScreenShotsNotEquals(); //nothing happened
//test succeed
}
You can also clear all stored errors by calling method ResetScreenshotFailFlag()
.
Example:
[Test]
public IEnumerator SomeTest1()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: true); //this line fails
FailIfScreenShotsNotEquals(); //test failed
}
[Test]
public IEnumerator SomeTest2()
{
yield return MakeScreenshotAndCompare("screen1", "reference1", 0.9f, dontFail: true); //this line fails
ResetScreenshotFailFlag(); // all error logs are cleared
FailIfScreenShotsNotEquals(); //nothing happened
//test succeed
}
Make reference screenshots instead of comparing
isTRUE
:
With this flag method MakeScreenshotAndCompare
just makes reference screenshots. This is handfull in case when you implement a lot of test with screenshot comparing but there are no reference screenshots created yet and you want to create them automatically. If you are sure that for the moment application behaviour is correct, you have to set Make reference screenshots instead of comparing
flag to true
and run all screenshot test you want to create reference screenshots for.
Then you should reset flag to let test run normally.
You can extend any of 4 common test actions classes (Check
, Wait
, Interact
and AsyncWait
) with new sets of actions using partial class modifier.
The following example shows how to create a simple assertation method, which retrieves GameObject
from hierarchy by path that is passed as an argument, checks whether it has user-defined custom component LevelButton
attached and checks the stars count on it. The method is placed its in own partial Check
class.
public static partial Check
{
...
public static void StarsCount(string path, int startCount) //path is a full path of GameObject in hierarchy on scene
{
var go = IsExist(path); //if object not exist, exception will be thrown and test failed
var levelButton = go.GetComponent<LevelButton>();
//lest fail test, if component doesn't present on the object
if (levelButton == null)
{
Assert.Fail("StarsCount: " + path + " object is exist, but LevelButton component not found.");
}
Assert.AreEqual(levelButton.StartCount, startCount, "Start Count is not equals.");
}
}
The method is now available to be used in tests as an action:
[UnityTest]
public IEnumerator SomeIntegrationTest()
{
// some code
Check.StarsCount("Level3/Button", 2);
}
Let's create a class that informs Flow Recorder when and how it should display the action created in the previous code snippet. The class is required to be inherited from ShowHelperBase
class and implement CreateGenerator()
method. CreateGenerator
method is responsible for defining which editable arguments the action has, their default values and how to generate its source code.
bool IsAvailable(GameObject go)
method is overridable. It typically indicates whether an assertation is available for the current GameObject
. By default it returns true
, even if GameObject
is null
. User can override method Camera GetCamera()
to specify which camera should be used by Flow Recorder to raytrace the desired GameObject
.
User can override method Camera GetCamera()
to specify which camera should be used by Flow Recorder to raytrace the desired GameObject
. See more below.
private class CheckStarsCount : ShowHelperBase
{
public override bool IsAvailable(GameObject go)
{
if (!go)
{
return false; //not available if game object is null
}
//not available if game object doen't containce LevelButton component
return go.GetComponent<LevelButton>() != null;
}
public override AbstractGenerator CreateGenerator(GameObject go)
{
var levelButton = go.GetComponent<LevelButton>();
//tell Flow Recorder about params and default values
return VoidMethod.Path(go).Int(levelButton.StartCount);
//if method is IEnumerator, use IEnumeratorMethod instead VoidMethod.
}
}
Let's bind our method to class.
[ShowInEditor(typeof(CheckStarsCount), "Check Start Count on Button")]
public static void StarsCount(string path, int startCount) //path is a full path of GameObject in hierarchy on scene
{
var go = IsExist(path); //if object not exist, exception will be thrown and test failed
var levelButton = go.GetComponent<LevelButton>();
//lest fail test, if component doesn't present on the object
if (levelButton == null)
{
Assert.Fail("StarsCount: " + path + " object is exist, but LevelButton component not found.");
}
Assert.AreEqual(levelButton.StartCount, startCount, "Start Count is not equals.");
}
The action can now be seen in Flow Recorder
actions list as Check Start Count on Button
upon clicking a GameObject
with LevelButton
component attached.
ShowInEditorAttribute
receive 3 arguments in constructor:
Type classType
- type ofShowHelperBase
class to be applied to the action;string description
- assertation's brief description (shown in recorder's dropdown list of assertations);bool isDefault
false - optional argument that indicates whether this assertation should be used to selectedGameObject
by default. In case of multiple default assertations available recorder resolves it by selecting one of them.
When user taps on a GameObject
on the game screen, Flow Recorder
looks through all assertation methods with a ShowInEditor
attribute. Flow Recorder
then finds suitable EditorHelpers
and passes the instance of selected GameObject
to their IsAvailable
methods. If any of those methods return true then appropriate assertation method is selected to be shown in the list of assertation methods in Flow Recorder
UI. Finally Flow Recorder
looks for a method with isDefault
value set to true and sets it as a selected assertation for the action.
Sometimes target GameObjects
contain child GameObjects
which can be raycast targets as well. This may lead to unpredicted behaviour in the Flow Recorder
. For example, you tap on a text label, but the recorder choses the background image containing GameObject
as a target instead. To address this issue Flow Recorder
passes target GameObject
to IsAvailabel
method and if it returns false
, Flow Recored
passes its parent GameObject
to IsAvailabel
method recursively, until it returns true
or parent is null
.
Here's an example of another assertation method which checks some specific game mechanics: assume we have a ScreenManager
MonoBehaviour
present on a scene with a string property of current active screen, and we want to check if lobby screen is active.
[ShowInEditor(typeof(CheckIsScreenActive), "Check Current Screen")]
public static void IsScreenActive(string screenName)
{
// FindAnyGameObject has overload to seach for GameObject with given type of component.
var screenManager = UITestUtils.FindAnyGameObject<ScreenManager>();
if (!screenManager)
{
Assert.Fail("IsScreenActive: no ScreenManager found");
}
Assert.AreEquals(screenManager.CurrentScreen, screenName);
}
private class CheckIsScreenActive : ShowHelperBase
{
public override AbstractGenerator CreateGenerator(GameObject go)
{
return VoidMethod.String(ScreenManager.CurrentScreen);
}
}
Example of a test with this assertation:
[UnityTest]
public IEnumerator SomeIntegrationTest()
{
// some code
Check.IsScreenActive("Lobby_Screen");
}
When user clicks on UI element (button, switch etc) Flow Recorder
receives click coordinates and uses UnityEngine.EventSystems.EventSystem
class to raycast by these coordinates to find the GameObject
user clicked. This works only with GameObjects
containing Unity UI elements. In case user needs to click on the GameObject
with no UI element attached he needs to define property in Class Helper
that returns specific camera to perform raycast from.
static class MapLocationClick
{
public static Camera Camera
{
get
{
// return custom game Camera to raycast to specific GameObject
return Map.mapCamera;
}
}
}
In some cases you want to ignore error or exception logs. Unity method LogAssert.Except()
wouldn't work because test runner has its own way to handle logs. You have to use class PermittedErrors
.
You can set errors to ignore in Setup() method or wherever in you test method with methods:
PermittedErrors.AddPermittedException(string message, string stacktrace = "", bool isExactMessage = false, bool isExactStacktrace = false)
PermittedErrors.AddPermittedError(string message, bool isExactMessage = false, bool isExactStacktrace = false)
AddPermittedError()
method is used to ignore log errors and AddPermittedException()
is used to ignore log exceptions.
message
- error messagestacktrace
- exception stacktrace. Only for exceptionsisExactMessage
- if true then error log message should be exact the same as message which is set to permitted errors. Otherwise log message should contains permitted message. Case sensitive.isExactStacktrace
- if true then exception log stacktrace should be exact the same as stacktrace which is set to permitted errors. Otherwise exception stacktrace should contains permitted stacktrace. Case sensitive.
You can clear ignored errors or exception by calling method PermittedErrors.Clear()
.