⚠ This project is still in active development, and APIs are subject to change. ⚠
Testing framework for GMS 2.3.1+ projects with useful features:
- Record Keeping - Test results are recorded and json exportable
- Crash Recovery - Resume progress after runner crashes
- Async Support - Easily test async events
- Tester Feedback - Easily compose instruction for testers and gather feedback
- Flexible - Use any assertion library you want
- Powerful Customization - Supports life cycle hooks, global options, and local options
Olympus is developed by Butterscotch Shenanigans ("Bscotch"). Check out our other project Ganary, which uses Olympus to run regression tests for GameMaker Runtime.
GameMaker Studio 2® is the property of Yoyo Games™. Butterscotch Shenanigans® and Olympus are not affiliated with Yoyo Games.
Import the resources in the "Olympus" group, such as using Stitch:
stitch merge --source-github=bscotch/olympus --if-folder-matches=OlympusCompose your test suite:
//Name your test suite
olympus_run("my suite name", function(){
//Add a unit test
olympus_add_test(
//Name your unit test
"my unit test name",
//Define the test assertion logic
function(){
var expected = "2";
var actual = "1";
if (actual != expected){
throw({
message: "Expected: " + expected + ". Actual: " + actual,
stacktrace: debug_get_callstack()
});
}
});
});Test record is written to file in the Save Area and a summary is shown in IDE output:
-------------------------
passed: 0
failed: 1
skipped: 0
crashed: 0
Record written to file as Olympus_records/my_suite_name.olympus.json
------------------------- - Olympus
Full API Reference is shown in the olympus_external_api script resource
The test suite summary data is a GML struct with the following shape:
{
"tallies": { //The tallies of unit tests results
"skipped": 0,
"crashed": 0,
"total": 1,
"passed": 0,
"failed": 1
},
"name": "my_suite_name", //The suite name defined by `olympus_run(suite_name)`
"tests": [
//Array of unit test summaries. See [Unit Test Summary](#unit-test-summary)
]
}You can access the most up to date version of this data through olympus_get_current_suite_summary().
You can also access this data at the start and the finish of a test suite by olympus_add_hook_before_suite_start() and olympus_add_hook_after_suite_finish().
You can set a function be executed right before a test suite starts with olympus_add_hook_before_suite_start().
The entire suite summary is passed to the function, so you can do something like iterating through all the added tests and announcing their names:
olympus_add_hook_before_suite_start(function(suite_summary){
show_debug_message("This suite contains the following tests:")
var tests = suite_summary.tests;
for (var i = 0; i < array_length(tests); i++){
var this_test = tests[i];
show_debug_message(this_test.name);
}
})You can set a function be executed after a test suite finishes with olympus_add_hook_after_suite_finish().
The entire suite summary is also passed to the function, so you can do something like annoucing the tallies of test results:
olympus_add_hook_after_suite_finish(function(suite_summary){
show_debug_message("Test completed.")
var tallies = suite_summary.tallies;
show_debug_message("total: " + tallies.total);
show_debug_message("skipped: " + tallies.skipped);
show_debug_message("crashed: " + tallies.crashed);
show_debug_message("passed: " + tallies.passed);
show_debug_message("failed: " + tallies.failed);
})The unit test summary is the elements in the suite summary struct's tests array, which is also accessible by calling the olympus_get_current_test_summaries() function.
{
"index": 0, //The unit test's index as the nth element of the suite's `tests` array
"name": "my unit test name", //The unit test name defined by `olympus_add_*(name)`
"status": "failed", //The unit test result
"millis": 4, //The time span of the unit test in milliseconds
"err": { //The error struct if unit the test did not pass
"message": "Expected: 2. Actual: 1",
"stacktrace": [
"demo_olympus_quick_start:10",
"_olympus_internal:485",
"_olympus_async_test_controller_Step_0:18",
]
}
}You can access this data at the start and the finish of each unit test by olympus_add_hook_before_each_test_start() and olympus_add_hook_after_each_test_finish().
Once you get a hold of the unit test summary struct, you can use the convenience function olympus_get_test_status() to access the status variable and olympus_get_test_name() to access the name variable.
You can set a function be executed before each unit test starts with olympus_add_hook_before_each_test_start().
The unit test summary is passed to the function, so you can do something like announcing the name of the unit test:
olympus_add_hook_before_each_test_start(function(unit_summary){
show_debug_message("Start testing: " + unit_summary.name)
})You can set a function be executed after each unit test finishes with olympus_add_hook_after_each_test_finish().
The unit test summary is also passed to the function, so you can do something like logging the error of the unit test if it did not pass:
olympus_add_hook_after_each_test_finish(function(unit_summary){
if (unit_summary.status != olympus_test_status.passed){
show_debug_message(unit_summary.err);
}
})GML's async events are mediated through objects. Taking http_get() as an example, we need some sort of mediator objects (let's call it obj_http_mediator) whose Async HTTP Event gives us the access to async_load when http_get() finally resolves:
///Pt 1
///obj_http_mediator Create Event
http_handle = http_get("https://google.com")
///obj_http_mediator Async HTTP Event
if async_load[?"id"] == http_handle{
show_debug_message("http status is: " + string(async_load[?"http_status"]) )
}With GML 2.3.1+, we can store script functions in variables and execute the functions by "calling" the variables:
///Pt 2
handler_function = function(the_async_load){
show_debug_message("http status is: " + string(the_async_load[?"http_status"]) )
}
handler_function(async_load); This language feature allows us to flexibly define what obj_http_mediator does with async_load. We start by storing the handler function into the instance variable handler_function:
///Pt 3
///Create Event
handler_function = function(){}
http_handle = http_get("https://google.com")
///Async HTTP Event
if async_load[?"id"] == http_handle{
//Parse the async_load, such as reading async_load [? "result"];
handler_function(async_load);
}When we spawn obj_http_mediator, we can reassign the variable handler_function to a new function:
///Pt 4
with instance_create_depth(0,0,0,demo_obj_http_mediator){
var new_handler_function = function(async_load_from_mediator){
show_debug_message("http status is: "+string(async_load_from_mediator[?"http_status"]))
}
handler_function = new_handler_function;
}Once you set up an obj_http_mediator as shown above, you can test http_get() with Olympus by following these steps:
- Wrap your async mediator object spawning logic in a function, and make sure that this function returns the mediator instance ID:
///Pt 1
var mediator_spawning_logic = function(){
return instance_create_depth(0,0,0,obj_http_mediator)
}- Define your
new_handler_functionof how to handle theasync_load. Note because Olympus packages the originalasync_loadinto theargumentarray, you have to retrieve it as the 0th element of the array:
///Pt 2
var new_handler_function = function(argument){
var async_load_from_mediator = argument[0];
var http_status = async_load_from_mediator[?"http_status"];
if (http_status == 200){
show_debug_message("Pinging Google succeeded.");
}
else{
throw("Expected 200. Got: " + string(http_status));
}
}- Let Olympus know the instance variable name of the handler function by constructing an options struct that has the variable name
resolution_callback_name
///Pt 3
var options_to_register_handler_function_name = {
resolution_callback_name: "handler_function"
}- Pass all these to
olympus_add_async_test():
///Pt 4
olympus_add_async_test("Test Pinging Google", mediator_spawning_logic, new_handler_function, options_to_register_handler_function_name);- Wrap all of these inside the
olympus_run()block:
///Pt 5
olympus_run("My Suite Name", function(){
//Define the logic to spawn the async mediator object and return its instance ID
var mediator_spawning_logic = function(){
return instance_create_depth(0,0,0,obj_http_mediator)
}
//Define your new_handler_function
var new_handler_function = function(response_array){
var async_load_from_mediator = response_array[0];
var http_status = async_load_from_mediator[?"http_status"];
if (http_status == 200){
show_debug_message("Pinging Google succeeded.");
}
else{
throw("Expected 200. Got: " + string(http_status));
}
}
//Register the mediator object's instance variable name of the handler function
var options_to_register_handler_function_name = {
resolution_callback_name: "handler_function"
}
//Add the test as an async test to the suite
olympus_add_async_test("Test Pinging Google", mediator_spawning_logic, new_handler_function, options_to_register_handler_function_name);
});Olympus will run all your added async tests sequentially, wait for each one to resolve before moving on to the next one.
For tasks such as confirming graphics/audio rendering, it may be difficult to verify with assertion logic. You can use olympus_add_async_test_with_user_feedback() to render the effect, serve a text propmt to the user, and let them decide whether the test passed or not.
The prompt uses the cross-platform supported get_string_async() method, which allows the user to pass or fail the test:
All the examples can be selected in the demo room creation code and run in the IDE with the demo config. You can only run one demo at a time as Olympus does not support concurrent suite running.
When adding async tests, Olympus needs to know the mediator object's instance variable name for the function that handles the async result. There are 3 ways to make the names known to Olympus:
As shown in step 5 of Testing Async with Olympus, we passed an options struct with the variable name resolution_callback_name to olympus_add_async_test to inform Olympus what the instance variable name is for the handler function:
var options_to_register_handler_function_name = {
resolution_callback_name: "handler_function"
}
olympus_add_async_test(..., options_to_register_handler_function_name); If all of your mediator objects use the same instance variable name for their async handler function, you can pass an options struct with the variable name global_resolution_callback_name to olympus_run to make that name known to Olympus:
var options_to_register_global_handler_function_name = {
global_resolution_callback_name: "handler_function"
}
olympus_run(..., options_to_register_global_handler_function_name); global_resolution_callback_name is set to "callback" by default, so if your mediator objects already use that name, you do not need to override the default.
NOTE: Each test's own resolution_callback_name option will take precedence to the suite's global_resolution_callback_name option.
olympus_test_resolve is a syntactic sugar that saves the hassel of having to define the resolution_callback_name or global_resolution_callback_name options. Taking the earlier obj_http_mediator example in the Background section, instead of:
///Async HTTP Event
if async_load[?"id"] == http_handle{
handler_function(async_load);
}You can just have:
///Async HTTP Event
if async_load[?"id"] == http_handle{
handler_function(async_load);
//`handler_function` must not mutate the content of `async_load`
olympus_test_resolve(async_load);
}Behind the scenes, olympus_test_resolve calls a function whose name is already known to Olympus, so you don't have to define the resolution_callback_name or global_resolution_callback_name options.
NOTE: The best practice is to make a copy of async_load to be passed to olympus_test_resolve so that we don't have to worry about olympus_test_resolve and handler_function interfere with each other.
Because the runner has to exit after uncaught exception occurs, a suite of tests are not guaranteed to complete if a particular test unit throws an uncaught exception or silently crashes.
Olympus deals with this by keeping track of the last test unit status and writing the progress to file. Upon crash and reboot, this allows the runner to unstuck itself by identifying the last running unit as the crash cause and skipping it to complete the test suite.
To enable this behavior, create an options struct with the variable name resume_previous_record and set it to true, and pass it to olympus_run():
var options_to_enable_crash_recovery = {
resume_previous_record: true
}
olympus_run(..., options_to_enable_crash_recovery); Sometimes we may want to pass shared variables between unit tests. This is doable as all the unit tests within olympus_run() have access to the same scope by default, so you can do something like this:
shared_variable_sum = 0;
olympus_run("shared variables test", function(){
olympus_add_test("sum should be 1", function(){
shared_variable_sum ++;
show_debug_message(string(shared_variable_sum)); //1
});
olympus_add_test("sum should be 2", function(){
shared_variable_sum ++;
show_debug_message(string(shared_variable_sum)); //2
})
})Alternatively, you can explicitly define what variables the tests should have access to by passing the options struct with the variable olympus_suite_options_context that points to a struct:
not_explicitly_defined_variable = "goodbye";
olympus_run("shared variables from custom context test", function(){
olympus_add_test("", function(){
show_debug_message(explicitly_shared_variable);
show_debug_message(not_explicitly_defined_variable); //Variable struct.not_explicitly_defined_variable not set before reading it.
});
}, {
olympus_suite_options_context: {
explicitly_shared_variable : "hello"
}
})Sometimes when an earlier unit test fails, we want to skip later unit tests. This can be done in 3 ways:
By passing an options struct as {bail_on_fail_or_crash: true} to olympus_run(), any unit test that fails or crashes will cause the rest of the unit tests to be skipped.
By passing an options struct as {dependency_names: ["test_name1", "test_name2"]} to any of the olympus_add*() APIs, the unit test will be skipped if any of its dependencies did not pass.
Unit tests added between olympus_test_dependency_chain_begin() and olympus_test_dependency_chain_end() will be treated as sequentially dependent on each other, while tests outside of the chain are not affected.
You can construct an options struct and pass to olympus_run() that will affect the test suite's global behavior across all unit tests:
| Name | Type | Default | Description |
|---|---|---|---|
| [olympus_suite_options_resume_previous_record] | boolean |
false |
Enabling this starts the suite from wherever the last run left off. Otherwise, the suite starts from the beginning. |
| [olympus_suite_options_skip_user_feedback_tests] | boolean |
false |
Enabling this skips tests that requires user feedback. |
| [olympus_suite_options_suppress_debug_logging] | boolean |
false |
Enabling this suppresses Olympus from logging to the IDE Output tab. |
| [olympus_suite_options_test_interval_milis] | number |
0 |
Adds a delay between each test. Useful if you want to allow an audio or a visual cue to be played between tests. |
| [olympus_suite_options_global_resolution_callback_name] | string |
"callback" |
Name of the instance variable for the resolution callback for all the mediator objects |
| [olympus_suite_options_global_rejection_callback_name] | string |
"reject" |
Name of the instance variable for the rejection callback for all the mediator objects |
| [olympus_suite_options_bail_on_fail_or_crash] | boolean |
false |
Enabling this will skip the rest of the tests if an earlier test fails or crashes |
| [olympus_suite_options_context] | struct |
The binding context for function_to_add_tests_and_hooks. The default uses the calling context. | |
| [olympus_suite_options_global_timeout_milliseconds] | number |
60000 | If any test is not able to resolve within this many milliseconds, the test will be failed. |
| [olympus_suite_options_allow_uncaught] | boolean |
false |
By default, Olympus catches uncaught error and record it. Enabling this allows uncaught error to be thrown instead and will stop recording test summaries or resuming unfinished records. |
| [olympus_suite_options_ignore_if_completed] | boolean |
false |
Enabling this will ignore re-running the suite if the suite has been completed previously. |
You can construct an options struct and pass to the olympus_add* APIs that will affect that specific unit test's behavior:
| Name | Type | Default | Description |
|---|---|---|---|
| [olympus_test_options_resolution_callback_name] | string |
If you have not defined a global_resolution_callback_name or want to overwrite that, specify it here | |
| [olympus_test_options_rejection_callback_name] | string |
If you have not defined a global_rejection_callback_name or want to overwrite that, specify it here | |
| [olympus_test_options_dependency_names] | string | string[] |
Names of tests whose failure will cause this test to be skipped | |
| [olympus_test_options_context] | struct |
The binding context for function_to_execute_synchronous_logic or function_to_spawn_object. The default uses the calling context. | |
| [olympus_test_options_resolution_context] | struct |
The binding context for function_to_execute_at_resolution. The default uses the calling context. | |
| [olympus_test_options_timeout_milliseconds] | number |
60000 | If this test is not able to resolve within this many milliseconds, the test will be failed. |
- Olympus uses
exception_unhandled_handler()to log uncaught errors. If you also usesexception_unhandled_handler(), make sure to re-assign your error handler function after the Olympus test suites conclude. - All unit tests must have unique names to support the dependency chaining. If you named two tests with the same name, the runner should throw an error on boot.
Please make sure to read the Contributing Guide before making a pull request.