terite/faketorio

Run automatic tests for your mod inside Factorio

Faketorio

Support library for automated testing of Factorio mods

Purpose

The problem with modding in Factorio (or any other script language based modding environment) is testing. Usually this is solved by a huge amount of manual testing. This amount grows with every new feature and every bugfix.. because in theory you have to test EVERYTHING EVERY.SINGLE.TIME (super annoying).

This is where automated testing comes in. I'm new to lua testing, but there are a couple unit testing frameworks available.

The problem with unit testing is that it only gets you so far. The most interesting aspect (how does it behave in the real game) is hard to mimic.

Enter Faketorio. The goal is to provide a standard system for running your mod against your local Factorio installation, execute tests INSIDE Factorio and then package your mod for release.

Installation

If you use LuaRocks for your lua modules it is as easy as running luarocks install faketorio.

If you do all the dependency management yourself you will have to download the files from this github repo and place them in a convenient location. Then require them with the proper path.

Usage

For a (very) short info on how to interact with faketorio type faketorio -h. For the long version keep reading :)

There are two main aspects of faketorio regarding your mod. The first is mod management and packaging and the second is testing.

Mod management

With Faketorio you can run and build your mod. To do this we need two things: a .faketorio config file in your home folder and a certain folder structure in your mod.

Folder structure

Faketorio assumes the following structure for your mod files:

- MyModFolder/
    - info.json             // the file with all your mod info
    - src/                  // all your mod source files go here (in as many subfolders as you like)
        - control.lua
        - data.lua
        - settings.lua
        - mymod.lua
    - locale/               // all your locale files as described in [aubergine10's guide to localization](https://github.com/aubergine10/Style/wiki/Localisation)
    - spec/                 // all your test files go here (as described by [busted](https://olivinelabs.com/busted/#defining-tests)
        - mod_spec.lua      // normal [busted](https://github.com/Olivine-Labs/busted) unit tests (or any other tests for that matter)
        - mod_feature.lua   // faketorio tests that are run inside of Factorio
    - target/               // temporary folder that faketorio uses to assemble files and folders

The Factorio specific files (control.lua, data.lua, settings.lua, ...) all go into the source folder without any additional subfolders.

.faketorio config file

Faketorio requires a config file to run. You can specify the location with the -c option. If no config path is provided faketorio will search for a file named .faketorio in the current folder. This file has to have three values configured.

factorio_mod_path = /home/jjurczok/.factorio/mods
factorio_run_path = /home/jjurczok/.local/share/Steam/steamapps/common/Factorio/bin/x64/factorio

# windows based example
factorio_run_path = D:\Spiele\Factorio\bin\x64\factorio.exe

faketorio_path = /usr/share/lua/5.2/faketorio

factorio_mod_path describes the folder where all your mods are. On my machine this is in the home folder, on windows it will be somewhere else factorio_run_path is the path to the executable (Factorio.exe on windows)

faketorio_path is the path where you installed faketorio itself. If you installed it via luarocks you can find this path with luarocks show faketorio or luarocks doc faketorio

Faketorio commands

With the configuration in place and the mod structured we can now start interacting with faketorio. Faketorio should be run from your mod folder. To do this open a terminal (or command line with start -> execute -> cmd on windows) and navigate to the folder where you do your development (ideally NOT inside the Factorio mods folder ;).

Now you can execute the faketorio commands to interact with your mod and Factorio.

• faketorio build
  This will simply create a properly named and filled folder in the target folder. This is the basis for all other commands and will be executed implicitly for you. You can use it to verify the contents of your packaged mod before it actually gets packaged.
• faketorio clean
  This command simply removes the target folder to give you a clean start (if in doubt about the contents and you want to force a regeneration of the mod)
• faketorio copy
  This command is used to reduce the develop -> copy mod -> start Factorio -> test -> close Factorio -> develop cycle. If you already have Factorio running this command will replace the mod in the Factorio mods folder with the version that you are currently developing. With this you can just click restart map in the game and have the newest version of the mod loaded. ATTENTION: changing locales/grafics requires a restart of the game as these resources are only loaded on game startup.
• faketorio package
  This command creates a properly named zipfile in the target folder that is named according to the Factorio mod naming conventions (ModName_Version). The information for this are extracted from your info.json. This file can then be uploaded to the Factorio mod portal.
• faketorio run
  This command does roughly the same as the build command. Except that it also copies the folder to your factorio mod folder and then starts Factorio. It will order Factorio to create a new map, load your mod and then run Factorio with that newly generated map. This gives you a clean game to test your mod. As Factorio loads mods that are not zipped over mods that are zipped you don't even have to change the version number in your info.json.
• faketorio test
  This command works like the run command except that it also adds all your faketorio feature files to the mod, enabeling you to run your automated tests inside Factorio. Simply wait for Factorio to finish loading and then type /faketorio into the debug console ingame. ATTENTION: This is not completely implemented yet!!

Tests

Now that we covered how to interact with Factorio itself it is time to talk about the real reason for this library. The actual testing.

The tests are inspired by busted and use roughly the same basic principles.

-- in mod_feature.lua
feature("My first feature", function()
    
    before_scenario(function()
        -- will be called before every scenario in this feature
        -- this is intended for setting up preconditions for this specific feature
    end)

    after_scenario(function()
        -- will be called after every scenario in this feature
        -- this is intended to bring the mod back into a state as it would be expected from the next test
    end)
    
    scenario("The first scenario", function()
        faketorio.click("todo_maximize_button")
        faketorio.log.info("my first test")
    end)

    scenario("The second scenario", function()
        faketorio.log.info("Scenario 2")
    end)
end)

As described in Folder Structure you can create feature files in your spec folder. These files have to follow the naming pattern <name>_feature.lua. It is best practice to have one feature per file.

In the scenarios you can basically write normal lua code to interact with your mod. Faketorio provides you with some additional functions that you can use.

Running tests ingame

To run your tests inside of Factorio simply invoke faketorio test in a terminal in the mod folder. Faketorio will generate a new map, copy your mod and the tests and starts Factorio.

As soon as you are in game you can open the debug console and enter /faketorio. Simply run the command and all your tests will be executed.

You will see a dialog popping up in the middle of the screen. The top bar indicates progress on feature level, the lower bar indicates progress on scenario level for the currently running feature.

If there are test errors they will be documented in the textbox below the progress bar after the testrun finishes.

Marking tests as success/failure

By default any scenario function that completes is counted as successful. If you want to mark your test as a failure (because some assertion did not work) you can just raise an error

-- in mod_feature.lua
feature("My first feature", function()
    
    scenario("The first scenario", function()
        faketorio.print("This test is successful.")
    end)

    scenario("The second scenario", function()
        faketorio.print("This test fails.")
        error("test failure")
    end)
end)

At the end of every test run you will receive a report to the console (and the logfile) indicating what worked and what did not. As usual a testrun is only successful if ALL tests pass!

TODO: provide screenshots of success and failure cases

TESTRUN FINISHED: SUCCESS
Run contained 1 features.

Interacting with Factorio

Of course just interacting with the data/tables of your mod is not enough. You also have to be able to mimic player behaviour. This is what the following functions are for.

faketorio.click(name)

To interact with your mod you need to be able to click on things ;) This is what the click function is for. The name parameter is the name of your gui element.

Faketorio will search through the four guis (left, top, center, goal) of the first player in the players list. If an element with the provided name is found it will raise an event for this element.

Currently only left mouse button clicks without modifiers (shift, alt, control) are supported.

faketorio.enter_text(name, text)

This function enters text into a text field. Simply provide a name to look for and faketorio will replace the .text attribute with the provided text.

faketorio.find_element_by_id(id, player)

Returns a gui element with the given id for the given player. Both parameters are mandatory.

logging

The log system functions are intended for debug output and if you want to report something to the user while the tests are running.

The system knows four different log levels. TRACE, DEBUG, INFO and WARN. The default log level is INFO. All messages that are logged with a level lower (read left in the list) as the current log level will be ignored.

Messages passed to the logging system will be printed to every player in Factorio. Additionally it will be written to the faketorio.log file in your Factorio script-output folder in your application directory.

To create log messages use one of the following functions

-- simple logging
faketorio.log.trace("my test debug message")
faketorio.log.debug("my test debug message")
faketorio.log.info("my test debug message")
faketorio.log.warn("my test debug message")

-- logging with parameter expansion (prints "my test pattern wololo.")
faketorio.log.trace("my test pattern %s.", {"wololo"})
faketorio.log.debug("my test pattern %s.", {"wololo"})
faketorio.log.info("my test pattern %s.", {"wololo"})
faketorio.log.warn("my test pattern %s.", {"wololo"})

-- to change the current log level and thus limiting the output during your tests use
faketorio.log.setTrace()
faketorio.log.setDebug()
faketorio.log.setInfo()
faketorio.log.setWarn()

mocks

Sometimes it is necessary to change the behavior of your mod, pretending certain external events happened. One example would be the user modified the mod settings. As the settings table is read only for the mod we can just mock the result.

Assuming you have a function like this

function myMod.is_setting_enabled(player)
    return settings.get_player_settings(player)["my-mod-setting"].value
end

we can now change the behavior of that function by mocking it:

local player = ...
myMod.is_setting_enabled(player) -- returns true (default value)

-- lets create a mock
when(myMod, "is_setting_enabled"):then_return(false)

myMod.is_setting_enabled(player) -- returns false (the mocked value)

myMod.is_setting_enabled:revert()
myMod.is_setting_enabled(player) -- returns true again (it calls the original function)

Credits

Faketorio is inspired by Factorio stdlib and the work Afforess did there to enhance the testing. Over time I hope to become as complete as this testing system.