Pytest Tests for Kaiserreich

Repository for .py tests for Kaiserreich with the possibility to set up Jenkins as a runner. It can be used 'as is' for Kaiserreich users, they only need to pass their system username in which doc folder the project is located and the name of the mod folder (see screenshots lower). For other HOI4 projects, it can be used as well but no support is provided.

The general idea of the project is to automate the scenarios testing that is almost impossible to verify otherwise (they can be checked manually via CWTools in some cases, but my solution benefits from all automation perks - it is never tired, fast and provides almost instant feedback). Tests are NOT running the game, instead, they parse and analyze the codebase. Repository is constantly updated - tests are mostly created upon finding bugs.

In-built multithreading support and high-performance optimization, current full runtime - around 3 minutes with 6 cores utilised.

Requirements - Python installation with pytest and pytest-xdist plugins.

Currently included tests

90+ tests, including:

Advisors tests

usage of non-unique advisor tokens test (it causes bugs all over the place. Advisors should have unique tokens)
usage of missing idea tokens in advisors activation
invalid advisors costs test
invalid advisors ledger lines test
invalid advisors not_already_hired_except_as lines values
advisors with invalid traits tests (all advisors slots)

Characters tests

missing characters test (finds all characters that are checked via 'character =' or 'has_character =' but never defined)
missing gfx portraits files test
unused characters test (characters that exist but never recruited)
duplicated characters test
characters without loc keys in their names test
characters with several advisor roles and missing not_already_hired_except_as lines
characters (unit leaders) with missing small portrait links
unit leaders with traits of unsupported type

Cosmetic tags tests:

unused cosmetic tags test (finds all cosmetic tags that are set but never checked)
unused cosmetic tags test - colors (finds all cosmetic tags colors that are never used)
missing cosmetic tags test (finds all cosmetic tags that are not set but checked)

Character flags tests:

unused character flags test (finds all character flags that are set but never checked)
missing character flags test (finds all character flags that are not set but checked)
cleared character flags test (finds all character flags that are not set but cleared)

Country flags tests:

unused country flags test (finds all country flags that are set but never checked)
missing country flags test (finds all country flags that are not set but checked)
cleared country flags test (finds all country flags that are not set but cleared)

Decisions tests:

decisions ai factor test (verifies that decisions and selectable missions have ai factors)
unused decisions categories test
duplicated decisions categories test
duplicated decisions test
war declaration decisions AI code test (checks if decisions have required AI code so AI performs better)

Events tests

triggered-only events that are not triggered from outside test (check for events that are not triggered from outside but should)
missing triggered events test
events with no pictures

Event targets tests

unused event targets test (finds all event targets that are set but never checked)
missing event targets test (finds event targets that are not set but checked)
cleared event targets test (finds all event targets that are not set but cleared)

Global flags tests:

unused global flags test (finds all global flags that are set but never checked)
missing global flags test (finds all global flags that are not set but checked)
cleared global flags test (finds all global flags that are not set but cleared)

Ideas tests:

unused ideas test (finds all ideas that are set but never checked)

Localization tests

localization files typo check (parses loc files and finds commonly misspelled words, as well as prints correct variant)
localization files syntax violations
localization files unused loc entries
Scripted loc brackets usage (finds oddities with vanilla scripted loc functions usage, such as missing brackets or unexpected following characters)
Unused scripted loc test
Usage of inline loc in scripted localisation files

OOB files tests:

unused OOB files (files with unit templates and/or division/ships spawns that are not used)
missing OOB files (files with unit templates and/or division/ships spawns that are missing)

Opinion modifiers tests:

unused opinion modifiers test (finds all opinion modifiers that are set but never checked)

Performance tests

a lot of tests related to unoptimized triggers usage
focuses/decisions/events that can be optimized by using ideology flags tests

Scripted triggers and effects tests:

unused scripted triggers test (finds all scripted triggers that are not used)
unused scripted effects test (finds all scripted effects that are not used)

State flags tests:

unused state flags test (finds all state flags that are set but never checked)
missing state flags test (finds all state flags that are not set but checked)
cleared state flags test (finds all state flags that are not set but cleared)

Syntax tests:

missing 'limit' expression in if/elif conditions test
usage of outdated syntax for armor equipment bonuses test (equipment -> chassis)
usage of DLC-locked armor chassis for non NSB owners and vice versa test (owners of NSB have specific armor equipment available; non-DLC players have their own. Checks if these two types of equipment are not mixed)
usage of outdated syntax for doctrines cost reduction test (tech_bonus -> doctrine_cost_reduction)
railways file test (verifies the expected and actual number of provinces provided in /map/railways.txt file)
usage of negative multiplication in ai factors test (negative multiplication is not what you want in 99% of cases)
usage of 4+ digits after decimal point in ai factors test (HOI4 supports only 3 digits after decimal point)
admirals and generals stats syntax tests (verifies generals and admirals have correct stats assigned)
usage of vanilla .dds icons in armour variants test
usage of incorrect equipment type in modifiers

Project development timeline on the Jenkins graph

Pytest instructions

It allows to run test locally, all you need is python installation and installed 'pytest' and 'pytest-xdist' plugins

Create venv with pytest installed - 'pip install pytest' or 'pipenv install pytest' if you use pipenv
Install pytest-xdist - 'pip install pytest-xdist' or 'pipenv install pytest-xdist' if you use pipenv
Clone repository with tests
Change directory to repository directory
Run:

pytest -v -s --tb=short "--username=xxx" "--mod_name=xxx" -n 6"

in console, replace username with your system username and mod_name with mod folder name, -n - number of your CPU cores

Pytest-Jenkins instructions

It allows to run tests automatically based on specific triggers

Create a Python Virtual Environment, install Pytest via 'pip install pytest' or 'pipenv install pytest' if you use pipenv, install 'pytest-xdist' - 'pip install pytest-xdist' or 'pipenv install pytest-xdist' if you use pipenv
Install Jenkins
Install Python Plugin and ShiningPanda Plugin Jenkins plugins:
Provide Git and Python paths in "Manage Jenkins/Global Tool Configuration/" (For Python - provide path to your virtual environment python.exe file)
Create a new Job (Freestyle project)
Configure the job

GitHub repository, branch - main
Build action - Custom Python Builder, path to your venv (not to exe), nature - Shell, command:

pytest -v -s --tb=short "--username=xxx" "--mod_name=xxx" -n 6 --junitxml TestResults.xml

(replace username with your system username and mod_name with mod folder name, -n - number of your CPU cores)

Post-Build Actions - Publish Junit test result report, test report xmls - *.xml

Setup the Build Triggers (or you can trigger the job manually)
Save the job