evil-minions is a load generator for Salt. It is used at SUSE for Salt, Uyuni and SUSE Manager scalability testing.
evil-minions is a program that monkey-patches salt-minion in order to spawn a number of additional simulated "evil" minions alongside the original one.
Evil minions will mimic the original by responding to commands from the Salt master by copying and minimally adapting responses from the original one. Responses are expected to be identical apart from a few details like the minion ID (it needs to be different in order for the Master to treat them as separate).
Evil minions are lightweight - hundreds can run on a modern x86 core.
sumaform users: follow sumaform-specific instructions
SUSE distros: install via RPM package
# replace openSUSE_Leap_15.0 below with a different distribution if needed
zypper addrepo https://download.opensuse.org/repositories/systemsmanagement:/sumaform:/tools/openSUSE_Leap_15.0/systemsmanagement:sumaform:tools.repo
zypper install evil-minions
Other distros: install via pip
git checkout https://github.com/moio/evil-minions.git
cd evil-minions
pip install -r requirements.txt
mkdir -p /etc/systemd/system/salt-minion.service.d
cp override.conf /etc/systemd/system/salt-minion.service.d
Starting salt-minion will automatically spawn any configured evil minions (10 by default). systemd is used in order to work correctly in case the minion service is restarted.
Every time the original minion receives and responds to a command, the command itself and the responses are "learned" by evil minions which will subsequently be able to respond to the same command. In practice, issuing the same command to all minions will work, eg.
salt '*' test.ping
evil minions will wait if presented with a command they have not learnt yet from the original minion.
Several parameters can be changed via commandline options in /etc/systemd/system/salt-minion.service.d/override.conf.
The number of evil minions can can be changed via the --count commandline switch.
Simulating minions is not very resource intensive:
- each evil-minon consumes ~2 MB of main memory, so thousands can be fit on a modern server
- ~1000 evil-minions can be simulated at full speed (or near full speed) on one modern x86_64 core (circa 2018)
- this means that a hypervisor vCPU, typically mapped to one HyperThread, will be able to support hundreds of evil minions
evil-minions combines multiprocessing and Tornado to fully utilize available CPUs.
By default, evil minions will respond as fast as possible, which might not be appropriate depending on the objectives of your simulation. To reproduce delays observed by the original minion from which the dump was taken, use the --slowdown-factor switch:
0.0, the default value, makes evil minions respond as fast as possible1.0makesevil-minionintroduce delays to match the response times of the original minion2.0makesevil-minionreact twice as slow as the original minion0.5makesevil-minionreact twice as fast as the original minion
By setting the random-slowdown-factor value, the evil minions will respond using the user defined slowdown-factor value plus a random extra delay expressed as a percentage of the original time, for example:
- with a
slowdown-factorof1.0and arandom-slowdown-factorof0.2, the evil-minions will reply with a slowdown-factor in the range of1.0and1.2. This value it's calculated per-evil-minion and is constant throughout the whole execution.
The default value of --random-slowdown-factor parameter is 0.
For more information, see --slowdown-factor section above.
Please, use evil-minions --help to get the detailed list.
- only the ZeroMQ transport is supported
- only
*and exact minion id targeting are supported at the moment - some Salt features are not faithfully reproduced:
mineevents,beaconevents, andstate.sls'sconcurrentoption - some Uyuni features are not currently supported: Action Chains