OpenBench is an open-source sequential probability ratio testing framework designed for self-play testing by both UCI and xboard compliant engines. OpenBench provides a lightweight interface and client which allows machines to work together to test changes to the engine for performance and stability. OpenBench currently has support for Fischer Random Chess.
You can join OpenBench's Discord server to join the discussion, see what developers are working on and talking about, or to find out how you can contribute to the project or become apart of it.
OpenBench is the primary testing framework used for the development of Ethereal. The primary instance of OpenBench can be found here. This OpenBench instance currently supports development for Ethereal, as well as for Demolito, Laser, Rubi, Weiss, FabChess, Igel, and Winter. Some engine authors are running their own public or private instances of OpenBench to support development for their engines.
OpenBench is heavily inspired by Fishtest. The project is powered by the Django Web Framework and Cutechess.
Install any version of python3
Install any POSIX compliant version of MinGW
Download a copy of the OpenBench Client file
sudo apt-get install python3 or yum install python3
sudo apt-get install gcc or yum install gcc
sudo apt-get install g++ or yum install g++
Download a copy of the OpenBench Client file
The client takes four arguments: Username, Password, Server, Threads. First create an account via the instance's webpage. The instance will also be the Server provided to the client. Threads tells the Client how many games can be run in parallel. This should be no more than your CPU count minus one. For hyperthreaded machines, where hyperthreading support is strong, you may use all threads provided by CPU minus one cores. The following would connect to the main instance using 4 threads or CPUs.
python3 OpenBench.py -U username -P password -S http://chess.grantnet.us/ -T 4
The client will create an Engines, Books, and PGNs directory. These are used to store compiled engines, downloaded opening books, and PGNs of the games played. By default, engine binaries that are 24 hours or older will be deleted, and recompiled if again needed. Additionally, PGNs are not saved by default. However, this is an option at the top of Client.py to enable this feature.
The only other thing needed by Client's is access to make. All engines are built using a make command. The Client will check for the compilers that may be needed, and only download workloads that it can compile. This allows a very minimal set of requirements to contribute CPU time.
When ./engine bench is run, the engine must provide a node count and nodes per second count after performing a search on some set of positions. The goal is that by making minor changes to the engine, the node count will vary. This acts as a hash for the engine. OpenBench performs a basic regex pattern to find the node count and nodes per second. Your engine should default to using a small amount of hash. This is because if a worker connects with 32 threads, it will run 32 benches in parallel.
OpenBench assumes that your program can be made simply via make EXE=<sha256>. Your makefile may be located anywhere, as that information is stored on the Server and sent to Clients. It is best for your makefile to default to a popcnt build, as not all workers are capable of using pext or bmi2 compiles. An advanced User may modify their Client.py to supply additional arguments to make.
The options provided to the engines in all OpenBench tests must start with "Threads=X Hash=Y". Your engine should support a provided Hash value. If your engine does not support a variable number of Threads, Cutechess will simply not pass on the information, and will report a warning which can be ignored safely.
Your engine should play nice when closed by error, by Cutechess, by python, or by any other means. This can be verified by checking for hanging processes. Generally this is not an issue, but poor code for reading input pipes can cause this to occur. This is crucial, as majority of machines connected to the main OpenBench instance are not constantly monitored.
Add your engine to OpenBench/config.py, similar to Ethereal's configuration. The test modes are fairly standard, and are derived from those of Fishtest. the NPS value is provided in order to scale CPUs of different speeds. Ethereal's speed of a 32-thread Ryzen 1950x. It is recommend to follow this scaling, so that the time controls have meaning without association with a particular CPU.
Create a user through the website, then through the Admin interface (found at /admin from your server), enable your profile. You may also set yourself as an approver so that you may approve tests written by others. Additionally, to bypass the cross-approval mechanism, you may set yourself as a superuser.
sudo apt-get install python3, pip3, git
pip3 install Django==3.0.5
pip3 install django-htmlmin
git clone https://github.com/AndyGrant/OpenBench
cd OpenBench
python3 manage.py makemigrations
python3 manage.py migrate
python3 manage.py migrate --run-syncdb
python3 manage.py createsuperuser
python3 manage.py runserver 127.0.0.1:8000