Open Source Tournament Manager Software for StarCraft: Broodwar AI Competitions
Created and maintained by David Churchill and Rick Kelly, organizers of the AIIDE Starcraft AI Competition
This software is a tool for running Starcraft AI bot tournaments using BWAPI. It uses a server-client architecture with one machine acting as a server and any number of other machines acting as clients. The tournament manager is written entirely in Java, and can be run on Windows 7 or higher, either on a physical machine or a virtual machine. All data sent and received is compressed and passed through Java sockets, so no special network configuration is required to run the software.
This repository includes precompiled server and client jar files, as well as the complete Java 7 source code. It also includes several required files for setup such as BWAPI .dll files which will automatically be configured and run for you. Also included are the bots and maps from the 2014-2016 AIIDE StarCraft AI Competitions (zipped). With these files you should be able to run a tournament as quickly as you can install StarCraft on all of your client machines!
This software regularly creates, deletes, and sends files over sockets, use it at your own risk.
The following video uses an older version of the tournament manager, but the set up procedure is still quite similar, and it demonstrates the main functionality of the software. The main difference from what you see in the video and the new software are that the settings files are now in JSON format, and the GUIs look slightly different. An updated video will be coming soon.
When running the software, one machine acts as a server for the tournament. The server is a central repository where all bot files (including file I/O) data, cumulative results, and replay files are stored. The server also monitors each client remotely and outputs results data that can be viewed in html. Tournament status can be viewed in real time via the server GUI.
The server program has a threaded component which monitors for new client connections and detects client disconnections, maintaining a current list of clients which can have one of the following statuses:
- READY - Client is free and ready to start a game of StarCraft
- STARTING - Client has started the StarCraft LAN lobby but the match has not yet begun
- RUNNING - Client is currently running a game of StarCraft
- SENDING - Client has finished the game and is sending results and data back to the server.
The server's main scheduling loop tries to schedule the next game from the games list every 2 seconds. Normally a new game can be started only if:
- two or more Clients are READY, and
- no clients are STARTING.
The reason no clients can be STARTING is to prevent multiple StarCraft game lobbies to be open on the same LAN, which may cause mis-scheduled games due to limitations in BWAPI versions previous to 4.2.0 on how we are able to join games automatically. If all bots use BWAPI 4.2.0 or higher there is an option in the server settings file to enable multiple games to start at the same time. Once these two conditions are met, the server sends the required bot files, map files, BWAPI version, and DLL injector to the client machines, specifying one client as the host and one as the away machine. Those clients' status are then set to STARTING.
Each client is handled by a separate thread in the server, and if the client is STARTING, RUNNING, or SENDING, it sends periodic status updates back to the server for remote monitoring. Data such as current game time, time-out information, map, game ID, etc are each updated once per second from each client to the server GUI. When a client finishes a game the results are sent back to the server along with file I/O data and replay files, which are all stored on the server. This process repeats until the tournament has finished.
Shutting down the server via the GUI will cause all client games to stop and all client software to shut down and clean up properly. The tournament can be resumed upon re-launching the server program as long as the results file, games list, and settings file do not change. If the server is shut down with games in progress (results not yet received by the server), those games will be rescheduled and played again if the same tournament is restarted.
The client software can be run on as many machines that are available on your LAN. After an initial setup of the client machine (installing StarCraft, etc.) the client software connects to the server machine to await instructions.
The client machine will stay idle until it receives instructions from the server that a game is to be run. Once the client receives the required files from the server, it ensures that no current StarCraft processes are running, records a current snapshot of the running processes on the client machine, writes the BWAPI settings file, and starts the game. When the game starts, a custom BWAPI Tournament Module is injected which outputs a GameState file to disk every few frames, which monitors the current state of StarCraft. The client software reads this file to check for various conditions such as bot time-outs, crashes, no game frame progression, and game termination. As the game is running, the client sends the contents of the GameState file to the server once per second to be monitored on the server GUI.
Once the game has terminated for any reason, the results of the game, replay files, and file I/O data are sent back to the server. Once the sending is complete, the client software shuts down any processes on the machine which were not running when the game began, to prevent things like crashed proxy bots or stray threads from hogging system resources from future games. StarCraft is shut down, the machine is cleaned of any files written during the previous game, and the client status is reported back to the server as READY.
The Tournament Manager comes with a stand-alone results parser that generates summary and detailed results using the games.txt, results.txt, and settings.json files found in the /server directory. The results parser is useful if you want to view HTML results for different tournaments you've previously run by swapping out the games list and results files.
Running a tournament using this software requires the following prerequisites:
- Microsoft Windows 7 (or higher) (Clients)
- Physical or Virtual Machines with minimum 2 CPU cores (Clients)
- StarCraft: BroodWar (Clients only)
- Microsoft VC++ Redistributables (See Below, Clients only)
- Any prerequisites for bots in the tournament, BWTA/BWTA2 DLLs, specific JDK versions required by Java bots, etc. (Clients only)
- Java JDK 8 (Clients and Server)
Download a zip file containing all Visual Studio redists and easy install script here: all_vcredist_x86.zip.
Download or clone the repository to any directory on your server machine that does not contain spaces. You will find the following directory structure:
TournamentManager/
client/ Client Directory
BWAPI.ini Default BWAPI settings file
client.jar Client jar file
client_settings.json Client settings file (modify this)
run_client.bat Script to run client
server/ Server Directory
bots/ Contains all files for each bot
botname/ Bot-specific directory
AI/ Where the .dll / proxy bot files go
read/ File I/O read directory
write/ File I/O write directory
html/ HTML for viewing results
css/ CSS files
javascript/ Javascript files
results/ Output folder for results data in JSON format
index.html Summary of tournament results
results.html Detailed results from each game
win_percentage_graph.html Win % over time for all bots
replays/ Replay storage directory *
required/ Required file storage directory
Required_BWAPI_374.zip BWAPI/Starcraft required files (BWAPI 3.7.4)
Required_BWAPI_401B.zip BWAPI/Starcraft required files (BWAPI 4.0.1 Beta)
Required_BWAPI_412.zip BWAPI/Starcraft required files (BWAPI 4.1.2)
Required_BWAPI_420.zip BWAPI/Starcraft required files (BWAPI 4.2.0)
Required_BWAPI_440.zip BWAPI/Starcraft required files (BWAPI 4.4.0)
games.txt Default tournament games list filename *
parse_results.bat Script to run results parser
results.txt Default tournament results filename *
results_parser.jar Parses existing results like server without running server
run_server.bat Script to run server
server.jar Server .jar file
server_settings.json Server settings file (modify this)
src/ Source Code Folder
packagename/ Source package directories
clean.bat Script to delete all .class files in subdirs
make.bat Script to compile / make jar files
*doesn't exist by default, but server will create
The tournament manager comes pre-compiled as 2 jar files (client/client.jar, server/server.jar), however if you want to compile the code you can use make.bat in the src/ folder, or any other build system you wish. The make.bat script will run clean.bat (delete .class files), compile the necessary Java files, create the required .jar files and put them into the correct sub-directories.
- Install Java JDK 8 or higher
- (Windows) Edit system PATH to include jdk/bin directory for javac and java
- Turn off firewall
- The following directory structure must exist for each bot in the tournament:
- server/bots/BotName/ - Bot directory
- server/bots/BotName/AI/ - Holds all AI files for each bot including BotName.dll
- server/bots/BotName/AI/BotName.dll - Bot .dll must be named the same as folder (the .dll file can be empty for a proxy bot, but must exist)
- server/bots/BotName/AI/run_proxy.bat - If bot is a proxy bot, this file must exist
- server/bots/BotName/read/ - Bot read directory
- server/bots/BotName/write/ - Bot write directory
- Put your map files inside the server/required/Required_*.zip files under the 'maps' folder
Note: The tournament manager comes with the bots and maps which competed in the 2014-2016 AIIDE StarCraft AI Competitions. Make sure that each bot and map you change is listed correctly in server/server_settings.json
Proxy Bots: The file server/bots/BotName/AI/run_proxy.bat is a file which must exist if your bot is listed as 'proxy' in server_settings.json, it will be run on the client machine immediately BEFORE StarCraft is launched, and must contain all code necessary to launch your proxy bot. All proxy bot files must be stored in the server/bots/BotName/AI/ directory (subdirectories allowed), since this is the folder which is copied to the client machine before a game starts.
- Edit server/server_settings.json to suit your tournament needs
- Run server/run_server.bat (Use 64-bit Java and -Xmx option in run-server.bat to set high memory limit for large tournaments (10,000+ games))
- If previous results exist, server will ask to resume tournament or delete previous results (under default configuration)
- If a games list file does not exist, server will prompt to generate one. It can generate a round robin tournament or a 1 vs all tournament based on the bots and maps in server_settings.json. Currently only round robin tournament games generation is supported, however you can edit the games list manually as long as you follow the existing syntax.
- Server will check if required files exist before launching GUI
- Once the GUI has started, you must manually launch client software on client machines
The Server GUI displays information about the tournament in the top bar, followed by a list of connected clients and their current status, and a log at the bottom. Game duration is given in normal speed Starcraft time, though the game is likely playing much faster (depending on server settings and client hardware).
The menu options under "Actions" allow you to:
- Generate detailed results (normally this is the only way to generate detailed results, unless automatic generation is turned on in the server settings),
- Send a command to all clients, or
- Request a screenshot of a client's screen.
The same client options, plus the options to kill clients or filter the log by client, can be found by selecting and right-clicking on one or more clients in the client list.
- Install StarCraft: BroodWar to a directory containing no spaces
- Upgrade StarCraft to version 1.16.1
- Install Microsoft Visual C++ Redists (zip file containing all Visual Studio redists and easy install script)
- Install Java JDK 8 or higher
- Edit system PATH to include jdk/bin directory for javac and java
- 32bit Windows: Edit registry so that HKLM\SOFTWARE has "Full Control" for current user
- 64bit Windows: Edit registry so that HKLM\SOFTWARE\Wow6432Node has "Full Control" for current user
- Copy client/ directory to client machine
- Edit client/client_settings.json to match your tournament setup
- Run client/run_client.bat
Note: Exactly one client instance can be run per machine. Clients can be run on a physical or a virtual machine (minimum 2 CPU cores), as long as your LAN settings can handle StarCraft's UDP communication. One client can be run on the same machine as the server. I regularly run sample tournaments with the server and one client running on a physical machine, with a 2nd client running in a VM.
The Client GUI has no options or actions you can take. It only displays information about the current game or status, and a log.
- Whenever a game result is received from a Tournament Manager client, the raw results are written to server/results.txt, and a results summary is written to server/html/results/results_summary.js (viewed at server/html/index.html)
- Detailed match results can be written manually via the Server GUI, or automatically whenever a game result is received if you turn on the option in the server settings file. It is off by default due to long processing times for very large tournaments.
- Replay files are saved to server/replays/BotName
- Bot read/write directories are stored in server/bots/BotName/(read|write)
Raw and detailed results contain a field called "gameEndType"/"End Type", which indicates if the game ended normally (NORMAL), or ended in some kind of crash or didn't start.
Game End Types:
- NORMAL
- GAME_STATE_NOT_UPDATED_60S: gamestate file wasn't updated for 60 seconds, but Starcraft is still running.
- STARCRAFT_CRASH: crash detected by Tournament Manager client, which means that Starcraft was running at some point, but later the client couldn't find the Starcraft process.
- GAME_STATE_NEVER_DETECTED: no game state file (output from TournamentModule once game is running) detected by time limit (60s), but StarCraft is running.
- STARCRAFT_NEVER_DETECTED: no game state file (output from TournamentModule once game is running) detected by time limit (60s), and StarCraft never detected running.
- NO_REPORT: only one report received; assigned after the fact (detailed results only); this could be caused by the TM client crashing, or a weird network error, etc.
If two clients report different end types, the one lower in this list is shown in the detailed results.
Note: Detecting when a bot crashes is difficult, so a crash is recorded whenever the game doesn't progress (gamestate file not updated) for more than a minute. In these cases the bot who recorded the higher frame count (meaning the game was running for longer) is declared the winner. This means it isn't possible to distinguish between a crash and a case of a bot taking more than a minute to process a single frame of the game.
Crashes in which the game never starts (frame count for both bots is zero) are not counted in the results summary in html/index.html.
Games that last more frames than gameFrameLimit in server_settings.json (default setting is equal to one hour at normal speed) are terminated, and the winner is the bot with the higher score.
These losses are reported as "Game Timeout" in the results summary and "Timeout" in the detailed results page.
If you are running a tournament from a Windows server, you can use PSExec to start your clients from the server rather than having to start the clients manually on each machine.
.bat file for starting clients (NOT INCLUDED IN THE REPOSITORY):
for /F "tokens=*" %%A in (client_list.txt) do PsExec.exe -i -d \\%%A cmd /S /C "cd c:\tm\client\ & java -jar client.jar client_settings.json"where client_list.txt contains something like:
192.168.1.102 -u username1 -p password1
192.168.1.103 -u username2 -p password2
If you're making changes to the software or your client settings you might want to use commands like this to transfer new versions to your client machines:
for /F "tokens=*" %%A in (client_list.txt) do PsExec.exe -i -d \\%%A -c -v ..\client\client.jar cmd /S /C ""
for /F "tokens=*" %%A in (client_list.txt) do PsExec.exe -i -d \\%%A cmd /S /C "move /Y c:\Windows\client.jar c:\TM\client\"All server configuration is done in /server/server_settings.json. This file must parse as valid JSON or the server will not start.
| Name | Value |
|---|---|
| bots |
Type: Array of json objects These are the bots that will play in the tournament. Each bot object must contain the following name/value pairs:
|
| maps |
Type: Array of strings Each round of the tournament will be played on these maps in the order they are listed in. The value should be the path to the map relative to the Starcraft directory; no spaces Example: "maps/aiide/(2)Benzene.scx" |
| gamesListFile |
Type: String Location of file with list of games to be played, relative to server.jar; No spaces. The user will be prompted to generate a new games list if the file does not already exist (i.e. if this is a new tournament). |
| resultsFile |
Type: String Location of tournament results file, relative to server.jar. No spaces. Raw results data returned from clients is stored in this file (one line for each client). Nice results are output by the server in the html/ directory. |
| detailedResults |
Type: Boolean Setting to true auto-generates detailed results whenever a game result is received. Generating detailed results gets slow for very large tournaments, so default is false. You can manually generate the results from the Actions menu in the server, which is recommended. |
| serverPort |
Type: Number Port to listen for clients on. This should match the port number in the client's ServerAddress setting. |
| clearResults |
Type: String Clear existing results on server start? Allowed values: "yes" | "no" | "ask" If "yes" then a new tournament is always started when the server is started. If "no" then an existing tournament will be resumed if possible. |
| startGamesSimultaneously |
Type: Boolean If set to true new games will be started while other games are still in the starting process (i.e. other Starcraft instances are in the lobby). If set to false only one game can be STARTING at a time. WARNING: This is only usable if all bots are using BWAPI version 4.2.0 or higher. If using older versions of BWAPI, bots will join any game in the lobby, leading to games with more than 2 players, and generally games that do not match. |
| tournamentType |
Type: String Allowed values: "AllVsAll" | "1VsAll"
|
| lobbyGameSpeed |
Type: String Allowed values: "Slowest" | "Slower" | "Slow" | "Normal" | "Fast" | "Faster" | "Fastest" This setting changes registry entries on the client machines so that the game speed slider in the game creation lobby is set appropriately. The actual game speed will be overridden by the tournamentModuleSettings.localSpeed setting, but the slider affects the number of latency frames in the game (the number of frames between a command and its execution). |
| enableBotFileIO |
Type: Boolean If set to true the server will wait for each round to complete before sarting the next round. Every time a round finishes the contents of 'BotName/write' will be copied to 'BotName/read'. Bots that implement learning from previous rounds will have access to the contents of the read directory in 'bwapi-data/read' on the client machine. If set to false the server will ignore round numbers when scheduling games, and never copy from 'write' to 'read'. |
| ladderMode |
Type: Boolean If set to true enables some alternate functionality designed to work with a persistant online ladder server in development by the authors of this project. Should not be used otherwise. |
| excludeFromResults |
Type: Array of strings Bots listed in this array will be excluded from the results summary and detailed results output, but games that include them will still be played. This feature is useful if you need to disqualify a bot from a tournament, or want to see the overall effects of a bot on the results. The values in this array should match bot names given in the array of competing bots. |
| tournamentModuleSettings |
Type: Object Tournament Module settings control the tournament module DLL which is injected into each Starcraft instance with BWAPI. It controls game speed, draws information to the screen, and outputs data about the game being played so that the client can tell if a bot has crashed, timed out, etc. |
| tournamentModuleSettings .localSpeed |
Type: Number BWAPI Local Speed; Calls BWAPI::Broodwar->setLocalSpeed(SpeedValue). Set to 0 to run games at the fastest speed possible. |
| tournamentModuleSettings .frameSkip |
Type: Number BWAPI Frame Skip; Calls BWAPI::Broodwar->setFrameSkip(SkipValue) This does nothing unless LocalSpeed is 0. |
| tournamentModuleSettings .gameFrameLimit |
Type: Number Game Frame Time Limit; Game stops when BWAPI::Broodwar->getFrameCount() > FrameLimit If gameFrameLimit is 0, no frame limit is used. Normal Starcraft speed is 24 frames per second. |
| tournamentModuleSettings .timeoutLimits |
Type: Array of json objects Each timeoutLimit object must contain the following name/value pairs:
|
| tournamentModuleSettings .drawBotNames |
Type: Boolean Set to true to draw bot names on the game screen. |
| tournamentModuleSettings .drawTournamentInfo |
Type: Boolean Set to true to draw tournament information on the game screen. |
| tournamentModuleSettings .drawUnitInfo |
Type: Boolean Set to true to draw unit information on the game screen. |
Example server_settings.json:
{
"bots": [
{"BotName": "UAlbertaBot", "Race": "Random", "BotType": "proxy", "BWAPIVersion": "BWAPI_420"},
{"BotName": "ExampleBot", "Race": "Protoss", "BotType": "dll", "BWAPIVersion": "BWAPI_412", "ClientRequirements": [{"Property": "GPU"}]}
],
"maps":
[
"maps/aiide/(2)Benzene.scx",
"maps/aiide/(2)Destination.scx"
],
"gamesListFile" : "games.txt",
"resultsFile" : "results.txt",
"detailedResults" : false,
"serverPort" : 1337,
"clearResults" : "ask",
"startGamesSimultaneously": false,
"tournamentType" : "AllVsAll",
"lobbyGameSpeed" : "Normal",
"enableBotFileIO" : true,
"ladderMode" : false,
"excludeFromResults" : ["ExampleBot"],
"tournamentModuleSettings":
{
"localSpeed" : 0,
"frameSkip" : 256,
"gameFrameLimit": 85714,
"timeoutLimits" :
[
{"timeInMS" : 55, "frameCount": 320},
{"timeInMS" : 1000, "frameCount": 10},
{"timeInMS" : 10000, "frameCount": 1}
],
"drawBotNames" : true,
"drawTournamentInfo": true,
"drawUnitInfo" : true
}
}| Name | Value |
|---|---|
| ClientStarcraftDir |
Type: String Directory of Starcraft on client machine; no spaces; end in "\" (backslashes must be escaped). |
| TournamentModule |
Type: String Location of BWAPI Tournament Module DLL, relative to Starcraft directory; no spaces. |
| ServerAddress |
Type: String IP address and port of server. Example: "192.168.1.100:1337" |
| ClientProperties (Optional) |
Type: Array of strings Features of this client that a bot can take advantage of if matched to its ClientRequirements in server settings. This array can be empty if you are not using the properties feature. |
Example client_settings.json:
{
"ClientStarcraftDir" : "C:\\TM\\Starcraft\\",
"TournamentModule" : "bwapi-data/TournamentModule.dll",
"ServerAddress" : "192.168.1.100:1337",
"ClientProperties" : ["GPU"]
}- Fix for client crash when there are extra files in the replay directory
- Fix for some actions that were mistakenly allowed by the Tournament Module in BWAPI 4.1.2 and 4.2.0 (Contribution by Chris Coxe)
- Client count added to server GUI
- Added win percentage by round to results
- Menu option to open results in browser
- New option in server settings to set StarCraft game lobby speed for all games
- Games list and results are stored in JSON format
- New field in detailed results: "Game End Type" provides more information about crashes
- For proxy bots, run_proxy.bat is now run from the Starcraft directory, rather than from Starcraft/bwapi-data/AI. This is more consistant with DLL bots for accessing the read and write directories. run_proxy.bat can still change the working directory if needed before running the bot.
- Bot requirements can have "!" before them to indicate that the client machine should not have that property
- Added BWAPI 4.4.0 support (Contribution by Chris Coxe)
- Fixed issue where crashes in which the game did not start were not outputting timer stats, causing error reading in results file on server.
- Fixed issue where 1vsALL tournament mode didn't work as intended.
- Added BWAPI 4.2.0 support.
- Allow multiple matches to start simultaneously by ensuring that hosts are always different (enable ONLY for BWAPI 4.2.0 bots and higher).
- ChaosLauncher is no longer used for injecting BWAPI; injectory is used instead.
- Settings files switched to JSON format.
- Added option to give properties to clients, (e.g. "GPU") and requirements to bots. Requirements and properties have to match for a client to run a game.
- Added Results Parser which outputs the results summary and detailed results of a tournament without having to start the server.
- Server GUI displays progress bar.
- Server GUI shows uptime counter.
- Logs include date as well as time.
- Server log can be filtered.
- Added right-click actions for clients: kill, take screenshot, run command, filter.
- Detailed tournament results and summary are now output in JSON format. HTML and CSS have been moved out of the tournament manager output so it is easier to format the results to your liking.
- Added win percentage graph to results output.
- Detailed results now say "unknown" for crashing bot where game didn't start.
- Added filters to Detailed Results Page.
- Added option to exclude a bot from results output in server settings