Update multi-variant engine to Fairy-Stockfish
niklasf opened this issue · 4 comments
Todo: Check that all variants are supported with Lichess rules, and that it's a good long term prospect: https://github.com/ianfab/Fairy-Stockfish
With fairy-stockfish/Fairy-Stockfish#264 having been merged, to my knowledge Fairy-Stockfish should support all lichess variants without any deviations in rules. At least my lichess bot https://lichess.org/@/Fairy-Stockfish using Fairy-Stockfish now no longer reports any illegal PV warnings, neither does cutechess in matches, and perft tests so far also all looked good. However, if you consider switching to Fairy-Stockfish I would recommend to focus the compatibility testing especially on atomic, racing kings, and horde, because that have been the most problematic with regard to checking of game end conditions or special rules, although I am not aware of any problems right now.
I also started to add more fine-grained perft tests (previously only had starting positions) to the CI test suite in https://github.com/ianfab/Fairy-Stockfish/blob/master/tests/perft.sh to better ensure stability, and further additions to that are of course very welcome, also of course if you have any further suggestions how to prevent functional regressions (apart from playing strength).
The latest playing strength comparisons against multi-variant Stockfish can be found at https://github.com/ianfab/Fairy-Stockfish/wiki/Playing-strength#compared-to-engine-opponents and http://www.variantfishtest.org:6543/tests. In general the playing strength difference for lichess variants is in the +-200 Elo range, not considering NNUE.
It should be noted that Fairy-Stockfish is not a specialized/optimized variant engine for specific variants, and was not designed or supposed to be a replacement/improvement of MV-SF. It is quite slow due to the overhead for fairy pieces and configurability of rules, causing around a 2-3x slowdown equivalent to around 100 Elo of playing strength. Furthermore, due to the vast number of supported variants, a highly specialized parameter tuning for specific variants like we did in MV-SF is basically out of scope. In total, I would say that a more specialized variant engine (like MV-SF) could normally be ~300 Elo stronger on average. However, this of course assumes similarly active development and usage of handcrafted evaluation. When using NNUE the advantage of more specialized parameter tuning basically gets irrelevant. Fairy-Stockfish's main advantage is that it is a rather generic framework for chess variants and therefore improvements and innovations (like NNUE) can relatively easily be applied to many variants, at the expense of some lack of further optimization that a specialized engine could provide.
I hope this information helps to evaluate the goals of this issue. If you need further info, let me know.
Thank you, that's exactly the information I was going to look for.
For reference: fairy-stockfish/Fairy-Stockfish#281
Released in v2.3.0, for now without NNUE for variants.