lichess-org/fishnet

stockfish is downloaded every time on restart

lenik-covert opened this issue · 7 comments

lenik-covert commented

Somehow the script (I'm running 1.18.2 / python version) thinks there's a new version of stockfish and re-downloads it again and again:

Dec 15 10:59:31 rincewind python[1438181]: Looking up stockfish-x86_64-bmi2 ...
Dec 15 10:59:31 rincewind python[1438181]: Latest release is tagged fishnet-20200613
Dec 15 10:59:31 rincewind python[1438181]: Found https://github.com/niklasf/Stockfish/releases/download/fishnet-20200613/stockfish-x86_64-bmi2
Dec 15 10:59:31 rincewind python[1438181]: Downloading stockfish-x86_64-bmi2 ...
Dec 15 10:59:37 rincewind python[1438181]: chmod +x stockfish-x86_64-bmi2

The file has the latest time, and since the file time is sent in headers, there might be something funny going on the server which serves the stockfish without actually checking if it's older. Or maybe the server version has a timestamp from the future -- I don't see any other explanation.

jim-perkins commented

Just a quick comment till Niklasf checks in. Believe version 1.x has been marked as depreciated. Version 2 is available to download.
EDIT- a3d0758

lenik-covert commented

ok, this affects not only fishnet, but puzzle makers, based on https://github.com/clarkerubber/Python-Puzzle-Creator and probably a bunch of other scripts that rely on downloading the latest stockfish from the github.

I can upgrage to 2.0, but I doubt it will really fix the original problem.

jim-perkins commented

Yea I dont know about that. Did run a quick test on my own instance of fishnet v2.x to see how often it was logging downloading a new version of stockfish. looks like 3 times. I know very little about stockfish but believe it will be updated more frequently since the usage of machine learning

jperkins@lichess:~$ journalctl -u fishnet2 | grep assets
Nov 29 14:42:50 lichess fishnet-x86_64-unknown-linux-gnu[18327]:   * New exe download url: "https://api.github.com/repos/niklasf/fishnet/releases/assets/28918026"
Dec 13 18:01:47 lichess fishnet-x86_64-unknown-linux-gnu[6636]:   * New exe download url: "https://api.github.com/repos/niklasf/fishnet/releases/assets/29509403"
Dec 14 04:50:49 lichess fishnet-x86_64-unknown-linux-gnu[18326]:   * New exe download url: "https://api.github.com/repos/niklasf/fishnet/releases/assets/29528674"
niklasf commented

This comes as a surprise, because I changed nothing (yet) about the distribution mechanism for fishnet 1. Maybe GitHub stopped supporting If-Modified-Since?

For fishnet, please update to 2.x. I am almost certain it will avoid the issue, because --auto-update there uses a much less fragile mechanism that is no longer time based.

For puzzle creation, https://github.com/clarkerubber/Python-Puzzle-Creator in unmaintained. From reading the code, it looks like a workaround would be to hack https://github.com/clarkerubber/Python-Puzzle-Creator/blob/6f7b446e3945125ee5afddd80878ca4be4592332/modules/fishnet/fishnet.py#L31 to update=False after the initial download. I'd recommend switching to https://github.com/ornicar/lichess-puzzler. It's Lichess's next generation puzzle generator.

The updates logged by @jim-perkins correspond to actual fishnet releases. Release notes can be found on this page: https://github.com/niklasf/fishnet/releases

lenik-covert commented

thank you, guys!

lenik-covert commented

@niklasf sorry to bother you with this...

thank you a lot for the references to the puzzle creators you gave, I have one question though, most of them are dedicated to work with lichess API, and for that reason require an API key. there's a page to generate API key in my lichess account, but I'm struggling to understand which features I should enable to 1) be able to download games 2) be able to upload puzzles back ? (the former is more important, the latter is probably not actually needed). Available features are:

  • Read preferences
  • Write preferences
  • Read email address
  • Read incoming challenges
  • Create, accept, decline challenges
  • Read private studies and broadcasts
  • Create, update, delete studies and broadcasts
  • Create tournaments
  • Read puzzle activity
  • Join, leave, and manage teams
  • Send private messages to other players
  • Play games with the board API
  • Play games with the bot API
niklasf commented

The new lichess-puzzler includes a server itself, so multiple instances can submit puzzles into the same database. As far as I know the actual public Lichess API (and tokens for it) are not involved. So you could either set up that server yourself or try to replace the submission mechanism with just printing the puzzle.

Maybe that would be a feature request over on that repository ... making the server optional.