Fishnet error on Windows: unexpected EOF
Rychveldir opened this issue · 11 comments
Rychveldir commented
Hi, running fishnet 2.6.0, for weeks now without issues. Today at 00:36 it started throwing errors:
I have tried updating, removing the .fishnet-stats file, removing the .ini file, no dice.
niklasf commented
Hi. Which version of Windows is this?
Rychveldir commented
Windows Server 2022 - 21H2 Build 20348.169
i have 2 identical servers running fishnet, only one has this issue
niklasf commented
Thanks.
I published fishnet v2.6.0 today, so that explains why it's happening now after working for a long time.
However I don't understand the cause. https://github.com/lichess-org/fishnet/releases/tag/v2.6.0 lists relevant changes, including an update of the Windows build environment, but Windows Server 2022 should be perfectly fine.
Can you please restart fishnet and send all output leading up to the error?
Then, in the meantime, it would be best to stop fishnet, so that it doesn't accept and fail jobs, or downgrade to v2.5.1 (and run without --auto-update).
Rychveldir commented
c:\Users\phil\Downloads>fishnet-x86_64-pc-windows-gnu.exe --auto-update
# _________ . .
# (.. \_ , |\ /|
# \ O \ /| \ \/ /
# \______ \/ | \ / _____ _ _ _ _ _
# vvvv\ \ | / | | ___(_)___| |__ | \ | | ___| |_
# \^^^^ == \_/ | | |_ | / __| '_ \| \| |/ _ \ __|
# `\_ === \. | | _| | \__ \ | | | |\ | __/ |_
# / /\_ \ / | |_| |_|___/_| |_|_| \_|\___|\__| 2.6.0
# |/ \_ \| /
# \________/ Distributed Stockfish analysis for lichess.org
### Updating ...
><> Checking for updates (--auto-update) ...
Checking target-arch... x86_64-pc-windows-gnu
Checking current version... v2.6.0
Checking latest released version... v2.5.1
><> Fishnet 2.6.0 is up to date
### Checking configuration ...
Endpoint: https://lichess.org/fishnet
Backlog: Join queue if user backlog >= 0ns or system backlog >= 0ns
CPU features: SSE2 | POPCNT | SSE41 | AVX2 | FAST_BMI2 | SF_SSE2 | SF_SSE41_POPCNT | SF_AVX2 | SF_BMI2
Engine: stockfish-x86-64-bmi2.exe (for GPLv3, run: fishnet-x86_64-pc-windows-gnu.exe license)
Cores: 47
### Running (CTRL-C to stop) ...
Resuming from ~/.fishnet-stats ...
[=====|== ] 47 cores, 70 queued, latest: https://lichess.org/D5uJ7CM2
E: Engine error: unexpected end of file
W: Worker 33 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#17
E: Engine error: unexpected end of file
W: Worker 16 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#34
E: Engine error: unexpected end of file
W: Worker 35 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#15
E: Engine error: unexpected end of file
W: Worker 12 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#38
E: Engine error: unexpected end of file
W: Worker 13 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#37
E: Engine error: unexpected end of file
E: Engine error: unexpected end of file
W: Worker 7 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#43
W: Worker 9 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#41
E: Engine error: unexpected end of file
W: Worker 39 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#42
E: Engine error: unexpected end of file
W: Worker 15 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#35
E: Engine error: unexpected end of file
W: Worker 28 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#22
E: Engine error: unexpected end of file
W: Worker 24 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#26
E: Engine error: unexpected end of file
W: Worker 29 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#21
E: Engine error: unexpected end of file
W: Worker 4 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#46
E: Engine error: unexpected end of file
E: Engine error: unexpected end of file
W: Worker 3 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#47
W: Worker 41 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#40
E: Engine error: unexpected end of file
W: Worker 45 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#36
E: Engine error: unexpected end of file
W: Worker 13 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#30
E: Engine error: unexpected end of file
W: Worker 11 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#39
E: Engine error: unexpected end of file
W: Worker 31 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#19
E: Engine error: unexpected end of file
W: Worker 40 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#41
E: Engine error: unexpected end of file
E: Engine error: unexpected end of file
W: Worker 46 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#35
W: Worker 1 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#49
E: Engine error: unexpected end of file
W: Worker 27 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#23
E: Engine error: unexpected end of file
W: Worker 25 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#25
E: Engine error: unexpected end of file
W: Worker 37 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#44
E: Engine error: unexpected end of file
W: Worker 44 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#37
E: Engine error: unexpected end of file
W: Worker 8 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#42
E: Engine error: unexpected end of file
W: Worker 5 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#45
E: Engine error: unexpected end of file
W: Worker 2 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#48
E: Engine error: unexpected end of file
W: Worker 34 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#16
E: Engine error: unexpected end of file
W: Worker 43 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#38
E: Engine error: unexpected end of file
W: Worker 22 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#28
E: Engine error: unexpected end of file
W: Worker 0 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#50
E: Engine error: unexpected end of file
W: Worker 30 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#20
E: Engine error: unexpected end of file
W: Worker 42 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#39
E: Engine error: unexpected end of file
W: Worker 23 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#27
E: Engine error: unexpected end of file
W: Worker 16 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#33
E: Engine error: unexpected end of file
W: Worker 6 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#44
E: Engine error: unexpected end of file
W: Worker 12 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#31
E: Engine error: unexpected end of file
W: Worker 21 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#29
E: Engine error: unexpected end of file
W: Worker 17 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#33
E: Engine error: unexpected end of file
W: Worker 10 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#40
E: Engine error: unexpected end of file
W: Worker 38 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#43
E: Engine error: unexpected end of file
W: Worker 20 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#30
E: Engine error: unexpected end of file
W: Worker 19 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#31
E: Engine error: unexpected end of file
W: Worker 36 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#45
E: Engine error: unexpected end of file
W: Worker 7 waiting for engine to shut down after error. Context: https://lichess.org/D5uJ7CM2#29
### Stopping soon. CTRL-C again to abort pending batches ...
E: Engine error: unexpected end of file
W: Worker 14 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#36
E: Engine error: unexpected end of file
W: Worker 26 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#24
E: Engine error: unexpected end of file
W: Worker 32 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#18
E: Engine error: unexpected end of file
W: Worker 18 waiting for engine to shut down after error. Context: https://lichess.org/NiZNDmjb#32
><> Stopping now.
c:\Users\phil\Downloads>
Rychveldir commented
Running on 2.5.1 now
Rychveldir commented
Ok soooo.. this is now happening on the other machine aswell after it updated itself.
><> fishnet/2.5.1: 4763 knps (nnue), 248.173 batches, 13.604.483 positions, 20.119.577.949.139 total nodes
[ | ] 5 cores, 0 queued, latest: https://lichess.org/E62jUbsF#13
[ | ] https://lichess.org/E62jUbsF finished (4742 knps)
[=====|===========> 5 cores, 25 queued, latest: https://lichess.org/d0BEq2JT#40
><> Checking for updates (--auto-update) ...
><> Fishnet updated to 2.6.1. Will restart soon
[ | ] 5 cores, 0 queued, latest: https://lichess.org/d0BEq2JT#15
[ | ] https://lichess.org/d0BEq2JT finished (4332 knps)
### Waiting 5s before restarting "C:\\Users\\phil\\Downloads\\fishnet-x86_64-pc-windows-gnu.exe" ...
C:\Users\grzadm08\Downloads># _________ . .
# (.. \_ , |\ /|
# \ O \ /| \ \/ /
# \______ \/ | \ / _____ _ _ _ _ _
# vvvv\ \ | / | | ___(_)___| |__ | \ | | ___| |_
# \^^^^ == \_/ | | |_ | / __| '_ \| \| |/ _ \ __|
# `\_ === \. | | _| | \__ \ | | | |\ | __/ |_
# / /\_ \ / | |_| |_|___/_| |_|_| \_|\___|\__| 2.6.1
# |/ \_ \| /
# \________/ Distributed Stockfish analysis for lichess.org
### Updating ...
><> Checking for updates (--auto-update) ...
Checking target-arch... x86_64-pc-windows-gnu
Checking current version... v2.6.1
Checking latest released version... v2.6.1
><> Fishnet 2.6.1 is up to date
### Checking configuration ...
Endpoint: https://lichess.org/fishnet
Backlog: Join queue if user backlog >= 0ns or system backlog >= 0ns
CPU features: SSE2 | POPCNT | SSE41 | AVX2 | FAST_BMI2 | SF_SSE2 | SF_SSE41_POPCNT | SF_AVX2 | SF_BMI2
Engine: stockfish-x86-64-bmi2.exe (for GPLv3, run: C:\Users\grzadm08\Downloads\fishnet-x86_64-pc-windows-gnu.exe license)
Cores: 5
### Running (CTRL-C to stop) ...
Resuming from ~/.fishnet-stats ...
><> fishnet/2.6.1: 2000 knps??? (nnue), 248.175 batches, 13.604.596 positions, 20.119.747.528.149 total nodes
><> fishnet/2.6.1: 2000 knps??? (nnue), 248.175 batches, 13.604.596 positions, 20.119.747.528.149 total nodes
><> fishnet/2.6.1: 2000 knps??? (nnue), 248.175 batches, 13.604.596 positions, 20.119.747.528.149 total nodes
[=====|===========> 5 cores, 36 queued, latest: https://lichess.org/4ZvQZACE
E: Engine error: unexpected end of file
W: Worker 4 waiting for engine to shut down after error. Context: https://lichess.org/4ZvQZACE#41
niklasf commented
At least it fails reliably 😅
So far I narrowed it down to the snapshot from https://github.com/lichess-org/fishnet/actions/runs/2009704414 working and v2.6.0 failing.
niklasf commented
Bisected to official-stockfish/Stockfish@6706263 (skipped official-stockfish/Stockfish@2b03723 and official-stockfish/Stockfish@48bf1a3).
Bad build: https://github.com/lichess-org/fishnet/actions/runs/2190274476
[fishnet 2.6.2-dev] Flags:
[fishnet 2.6.2-dev] CXX: x86_64-w64-mingw32-c++
[fishnet 2.6.2-dev] CXXFLAGS: -fno-asynchronous-unwind-tables -DNNUE_EMBEDDING_OFF -Wall -Wcast-qual -fno-exceptions -std=c++17 -pedantic -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_AVX512 -mavx512f -mavx512bw -DUSE_VNNI -mavx512vnni -mavx512dq -mavx512vl -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2 -flto=auto
[fishnet 2.6.2-dev] LDFLAGS: -static -fno-asynchronous-unwind-tables -DNNUE_EMBEDDING_OFF -Wall -Wcast-qual -fno-exceptions -std=c++17 -pedantic -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_AVX512 -mavx512f -mavx512bw -DUSE_VNNI -mavx512vnni -mavx512dq -mavx512vl -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2 -flto=auto -save-temps
Good build: https://github.com/lichess-org/fishnet/actions/runs/2189931249
[fishnet 2.6.2-dev] Flags:
[fishnet 2.6.2-dev] CXX: x86_64-w64-mingw32-c++
[fishnet 2.6.2-dev] CXXFLAGS: -fno-asynchronous-unwind-tables -DNNUE_EMBEDDING_OFF -Wall -Wcast-qual -fno-exceptions -std=c++17 -pedantic -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -DUSE_AVX2 -mavx2 -DUSE_AVX512 -mavx512f -mavx512bw -DUSE_VNNI -mavx512vnni -mavx512dq -mavx512vl -DUSE_SSE41 -msse4.1 -DUSE_SSSE3 -mssse3 -DUSE_SSE2 -msse2 -DUSE_PEXT -mbmi2
[fishnet 2.6.2-dev] LDFLAGS: -static
niklasf commented
Difference appears to be -flto=auto, so this failure looks like official-stockfish/Stockfish#2769. Will try a few things before giving up and removing -flto=auto ...
niklasf commented
Ok, this was a huge pain, because every change required waiting on the half-hour GitHub workflow build (the issue only happens when downloading a binary built on a different machine), but I think I got a workaround now.
Published v2.6.2. Please update and let me know if it works.
Rychveldir commented
v2.6.2 running fine on both machines without problems now!