Windows host configuration for Holesky testnet

Question

Windows host configuration for Holesky testnet

jakubgs opened this issue 7 months ago · 18 comments

Since Holesky is the next long-running testnet and Prater is essentially dead it's time to deploy Beacon Nodes on Windows.

Answer 1 · 2024-03-26T12:58:29.000Z

Host was re-purposed from Prater testnet and reinstalled by Innova support:
https://client.innovahosting.net/viewticket.php?tid=047561&c=VMuZGsvh

Answer 2 · 2024-03-26T12:59:22.000Z

Initially there were build issues on Windows Server 2019 Essentials:

status-im/nimbus-eth2#5507

But those were not reproducible on Windows Server 2019 Standard, instead a segmentation fault was identified:

status-im/nimbus-eth2#6139

But that was mitigated by downgrading GCC from 13.2.0 to 11.2.0.

Answer 3 · 2024-03-26T13:01:38.000Z

Based on recommendation from @cheatfate I tried using a ReFS partition instead of NTFS to improve performance:

Which should be available as a format option since Windows Server 2012:

Answer 4 · 2024-03-26T13:03:08.000Z

I had to install HPE Smart Storage Administrator (HPE SSA) to manage the logical volumes and was able to use to 1.6 TB SSDs to create a RAID0 volume:

Answer 5 · 2024-03-26T13:04:06.000Z

And indeed that new logical volume could be formatted as ReFS:

Used 4kb Allocation Unit size based on small page size of SQLite:

Since the SQLite database file format was designed (in 2003) the default page size for new databases has been 1024 bytes. This was a reasonable choice in 2003. But on modern hardware, a 4096 byte page is a faster and better choice. So, beginning with SQLite version 3.12.0 (2016-03-29)) the default page size for new database files has been increased to 4096 bytes.

https://www.sqlite.org/pgszchng2016.html

Answer 6 · 2024-03-26T13:05:36.000Z

There is definitely a performance difference, though some of it is definitely due to use of RAID0:

`C:\` (800 GB LV)	`D:\` (3 TB RAID0 LV)

Not the greatest comparison, but ReFS definitely has an impact.

Answer 7 · 2024-03-26T20:01:47.000Z

Here's the changes:

infra-role-beacon-node-windows#462e2dc5 - meta: use full names of Ansible roles
infra-role-beacon-node-windows#e9aa3604 - config: fix location of data path variable
infra-role-beacon-node-windows#8fafee59 - build: fix setting safe.directory for admin
infra-role-beacon-node-windows#593144c2 - consul: fix variable for wireguard IP address
infra-role-beacon-node-windows#5c171bf8 - service: specify correctly path for service definition
infra-role-beacon-node-windows#8e207124 - config: fix setting data folder permissions
infra-role-beacon-node-windows#832cc4ca - install: add assert to avoid GCC 13.2.0 version
infra-nimbus#6aeb2046 - nimbus.holesky: deploy windows-01 host with nodes
infra-nimbus#467eee3d - nimbus.holesky: use ReFS volume mounted as D drive
infra-nimbus#19e3ede1 - nimbus.holesky: add libp2p node on windows-01 host
infra-nimbus#d62f1960 - nimbus.holesky: set missing fact_path for windows-01
infra-nimbus#dfebbdd9 - requirements: bump windows beacon-node role

Use of ReFS volume probably forced approach to setting permissions of data folder, since it has to be very specific or you get:

{"lvl":"FAT","ts":"2024-03-26 08:49:01.793-07:00","msg":"Data folder has insecure ACL","path":"D:\\beacon-node-holesky-stable\\data"}

Answer 8 · 2024-03-26T20:55:39.000Z

Actually, the old build failures I described here:

status-im/nimbus-eth2#5507

Started appearsing for unstable node:

which gcc &>/dev/null || { echo "C compiler (gcc) not installed. Aborting."; exit 1; }
gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -o wingenminiupnpcstrings.exe wingenminiupnpcstri
ngs.c
gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -DMINIUPNP_STATICLIB -c -o minixml.o src/minixml.
c
gcc -Os -Wall -W -Wstrict-prototypes -DNDEBUG -D_WIN32_WINNT=0x501 -Iinclude -I. -DMINIUPNP_STATICLIB -c -o igd_desc_parse.o src/i
gd_desc_parse.c
gcc: fatal error: cannot execute 'cc1': CreateProcess: No such file or directory
compilation terminated.
make[1]: *** [Makefile.mingw:121: wingenminiupnpcstrings.exe] Error 1
make[1]: *** Waiting for unfinished jobs....
gcc: fatal error: cannot execute 'cc1': CreateProcess: No such file or directory
compilation terminated.
make[1]: *** [Makefile.mingw:101: minixml.o] Error 1
gcc: fatal error: cannot execute 'cc1': CreateProcess: No such file or directory
compilation terminated.
make[1]: *** [Makefile.mingw:101: igd_desc_parse.o] Error 1
make[1]: Leaving directory 'D:/beacon-node-holesky-unstable/repo/vendor/nim-nat-traversal/vendor/miniupnp/miniupnpc'
make: *** [vendor/nimbus-build-system/makefiles/targets.mk:134: libminiupnpc.a] Error 2

Answer 9 · 2024-03-26T20:59:28.000Z

I managed to fix it by copying these files from stable repo that built fine:

nimbus@windows-01 MINGW64 .../miniupnp/miniupnpc ((97d928b...))
$ ls -l wingenminiupnpcstrings.exe
-rwxr-xr-x 1 nimbus 197121 236465 Mar 25 12:23 wingenminiupnpcstrings.exe*

Answer 10 · 2024-03-26T23:00:25.000Z

Interestingly, after I cleaned up miniupnpc:

nimbus@windows-01 MINGW64 .../miniupnp/miniupnpc ((f5d0e49...))
$ g clean -fdx
Removing .Makefile.mingw.swp
Removing addr_is_reserved.o
Removing connecthostport.o
Removing igd_desc_parse.o
Removing minisoap.o
Removing minissdpc.o
Removing miniupnpc.o
Removing miniupnpcstrings.h
Removing miniwget.o
Removing minixml.o
Removing portlistingparse.o
Removing rc_version.h
Removing receivedata.o
Removing upnpcommands.o
Removing upnpdev.o
Removing upnperrors.o
Removing upnpreplyparse.o

It started failing again, so i removed the PATH=".:${PATH}" part from makefiles/targets.mk and it worked:

nimbus@windows-01 MINGW64 /d/beacon-node-holesky-libp2p/repo (nim-libp2p-auto-bump-unstable)
$ make libminiupnpc.a --debug
Reading makefiles...
Updating makefiles....
Updating goal targets....
 File 'libminiupnpc.a' does not exist.
   File 'sanity-checks' does not exist.
  Must remake target 'sanity-checks'.
  Successfully remade target file 'sanity-checks'.
Must remake target 'libminiupnpc.a'.
Successfully remade target file 'libminiupnpc.a'.

No idea why it's necessary at all.

Answer 11 · 2024-03-26T23:08:43.000Z

Made PR with a fix:

status-im/nimbus-build-system#80

Answer 12 · 2024-03-26T23:14:56.000Z

But currently libp2p build fails with:

D:\beacon-node-holesky-libp2p\repo\beacon_chain\networking\eth2_network.nim(2305, 42) template/generic instantiation of `topic` from here
D:\beacon-node-holesky-libp2p\repo\beacon_chain\networking\eth2_network.nim(2299, 11) Error: undeclared field: 'topicIds' for type messages.Message [type declared in D:\beacon-node-holesky-libp2p\repo\vendor\nim-libp2p\libp2p\protocols\pubsub\rpc\messages.nim(40, 3)]                                                            
make: *** [Makefile:448: nimbus_beacon_node] Error 1

I can reproduce this failure on Linux tho so it doesn't seem Windows related.

Answer 13 · 2024-03-26T23:15:42.000Z

But at least stable/testing/unstable running:

Answer 14 · 2024-03-27T08:20:50.000Z

Still seeing a symlinking permissions issue:

Build completed successfully: build/nimbus_beacon_node                                                                                                         
 >>> Install binaries...                                                                                                                                       
ln: failed to create symbolic link 'D:\beacon-node-holesky-unstable\bin/nimbus_beacon_node.exe': Permission denie

Answer 15 · 2024-03-27T13:32:53.000Z

Fixed bin folder permissions:

infra-role-beacon-node-windows#a10542e5 - config: create bin folder as nimbus user

All nodes are up:

I consider this done.

Answer 16 · 2024-03-27T15:14:53.000Z

Forgot to purchase a license. This looks like the best option: Standard for 8 cores for 70 CHF:
https://thekeystore.ch/en/products/microsoft-windows-server-2019-standard-8-core

Answer 17 · 2024-03-27T15:24:42.000Z

Done:

But when I tried to use the key i got:

Gonna contact their support.

Answer 18 · 2024-03-27T18:26:29.000Z

Got a replacement key, we're back in business: