Ethereum 2.0 staking

Introduction

Steps to install and configure Arch Linux server running Geth, Ethereum 2.0 beacon chain and validator along with other services for monitoring (Prometheus and Grafana).

The server is connected to a router (with proprietary SW).

Hardware

server
Asus PN50 r8 4800U
32GB DDR4 memory (2x16)
2TB NVMe M.2 Samsung 970 EVO Plus SSD

In order to update BIOS, follow this video. Also, configure the server to boot on AC power (APM settings).

router
Huawei HG8145V5

The router uses its own proprietary system.

UPS
APC Back-UPS Pro

Arch Linux installation

Follow the installation guide and/or watch this video.

Disk partitions

mount point	size	type	file system
`/boot`	512MB	EFI system partition	FAT-32
`/`	10GB	Linux filesystem	Ext4
`/home`	~1.8TB	Linux filesystem	Ext4

/etc/fstab:

# /dev/nvme0n1p2 UUID=c9e84601-3828-4133-9820-f939c6fe8289
/dev/nvme0n1p2      	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p1 UUID=3097-5993
/dev/nvme0n1p1      	/boot     	vfat      	rw,relatime,fmask=0022, ...

# /dev/nvme0n1p3 UUID=35b11765-d2e8-4147-8651-8adbbcb40992
/dev/nvme0n1p3      	/home     	ext4      	rw,relatime	0 2

All partitions are mounted during the installation (in /mnt).

Bootloader installation

pacman -S grub efibootmgr dosfstools os-prober mtools
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=grub_uefi --recheck

Install amd-ucode and generate grub configuration:

pacman -S amd-ucode
grub-mkconfig -o /boot/grub/grub.cfg

Network configuration

Wireless and bluetooth interfaces are disabled in BIOS. There are only loopback and enp2s0f0 (wired network interface) available.

enp2s0f0 is configured to use DHCP, /etc/systemd/network/wired.netword:

[Match]
Name=enp2s0f0

[Network]
DHCP=yes

Boot log check

Check for any errors/warnings with journalctl -kx:

Dec 13 08:52:15 asus kernel: GPT:Primary header thinks Alt. header is not at the end of the disk.
Dec 13 08:52:15 asus kernel: GPT:1397399 != 7827391
Dec 13 08:52:15 asus kernel: GPT:Alternate GPT header not at the end of the disk.
Dec 13 08:52:15 asus kernel: GPT:1397399 != 7827391
Dec 13 08:52:15 asus kernel: GPT: Use GNU Parted to correct GPT errors.

The above error can be fixed with gdisk, enable expert options and select j command.

RAM frequency check

Make sure RAM memory runs on the right frequency (there are some issues reported with RAM running at lower frequency). Use dmidecode -t memory or lshw -short -C memory:

H/W path            Device          Class          Description
==============================================================
/0/0                                memory         64KiB BIOS
/0/30                               memory         32GiB System Memory
/0/30/0                             memory         16GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered) 3200 MHz (0.3 ns)
/0/30/1                             memory         16GiB SODIMM DDR4 Synchronous Unbuffered (Unregistered) 3200 MHz (0.3 ns)
/0/32                               memory         512KiB L1 cache
/0/33                               memory         4MiB L2 cache
/0/34                               memory         8MiB L3 cache

Time service

Enable network time synchronisation:

timedatectl set-ntp true

Start and enable systemd-timesyncd.service:

systemctl start systemd-timesyncd
systemctl enable systemd-timesyncd

Make sure time, timezone and NTP service is active, use:

timedatectl status

DNS service

Start and enable systemd-resolved.service:

systemctl start systemd-resolved
systemctl enable systemd-resolved

Users and groups

user	groups	purpose	home directory
`staker`	`staker`, `wheel`	SSH login, general access	`/home/staker`
`eth1`	`eth1`	Ethereum 1 client	`/home/eth1`
`beacon`	`beacon`	Ethereum 2 beacon chain	`/home/beacon`
`validator`	`validator`	Ethereum 2 validator	`/home/validator`

SSH service

Amend the /etc/ssh/sshd_config based on this guide:

Port 22000
AddressFamily any
ListenAddress 0.0.0.0

# Ciphers and keying
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
#RekeyLimit default none
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
# The available Key Exchange algorithms
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Authentication:

AllowUsers staker

Protocol 2

PubkeyAuthentication yes

HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key

PermitRootLogin no

#LoginGraceTime 2m
#StrictModes yes
MaxAuthTries 6
MaxSessions 6

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to no to disable s/key passwords
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no # pam does that
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	/usr/lib/ssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

The staker user is allowed to connect to the SSH server using public key authentication. Therefore, the SSH public key of this user must be present in staker’s home directory in .ssh/authorized_keys file.

UPS service

This service communicates with UPS. If there is a power failure, the UPS sends a signal to the server once it starts running on batteries. It can also instruct the server to shut down in case there is low power in UPS’s batteries. Follow this guide to configure apcupsd service.

Network speed limiter

TODO

Router configuration

DHCP static IP

The server is always assigned 192.168.1.100 address.

Port forwarding

The router’s port forwarding configuration (the destination IP is the server’s statically assigned IP by DHCP):

service	from port	to port	TCP	UDP
SSH server	22000	22000	x
Geth client	30303	30303	x	x
Prysm beacon chain	12000	12000		x
Prysm beacon chain	13000	13000	x

Dynamic DNS

In case of dynamic public IP address, the router can change its WAN IP with each restart. Therefore, it’s a good idea to configure DDNS.

For example, use dynu.com and configure the router following this guide.

Firewall

TODO

Ethereum 1.0 client (`geth`)

Installation

Install the geth client using pacman:

pacman -S go-ethereum

Configuration

/home/eth1/mainnet/geth/geth.toml:

[Eth]
NetworkId = 1
SyncMode = "fast"
LightPeers = 20

[Node]
DataDir = "/home/eth1/mainnet/geth/data"
HTTPHost = "127.0.0.1"
HTTPPort = 8545
HTTPVirtualHosts = ["localhost"]
HTTPModules = ["net", "web3", "eth"]

[Node.P2P]
ListenAddr = ":30303"
MaxPeers = 20

Note that LightPeers and MaxPeers are both reduced to 20. It’s in order to limit network traffic after the node is synced.

/usr/lib/systemd/system/geth-mainnet.service:

[Unit]
Description=Mainnet geth client
Requires=network.target
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=eth1
Group=eth1
ExecStart=/usr/bin/geth --cache 2048 --pprof --metrics --config /home/eth1/mainnet/geth/geth.toml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

The --pprof and --metrics options are needed for Prometheus metrics on port 6060.

Operation

Start and enable the service:

systemctl start geth-mainnet
systemctl enable geth-mainnet

Open ports:

purpose	port	TCP	UDP
HTTP API	8545	x
P2P/sync	30303	x	x
Prometheus metrics	6060	x

Check open ports with ss -tunlp | grep geth.

Updates are done using pacman -Syu and rebooting the server.

To improve:

logs are not in journald format.

Ethereum 2.0 beacon chain (`prysm`)

Installation

Install the beacon chain client using the intallation script in /home/beacon/mainnet/prysm-bin:

mkdir -p mainnet/prysm-bin && cd mainnet/prysm-bin
curl https://raw.githubusercontent.com/prysmaticlabs/prysm/master/prysm.sh --output prysm.sh && chmod +x prysm.sh

NOTE: The above commands don’t install the beacon chain client, but when the prysm.sh script is run for the first time it downloads the executable.

Configuration

/home/beacon/mainnet/prysm-beacon/beacon.yaml:

datadir: "/home/beacon/mainnet/prysm-beacon/data"
http-web3provider: "http://127.0.0.1:8545"
fallback-web3provider:
- "https://mainnet.infura.io/..."
- "https://eth-mainnet.alchemyapi.io/..."
log-format: text
mainnet: true
accept-terms-of-use: true

Note that after the primary API provider goes down, there are a couple of fallback API providers. If the primary provider goes up again, it will be switched to it automatically.

/usr/lib/systemd/system/prysm-beacon-mainnet.service:

[Unit]
Description=Mainnet beacon chain prysm client
Requires=network.target
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
User=beacon
Group=beacon
ExecStart=/home/beacon/mainnet/prysm-bin/prysm.sh beacon-chain --config-file /home/beacon/mainnet/prysm-beacon/beacon.yaml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Operation

Start and enable the service:

systemctl start prysm-beacon-mainnet
systemctl enable prysm-beacon-mainnet

Open ports:

purpose	port	TCP	UDP
P2P/sync	12000		x
P2P/sync	13000	x
gRPC gateway	3500	x
RPC API	4000	x
Prometheus metrics	8080	x

Check open ports with ss -tunlp | grep beacon.

Updates are done using the prysm.sh script. With each start it checks for the latest version. If there is a newer version, the script downloads it and runs it. Both the prysm-validator-mainnet.service and prysm-beacon-mainnet.service should be updated at the same time. This can be done with:

systemctl stop prysm-validator-mainnet.service prysm-beacon-mainnet.service
systemctl daemon-reload
systemctl start prysm-beacon-mainnet.service
systemctl start prysm-validator-mainnet.service

To improve:

add fallback web3 providers;
logs in journald format lack more details (using text instead);
it would be nicer to have a dedicated package for Arch Linux.

Ethereum 2.0 validator (`prysm`)

Installation

Install the beacon chain client using the intallation script in /home/validator/mainnet/prysm-bin:

mkdir -p mainnet/prysm-bin && cd mainnet/prysm-bin
curl https://raw.githubusercontent.com/prysmaticlabs/prysm/master/prysm.sh --output prysm.sh && chmod +x prysm.sh

NOTE: The above commands don’t install the validator, but when the prysm.sh script is run for the first time it downloads the executable.

Validator keys

Prepare deposit script

Download eth2.0-deposit-cli make sure the SHA256 checksum matches:

sha256sum eth2deposit-cli-ed5a6d3-PLATFORM-amd64.tar.gz

Decompress the file:

tar -xvzf eth2deposit-cli-ed5a6d3-PLATFORM-amd64.tar.gz

Generate validator keys

./deposit new-mnemonic --num_validators 2 --chain mainnet

The above command generated validator-keys directory, which has deposit_data-*.json file, and keystore-m_*.json file for each validator.

Import the generated keys into Prysm wallet

Password used to generate validator keys in the previous step is needed):

./prysm-bin/prysm.sh validator accounts import --keys-dir=path/to/validator_keys

Set the wallet’s path to be /home/validator/mainnnet/prysm-wallet. The wallet has its own password, which is store in plaintext in /home/validator/mainnet/prysm-wallet-password.txt. Make sure both the wallet directory and the password file have the right permissions set:

chmod -R 700 prysm-wallet
chmod -R 600 prysm-wallet-password.txt

It’s also possible to import additional keys:

./prysm-bin/prysm.sh validator accounts import --keys-dir=path/to/validator_keys --wallet-dir=prysm-wallet

Send transactions to deposit contract

Go to ETH2 launch pad and upload the deposit_data-*.json file that will generate transactions in Metamask (one transaction per stake) for Ethereum deposit contract.

The validators become active after:

2048 ETH 1 blocks;
32 ETH 2 epochs;
validators may wait in the validator queue (4 validators are added per epoch, which is 900 per day).

If the validator queue is empty, the minimal waiting time is ~11.5 hours:

2048 blocks = 2048 x 14 seconds = 28,672 seconds = ~8 hours
32 Epochs = 32 x 6.4 minutes = 204.8 minutes = ~3.5 hours

Configuration

/home/validator/mainnet/prysm-validator/validator.yaml:

datadir: "/home/validator/mainnet/prysm-validator/data"
graffiti: "INSERT_GRAFFITI_HERE"
beacon-rpc-provider: "127.0.0.1:4000"
wallet-dir: "/home/validator/mainnet/prysm-wallet"
wallet-password-file: "/home/validator/mainnet/prysm-wallet-password.txt"
log-format: text
mainnet: true
web: false
accept-terms-of-use: true

/usr/lib/systemd/system/prysm-validator-mainnet.service:

[Unit]
Description=Mainnet validator prysm client
Requires=prysm-beacon-mainnet.service
After=prysm-beacon-mainnet.service
StartLimitIntervalSec=0

[Service]
Type=simple
User=validator
Group=validator
ExecStart=/home/validator/mainnet/prysm-bin/prysm.sh validator --config-file /home/validator/mainnet/prysm-validator/validator.yaml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Operation

Start and enable the service:

systemctl start prysm-validator-mainnet
systemctl enable prysm-validator-mainnet

Open ports:

purpose	port	TCP	UDP
Prometheus metrics	8081	x

Note that when web: true the port 7500 is open to access web interface.

Check open ports with ss -tunlp | grep validator.

Check the section about the beacon chain to find out how to proceed with the update.

To improve:

logs in journald format lack more details (using text instead);
it would be nicer to have a dedicated package for Arch Linux.

System monitoring

Grafana ETH 2 dashboard

Follow eth2-grafana for installation and configuration. There are a few differences described below.

Use `pacman`

Install prometheus, prometheus-blackbox-exporter, prometheus-node-exporter and grafana using pacman:

pacman -S prometheus prometheus-blackbox-exporter prometheus-node-exporter grafana

Use the configuration from eth2-grafana, start and enable the services:

systemctl daemon-reload
systemctl enable prometheus-node-exporter prometheus-blackbox-exporter prometheus grafana
systemctl start prometheus-node-exporter prometheus-blackbox-exporter prometheus grafana

[2021-04-11 Sun] There is an issue with prometheus-node-exporter. It’s fixed by adding --no-collector.rapl into /usr/lib/systemd/system/prometheus-node-exporter.service (ExecStart).

`json_exporter`

This package is not in the package repository, so we need to build, install and configure it manually.

Download and build json_exporter (as root in home directory):

pacman -S make go
git clone https://github.com/prometheus-community/json_exporter
cd json_exporter && make build

Create json_exporter group and user:

groupadd -r json_exporter
useradd -r -d / -s /usr/bin/nologin -M -g json_exporter -M json_exporter

Copy the json_exporter binary into /usr/bin:

cp json_exporter /usr/bin/prometheus-json-exporter

Copy the following configuration into /etc/prometheus/json-exporter.yml:

metrics:
- name: ethusd
  path: $.ethereum.usd
  help: Ethereum (ETH) price in USD

Add the following systemd config into /usr/lib/systemd/system/prometheus-json-exporter.service:

[Unit]
Description=Prometheus JSON Exporter
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
User=json_exporter
Group=json_exporter
ExecStart=/usr/bin/prometheus-json-exporter --config.file=/etc/prometheus/json-exporter.yml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Start and enable the service:

systemctl daemon-reload
systemctl start prometheus-json-exporter
systemctl enable prometheus-json-exporter

Grafana ETH 1 dashboard

The configuration for Prometheus has to contain:

- job_name: 'geth node'
  scrape_interval: 5s
  metrics_path: /debug/metrics/prometheus
  static_configs:
  - targets: ['127.0.0.1:6060']

Import dashboard for geth from here.

Grafana HW information dashboard

Import another dashboard from here. It enables very detailed HW monitoring.

Grafana system logs

In order to view system logs in Grafana, the loki and promtail must be installed:

pacman -S loki promtail

The configuration for promtail (/etc/loki/promtail.yaml) must include journal logs (not the default one):

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

The configuration for loki is unchanged.

Both services must be started and enabled:

systemctl daemon-reload
systemctl start promtail
systemctl enable promtail
systemctl start loki
systemctl enable loki

The loki service works as a datasource for Grafana and it is added in Grafana’s UI (http://localhost:3100). The system logs are accessible through “Explore”, where datasource is “Loki”.

Open ports related to Grafana

service	port	TCP
`prometheus`	9090	x
`prometheus-blackbox-exporter`	9115	x
`prometheus-node-exporter`	9100	x
`prometheus-json-exporter`	7979	x
`grafana-server`	3000	x
`promtail` HTTP	9080	x
`promtail` ?	39603	x
`loki` HTTP	3100	x
`loki` gRPC	9095	x

Grafana alerts to Telegram

TODO

Prysm validator web interface

The validator has its own web interface running on port 7500. Although, it’s disabled at the moment (not very useful).

Remote access

SSH client is configured as follows (.ssh/config):

Host *
    # Send "ping" each 60s if inactive
    ServerAliveInterval 60
    VisualHostKey yes
    # TCP connection multiplexing
    ControlMaster auto
    ControlPath ~/.ssh/multiplex/%r@%h:%p
    ControlPersist 1

Host ethserver
    HostName YOUR_IP/DOMAIN
    User staker
    Port 22000
    # Grafana
    LocalForward 3000 localhost:3000
    # Prysm validator web
    LocalForward 7500 localhost:7500
    # Prysm beacon chain gRPC
    LocalForward 3500 localhost:3500
    # Prysm beacon chain Prometheus metrics
    LocalForward 8080 localhost:8080
    # Prysm validator Prometheus metrics
    LocalForward 8081 localhost:8081

After connecting to the server using ssh ethserver, a browser can access Grafana (http://localhost:3000) and Prysm validator web interface (http://localhost:7500).

jur0/eth-staking

Ethereum 2.0 staking

Introduction

Hardware

Arch Linux installation

Disk partitions

Bootloader installation

Network configuration

Boot log check

RAM frequency check

Time service

DNS service

Users and groups

SSH service

UPS service

Network speed limiter

Router configuration

DHCP static IP

Port forwarding

Dynamic DNS

Firewall

Ethereum 1.0 client (geth)

Installation

Configuration

Operation

Ethereum 2.0 beacon chain (prysm)

Installation

Configuration

Operation

Ethereum 2.0 validator (prysm)

Installation

Validator keys

Prepare deposit script

Generate validator keys

Import the generated keys into Prysm wallet

Send transactions to deposit contract

Configuration

Operation

System monitoring

Grafana ETH 2 dashboard

Use pacman

json_exporter

Grafana ETH 1 dashboard

Grafana HW information dashboard

Grafana system logs

Open ports related to Grafana

Grafana alerts to Telegram

Prysm validator web interface

Remote access

Ethereum 1.0 client (`geth`)

Ethereum 2.0 beacon chain (`prysm`)

Ethereum 2.0 validator (`prysm`)

Use `pacman`

`json_exporter`