Teku Validator client unable to find keys
Closed this issue · 8 comments
Describe the bug
Teku Validator client unable to find validator keys
To Reproduce
Start Sedge with network Gnosis and Teku for CL client. once synced start the validator.
had sedge started up a day ago, EL and CL clients synced okay. prompted to start the validator (at this point did not generate keys yet)
Not able to access the 'validator-data' folder until I changed permissions
keys generated restart validator client
sedge keys --network gnosis
places keys in ./keystore/validator_keys
and password.txt in ./keystore
Teku is looking for keys in --validator-keys=/data/keys:/data/passwords
I think Teku requires password file per validator with same name structure but .txt
extension
Tried to Import validators
ran the container to import the keys, and restarted the validator client
same error and looks like the validators are imported to
./validator-data/validators
Hello @GLCNI, we are looking into your issue now!
Could you please provide us the docker-compose.yml
file under the docker-compose-scripts
folder?
docker-compose.yml
version: "3.9"
services:
execution:
stop_grace_period: 30s
container_name: execution-client
restart: unless-stopped
image: ${EC_IMAGE_VERSION}
networks:
- sedge
volumes:
- ${EC_DATA_DIR}:/nethermind/data
- ${EC_JWT_SECRET_PATH}:/tmp/jwt/jwtsecret
ports:
- 30303:30303/tcp
- 30303:30303/udp
expose:
- 8545
- 8551
command:
- --config=${EL_NETWORK}
- --datadir=/nethermind/data
- --log=${NETHERMIND_LOG_LEVEL}
- --Sync.SnapSync=${EC_SNAP_SYNC_ENABLED}
- --JsonRpc.Enabled=true
- --JsonRpc.Host=0.0.0.0
- --JsonRpc.Port=8545
- --JsonRpc.EnabledModules=${EC_ENABLED_MODULES}
- --JsonRpc.JwtSecretFile=/tmp/jwt/jwtsecret
- --JsonRpc.EngineHost=0.0.0.0
- --JsonRpc.EnginePort=8551
- --Network.DiscoveryPort=30303
- --HealthChecks.Enabled=${NETHERMIND_HEALTH_CHECKS_ENABLED}
- --Pruning.CacheMb=${NETHERMIND_PRUNING_CACHEMB}
logging:
driver: json-file
options:
max-size: 10m
max-file: "10"
consensus:
stop_grace_period: 30s
container_name: consensus-client
restart: unless-stopped
image: ${CC_IMAGE_VERSION}
networks:
- sedge
volumes:
- ${CC_DATA_DIR}:/var/lib/teku
- ${CC_JWT_SECRET_PATH}:/tmp/jwt/jwtsecret
ports:
- 9000:9000/tcp
- 9000:9000/udp
- 5054:5054/tcp
expose:
- 4000
command:
- --log-destination=CONSOLE
- --logging=${CC_LOG_LEVEL}
- --network=${CL_NETWORK}
- --p2p-discovery-bootnodes=${CL_BOOTNODES}
- --p2p-enabled=true
- --p2p-port=9000
- --p2p-peer-upper-bound=${CC_PEER_COUNT}
- --rest-api-enabled=true
- --rest-api-host-allowlist=*
- --rest-api-interface=0.0.0.0
- --rest-api-port=4000
- --rest-api-cors-origins=*
- --rest-api-docs-enabled=false
- --data-beacon-path=/var/lib/teku/beacon
- --data-path=/var/lib/teku
- --data-storage-archive-frequency=2048
- --data-storage-mode=PRUNE
- --data-storage-non-canonical-blocks-enabled=false
- --ee-endpoint=${EC_AUTH_URL}
- --validators-proposer-default-fee-recipient=${CL_FEE_RECIPIENT}
- --ee-jwt-secret-file=/tmp/jwt/jwtsecret
- --metrics-enabled=true
- --metrics-host-allowlist=*
- --metrics-interface=0.0.0.0
- --metrics-port=5054
- --initial-state=${CHECKPOINT_SYNC_URL}/eth/v2/debug/beacon/states/finalized
logging:
driver: json-file
options:
max-size: 10m
max-file: "10"
validator-import:
container_name: validator-import-client
build:
context: github.com/NethermindEth/teku-init-validator
volumes:
- ${VL_DATA_DIR}:/data
- ${KEYSTORE_DIR}:/keystore
validator:
container_name: validator-client
image: ${VL_IMAGE_VERSION}
depends_on:
validator-import:
condition: service_completed_successfully
networks:
- sedge
ports:
- 5056:5056
volumes:
- ${VL_DATA_DIR}:/data
command:
- validator-client
- --beacon-node-api-endpoint=${CC_API_URL}
- --data-path=/data
- --log-destination=CONSOLE
- --validators-graffiti=${GRAFFITI}
- --validator-keys=/data/keys:/data/passwords
- --metrics-enabled=true
- --metrics-host-allowlist=*
- --metrics-interface=0.0.0.0
- --metrics-port=5056
- --validators-proposer-default-fee-recipient=${CL_FEE_RECIPIENT}
logging:
driver: json-file
options:
max-size: 10m
max-file: "10"
networks:
sedge:
name: sedge_network
.env file
# --- Global configuration ---
EL_NETWORK=xdai
CL_NETWORK=gnosis
# --- Execution Layer - Execution Node - configuration ---
EC_IMAGE_VERSION=nethermind/nethermind:1.14.3
NETHERMIND_LOG_LEVEL=INFO
EC_ENABLED_MODULES=[Web3,Eth,Subscribe,Net,]
EC_METRICS_ENABLED=false
EC_NODENAME=Nethermind
NETHERMIND_METRICS_PUSH_GATEWAY_URL=http://localhost:9090/metrics
NETHERMIND_HEALTH_CHECKS_ENABLED=false
NETHERMIND_PRUNING_CACHEMB=2048
EC_DATA_DIR=./execution-data
EC_SNAP_SYNC_ENABLED=false
EC_JWT_SECRET_PATH=/home/glc/docker-compose-scripts/jwtsecret
# --- Consensus Layer - Beacon Node - configuration ---
CC_PEER_COUNT=50
CC_LOG_LEVEL=info
EC_API_URL=http://execution:8545
EC_AUTH_URL=http://execution:8551
CC_INSTANCE_NAME=Teku
CC_IMAGE_VERSION=consensys/teku:22.9.0
CC_DATA_DIR=./consensus-data
CC_JWT_SECRET_PATH=/home/glc/docker-compose-scripts/jwtsecret
CL_FEE_RECIPIENT=< >
CL_BOOTNODES="enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCC>
CHECKPOINT_SYNC_URL=https://rpc-gbc.gnosischain.com
# --- Consensus Layer - Validator Node - configuration ---
CC_API_URL=http://consensus:4000
GRAFFITI=nethermind-teku
VL_LOG_LEVEL=info
VL_INSTANCE_NAME=TekuValidator
VL_IMAGE_VERSION=consensys/teku:22.9.0
KEYSTORE_DIR=./keystore
VL_DATA_DIR=./validator-data
Thanks!
Can you please run this command:
ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'
and let me know if the output is the same that the one above:
.
|-consensus-data
|---beacon
|-----db
|-----kvstore
|---validator
|-execution-data
|---keystore
|---logs
|---nethermind_db
|-----xdai
|-------blockInfos
|-------blocks
|-------bloom
|-------canonicalHashTrie
|-------code
|-------discoveryNodes
|-------headers
|-------metadata
|-------peers
|-------receipts
|-------state
|---------0
|-------witness
|-keystore
|---validator_keys
|-validator-data
|---keys
|---passwords
|---validator
I have been trying in different moments, and for example, I tried to run the validator by hand without waiting to sync and it ran without issues, with v0.4.0
.
I follow the next commands:
- Run
sedge keys --network gnosis
- Run
sedge cli --network gnosis -c teku -v teku
On the process of sync, I wait until I reached a point like this:
2022-11-23 14:41:36 -- [INFO] [GetRequest] Retrying request
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is syncing
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:41:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:46:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:51:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 14:56:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:01:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:06:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of consensus nodes...
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.3:4000 is synced
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Tracking sync progress of execution nodes...
2022-11-23 15:11:52 -- [INFO] [ETH2 Monitor] [TrackSync] Endpoint http://172.26.0.4:8545 is syncing
That means that both nodes, execution, and consensus are syncing, open another terminal and:
3. Run cd docker-compose-scripts
and docker compose up validator
, and this is the output
Attaching to validator-client
validator-client | 2022-11-23 15:05:34,344 main INFO Configuring logging for destination: console
validator-client | 2022-11-23 15:05:34,404 main INFO Logging includes events: true
validator-client | 2022-11-23 15:05:34,406 main INFO Logging includes validator duties: true
validator-client | 2022-11-23 15:05:34,406 main INFO Logging includes color: true
validator-client | 2022-11-23 15:05:34,501 main INFO Include P2P warnings set to: false
validator-client | 2022-11-23 15:05:37.494 INFO - Ignoring non-configurable constants supplied in network configuration: BLS_WITHDRAWAL_PREFIX, TARGET_AGGREGATORS_PER_COMMITTEE, RANDOM_SUBNETS_PER_VALIDATOR, EPOCHS_PER_RANDOM_SUBNET_SUBSCRIPTION, DOMAIN_BEACON_PROPOSER, DOMAIN_BEACON_ATTESTER, DOMAIN_RANDAO, DOMAIN_DEPOSIT, DOMAIN_VOLUNTARY_EXIT, DOMAIN_SELECTION_PROOF, DOMAIN_AGGREGATE_AND_PROOF, DOMAIN_SYNC_COMMITTEE, DOMAIN_CONTRIBUTION_AND_PROOF, TARGET_AGGREGATORS_PER_SYNC_SUBCOMMITTEE, DOMAIN_SYNC_COMMITTEE_SELECTION_PROOF, SYNC_COMMITTEE_SUBNET_COUNT
validator-client | 2022-11-23 15:05:37.527 INFO - Ignoring unknown items in network configuration: DOMAIN_APPLICATION_BUILDER
validator-client | 2022-11-23 15:05:39.889 INFO - Teku version: teku/v22.9.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17
validator-client | 2022-11-23 15:05:39.901 INFO - This software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
validator-client | 2022-11-23 15:05:40.485 INFO - validator-api-enabled is false, not starting rest api.
validator-client | 2022-11-23 15:05:40.505 INFO - Storing validator data in: /data/validator
validator-client | 2022-11-23 15:05:40.540 INFO - Using optimized BLST library
validator-client | 2022-11-23 15:05:40.586 INFO - Starting metrics http service on 0.0.0.0:5056
validator-client | 2022-11-23 15:05:41.019 INFO - BLS: loaded BLST library
validator-client | 2022-11-23 15:05:41.112 INFO - Loading 5 validator keys...
validator-client | 2022-11-23 15:05:41.737 INFO - Metrics service started and listening on 0.0.0.0:5056
validator-client | 2022-11-23 15:05:46.053 INFO - Loaded 5 Validators: b7bdb9b, 8db4e97, 8292e87, a35007c, 885583c
validator-client | 2022-11-23 15:05:46.270 INFO - Starting EventSource client using URI: http://consensus:4000/eth/v1/events?topics=head
validator-client | 2022-11-23 15:05:46.320 INFO - Connected to EventSource stream.
validator-client | 2022-11-23 15:05:46.346 INFO - Validator *** Successfully connected to beacon node event stream
validator-client | 2022-11-23 15:05:46.506 WARN - Unable to retrieve status for validator b7bdb9b.
validator-client | 2022-11-23 15:05:46.506 WARN - Unable to retrieve status for validator 8db4e97.
validator-client | 2022-11-23 15:05:46.507 WARN - Unable to retrieve status for validator 8292e87.
validator-client | 2022-11-23 15:05:46.507 WARN - Unable to retrieve status for validator a35007c.
validator-client | 2022-11-23 15:05:46.508 WARN - Unable to retrieve status for validator 885583c.
validator-client | 2022-11-23 15:05:48.523 INFO - Not performing attestation duties for slot 6044521 because epoch 377782 is too far ahead of the current epoch UNDEFINED
validator-client | 2022-11-23 15:05:50.151 WARN - Validator *** Slashing protection not loaded for validators: b7bdb9b, 885583c, 8db4e97, a35007c, 8292e87
^CGracefully stopping... (press Ctrl+C again to force)
Will try your compose, check the differences, and see what is the source of the problem. WIll keep you updated, Thanks for letting us know!
Ran the command, this is the output
|-consensus-data
|---beacon
|-----db
|-----kvstore
|---validator
|-execution-data
|---keystore
|---logs
|---nethermind_db
|-----xdai
|-------blockInfos
|-------blocks
|-------bloom
|-------canonicalHashTrie
|-------code
|-------discoveryNodes
|-------headers
|-------metadata
|-------peers
|-------receipts
|-------state
|---------0
|-------witness
|-keystore
|---validator_keys
|-validator-data
|---validator
|---validators
|-----0x81854d896dd31edc5fe805227a607d5cbf09092f14f24e7054c13f9068116ce86a8607b59572d6dfd65d59ff8ac23c65
|-----0x87325850383845784da9e02a6b660378f106ca58ae7f511a5fe5508f12928102b605ded3b4efc5901b4cc9854fe70894
|-----0xa3c65ebf5dba6f1e2b1f9fe4860ca0ac937dc15dc8d501d000743f16bf02b9e18df5da98b5302ac092678044e5cd42af
|-----0xb1bbc475c1db718e40e7a15c3da2778dcc96602d92cc0a78232b11ca8d6d4d9eeb62d58cb3a1f1f05f4facf6a10857ec
Can you please provide me the args to sedge cli
you are using? because the compose that is generated is different from the one I am generating locally, and I am using basic config.
Hey @GLCNI I have been testing, a few things, and all of them worked for me, here is what I did overall:
- Generate keys for gnosis using
sedge keys --network gnosis
- Run sedge cli only to generate the compose, without running it.
sedge cli --network gnosis -c teku
- Deleted my compose, and copy the one provided by you
- Deleted my
.env
file and copy the one you provided, just modifying the path for my local machine, changing only the usernames. - Run these commands on this order:
docker compose up execution -d
docker compose up execution -d
docker compose up validator-import
dicker compose up validator
And everything works.
I am running it on a Ubuntu machine.
Can you please run the docker compose with that .env in a new folder, generated using sedge cli
? or cleaning the validator data or, the sedge folder. After that, run the validator-import container docker compose validator-import
that will move for you all the keystores to the right location for teku, and prepare the environment.
After that, run the validator and please, let us know if the issue persists.
Thanks in advance!
Solved: @stdevMac deleted the validator data folder and re-imported validator keys,
Problem:
it was using a cached image for validator import from a previous build, likely lighthouse so had to run
docker system prune -a
before this would work. because I've been testing with different clients I had this issue, but this might be an issue when switching clients.
github.com/NethermindEth/teku-init-validator
which works fine, keys detected in teku validator client