Cannot run both containers at the same time
fvictorio opened this issue · 3 comments
Starting the Rpcdaemon container or the Erigon container separately works fine, but if I start both of them, the second one won't work.
To Reproduce
- Install erigon package version
0.1.14 (v2021.10.03 upstream) - Set this configuration option:
RPCDAEMON_EXTRA_OPTS:--http.api=eth,erigon,web3,net,debug,trace,txpool- Public port mapping: 8545 -> 8545
- Start both containers
If the Rpcdaemon container is started first, the Erigon container won't work, and these are the logs:
Erigon logs
[INFO] [11-03|10:18:47.530] Starting metrics server addr=http://0.0.0.0:6060/debug/metrics/prometheus
[INFO] [11-03|10:18:47.530] Starting pprof server cpu="go tool pprof -lines -http=: http://0.0.0.0:6061/debug/pprof/profile?seconds=20" heap="go tool pprof -lines -http=: http://0.0.0.0:6061/debug/pprof/heap"
[INFO] [11-03|10:18:47.530] Build info git_branch=heads/v2021.10.03 git_tag=v2021.10.03 git_commit=a481376a81d1bba61b12502634aeccd2ab767d9c
[INFO] [11-03|10:18:47.530] Starting Erigon on Ethereum mainnet...
[INFO] [11-03|10:18:47.531] Maximum peer count ETH=100 total=100
[INFO] [11-03|10:18:47.531] Set global gas cap cap=50000000
[INFO] [11-03|10:18:47.565] Opening Database label=chaindata path=/home/erigon/.local/share/chaindata
mdbx_lck_seize:29459 lock-against-without-lck, err 11
[EROR] [11-03|10:18:47.566] Erigon startup err="mdbx_env_open: resource temporarily unavailable, label: chaindata, trace: [github.com/ledgerwatch/erigon-lib/kv/mdbx.MdbxOpts.Open github.com/ledgerwatch/erigon/node.OpenDatabase.func1 github.com/ledgerwatch/erigon/node.OpenDatabase github.com/ledgerwatch/erigon/eth.New github.com/ledgerwatch/erigon/turbo/node.RegisterEthService github.com/ledgerwatch/erigon/turbo/node.New main.runErigon github.com/urfave/cli.HandleAction github.com/urfave/cli.(*App).Run main.main]"
If the Erigon container is started first, the Rpcdaemon has these logs:
Rpcdaemon logs
[INFO] [11-03|10:24:05.816] Starting metrics server addr=http://0.0.0.0:6060/debug/metrics/prometheus
[EROR] [11-03|10:24:05.816] Could not connect to DB error="mdbx_env_open: resource temporarily unavailable, label: chaindata, trace: [github.com/ledgerwatch/erigon-lib/kv/mdbx.MdbxOpts.Open github.com/ledgerwatch/erigon/cmd/rpcdaemon/cli.RemoteServices main.main.func1 github.com/spf13/cobra.(*Command).execute github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra.(*Command).ExecuteContext main.main runtime.main runtime.goexit]"
Expected behavior
Both containers should work together, since otherwise connecting to the node via JSON-RPC is not possible.
DAppNode version:
Core DAppNode Packages versions
bind.dnp.dappnode.eth: 0.2.6
core.dnp.dappnode.eth: 0.2.48
dappmanager.dnp.dappnode.eth: 0.2.43, commit: 4044dae4
https.dnp.dappnode.eth: 0.1.1
ipfs.dnp.dappnode.eth: 0.2.15
vpn.dnp.dappnode.eth: 0.2.8, commit: f9a8743e
wifi.dnp.dappnode.eth: 0.2.8
System info
dockerComposeVersion: 1.25.5
dockerServerVersion: 20.10.2
dockerCliVersion: 20.10.2
os: debian
versionCodename: bullseye
architecture: amd64
kernel: 5.9.0-4-amd64
Disk usage: 58%
- Package version: 0.1.14 (v2021.10.03 upstream)
- OS: DAppNodeXtreme
Hey! I just came across this issue after having the same trouble running Erigon in Kubernetes, and found the solution.
The two binaries are meant to share memory, this is why there is contention over the database when they can't (in containers).
The solution is to allow them to share a process namespace, with docker you can do this with:
# Where X is the same
docker run --pid=X ... # erigon main
docker run --pid=X ... # erigon rpc
Sorry I'm not familiar with docker compose but I'm sure it's fairly straightforward.
If you're coming across this and running the nodes in Kubernetes, the equivalent is to run the containers in the same pod, and set pod.spec.shareProcessNamespace: true.
We have unified the services, this problem cant happen again.