TimeoutError: IPC socket file has not appeared in 30 seconds

Question

TimeoutError: IPC socket file has not appeared in 30 seconds

jtatman opened this issue 4 years ago · 7 comments

Trinity version: Trinity/v0.1.0a36/linux/cpython3.8.3
OS: linux - ubuntu bionic
Python version (output of python --version): Python 3.8.3
Environment (output of pip freeze):

aiohttp==3.6.0
anyio==1.3.1
argcomplete==1.12.2
asks==2.4.10
asn1crypto==1.4.0
async-exit-stack==1.0.1
async-generator==1.10
async-lru==0.1.0
async-service==0.1.0a8
async-timeout==3.0.1
asyncio-cancel-token==0.2.0
asyncio-run-in-process==0.1.0a7
attrs==20.3.0
backcall==0.2.0
base58==1.0.3
bitarray==1.2.2
blake2b-py==0.1.3
bloom-filter==1.3
cached-property==1.5.2
cachetools==3.1.1
certifi==2020.12.5
cffi==1.14.4
chardet==3.0.4
cloudpickle==1.6.0
coincurve==10.0.0
cryptography==2.8
cytoolz==0.11.0
decorator==4.4.2
eth-abi==2.1.1
eth-account==0.5.4
eth-bloom==1.0.3
eth-hash==0.2.0
eth-keyfile==0.5.1
eth-keys==0.3.3
eth-rlp==0.2.1
eth-typing==2.2.2
eth-utils==1.9.5
fastecdsa==1.7.5
h11==0.11.0
hexbytes==0.2.1
idna==2.7
ifaddr==0.1.7
ipfshttpclient==0.6.1
ipython==7.9.0
ipython-genutils==0.2.0
jedi==0.17.2
jsonschema==3.0.1
lahja==0.16.0
libp2p==0.1.5
lru-dict==1.1.6
lxml==4.6.2
multiaddr==0.0.9
multidict==4.7.6
mypy==0.790
mypy-extensions==0.4.3
netaddr==0.8.0
netdisco==2.8.2
netifaces==0.10.9
noiseprotocol==0.3.1
outcome==1.1.0
parsimonious==0.8.1
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
plyvel==1.2.0
prometheus-client==0.7.1
prompt-toolkit==2.0.10
protobuf==3.14.0
psutil==5.8.0
ptyprocess==0.6.0
py-ecc==1.7.1
py-evm==0.3.0a17
pycparser==2.20
pycryptodome==3.9.9
pyethash==0.1.27
pyformance==0.4
Pygments==2.7.3
pymultihash==0.8.2
PyNaCl==1.3.0
pyrsistent==0.16.1
pysha3==1.0.2
python-dateutil==2.8.1
python-snappy==0.5.4
requests==2.25.1
rlp==1.2.0
rpcudp==3.0.0
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.2
six==1.15.0
sniffio==1.2.0
sortedcontainers==2.3.0
SQLAlchemy==1.3.22
ssz==0.2.4
termcolor==1.1.0
toolz==0.11.1
traitlets==5.0.5
trie==2.0.0a1
trinity==0.1.0a36
trio==0.13.0
trio-typing==0.3.0
typed-ast==1.4.1
typing-extensions==3.7.4.2
u-msgpack-python==2.7.1
uPnPClient==0.0.8
urllib3==1.26.2
uvloop==0.14.0
varint==1.0.2
wcwidth==0.2.5
web3==5.11.1
websockets==8.1
yarl==1.6.3
zeroconf==0.28.7

What is wrong?

Full Output:

$ trinity
INFO 2020-12-19 04:19:53,942 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-19 04:19:53,943 trinity Started main process (pid=26939)
INFO 2020-12-19 04:19:53,945 trinity Trinity/v0.1.0a36/linux/cpython3.8.3
INFO 2020-12-19 04:19:53,946 trinity Trinity DEBUG log file is created at /home/james/.local/share/trinity/mainnet/logs-eth1/trinity.log
INFO 2020-12-19 04:19:53,948 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7ff35f48b070> server over IPC socket: /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/cytoolz/compatibility.py:2: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.
warnings.warn("The toolz.compatibility module is no longer "
/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/async_lru.py:96: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def _wait_closed(wrapped, *, return_exceptions, loop):
/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/async_lru.py:191: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def wrapped(*fn_args, **fn_kwargs):
/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/aiohttp/helpers.py:105: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def noop(*args, **kwargs): # type: ignore
INFO 2020-12-19 04:20:01,921 trinity Started DB server process (pid=26945)
ERROR 2020-12-19 04:20:31,948 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-19 04:20:31,956 asyncio_run_in_process Child process pid=26945 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/lib/python3.8/argparse.py", line 2521, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/lib/python3.8/argparse.py", line 2508, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-19 04:20:31,964 root Error during trinity run
Traceback (most recent call last):
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/home/james/python/trinity-eth/.venv/lib/python3.8/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/lib/python3.8/argparse.py", line 2521, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/lib/python3.8/argparse.py", line 2508, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-19 04:20:31,972 root Shutting down Trinity
WARNING 2020-12-19 04:20:31,974 root Removed dangling IPC socket file /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

How can it be fixed

I tried to rebuild environments using 3.6 and 3.7, and got the same issue. Wondering why the database server doesn't seem to start in time - could be my system response time, naturally, but other applications are not affected nor resources scarce. Multiple attempts show same results.

Answer 1 · 2020-12-19T12:49:09.000Z

Hi @jtatman This may be linked to #1826 where @carver details a workaround to allow the DB to be ready.

Answer 2 · 2020-12-20T11:53:23.000Z

Thank you for the prompt reply - after looking that issue over and trying again with that issue in mind, and following those workaround instructions, this is what I see:

$ trinity db-shell
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/bin/trinity", line 8, in
sys.exit(main())
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/main.py", line 30, in main
(Eth1AppConfig,)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 376, in main_entry
args.func(args, trinity_config)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/component.py", line 89, in run_shell
with get_eth1_shell_context(config.database_dir, trinity_config) as context:
File "/home/james/miniconda3/envs/trinity/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/console.py", line 161, in get_eth1_shell_context
head = chaindb.get_canonical_head()
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 133, in get_canonical_head
return self._get_canonical_head(self.db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 137, in _get_canonical_head
canonical_head_hash = cls._get_canonical_head_hash(db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 145, in _get_canonical_head_hash
raise CanonicalHeadNotFound("No canonical head set for this chain")
eth.exceptions.CanonicalHeadNotFound: No canonical head set for this chain

possible issue

After poking around a bit in the issues, it seems I may not be specifying the sync correctly, or otherwise not launching correctly. To check the database issue, I changed the --trinity-root-dir option to a custom dir and re-initialized from scratch, but got the same error. I see the error generated in the header files:

$ grep -i "no canonical head" /home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py
f"No canonical header for block number #{block_number}"

This seems to pertain to no sync block being specified - if I'm not mistaken. Am I supposed to sync first using only headers or beam options perhaps? Just a wild thought, probably. I see there's an option for forcing the beam block which is referred to in some issues that have been closed (--force-beam-block block#).

I am in the process of checking the docker instructions as well just to see if I can get something working as a reference point, which may be the other solution, but I'd love to help debug the standard launch instructions.

Answer 3 · 2020-12-20T11:55:51.000Z

As a further puzzle, I seem to get the same error using the docker image as such:

$ docker run -it ethereum/trinity
Unable to find image 'ethereum/trinity:latest' locally
latest: Pulling from ethereum/trinity
4ae16bd47783: Pull complete
bbab4ec87ac4: Pull complete
2ea1f7804402: Pull complete
96465440c208: Pull complete
6ac892e64b94: Pull complete
5b3ec9e84adf: Pull complete
317202007d7c: Pull complete
ba1ee226143f: Pull complete
e33fb8e92c2f: Pull complete
fd7d3ea578ae: Pull complete
81e93632b259: Pull complete
183569d6e84a: Pull complete
5bb1b0cea1bd: Pull complete
35f2bbdb1b61: Pull complete
742b322527f6: Pull complete
Digest: sha256:5329cb680cd5ee1cc0a204abc0deafd1c25812e2921babba3ac8e80ec2f01e81
Status: Downloaded newer image for ethereum/trinity:latest
INFO 2020-12-20 11:51:39,634 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 11:51:39,636 trinity Started main process (pid=1)
INFO 2020-12-20 11:51:39,639 trinity Trinity/v0.1.0a36/linux/cpython3.7.4
INFO 2020-12-20 11:51:39,640 trinity Trinity DEBUG log file is created at /root/.local/share/trinity/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 11:51:39,645 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7fe8f4579ed0> server over IPC socket: /root/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
INFO 2020-12-20 11:51:49,061 trinity Started DB server process (pid=12)
ERROR 2020-12-20 11:52:19,112 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 11:52:19,122 asyncio_run_in_process Child process pid=12 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/usr/src/app/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/usr/src/app/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/usr/local/lib/python3.7/argparse.py", line 2501, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/usr/local/lib/python3.7/argparse.py", line 2488, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 11:52:19,152 root Error during trinity run
Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/usr/src/app/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/usr/src/app/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/usr/local/lib/python3.7/argparse.py", line 2501, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/usr/local/lib/python3.7/argparse.py", line 2488, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 11:52:19,158 root Shutting down Trinity
WARNING 2020-12-20 11:52:19,164 root Removed dangling IPC socket file /root/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

Answer 4 · 2020-12-20T12:00:00.000Z

And then subsequently following the instructions from this page - https://trinity-client.readthedocs.io/en/latest/cookbook.html -> Starting from a trusted checkpoint - I tried again from the command line (without docker intervention):

trinity --sync-from-checkpoint eth://block/byhash/0xa65877df954e1ff2012473efee8287252eee956c0d395a5791f1103a950a1e21?score=15,835,269,727,022,672,760,774
INFO 2020-12-20 05:54:39,595 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 05:54:39,596 trinity Started main process (pid=4511)
INFO 2020-12-20 05:54:39,597 trinity Trinity/v0.1.0a36/linux/cpython3.7.9
INFO 2020-12-20 05:54:39,598 trinity Trinity DEBUG log file is created at /home/james/.local/share/trinity/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 05:54:39,600 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7f6acf426cd0> server over IPC socket: /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/cytoolz/compatibility.py:6: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.
category=DeprecationWarning)
INFO 2020-12-20 05:54:46,906 trinity Started DB server process (pid=4518)
ERROR 2020-12-20 05:55:16,928 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 05:55:16,935 asyncio_run_in_process Child process pid=4518 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 05:55:16,953 root Error during trinity run
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 05:55:16,956 root Shutting down Trinity
WARNING 2020-12-20 05:55:16,958 root Removed dangling IPC socket file /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

so...

It appears to be consistent over differerent approaches...

Answer 5 · 2020-12-20T15:44:41.000Z

Hi @jtatman

$ trinity db-shell
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

Just a thought that crossed my mind: is it possible that your default trinity root (~/.local/share/trinity) contained an incompatible DB (for ex, from an older version) ?

Now for your attempt with a custom folder:

To check the database issue, I changed the --trinity-root-dir option to a custom dir and re-initialized from scratch, but got the same error. I see the error generated in the header files:

$ grep -i "no canonical head" /home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py
f"No canonical header for block number #{block_number}"

Would it be possible to post a bit more of the corresponding logs ? (so, with this command: trinity --trinity-root-dir <custom empty folder> ?) I wonder if this is really the same error or if there is a different root cause this time ...

Answer 6 · 2020-12-20T19:57:14.000Z

So, first apologies for the torrent of possibly confusing info. Let me cycle through in a bit more organized fashion if possilble... I should preface this by saying this is a new install (on this machine), using pip, through miniconda venv... so it is unlikely in my mind that there is an older copy of the database - this is my rationale for changing the directory for root-dir to get a fresh copy of the database.

#####first, the original command with no arguments - this was the first thing done:

$ conda activate trinity
$ trinity db-shell

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/bin/trinity", line 8, in
sys.exit(main())
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/main.py", line 30, in main
(Eth1AppConfig,)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 376, in main_entry
args.func(args, trinity_config)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/component.py", line 89, in run_shell
with get_eth1_shell_context(config.database_dir, trinity_config) as context:
File "/home/james/miniconda3/envs/trinity/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/console.py", line 161, in get_eth1_shell_context
head = chaindb.get_canonical_head()
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 133, in get_canonical_head
return self._get_canonical_head(self.db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 137, in _get_canonical_head
canonical_head_hash = cls._get_canonical_head_hash(db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 145, in _get_canonical_head_hash
raise CanonicalHeadNotFound("No canonical head set for this chain")
eth.exceptions.CanonicalHeadNotFound: No canonical head set for this chain

#####second, explicitly coding the path to the current database

$ trinity --data-dir ~/.local/share/trinity/mainnet db-shell
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/bin/trinity", line 8, in
sys.exit(main())
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/main.py", line 30, in main
(Eth1AppConfig,)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 376, in main_entry
args.func(args, trinity_config)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/component.py", line 89, in run_shell
with get_eth1_shell_context(config.database_dir, trinity_config) as context:
File "/home/james/miniconda3/envs/trinity/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/console.py", line 161, in get_eth1_shell_context
head = chaindb.get_canonical_head()
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 133, in get_canonical_head
return self._get_canonical_head(self.db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 137, in _get_canonical_head
canonical_head_hash = cls._get_canonical_head_hash(db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 145, in _get_canonical_head_hash
raise CanonicalHeadNotFound("No canonical head set for this chain")
eth.exceptions.CanonicalHeadNotFound: No canonical head set for this chain

#####third, this is the output when changing --trinity-root-dir option, both without and with db-shell option:

$ trinity --trinity-root-dir ~/python/trinity-eth/
INFO 2020-12-20 05:24:45,895 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 05:24:45,896 trinity Started main process (pid=3282)
INFO 2020-12-20 05:24:45,898 trinity Trinity/v0.1.0a36/linux/cpython3.7.9
INFO 2020-12-20 05:24:45,899 trinity Trinity DEBUG log file is created at /home/james/python/trinity-eth/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 05:24:45,901 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7f62c22c7a90> server over IPC socket: /home/james/python/trinity-eth/mainnet/ipcs-eth1/logging.ipc
/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/cytoolz/compatibility.py:6: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.
category=DeprecationWarning)
INFO 2020-12-20 05:24:54,021 trinity Started DB server process (pid=3290)
ERROR 2020-12-20 05:25:24,035 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 05:25:24,041 asyncio_run_in_process Child process pid=3290 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 05:25:24,071 root Error during trinity run
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 05:25:24,072 root Shutting down Trinity
WARNING 2020-12-20 05:25:24,074 root Removed dangling IPC socket file /home/james/python/trinity-eth/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete
#####with db option here
$ trinity --trinity-root-dir ~/python/trinity-eth/ db-shell
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/bin/trinity", line 8, in
sys.exit(main())
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/main.py", line 30, in main
(Eth1AppConfig,)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 376, in main_entry
args.func(args, trinity_config)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/component.py", line 89, in run_shell
with get_eth1_shell_context(config.database_dir, trinity_config) as context:
File "/home/james/miniconda3/envs/trinity/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/console.py", line 161, in get_eth1_shell_context
head = chaindb.get_canonical_head()
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 133, in get_canonical_head
return self._get_canonical_head(self.db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 137, in _get_canonical_head
canonical_head_hash = cls._get_canonical_head_hash(db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 145, in _get_canonical_head_hash
raise CanonicalHeadNotFound("No canonical head set for this chain")
eth.exceptions.CanonicalHeadNotFound: No canonical head set for this chain

#####fourth, what probably amounts to nonsense by adding option --force-beam-block-number to somehow get through the "cannonicalize" errors
$ trinity --trinity-root-dir ~/python/trinity-eth/ --force-beam-block-number 0
INFO 2020-12-20 05:36:16,096 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 05:36:16,097 trinity Started main process (pid=3963)
INFO 2020-12-20 05:36:16,098 trinity Trinity/v0.1.0a36/linux/cpython3.7.9
INFO 2020-12-20 05:36:16,099 trinity Trinity DEBUG log file is created at /home/james/python/trinity-eth/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 05:36:16,100 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7f9b5bed4790> server over IPC socket: /home/james/python/trinity-eth/mainnet/ipcs-eth1/logging.ipc
/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/cytoolz/compatibility.py:6: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.
category=DeprecationWarning)
INFO 2020-12-20 05:36:22,784 trinity Started DB server process (pid=3970)
ERROR 2020-12-20 05:36:52,812 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 05:36:52,819 asyncio_run_in_process Child process pid=3970 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 05:36:52,833 root Error during trinity run
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 05:36:52,837 root Shutting down Trinity
WARNING 2020-12-20 05:36:52,839 root Removed dangling IPC socket file /home/james/python/trinity-eth/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

$ trinity --trinity-root-dir ~/python/trinity-eth/ --force-beam-block-number 0 db-shell
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 143, in _get_canonical_head_hash
return Hash32(db[SchemaV1.make_canonical_head_hash_lookup_key()])
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/backends/level.py", line 58, in getitem
raise KeyError(key)
KeyError: b'v1:canonical_head_hash'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/bin/trinity", line 8, in
sys.exit(main())
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/main.py", line 30, in main
(Eth1AppConfig,)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 376, in main_entry
args.func(args, trinity_config)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/component.py", line 89, in run_shell
with get_eth1_shell_context(config.database_dir, trinity_config) as context:
File "/home/james/miniconda3/envs/trinity/lib/python3.7/contextlib.py", line 112, in enter
return next(self.gen)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/components/builtin/attach/console.py", line 161, in get_eth1_shell_context
head = chaindb.get_canonical_head()
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 133, in get_canonical_head
return self._get_canonical_head(self.db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 137, in _get_canonical_head
canonical_head_hash = cls._get_canonical_head_hash(db)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/eth/db/header.py", line 145, in _get_canonical_head_hash
raise CanonicalHeadNotFound("No canonical head set for this chain")
eth.exceptions.CanonicalHeadNotFound: No canonical head set for this chain

#####fifth, trying to sanity check through docker:

$ docker run -it ethereum/trinity
Unable to find image 'ethereum/trinity:latest' locally
latest: Pulling from ethereum/trinity
4ae16bd47783: Pull complete
bbab4ec87ac4: Pull complete
2ea1f7804402: Pull complete
96465440c208: Pull complete
6ac892e64b94: Pull complete
5b3ec9e84adf: Pull complete
317202007d7c: Pull complete
ba1ee226143f: Pull complete
e33fb8e92c2f: Pull complete
fd7d3ea578ae: Pull complete
81e93632b259: Pull complete
183569d6e84a: Pull complete
5bb1b0cea1bd: Pull complete
35f2bbdb1b61: Pull complete
742b322527f6: Pull complete
Digest: sha256:5329cb680cd5ee1cc0a204abc0deafd1c25812e2921babba3ac8e80ec2f01e81
Status: Downloaded newer image for ethereum/trinity:latest
INFO 2020-12-20 11:51:39,634 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 11:51:39,636 trinity Started main process (pid=1)
INFO 2020-12-20 11:51:39,639 trinity Trinity/v0.1.0a36/linux/cpython3.7.4
INFO 2020-12-20 11:51:39,640 trinity Trinity DEBUG log file is created at /root/.local/share/trinity/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 11:51:39,645 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7fe8f4579ed0> server over IPC socket: /root/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
INFO 2020-12-20 11:51:49,061 trinity Started DB server process (pid=12)
ERROR 2020-12-20 11:52:19,112 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 11:52:19,122 asyncio_run_in_process Child process pid=12 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/usr/src/app/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/usr/src/app/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/usr/local/lib/python3.7/argparse.py", line 2501, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/usr/local/lib/python3.7/argparse.py", line 2488, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 11:52:19,152 root Error during trinity run
Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/usr/src/app/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/app/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/usr/src/app/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/usr/local/lib/python3.7/argparse.py", line 2501, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/usr/local/lib/python3.7/argparse.py", line 2488, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 11:52:19,158 root Shutting down Trinity
WARNING 2020-12-20 11:52:19,164 root Removed dangling IPC socket file /root/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

#####last, through following the cookbook (https://trinity-client.readthedocs.io/en/latest/cookbook.html) I tried to initialized from a specific block, to sort through the cannonical head issue, or just try something different:

$ trinity --sync-from-checkpoint eth://block/byhash/0xa65877df954e1ff2012473efee8287252eee956c0d395a5791f1103a950a1e21?score=15,835,269,727,022,672,760,774
INFO 2020-12-20 05:54:39,595 trinity
______ _ _ __
/_ /() () / __
/ / / / / __ / / __/ / / /
/ / / / / / / / / / // // /
// // /// ///_/_, /
/___/
INFO 2020-12-20 05:54:39,596 trinity Started main process (pid=4511)
INFO 2020-12-20 05:54:39,597 trinity Trinity/v0.1.0a36/linux/cpython3.7.9
INFO 2020-12-20 05:54:39,598 trinity Trinity DEBUG log file is created at /home/james/.local/share/trinity/mainnet/logs-eth1/trinity.log
INFO 2020-12-20 05:54:39,600 IPCListener Starting <trinity._utils.logging.IPCListener object at 0x7f6acf426cd0> server over IPC socket: /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/cytoolz/compatibility.py:6: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.
category=DeprecationWarning)
INFO 2020-12-20 05:54:46,906 trinity Started DB server process (pid=4518)
ERROR 2020-12-20 05:55:16,928 trinity Timeout waiting for database to start. Exiting...
usage: trinity [-h]
trinity: error: Timed out waiting for database start
WARNING 2020-12-20 05:55:16,935 asyncio_run_in_process Child process pid=4518 failed to exit cleanly. Sending SIGKILL
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/asyncio_run_in_process/run_in_process.py", line 274, in _open_in_process
yield proc
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
ERROR 2020-12-20 05:55:16,953 root Error during trinity run
Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 262, in _run
wait_for_ipc(boot_info.trinity_config.database_ipc_path)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/_utils/ipc.py", line 29, in wait_for_ipc
raise TimeoutError("IPC socket file has not appeared in %d seconds!" % timeout)
TimeoutError: IPC socket file has not appeared in 30 seconds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 309, in run
loop.run_until_complete(_run(boot_info, get_base_db_fn, component_manager_manager))
File "uvloop/loop.pyx", line 1450, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1443, in uvloop.loop.Loop.run_until_complete
File "uvloop/loop.pyx", line 1351, in uvloop.loop.Loop.run_forever
File "uvloop/loop.pyx", line 519, in uvloop.loop.Loop._run
File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
File "/home/james/miniconda3/envs/trinity/lib/python3.7/site-packages/trinity/bootstrap.py", line 265, in _run
argparse.ArgumentParser().error(message="Timed out waiting for database start")
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2508, in error
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
File "/home/james/miniconda3/envs/trinity/lib/python3.7/argparse.py", line 2495, in exit
_sys.exit(status)
SystemExit: 2
INFO 2020-12-20 05:55:16,956 root Shutting down Trinity
WARNING 2020-12-20 05:55:16,958 root Removed dangling IPC socket file /home/james/.local/share/trinity/mainnet/ipcs-eth1/logging.ipc
Trinity shutdown complete

#####this was the progression exactly as entered, in time series - hopefully this might clarify

Answer 7 · 2020-12-25T09:40:26.000Z

For the sake of saying so, I'm not sure if this exact issue is reproducible on other architectures. On osx, the install went forward as you suggested - the first time, the timeout occurred, then I opened with db-shell, and it opened the database once, then I was able to start the trinity client. If there is any other testing I can do, or if I can help with bleeding edge, please let me know if you have a chance.