[dev_network] set_primary_validator failed due to Connection refused
DucPhamTV opened this issue · 6 comments
Bug Description
set_primary_validator failed due to Connection refused
Steps to Reproduce
update .env, set PUBLIC_IP_ADDRESS=127.0.0.1
docker-compose up
$ docker-compose up
bank_1 | Waiting for PV...
pv_1 | HTTP GET /config 200 [0.29, 172.19.0.5:56118]
bank_1 | PV started
bank_1 | Bank initialization complete
bank_1 | Waiting for self...
bank_1 | HTTP GET /config 200 [0.47, 127.0.0.1:47972]
bank_1 | self started
But at step set_primary_validator, it raised the error:
...
bank_1 | During handling of the above exception, another exception occurred:
bank_1 |
bank_1 | Traceback (most recent call last):
bank_1 | File "/usr/local/lib/python3.8/site-packages/thenewboston/base_classes/fetch_primary_validator_config.py", line 101, in handle
bank_1 | config = self.fetch_validator_config()
bank_1 | File "/usr/local/lib/python3.8/site-packages/thenewboston/base_classes/fetch_primary_validator_config.py", line 82, in fetch_validator_config
bank_1 | results = fetch(url=url, headers={})
bank_1 | File "/usr/local/lib/python3.8/site-packages/thenewboston/utils/network.py", line 8, in fetch
bank_1 | response = requests.get(url, headers=headers)
bank_1 | File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
bank_1 | return request('get', url, params=params, **kwargs)
bank_1 | File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
bank_1 | return session.request(method=method, url=url, **kwargs)
bank_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
bank_1 | resp = self.send(prep, **send_kwargs)
bank_1 | File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
bank_1 | r = adapter.send(request, **kwargs)
bank_1 | File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
bank_1 | raise ConnectionError(e, request=request)
bank_1 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /config (Caused by NewConnectionError('<urllib3.connection.
HTTPConnection object at 0x7f92bbe81100>: Failed to establish a new connection: [Errno 111] Connection refused'))
Expected behavior
other containers can get config from PV.
Actual behavior
Connection refuse when getting config from PV.
OS and Browser
Ubuntu-20.04
docker-compose version 1.25.0
docker version 19.03.8,
127.0.0.1:8001 is unreachable inside a container. but pv:8000 is reachable.
$ docker exec -it bank_bank_1 curl http://pv:8000 -v
* Trying 172.19.0.4:8000...
* Connected to pv (172.19.0.4) port 8000 (#0)
> GET / HTTP/1.1
> Host: pv:8000
> User-Agent: curl/7.69.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept, Cookie, Origin
< Allow: GET, HEAD, OPTIONS
< Server-Timing: TimerPanel_utime;dur=7.183999999999635;desc="User CPU time", TimerPanel_stime;dur=3.8770000000000193;desc="System CPU time", TimerPanel_total;dur=11.060999999999655;desc="Total CPU time", TimerPanel_total_time;dur=11.115074157714844;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Frame-Options: DENY
< Content-Length: 575
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
<
* Connection #0 to host pv left intact
{"accounts":"http://pv:8000/accounts","bank_blocks":"http://pv:8000/bank_blocks","bank_confirmation_services":"http://pv:8000/bank_confirmation_services","banks":"http://pv:8000/banks","clean":"http://pv:8000/clean","confirmation_blocks":"http://pv:8000/confirmation_blocks","crawl":"http://pv:8000/crawl","config":"http://pv:8000/config","primary_validator_updated":"http://pv:8000/primary_validator_updated","upgrade_request":"http://pv:8000/upgrade_request","confirmation_block_history":"http://pv:8000/confirmation_block_history","validators":"http://pv:8000/validators"}
And
$ docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v
* Trying 127.0.0.1:8001...
* connect to 127.0.0.1 port 8001 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8001: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 8001: Connection refused
I'm thinking about using docker network IP instead of PUBLIC_IP_ADDRESS=127.0.0.1 ...
@DucPhamTV probably something is wrong with docker proxy or some other process is listening on 8001 port ...
also, what I would like to see
delete exsiting volumes
start docker-compose up
wait for couple of mins, and copy the ENTIRE log and attach here as a file
also, on you host machine do netstat -tap and paste a responce here
also docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v is wrong
this cmd should be run from host machine curl http://127.0.0.1:8001
@DucPhamTV fixed, try again
@vosi .It still doesn't work for me. I don't see new update on master branch related to this issue. Just a new commit about update database names. Here is log when docker compose up is provisioned docker-compose-log
Here is netstat -tap. filtered unrelated information.
tcp6 0 0 [::]:8001 [::]:* LISTEN -
tcp6 0 0 [::]:8002 [::]:* LISTEN -
tcp6 0 0 [::]:8003 [::]:* LISTEN -
tcp6 0 0 [::]:8004 [::]:* LISTEN -
Only PV listens on 8001. I can access it by my browser.
also docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v is wrong
AFAIK, What exactly set_primary_validator is doing this, except that it uses python requests, I used curl.
I know inside a container, It can't reach 127.0.0.1:8001 on host. Since .env, set PUBLIC_IP_ADDRESS=127.0.0.1 will cause this thing happen.
ah .. .right, the thing is, my docker setup is on a dedicated server, haven't tried provisioning the network on a setup with a local docker host
I suppose a simple fix would be to use some 10/8 or 172.16/12 or 192.168/16 network
I'll update the doc shortly