race condition while spawning proxies
Opened this issue · 0 comments
shadeofblue commented
When multiple proxies are run within an app and they try to bind to a free port, they find the port already taken by another proxy.
As proxy is using a app port from descriptor and binds it to free port on requestor machine, it happens to be already used by another proxy.
We need make proxies not clash into each other, as a race condition, and reserve different free ports.
Example app:
meta:
name: "Gas scanner demo app"
author: "GolemFactory"
version: "0.1.0"
payloads:
mongo:
runtime: "vm"
params:
image_hash: "69dec7e8e106143d317516580d8e8f36ec014f484eb73b79a042a1f8"
backend:
runtime: "vm/manifest"
params:
manifest: ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgImNyZWF0ZWRBdCI6ICIyMDIyLTA5LTI5VDAwOjAwOjAwLjAwMDAwMFoiLAogICAgImV4cGlyZXNBdCI6ICIyMTAwLTAxLTAxVDAwOjAwOjAwLjAwMDAwMFoiLAogICAgInBheWxvYWQiOiBbCiAgICAgICAgewogICAgICAgICAgInBsYXRmb3JtIjogewogICAgICAgICAgICAiYXJjaCI6ICJ4ODZfNjQiLAogICAgICAgICAgICAib3MiOiAibGludXgiCiAgICAgICAgICB9LAogICAgICAgICAgInVybHMiOiBbCiAgICAgICAgICAgICJodHRwOi8veWFjbjIuZGV2LmdvbGVtLm5ldHdvcms6ODAwMC9kb2NrZXItZ2FzX3NjYW5uZXJfYmFja2VuZF9pbWFnZS1sYXRlc3QtM2Y2ZmY2Nzc3My5ndm1pIgogICAgICAgICAgXSwKICAgICAgICAgICJoYXNoIjogInNoYTM6N2U2NzFiYmNiZjE1Njg2Mjc4Y2Y3MmY3Njg0N2FjZjQ0MTQ5ODBhOTkzZTQ4MjczODYwZjIwNmIiCiAgICAgICAgfQogICAgICBdLAogICAgICAiY29tcE1hbmlmZXN0IjogewogICAgICAidmVyc2lvbiI6ICIwLjEuMCIsCiAgICAgICJzY3JpcHQiOiB7CiAgICAgICAgImNvbW1hbmRzIjogWwogICAgICAgICAgInJ1biAuKiIKICAgICAgICBdLAogICAgICAgICJtYXRjaCI6ICJyZWdleCIKICAgICAgfSwKICAgICAgIm5ldCI6IHsKICAgICAgICAiaW5ldCI6IHsKICAgICAgICAgICJvdXQiOiB7CiAgICAgICAgICAgICJwcm90b2NvbHMiOiBbCiAgICAgICAgICAgICAgImh0dHAiCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgICJ1cmxzIjogWwogICAgICAgICAgICAgICAgImh0dHA6Ly9ib3IuZ29sZW0ubmV0d29yayIKICAgICAgICAgICAgXQogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgfQogIH0=
frontend:
runtime: "vm"
params:
image_hash: "49dd38a6da1d6de32f6a7c51f909e104c717bbb27afc24f2beb9da81"
nodes:
mongo:
payload: "mongo"
init:
- run:
args: ["/bin/bash", "-c", "/usr/local/bin/docker-entrypoint.sh /usr/bin/mongod --bind_ip_all &"]
network: "default"
ip:
- "192.168.0.2"
tcp_proxy:
ports:
- "27017"
backend:
payload: "backend"
init:
- run:
args: [
"/bin/bash",
"-c",
"echo -e \"PROVIDER_ADDRESS=http://bor.golem.network/\n
MONGO_DB_CONNECTION_STRING=mongodb://192.168.0.2:27017\n
MONGO_DB_NAME=GasUsage\n
AGGREGATOR_DELAY_START=60\n
AGGREGATOR_DELAY_SECONDS='60'\n
COLD_START_BLOCK='-1'\n
WATCHDOG_START_AGGREGATOR='node dist/gas_scanner_aggregator.js'\n
WATCHDOG_START_SERVER='node dist/gas_scanner_server.js'\n
WATCHDOG_START_COMMAND='node dist/gas_scanner_main.js --fillMissingBlocks --clearDatabase'\n
WATCHDOG_AFTER_KILL_DELAY_MS=5000\n
WATCHDOG_AFTER_START_DELAY_MS=15000\n
WATCHDOG_ALLOWED_SECONDS_BEHIND=30\n
WATCHDOG_CHECK_EVERY_MS=250000\n
SERVER_LISTEN_PORT=7888\n
SERVER_CACHE_VALIDITY=3000\n\" > .env"
]
- run:
args: ["/bin/bash", "-c", "cp .env dist/.env"]
- run:
args: ["/bin/bash", "-c", "node dist/gas_scanner_watchdog.js > stdout 2> stderr &"]
- run:
args: ["/bin/bash", "-c", "curl -H \"Content-Type: application/json\" -d '{\"jsonrpc\": \"2.0\", \"method\": \"eth_blockNumber\", \"params\": [], \"id\": 0}' http://bor.golem.network/"]
- run:
args: ["/bin/bash", "-c", "sleep 60"]
- run:
args: ["/bin/bash", "-c", "cat stdout"]
- run:
args: ["/bin/bash", "-c", "cat stderr"]
http_proxy:
ports:
- "7888"
tcp_proxy:
ports:
- "7888"
network: "default"
ip:
- "192.168.0.3"
depends_on:
- "mongo"
frontend:
payload: "frontend"
init:
- run:
args: ["/bin/bash", "-c", "chmod a+x /"]
- run:
args: ["/bin/bash", "-c", "/bin/run.sh 192.168.0.3 &"]
http_proxy:
ports:
- "80"
network: "default"
ip:
- "192.168.0.4"
depends_on:
- "backend"
networks:
default:
ip: "192.168.0.0/24"
traceback:
[2022-12-06T11:37:50.330+0100 INFO yapapi.summary] Golem engine has shut down
Traceback (most recent call last):
File "/home/blue/.envs/dapp-runner/bin/dapp-runner", line 6, in <module>
sys.exit(_cli())
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/blue/PycharmProjects/dapp-runner/dapp_runner/__main__.py", line 125, in start
start_runner(config_dict, dapp_dict, **kwargs)
File "/home/blue/PycharmProjects/dapp-runner/dapp_runner/runner/__init__.py", line 145, in start_runner
loop.run_until_complete(task)
File "/home/blue/.pyenv/versions/3.9.12/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home/blue/PycharmProjects/dapp-runner/dapp_runner/runner/__init__.py", line 105, in _run_app
await r.stop()
File "/home/blue/PycharmProjects/dapp-runner/dapp_runner/runner/runner.py", line 300, in stop
await asyncio.gather(*self._tasks)
File "/home/blue/PycharmProjects/dapp-runner/dapp_runner/runner/runner.py", line 92, in _start_local_http_proxy
await proxy.run()
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/yapapi/contrib/service/http_proxy.py", line 275, in run
await site.start()
File "/home/blue/.envs/dapp-runner/lib/python3.9/site-packages/aiohttp/web_runner.py", line 121, in start
self._server = await loop.create_server(
File "/home/blue/.pyenv/versions/3.9.12/lib/python3.9/asyncio/base_events.py", line 1506, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8081): address already in use