[BUG] Docker Network Exposed To Entire Flux Node
flikites opened this issue · 8 comments
Each Flux node operates with only one docker network. This network can be accessed by any app deployed to the node. This means that all apps on the flux node can communicate regardless of app owner, which is a very serious security concern.
(Copied from Discord)
If you are worried that the multi-network setup would break some apps because they are relying on the behavior of the current single network, we could potentially introduce a boolean called use_multiple_network to the app specification. All current apps need to update the spec to get new behavior. All newly registered apps should have this field set to True by default.
IMO, the default expectation is All components within one app should be placed in one docker network.
Another potential expectation is Deployer can specify which network an app should be in, similar to how you could specify the network for each service in Docker's stack.yaml. However, I think this use case should be supported by merging multiple apps into one app so that components in these apps can be in the same docker network (I feel like this might also be easier from apps scheduling point of view). Note that the maximum number of components per app is 5 currently (which means you can put more than 5 components in a single docker network).
Method from discord here.
Think it would be good to change the label on this issue - it's a feature, and was coded this way by design.
(Copied from discord)
My thoughts are that as it's done with the new app spec it's fine. People on v5 have single network, people deploying on v6 will have the new isolated network. Assume each app subnet wouldn't be routable to others.
I can't see any legitimate use cases for single network. As long as apps on different networks are able to route out via the wan to each other. In case one app is reliant on another app and they happen to be on the same machine. I assume this is okay as the wan IP goes on the docker interface so hairpining should be fine.
So I guess the issue is around educating people who are deploying on v6 how to use the Flux network, I.e using component / app names (it should be stickied on this channel right?) dns and ptr records.
For example:
dig -x `ifconfig eth0 | grep 'inet' | awk '{print $2}'` +short | cut -d'.' -f1
This will get you the container name and app name from inside the container. Set this as an env var and pass that to you app.
Awesome though, this is a good step forward for usability on the network
To get the above to work, on Debian based machines you need to install:
apt update && apt install net-tools iputils-ping dnsutils
Also think it's worth thinking about DNS (container names) at the same time. As it will become harder and harder to change this in the future as more people rely on it.
Current state - it's quite hard programatically to derive component name / app name from within an app. You have to strip the leading "flux" from the ptr record and split the string on hypens. It's messy.
Adding custom networks - this will change the PTR record schema. RIght now, you drop the netowork name from the ptr record as it doens't signify anything. Here is an example:
; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> -x 172.15.0.3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65522
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;3.0.15.172.in-addr.arpa. IN PTR
;; ANSWER SECTION:
3.0.15.172.in-addr.arpa. 600 IN PTR fluxalephiumnode_AlephiumNode1669304356090.fluxDockerNetwork.
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Thu Nov 24 17:06:38 UTC 2022
;; MSG SIZE rcvd: 138
Once private networks come in - the fluxDockerNetwork will get replaced with the network name. Which, would make sense if it was the app name. In a perfect world, this is how I'd want to see it:
<component_name>.<app_name>
Or, even better, and probably a bit of a dream as nodes don't have identifiers (that I'm aware) <component_name>.<node_name>.<app_name>
This would remove the leading flux and make the ptr more easily parsable by apps. I understand why the leading flux is there, and most likely, that's another issue. (identifying containers that are supposed to be running under Flux)
On a side note, is it possible to identify a specific component on a specific node via the public app.runonflux.io domain? I have some things I'm doing with certificates and this would greatly help out
Or better yet, instead of having to use the ptr, if this gave the container name instead:
root@f3339897a183:/# hostname
f3339897a183
Thought I would point out why the dns / naming is important to me. Most dapps that are built on Flux right now are replicated apps. I.e, they are deployed in separation and run isolated on each Fluxnode.
I'm more working towards actual distributed apps, that rely on each deployed node to be a part of a single bigger dapp across nodes.
The above commits are introducing fluxDockerNetwork_APPNAME docker network. Each application is now attached to this private network.
The network is in a scheme of
Subnet: 172.${number}.0.0/16,
Gateway: 172.${number}.0.1,
where number is 50 + X. X is a current number of instaleld apps on a particular node. I could not figure out any determinstic way of how to use this number and ensure that it is available. So it can happen that on one node, the gateway will be 172.50.0.1 and on another it can be 172.52.0.1.
Thank you for your comments
The above commits are introducing fluxDockerNetwork_APPNAME docker network. Each application is now attached to this private network. The network is in a scheme of Subnet:
172.${number}.0.0/16, Gateway:172.${number}.0.1, where number is 50 + X. X is a current number of instaleld apps on a particular node. I could not figure out any determinstic way of how to use this number and ensure that it is available. So it can happen that on one node, the gateway will be 172.50.0.1 and on another it can be 172.52.0.1.Thank you for your comments
Great work!