Firewall rules using VPC as target should allow/deny traffic based on private IP, not external IP
Closed this issue · 4 comments
While investigating oxidecomputer/omicron#3373, I've dug in a bit more on just the default firewall rule behavior (w/o making any changes to the VPC fw rules). This is the opte firewall setup for a brand new project and default VPC (and no change to the fw rules):
BRM44220011 # /opt/oxide/opte/bin/opteadm list-layers -p opte6
NAME RULES IN RULES OUT DEF IN DEF OUT FLOWS
gateway 1 5 deny deny 0
firewall 3 0 deny stateful allow 0
router 0 2 allow deny 0
nat 1 2 allow allow 0
overlay 1 1 deny deny 0
BRM44220011 # /opt/oxide/opte/bin/opteadm dump-layer firewall -p opte6
Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.20.17.42 64834 172.30.0.5 5201 0 no-op
Outbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.30.0.5 5201 172.20.17.42 64834 1 no-op
Inbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
2 65534 1 inner.ip.proto=TCP "Stateful Allow"
inner.ulp.dst=22
1 65534 1 meta: vni=7524168 "Stateful Allow"
0 65534 0 inner.ip.proto=ICMP "Stateful Allow"
DEF -- 0 -- "deny"
Outbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
DEF -- 47 -- "stateful allow
I created an instance in it and ran an iperf3 server on it.
ubuntu@default-fwrules:~$ iperf3 -s -D
ubuntu@default-fwrules:~$ exit
logout
Connection to 172.20.26.45 closed.
pisces-2:docs angela$ nc -vz 172.20.26.45 5201
Connection to 172.20.26.45 port 5201 [tcp/targus-getdata1] succeeded!
Based on the default rules, the only port accessible on tcp should be 22. So it'd look like the deny-all default is not taking effect.
Next, I tried disabling the allow-ssh rule and saw that the change was reflected in the opte port:
BRM44220011 # /opt/oxide/opte/bin/opteadm dump-layer firewall -p opte6
Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.20.17.42 64897 172.30.0.5 22 0 no-op
Outbound Flows
----------------------------------------------------------------------
PROTO SRC IP SPORT DST IP DPORT HITS ACTION
TCP 172.30.0.5 22 172.20.17.42 64897 1 no-op
Inbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
4 65534 1 meta: vni=7524168 "Stateful Allow"
3 65534 0 inner.ip.proto=ICMP "Stateful Allow"
DEF -- 0 -- "deny"
Outbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
DEF -- 50 -- "stateful allow"
Afterwards, I was still able to SSH to the vm. This further confirms that there is no firewall at all.
If you mean the Unified Flow Table, then that can be done with opteadm clear-uft.
After digging in a bit more, I think the issue here isn't that there was no firewall in effect. If I disable the "allow-internal-inbound" rule that is meant to allow all traffic within the same VPC, I am no longer allowed to tcp-connect to ports from my workstation, except for port 22.
It seems that the external IP range is inadvertently filtered in the same way as internal IP range. In other words, the allow-internal-inbound which has VPC as target and source (host filter) allows all traffic between IPs on the same external subnet as well. Since my workstation is on a 172.20.x.x IP address, it's considered being on the same subnet as the VM's 172.20.26.x external IP.
The only way I can actually achieve "allow-internal-inbound" is to create a rule that uses a specific subnet as the source and target, resulting in rules that look like these in opte:
Inbound Rules
----------------------------------------------------------------------
ID PRI HITS PREDICATES ACTION
173 0 0 inner.ip.proto=UDP "Stateful Allow"
inner.ip6.src=fdaf:4243:35d6::/64
172 0 0 inner.ip.proto=UDP "Stateful Allow"
inner.ip.src=172.30.0.0/22
171 0 0 inner.ip.proto=TCP "Stateful Allow"
inner.ip6.src=fdaf:4243:35d6::/64
170 0 0 inner.ip.proto=TCP "Stateful Allow"
inner.ip.src=172.30.0.0/22
169 65534 0 inner.ip.proto=ICMP "Stateful Allow"
DEF -- 140 -- "deny"
I am not sure if the vpc source/target should filter based on the external IP address. If user needs to restrict access to VMs' external IP, they probably should do so by specifying specific subnet CIDR or IP addresses as the target. (The fact that we have a default rule named "allow-internal-inbound" seems to suggest that it's for allowing traffic on private IP only.)
It seems like we need to be able to discern between boundary services ingress traffic and internal VPC traffic. Because these are both on the same Geneve VNI we need additional information. At first glance, there are two options.
- Use Geneve TLVs. Every ingress packet that gets NAT'd by boundary services would also have a TLV added to the packet indicating this is from an external source.
- Use the source address of the underlay packet. This address is not a normal underlay address in the RFD 63 sense. It has its own (for better or worse) addressing scheme that is currently just an ad-hoc
fd00:99::1/64address. When we get to multi-switch this will shift to an anycast address or a pair of addresses on different /64s. But at any rate, it will be distinguishable from sled underlay addresses.
Option 1 is where we ultimately want to be. If complications arise and other critical path work takes precedence option 2 may be an ok path to take.