Port mapping endpoint NAT policy not getting applied with L2Bridge HNS network

[rawahars]

Describe the bug
We are trying to implement port mapping functionality on L2Bridge HNS network in a CNI plugin which would be invoked by the container runtime. We saw that the same support was added to win-bridge OSS CNI plugin in the PR containernetworking/plugins#475.

The end to end workflow is-

• Run pause container in none network mode which creates a new namespace for the pods
• Invoke the CNI plugin which configures the network namespace
  • If not done, create a new L2Bridge HNS network with the primary host interface
  • Create a new Endpoint from the above HNS network.
  • Attach the endpoint to the container namespace
• Run other containers which belong to the same pod

To implement this port mapping functionality, we are adding the NAT HNS policy to the HNS Endpoint before creation (same as above PR)-

{
  "Type": "NAT",
  "InternalPort": 80,
  "ExternalPort": 8081,
  "Protocol": "TCP"
}

However, when the endpoint is created, the HNS policy is not getting applied i.e. the exposed port in host namespace is not able to access the web server running inside the container namespace.

We also validated using netstat -a command that the exposed port on host was not Listening.

To Reproduce
The code/config used in the CNI for creating the network and endpoint is as below-

// Create HNS Network with the following config
n = hcsshim.HNSNetwork{
		Name:               nw.NetworkName,
		Type:               "L2Bridge",
		NetworkAdapterName: nw.AdapterName,

		Subnets: []hcsshim.Subnet{
			{
				AddressPrefix:  nw.SubnetPrefix,
				GatewayAddress: nw.Gateway,
			},
		},
	}

	hnsResponse, err := n.Create()

// Create HNS Endpoint with the following config
hnsEndpoint := hcsshim.HNSEndpoint{
		Name:               endpointName,
		VirtualNetworkName: nw.NetworkName,
	}
portMapPolicy, err := json.Marshal(NatPolicy{
		Type:                 "NAT",
		ExternalPort:         uint16(ep.PortMap.HostPort),
		InternalPort:         uint16(ep.PortMap.ContainerPort),
		Protocol:             "TCP",
		ExternalPortReserved: true,
	})

hnsEndpoint.Policies = append(hnsEndpoint.Policies, portMapPolicy)

outboundNatPolicy, err := json.Marshal(hcsshim.OutboundNatPolicy{
	Policy: hcsshim.Policy{Type: hcsshim.OutboundNat},
})

hnsEndpoint.Policies = append(hnsEndpoint.Policies, outboundNatPolicy)

hnsResponse, err := hnsEndpoint.Create()

Expected behavior
We expect that upon attaching the HNS Endpoint to the container namespace, we would be able to access the web server running inside container namespace using the exposed port on host.

Configuration:

• Edition: Windows Server Full edition
• Base Image being used: Windows Server 2019 and Windows Server 2022
• Container engine: docker as well as containerd
• Container Engine version: docker version 20.10.21 and containerd version 1.6.6

Additional context
To determine if something was amiss in our setup, we created a setup using docker with the following commands-

docker network create -d l2bridge --subnet=10.0.1.0/24 -o com.docker.network.windowsshim.dnsservers=10.0.0.2 --gateway=10.0.1.1 -o com.docker.network.windowsshim.enable_outboundnat=true winl2bridge

docker run --rm -d -p 8081:80 --net winl2bridge mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019

In this setup, we saw that the exposed port 8081 was listening in the host namespace and we could access the webserver inside container using the exposed port.

We compared the HNS configurations of both the endpoints i.e. one we created using our CNI vs the one created by docker and both were exactly same.

[sbangari]

@rawahars This issue is caused by a missing route on the host network namespace. There are two options to get around this problem.

1. We can create the route manually on the node (not preferred)
2. Specify an additional flag while creating the NAT policy (only available in V2 version of HCN API) and the route is automatically created by Windows platform (preferred) https://pkg.go.dev/github.com/Microsoft/hcsshim@v0.9.9/hcn#PortMappingPolicySetting

Please refer to this similar bug for more details:
Azure/azure-container-networking#1863

[rawahars]

@sbangari Thanks for the response.

Can you please let me know which route policy needs to be created here in the host?
Is it for the host primary IP address?

[daschott]

Can you try with specifying portMapping policy with flag set?

rawPolicy, _ := json.Marshal(&hcn.PortMappingPolicySetting{
			ExternalPort: 8081,
			InternalPort: 80,
			Protocol:     6,
			Flags:        hcn.NatFlagsLocalRoutedVip,
		})
		
		hnsv2Policy, _ := json.Marshal(&hcn.EndpointPolicy{
			Type:     hcn.PortMapping,
			Settings: rawPolicy,
		})

Or in powershell:

$type = "PortMapping";

$pMapPolicy = @{
InternalPort = 80;
ExternalPort = 8081;
Protocol = 6;
Flags = 1;
};

$settings = @{
Type = $type;
Settings = $pMapPolicy;
};
Add-HnsEndpointPolicy -Endpoints $id -Policies @($settings)

Also FYI CNI plugin equivalent that uses v2 code can be found under: aka.ms/wincni

[rawahars]

@daschott I did try the above Powershell with the created endpoint but it failed with the error Invalid policy type.

We are in an unfortunate position where we need to create these endpoints via HNS V1 since our CNI also needs to support both containerd and docker runtime. Essentially, we need to support-

• With containerd, the namespace is created and then we use the namespace id to attach the endpoints
• With docker, the pause container is started in none mode which then creates the namespace and we configure it.

Therefore, we create HNS Network and endpoints via V1 commands and then either use hcsshim.HotAttachEndpoint for docker or hcn.AddNamespaceEndpoint for containerd.

I went through moby code and it seems like docker is also using HNS v1 but the port mapping works in that case.
Also, win-bridge CNI was working with HNS V1 before V2 support was added.

In our CNI, we do add the Route policies-

{DestinationPrefix=10.0.1.81/32; NeedEncap=True; Type=ROUTE}

Is there a way we can achieve this via V1 commands?

[daschott]

Here are sample commands that I used. It seems to be working for me e.g. when I try to curl <host_ip>:8080
Can you try this and see if it is still showing Invalid policy type? Important to add "flags = 1" in portmappings policy.

# Import module
Start-BitsTransfer https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/hns.v2.psm1
ipmo .\hns.v2.psm1

# Create network 
docker network create -d l2bridge --subnet 172.16.0.0/24 -o com.docker.network.windowsshim.dnsservers=168.63.129.16 --gateway=172.16.0.1 winl2bridge
$network = get-hnsnetwork | ? Name -Like $(docker network inspect --format='{{.ID}}' winl2bridge)

# Repro issue

# Create container, endpoint, attach
docker run -d --rm --name c1 --hostname c1 --network none mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
$compartmentId = docker exec "c1" powershell.exe "Get-NetCompartment | Select -ExpandProperty CompartmentId"
$hnsEndpoint = New-HnsEndpoint -NetworkId $network.ID -Name my_ep -IPAddress 172.16.0.172 -Verbose
Attach-HnsHostEndpoint -EndpointID $hnsEndpoint.Id -CompartmentID $compartmentId

# Attach portMapping policy
$id=$hnsEndpoint.ID

$type = "PortMapping";
$pMapPolicy = @{
InternalPort = 80;
ExternalPort = 8080;
Protocol = 6;
Flags = 1;
};

$settings = @{
Type = $type;
Settings = $pMapPolicy;
};

Add-HnsEndpointPolicy -Endpoints $id -Policies @($settings)

[rawahars]

@daschott Ah I think the reason for it to be working is that the HNS Powershell module is based on HNS V2 schema and therefore, it would work.

I tried it on the HNS endpoint which I created via HNS V1. The policy addition failed there.

[rawahars]

Is there a way we can accomplish this using HNS v1 API similar to how docker does it or win-bridge CNI plugin?

[jterry75]

@sbangari - How is Docker doing this? In Libnetwork CreateEndpoint is done here. Which you can see creates a NatPolicy here just like the example above. You end up with an endpoint, that has a nat policy on it at creation. Then you HotAttach it here. This is all HNSv1. Where is the missing "route in the host network namespace". Where is Docker adding that? If you can show us the missing modify call we are happy to add it.

[jterry75]

@daschott - Can you remind me when HCN (HNSv2) was added. My brain cant seem to remember. Is HCN fully supported on WS2019.

Since in the Docker case, the network is created via HNSv1, I assume it's then totally valid based on your sample above to use HCN to create an endpoint, apply policies, and use HotAttach (HNSv1) to attach v2 created Endpoint to an HNSv1 Network?

[jterry75]

@daschott - I read the code of exactly your commands. You are doing something different than Docker even that I can tell. Attach-HnsHostEndpoint. What is "HostAttach" versus "ContainerAttach"? We are certainly not doing an HNSCall with with a System type and CompartmentId. We are doing an HNSCall with ContainerId, ContainerType, and EndpointId. Where is Docker/winbridge etc doing a HostAttach?

[daschott]

V2 is supported on 2019. We are not adding features to old V1 API and recommend you use V2. Kubernetes and products we actively support and work on are all using V2 at this point. Is there any reason V2 is not feasible?

Can you apply the above policy on a container with running endpoint? I think it may still work if you created V1 endpoint or used another call to attach endpoint to container. Are you saying that is failing or working now?

I can also try on my setup if there are continued challenges, but will likely not have much time until sometime next week at earliest.

[rawahars]

@daschott We were able to get it working by using HNS V2. I will close this issue once we have completed all the validations.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@MikeZappa87, please provide an update or close this issue.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@MikeZappa87, please provide an update or close this issue.