kata-containers/runtime

Kata on MSFT Win10 WSL2

networkop opened this issue · 5 comments

networkop commented

I'm trying to run docker with kata on WSL2 with KVM support and I'm getting the following error:

docker: Error response from daemon: OCI runtime create failed: Failed to add filter for index 18 : no such file or directory: unknown

I'm running a custom kernel with

CONFIG_VHOST_VSOCK=y

Please see the kata.log

The biggest issue with WSL2 is that it doesn't run systemd, however it would be very interesting to be able to run docker+kata+(firecracker|clh|qemu) on a WSL2 VM.

fidencio commented

@networkop, let me redirect this issue to the appropriate repo, as it's happening with kata-containers 1.x.

  • 👍1
fidencio commented

I, myself, never tried to run kata-containers using WSL2 so I'm not of much help (sorry!), but I'd be interested to know in all possible logs you can get from it.

kata-containers itself doesn't require systemd, so you should be good to go there ... however docker does.
Out of curiosity, how did you deploy kata-containers? Was it using kata-deploy?

networkop commented

@fidencio thanks, I didn't realise it was the wrong repo. Sorry, I'm still getting used to the repo layout.

as for the deployment, I installed docker using the official instructions for ubuntu, installed kata as static binaries from here into the /opt directory and pointed docker at kata-fc as one of the available runtimes:

{
  "runtimes": {
    "kata-runtime": {
      "path": "/opt/kata/bin/kata-clh"
    }
  }
}

Docker is started with the init script service docker start and I also had to start a syslog daemon to satisfy kata's logging requirements. Here's the slightly redacted output of my docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux kata-runtime
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.16.3-microsoft-standard-WSL2+
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB
 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
networkop commented

Adding some more logs. Here's the command I'm trying to run:

$ docker run --runtime=kata-runtime --rm busybox -uname -r
docker: Error response from daemon: OCI runtime create failed: Failed to add filter for index 68 : no such file or directory: unknown.

I've tried to run this with all 3 available hypervisors (FC, CLH and QEMU) and getting the same error in all 3 cases.

strace.log

And here's the syslog with all debug options turned on (firecracker as runtime):

May  2 10:14:24 DESKTOP kernel: [ 7067.292267] docker0: port 1(veth56189fe) entered blocking state
May  2 10:14:24 DESKTOP kernel: [ 7067.292269] docker0: port 1(veth56189fe) entered disabled state
May  2 10:14:24 DESKTOP kernel: [ 7067.292292] device veth56189fe entered promiscuous mode
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.3621666+01:00" level=info msg="loaded configuration" arch=amd64 command=create file=/opt/kata/share/defaults/kata-containers/configuration-fc.toml format=TOML name=kata-runtime pid=3744 source=katautils
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.3623243+01:00" level=info msg="VSOCK supported, configure to not use proxy" arch=amd64 command=create name=kata-runtime pid=3744 source=katautils
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.3623736+01:00" level=info arch=amd64 arguments="\"create --bundle /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 --pid-file /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67/init.pid 4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67\"" command=create commit=b967088a667018b7468a9f93d48cb81650e0dfa4 name=kata-runtime pid=3744 source=runtime version=1.12.1
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.363355+01:00" level=info msg="shm-size detected: 67108864" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime pid=3744 source=virtcontainers subsystem=oci
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.364194+01:00" level=info msg="create netns" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime netns=/var/run/netns/cnitest-89e20f39-5cc3-2577-e313-c4fb4c639341 pid=3744 source=katautils
May  2 10:14:24 DESKTOP kernel: [ 7067.512881] eth0: renamed from vethab06b0e
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5839534+01:00" level=info msg="FC not ready, queueing device" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5845079+01:00" level=info msg="Endpoints found after scan" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 endpoints="[0xc0000c6840]" name=kata-runtime pid=3744 source=virtcontainers subsystem=network
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5849938+01:00" level=info msg="Attaching endpoint" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 endpoint-type=virtual hotplug=false name=kata-runtime pid=3744 source=virtcontainers subsystem=network
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5863471+01:00" level=error msg="Error bridging virtual endpoint" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="Failed to add filter for index 74 : no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=network
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5864379+01:00" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime pid=3744 sandbox=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 source=virtcontainers subsystem=sandbox
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5864889+01:00" level=error msg="Failed to umount resource" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5865201+01:00" level=error msg="Failed to umount resource" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5865731+01:00" level=error msg="Failed to umount resource" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5866312+01:00" level=error msg="Failed to umount resource" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5866588+01:00" level=error msg="Failed to umount resource" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 error="no such file or directory" name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5867225+01:00" level=info arch=amd64 cleaningJail=/run/vc/firecracker/4bb24e7ee549a408351353748629d8dc command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime pid=3744 source=virtcontainers subsystem=firecracker
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5867646+01:00" level=info msg="cleanup agent" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime path=/run/kata-containers/shared/sandboxes/4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67/shared pid=3744 source=virtcontainers subsystem=kata_agent
May  2 10:14:24 DESKTOP kata-runtime[3744]: time="2021-05-02T10:14:24.5869962+01:00" level=error msg="Failed to add filter for index 74 : no such file or directory" arch=amd64 command=create container=4bb24e7ee549a408351353748629d8dcaf27735acfe4cf1679850984879fae67 name=kata-runtime pid=3744 source=runtime
May  2 10:14:24 DESKTOP kernel: [ 7067.583140] IPv6: ADDRCONF(NETDEV_CHANGE): veth56189fe: link becomes ready
May  2 10:14:24 DESKTOP kernel: [ 7067.583156] docker0: port 1(veth56189fe) entered blocking state
May  2 10:14:24 DESKTOP kernel: [ 7067.583157] docker0: port 1(veth56189fe) entered forwarding state
May  2 10:14:24 DESKTOP kernel: [ 7067.590716] eth0: Caught tx_queue_len zero misconfig
May  2 10:14:24 DESKTOP kernel: [ 7067.614536] vethab06b0e: renamed from eth0
May  2 10:14:24 DESKTOP kernel: [ 7067.733378] docker0: port 1(veth56189fe) entered disabled state
May  2 10:14:24 DESKTOP kernel: [ 7067.753993] docker0: port 1(veth56189fe) entered disabled state
May  2 10:14:24 DESKTOP kernel: [ 7067.756360] device veth56189fe left promiscuous mode
May  2 10:14:24 DESKTOP kernel: [ 7067.756365] docker0: port 1(veth56189fe) entered disabled state
docker: Error response from daemon: OCI runtime create failed: Failed to add filter for index 74 : no such file or directory: unknown.

So it seems like it has something to do with networking. The index must be referring to the intfIndex of the veth link that docker creates. I can see that counter increasing with each attempt:

$ docker run --runtime=kata-runtime --rm busybox -uname -r
docker: Error response from daemon: OCI runtime create failed: Failed to add filter for index 88 : no such file or directory: unknown.
$ docker run --runtime=kata-runtime --rm busybox -uname -r
docker: Error response from daemon: OCI runtime create failed: Failed to add filter for index 90 : no such file or directory: unknown.

My kata configuration files are all default, so internetworking model is:

internetworking_model="tcfilter"
fidencio commented

This issue is being automatically closed as Kata Containers 1.x has now reached EOL (End of Life). This means it is no longer being maintained.

Important:

All users should switch to the latest Kata Containers 2.x release to ensure they are using a maintained release that contains the latest security fixes, performance improvements and new features.

This decision was discussed by the @kata-containers/architecture-committee and has been announced via the Kata Containers mailing list:

If you believe this issue still applies to Kata Containers 2.x, please open an issue against the Kata Containers 2.x repository, pointing to this one, providing details to allow us to migrate it.