Deployment steps for k8s (and probably others)
Closed this issue · 17 comments
Hi guys,
It's unclear how to specify to px-dev which block devices it should use to store data/volumes on per node basis.
@AleksandrNull was this resolved? In general, you specify the disks to use in /etc/pwx/config.json
https://github.com/portworx/px-dev/blob/master/conf/config.json
Hi, @jvinod.
Yes, I found disk related things on ubuntu/docker related docs, not in k8s ones, so it's clear for now. But right now I'm stuck with px.ko module error during start of px-dev container on k8s cluster.
@AleksandrNull can you give us:
- the output of uname -a
- ls -l /usr/src
- the the docker run .. command that you started px with.
Thanks
Yep. Should I put it here ?
sure
root@ubuntu:# uname -a# ls -l /usr/src
Linux ubuntu 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:
total 8
drwxr-xr-x 24 root root 4096 Jan 27 2016 linux-headers-3.13.0-76
drwxr-xr-x 7 root root 4096 Jan 27 2016 linux-headers-3.13.0-76-generic
root@ubuntu:# docker run --restart=always --name px-dev -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd -v /dev:/dev -v /etc/pwx:/etc/pwx -v /opt/pwx/bin:/export_bin -v /var/lib/kubelet/plugins/volume/exec/pxflexvolume:/export_flexvolume:shared -v /var/run/docker.sock:/var/run/docker.sock -v /var/cores:/var/cores -v /var/lib/kubelet:/var/lib/kubelet:shared --ipc=host portworx/px-dev:latest
Hmm, should I have headers and sources from current kernel ? :) If that's a problem then I know what to do :)
Ok, got it. It thinks that I run it on rhel/centos:
Sat Aug 27 07:15:44 UTC 2016 : Running on Linux ubuntu 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Failed to enable rt scheduler
checking /usr/src/kernels/3.13.0-93-generic
checking /usr/src/linux-headers-3.13.0-93-generic
checking /lib/modules/3.13.0-93-generic/build
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
- base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
- updates: mirror.cogentco.com
No package kernel-headers-3.13.0-93-generic available.
Error: Nothing to do
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile - base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
- updates: mirror.cogentco.com
No package kernel-devel-3.13.0-93-generic available.
Error: Nothing to do
checking /usr/src/kernels/3.13.0-93-generic
checking /usr/src/linux-headers-3.13.0-93-generic
checking /lib/modules/3.13.0-93-generic/build
Checking short path for KVERSION 3.13.0-93.el7.x86_64
checking /usr/src/kernels/3.13.0-93.el7.x86_64
checking /usr/src/linux-headers-3.13.0-93.el7.x86_64
checking /lib/modules/3.13.0-93.el7.x86_64/build
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile - base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
- updates: mirror.cogentco.com
No package kernel-headers-3.13.0-93.el7.x86_64 available.
Error: Nothing to do
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile - base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
- updates: mirror.cogentco.com
No package kernel-devel-3.13.0-93.el7.x86_64 available.
Error: Nothing to do
checking /usr/src/kernels/3.13.0-93.el7.x86_64
checking /usr/src/linux-headers-3.13.0-93.el7.x86_64
checking /lib/modules/3.13.0-93.el7.x86_64/build
Failed to install/locate kernel headers for 3.13.0-93.el7.x86_64
^^^^ that's from px-dev container logs ;)
Can you run :
sudo apt-get install linux-headers-$(uname -r)
docker restart px-dev
Normally you don't need to do that. We would take care of it automatically. But it looks like your machine diverged from its original kernel?
@jvinod
Already done that and it's not helped. Found out that px-dev scripts/binaries detect my system as rhel/centos not ubuntu.
Was it able to install the headers? If so /usr/src should have headers that map to uname -r . With the output you posted, it does not: you're running 3.13.0-93-generic but you have headers for 3.13.0-76-generic
px-dev does not detect the system as centos.
We're working on making these corner cases painless..
root@ubuntu:# ls -l /usr/src# uname -a
total 16
drwxr-xr-x 24 root root 4096 Jan 27 2016 linux-headers-3.13.0-76
drwxr-xr-x 7 root root 4096 Jan 27 2016 linux-headers-3.13.0-76-generic
drwxr-xr-x 24 root root 4096 Aug 27 07:13 linux-headers-3.13.0-93
drwxr-xr-x 7 root root 4096 Aug 27 07:14 linux-headers-3.13.0-93-generic
root@ubuntu:
Linux ubuntu 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
----px-dev-logs----------
root@ubuntu:~# docker logs px-dev
Sat Aug 27 07:14:27 UTC 2016 : Running on Linux ubuntu 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Failed to enable rt scheduler
/docker-entry-point.sh: line 168: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
checking /usr/src/kernels/3.13.0-93-generic
checking /usr/src/linux-headers-3.13.0-93-generic
checking /lib/modules/3.13.0-93-generic/build
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
- base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
- updates: mirror.cogentco.com
No package kernel-headers-3.13.0-93-generic available.
Error: Nothing to do
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile - base: centos.aol.com
- epel: mirror.symnds.com
- extras: mirror.cogentco.com
-
updates: mirror.cogentco.com
No package kernel-devel-3.13.0-93-generic available.
Error: Nothing to do
It looks like yum, not apt-get for me. It wouldn't be able to retrieve these headers. I hope we're talking about ls -l /usr/src and uname -a at host and not inside of px-dev container ?
oh great:) I should have looked more closely at your docker run command:
Could you add to it -v /usr/src:/usr/src . I fixed the incorrect documentation.
You would need to docker stop px-dev, docker rm px-dev and then run again.
Regarding rhel/centos: We run in a centos based container (therefore yum) - and yes that is problematic to pull ubuntu distro headers. We're working on fixing that.
docker run --restart=always --name px-dev -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd -v /dev:/dev -v /etc/pwx:/etc/pwx -v /usr/src:/usr/src -v /opt/pwx/bin:/export_bin -v /var/lib/kubelet/plugins/volume/exec/px~flexvolume:/export_flexvolume:shared -v /var/run/docker.sock:/var/run/docker.sock -v /var/cores:/var/cores -v /var/lib/kubelet:/var/lib/kubelet:shared --ipc=host portworx/px-dev:latest
Yup!
That's the thing! Now it works!
root@ubuntu:~# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 838f3bbb-19e2-47c1-b0bc-46bfc65bfed9
IP: 172.16.96.163
Local Storage Pool: 1 device
Device Path Media Type Size Last-Scan
1 /dev/vdb STORAGE_MEDIUM_MAGNETIC 195 GiB 27 Aug 16 07:47 UTC
total - 195 GiB
Cluster Summary
Cluster ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5
Node IP: 172.16.96.163 - Capacity: 516 KiB/196 GiB Online (This node)
Node IP: 172.16.96.165 - Capacity: 516 KiB/196 GiB Online
Global Storage Pool
Total Used : 1.0 MiB
Total Capacity : 392 GiB
Thanks a lot, will test it further :) Is there any chance to get a possibility to test it on larger that 3 device scale + cache/ssd tiering ? ;)
Great :) Look forward to your feedback.
You'd need the enterprise edition to lift the cluster size + volume restrictions.
@jvinod
Ok, got it. Will test 3x3 mode then. Thanks and have a good night. I really appreciate your help!