Vega GPU not recognized inside container
evilmav opened this issue · 7 comments
evilmav commented
Running Arch linux, kernel 5.5.11. rocminfo on the host does show both agents and I can use OpenCL, but miopen currently fails to build, so I went for docker. Inside rocm-terminal container however, rocminfo only shows my CPU as an agent.
/dev/kfd under current arch is not in group video, but render. The container doesn't have it defined and shows 990 as a group:
rocm-user@MavsArch:~$ ll /dev
...
crw-rw-rw- 1 root **990** 242, 0 Mar 26 17:29 kfd
I therefore call the container as follows for testing:
docker run -it --network=host --device=/dev/kfd --device=/dev/dri --ipc=host --shm-size 8G --group-add video --group-add 990 --security-opt seccomp=unconfined rocm/rocm-terminal
rocminfo inside container:
$ rocminfo
sh: 1: lsmod: not found
ROCk module is NOT loaded, possibly no GPU devices
Failed to get user name to check for video group membership
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
==========
HSA Agents
==========
*******
Agent 1
*******
Name: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Marketing Name: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 0
Compute Unit: 4
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 15825980(0xf17c3c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 15825980(0xf17c3c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
ISA Info:
N/A
*** Done ***
Interestingly, rocm-smi sees the GPU:
$ rocm-smi
========================ROCm System Management Interface========================
================================================================================
GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 34.0c 4.0W 852Mhz 167Mhz 13.73% auto 165.0W 5% 0%
================================================================================
==============================End of ROCm SMI Log ==============================
rocminfo on host:
% rocminfo
ROCk module is loaded
maverick is member of video group
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
==========
HSA Agents
==========
*******
Agent 1
*******
Name: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Marketing Name: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 0
Compute Unit: 4
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 15825980(0xf17c3c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 15825980(0xf17c3c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
ISA Info:
N/A
*******
Agent 2
*******
Name: gfx900
Marketing Name: Vega 10 XL/XT [Radeon RX Vega 56/64]
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 4096(0x1000)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
Chip ID: 26751(0x687f)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1590
BDFID: 768
Internal Node ID: 1
Compute Unit: 56
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: FALSE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 8372224(0x7fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: FALSE
Pool 2
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Acessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx900
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
Also on host:
% lsmod | grep amdgpu
amdgpu 5279744 54
gpu_sched 36864 1 amdgpu
i2c_algo_bit 16384 1 amdgpu
ttm 122880 1 amdgpu
drm_kms_helper 237568 1 amdgpu
drm 557056 24 gpu_sched,drm_kms_helper,amdgpu,ttm
Would appreciate any assistance...
lolzballs commented
I was experiencing this issue too on Arch Linux. I tried a variety of docker flags before settling on --privileged, which seems to work. I'm not sure about the security implications of this and why its necessary but for now it works for me.
Here's the full command: docker run -it --rm --device=/dev/kfd --device=/dev/dri --privileged rocm/tensorflow:rocm3.1-tf2.1-python3
aligirayhanozbay commented
Debian bullseye with the 5.5 kernel here:
Linux ***** 5.5.0-1-amd64 #1 SMP Debian 5.5.13-2 (2020-03-30) x86_64 GNU/Linux
I had the same issue as you and your solution fixed it, except I did not have rocm-dkms installed on the host (fails to build on this kernel version anyway). When the --privileged option is not passed I received the same errors as you. With the rocm/tensorflow:latest image when executing docker run -it -p 8880:8880 -v ~/container-folders/tf:/persistent-tf/ -v ~/container_folders/:/persistent-common --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/tensorflow:latest bash
rocm-smi works fine
========================ROCm System Management Interface========================
================================================================================
GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 37.0c 27.0W 808Mhz 350Mhz 21.96% auto 250.0W 3% 8%
================================================================================
==============================End of ROCm SMI Log ==============================
rocminfo fails to recognize my Radeon VII
sh: 1: lsmod: not found
ROCk module is NOT loaded, possibly no GPU devices
Failed to get user name to check for video group membership
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD Ryzen 7 2700 Eight-Core Processor
Marketing Name: AMD Ryzen 7 2700 Eight-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3200
BDFID: 0
Internal Node ID: 0
Compute Unit: 16
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 32899232(0x1f600a0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 32899232(0x1f600a0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
ISA Info:
N/A
*** Done ***
Starting tensorflow fails with the following error:
root@94406f3c07fa:/root# python3
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> s = tf.constant([3])
2020-04-22 00:01:54.375262: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
2020-04-22 00:01:54.399379: E tensorflow/stream_executor/rocm/rocm_driver.cc:975] could not retrieve ROCM device count: HIP_ERROR_NoDevice
2020-04-22 00:01:54.406259: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3199545000 Hz
2020-04-22 00:01:54.407303: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3f9e1d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-22 00:01:54.407326: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-04-22 00:01:54.408750: E tensorflow/stream_executor/rocm/rocm_driver.cc:975] could not retrieve ROCM device count: HIP_ERROR_NoDevice
After I added the --privileged argument everything worked fine (and tensorflow on this container starts much faster than previous images or on bare metal on Nvidia!)
rocm-smioutput:
========================ROCm System Management Interface========================
================================================================================
GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 37.0c 25.0W 808Mhz 350Mhz 21.96% auto 250.0W 3% 4%
================================================================================
==============================End of ROCm SMI Log ==============================
rocminfooutput:
sh: 1: lsmod: not found
ROCk module is NOT loaded, possibly no GPU devices
Failed to get user name to check for video group membership
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD Ryzen 7 2700 Eight-Core Processor
Marketing Name: AMD Ryzen 7 2700 Eight-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3200
BDFID: 0
Internal Node ID: 0
Compute Unit: 16
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 32899232(0x1f600a0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 32899232(0x1f600a0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: TRUE
ISA Info:
N/A
*******
Agent 2
*******
Name: gfx906
Marketing Name: Vega 20
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 4096(0x1000)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
Chip ID: 26287(0x66af)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1801
BDFID: 10496
Internal Node ID: 1
Compute Unit: 60
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: FALSE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16760832(0xffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Acessible by all: FALSE
Pool 2
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Acessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx906
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
- Tensorflow output:
root@f6b537bba1c0:/root# python3
Python 3.6.9 (default, Nov 7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> s = tf.constant([3])
2020-04-22 00:04:40.840617: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
2020-04-22 00:04:40.898834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1573] Found device 0 with properties:
pciBusID: 0000:29:00.0 name: Vega 20 ROCm AMD GPU ISA: gfx906
coreClock: 1.801GHz coreCount: 60 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: -1B/s
2020-04-22 00:04:40.936718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocblas.so
2020-04-22 00:04:40.938159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libMIOpen.so
2020-04-22 00:04:40.940482: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocfft.so
2020-04-22 00:04:40.940665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocrand.so
2020-04-22 00:04:40.940764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-22 00:04:40.946171: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3199545000 Hz
2020-04-22 00:04:40.947017: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x6520fa0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-22 00:04:40.947046: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-04-22 00:04:40.948107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1573] Found device 0 with properties:
pciBusID: 0000:29:00.0 name: Vega 20 ROCm AMD GPU ISA: gfx906
coreClock: 1.801GHz coreCount: 60 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: -1B/s
2020-04-22 00:04:40.948141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocblas.so
2020-04-22 00:04:40.948149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libMIOpen.so
2020-04-22 00:04:40.948157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocfft.so
2020-04-22 00:04:40.948166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library librocrand.so
2020-04-22 00:04:40.948227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-04-22 00:04:40.948281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-22 00:04:40.948291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-04-22 00:04:40.948295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-04-22 00:04:40.948467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15306 MB memory) -> physical GPU (device: 0, name: Vega 20, pci bus id: 0000:29:00.0)
>>>
CyberCyclone commented
I'm having this issue with Ubuntu 20.04 as the host. The --privileged flag has no effect. I upgraded from Ubuntu 19.04 since it had the same issue and apparently Ubuntu 20 has better AMD support, but that doesn't seem to have done anything.
I've followed all the instructions from here: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master/quick-start.md with no issues. When I run sudo /opt/rocm/bin/rocminfo within the Docker container, it gives me this:
sh: 1: lsmod: not found
ROCk module is NOT loaded, possibly no GPU devices
Failed to get user name to check for video group membership
And then lists the CPU and 1 GPU with the name "gfx900" (Vega64).
aligirayhanozbay commented
@CyberCyclone
I get the same 'ROCk module is NOT loaded, possibly no GPU devices' after running rocminfo, and GPU is displayed below it like in your case with the --privileged flag. However, I was able to run tensorflow and pytorch on the GPU despite this. Perhaps it'd be worth your while to download the tensorflow latest image and see if it works? If it works the problem may be just with the reporting utility and whatever use case you may have will work, as was the case for me with Tensorflow/Pytorch.
CyberCyclone commented
@aligirayhanozbay Are you sure its running with GPU? I can get it to run, but it only runs on the CPU. The GPU load is 0%.
aligirayhanozbay commented
@CyberCyclone I'm fairly sure - running a little snippet of code like this, I see 100% GPU utilization and the GPU clock shoots up to 1800MHz (stock max clock of the VII) monitored using radeontop outside the container:
import torch
q = torch.rand(10000,10000).cuda()
while True:
r = torch.einsum('ij,jk->',q,q)forgetso commented
@CyberCyclone You can get rid of the error in rocminfo by installing kmod.
Run the following in the docker container:
apt-get update -y
apt-get install kmod
Presumably kmod contains lsmod or installs it or something. The point is, lsmod is missing from the container (at least under root). You will then get:
/opt/rocm/bin/rocminfo
ROCk module is loaded
Able to open /dev/kfd read-write