Get CPU infos on docker container
TECO-Octo opened this issue · 9 comments
- CodeCarbon version: 2.3.4
- Python version: 3.8.16
- Operating System: Linux
Description
I wanted to track the emissions of a scope of code inside a docker container.
Though it is not able to find and track the CPU, thus it is falling back on constant mode.
What I Did
I used codecarbon task manager in a small portion of code (in my case, during the inference of a model).
Here are the logs from my docker container
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] [setup] RAM Tracking...
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] [setup] GPU Tracking...
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] No GPU found.
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] [setup] CPU Tracking...
2024-04-22 16:57:32 [codecarbon WARNING @ 14:57:32] No CPU tracking mode found. Falling back on CPU constant mode.
2024-04-22 16:57:32 [codecarbon WARNING @ 14:57:32] We were unable to detect your CPU using the `cpuinfo` package. Resorting to a default power consumption of 85W.
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] CPU Model on constant consumption mode: Unknown
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] >>> Tracker's metadata:
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] Platform system: Linux-6.6.16-linuxkit-aarch64-with-glibc2.17
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] Python version: 3.8.16
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] CodeCarbon version: 2.3.4
2024-04-22 16:57:32 [codecarbon INFO @ 14:57:32] Available RAM : 7.657 GB
I wonder if there is a way to allow CPU infos access on a docker container to avoid the falling back mode in codecarbon.
Hello,
Yes, you could run the container in privileged mode to allow it to read /sys to use the RAPL files of the hosts.
But do this only on container you trust !
Can you try with:
volumes:
- type: bind
source: /proc
target: /proc
- type: bind
source: /sys/class/powercap
target: /sys/class/powercap
I see this on https://github.com/hubblo-org/scaphandre/blob/e09dfcc7abd28291e783643e366393325fad3f6d/docker-compose/docker-compose.yaml#L26 from https://hubblo-org.github.io/scaphandre-documentation/tutorials/getting_started.html
Hello!
/sys/class/powercap
does not exist for me as I run the docker from a MacOS device (Apple Silicon M1 chip).
Hence I followed the methodology from this link for Apple Silicon Chips
I modified the docker-compose as such
privileged: true
volumes:
- type: bind
source: /proc
target: /proc
- type: bind
source: /usr/bin/powermetrics
target: /usr/bin/powermetrics
But still encounter the same issue (switching to falling back mode)
Ok, so I don't think it's possible because for Powermetrics we don't just read a file but executing a Mac application so it probably can't work in Docker.
@LuisBlanche any idea ?
Hi, yes I think it might be complicated indeed. I'll have a better look at it when I can, but it's been a common issue with people trying to use codecarbon
in Docker. One suggestion I could give is to run codecarbon monitor
outside docker right before running it, but I don't know if it fits your use case
Maybe using this doc about docker runtime metrics we could try to do something but i will only give us CPU usage % and not task level consumption
Thanks for your answers! Yes, sadly it seems complicate as for now.
I will keep in touch with you guys if you come up with something else!