Missing blkio and memory cgroup support on 3B+ and 3B, 5.4.42-v7+ kernel
GeiserX opened this issue · 19 comments
GeiserX commented
Describe the bug
When I try to do anything related to containers in RPi, the software fails and messages that there is no memory cgroup. This issue comes from here: k3s-io/k3s#1825
To reproduce
5.4.42-v7+ kernel, Raspbian 10 buster, only affecting 3B+ and 3B versions, at least not in the Pi 4.
Expected behaviour
Software to install successfully and allow management of containers
Actual behaviour
Printing of error messages related to cgroups
System
System Information
------------------
Raspberry Pi 3 Model B Plus Rev 1.3
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
Raspberry Pi reference 2020-02-13
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 5f884374b6ac6e155330c58caa1fb7249b8badf1, stage2
Linux K3sNode2 5.4.42-v7+ #1319 SMP Wed May 20 14:06:11 BST 2020 armv7l GNU/Linux
Revision : a020d3
Serial : 00000000a3b1c042
Model : Raspberry Pi 3 Model B Plus Rev 1.3
Throttled flag : throttled=0x0
Camera : supported=0 detected=0
Videocore information
---------------------
May 19 2020 22:31:46
Copyright (c) 2012 Broadcom
version ba3a9fd1ea954b440ddea12f8f2b37b7ec1a2e23 (clean) (release) (start_cd)
alloc failures: 0
compactions: 0
legacy block fails: 0
Filesystem information
----------------------
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 61112892 2034076 56564348 4% /
devtmpfs 466624 0 466624 0% /dev
tmpfs 499904 0 499904 0% /dev/shm
tmpfs 499904 6612 493292 2% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 499904 0 499904 0% /sys/fs/cgroup
/dev/mmcblk0p1 258095 54326 203769 22% /boot
tmpfs 99980 0 99980 0% /run/user/1000
Filename Type Size Used Priority
Package version information
---------------------------
raspberrypi-ui-mods:
Installed: (none)
raspberrypi-sys-mods:
Installed: 20191105
openbox:
Installed: (none)
lxpanel:
Installed: (none)
pcmanfm:
Installed: (none)
rpd-plym-splash:
Installed: (none)
Networking Information
----------------------
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x
inet6 y::y.y.y.y prefixlen 64 scopeid 0x20<link>
ether m.m.m.m txqueuelen 1000 (Ethernet)
RX packets 560 bytes 78962 (77.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 231 bytes 30931 (30.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet x.x.x.x netmask x.x.x.x
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wlan0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether m.m.m.m txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
USB Information
---------------
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 3, If 0, Class=Hub, Driver=hub/3p, 480M
|__ Port 1: Dev 4, If 0, Class=Vendor Specific Class, Driver=lan78xx, 480M
Display Information
-------------------
Running Legacy framebuffer
Display: 3
state 0x120000 [TV is off]
device_name=Unk-Composite dis
config.txt
----------
aphy_params_current=819
arm_freq=1400
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x331df0
desired_osc_freq_boost=0x3c45b0
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
dphy_params_current=547
enable_tvout=1
force_eeprom_read=1
force_pwm_open=1
framebuffer_depth=16
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
init_uart_clock=0x2dc6c00
lcd_framerate=60
over_voltage_avs=50000
over_voltage_avs_boost=0x249f0
pause_burst_frames=1
program_serial_random=1
sdram_freq=450
total_mem=1024
hdmi_force_cec_address:0=65535
hdmi_force_cec_address:1=65535
hdmi_pixel_freq_limit:0=0x9a7ec80
device_tree=-
overlay_prefix=overlays/
hdmi_cvt:0=
hdmi_cvt:1=
hdmi_edid_filename:0=
hdmi_edid_filename:1=
hdmi_timings:0=
hdmi_timings:1=
cmdline.txt
-----------
coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 console=ttyS0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
raspi-gpio settings
-------------------
raspinfo: line 166: raspi-gpio: command not found
vcdbg log messages
------------------
Unable to determine the value of __LOG_START
Unable to read logging_header from 0x00000000
dmesg log
---------
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.4.42-v7+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1319 SMP Wed May 20 14:06:11 BST 2020
[ 0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x3a400000, size 64 MiB
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] On node 0 totalpages: 255488
[ 0.000000] Normal zone: 1996 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 255488 pages, LIFO batch:63
[ 0.000000] percpu: Embedded 20 pages/cpu s49792 r8192 d23936 u81920
[ 0.000000] pcpu-alloc: s49792 r8192 d23936 u81920 alloc=20*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 253492
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 console=ttyS0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 933252K/1021952K available (8192K kernel code, 679K rwdata, 2524K rodata, 1024K init, 824K bss, 23164K reserved, 65536K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] ftrace: allocating 27876 entries in 55 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] random: get_random_bytes called from start_kernel+0x328/0x4e8 with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 19.20MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[ 0.000007] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[ 0.000022] Switching to timer-based delay loop, resolution 52ns
[ 0.000292] Console: colour dummy device 80x30
[ 0.000888] printk: console [tty1] enabled
[ 0.000954] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[ 0.001003] pid_max: default: 32768 minimum: 301
[ 0.001349] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.001394] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.002586] CPU: Testing write buffer coherency: ok
[ 0.003154] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.004101] Setting up static identity map for 0x100000 - 0x10003c
[ 0.004319] rcu: Hierarchical SRCU implementation.
[ 0.005008] smp: Bringing up secondary CPUs ...
[ 0.006126] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.007366] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[ 0.008506] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[ 0.008660] smp: Brought up 1 node, 4 CPUs
[ 0.008747] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[ 0.008774] CPU: All CPU(s) started in HYP mode.
[ 0.008799] CPU: Virtualization extensions available.
[ 0.009766] devtmpfs: initialized
[ 0.026451] VFP support v0.3: implementor 41 architecture 3 part 40 variant 3 rev 4
[ 0.026740] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.026796] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.029833] pinctrl core: initialized pinctrl subsystem
[ 0.031072] NET: Registered protocol family 16
[ 0.035410] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[ 0.041833] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 0.041873] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 0.042119] Serial: AMBA PL011 UART driver
[ 0.044082] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[ 0.060127] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-05-19 22:31, variant start_cd
[ 0.070141] raspberrypi-firmware soc:firmware: Firmware hash is ba3a9fd1ea954b440ddea12f8f2b37b7ec1a2e23
[ 0.124875] bcm2835-dma 3f007000.dma: DMA legacy API manager, dmachans=0x1
[ 0.127048] SCSI subsystem initialized
[ 0.127345] usbcore: registered new interface driver usbfs
[ 0.127427] usbcore: registered new interface driver hub
[ 0.127576] usbcore: registered new device driver usb
[ 0.129705] clocksource: Switched to clocksource arch_sys_counter
[ 1.284797] VFS: Disk quotas dquot_6.6.0
[ 1.284944] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 1.285143] FS-Cache: Loaded
[ 1.285414] CacheFiles: Loaded
[ 1.296708] thermal_sys: Registered thermal governor 'step_wise'
[ 1.297093] NET: Registered protocol family 2
[ 1.298148] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[ 1.298217] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[ 1.298357] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 1.298567] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.298751] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 1.298825] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[ 1.299135] NET: Registered protocol family 1
[ 1.299893] RPC: Registered named UNIX socket transport module.
[ 1.299924] RPC: Registered udp transport module.
[ 1.299950] RPC: Registered tcp transport module.
[ 1.299975] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.301817] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 7 counters available
[ 1.305104] Initialise system trusted keyrings
[ 1.305367] workingset: timestamp_bits=30 max_order=18 bucket_order=0
[ 1.315906] FS-Cache: Netfs 'nfs' registered for caching
[ 1.316562] NFS: Registering the id_resolver key type
[ 1.316634] Key type id_resolver registered
[ 1.316661] Key type id_legacy registered
[ 1.316697] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1.317954] Key type asymmetric registered
[ 1.317983] Asymmetric key parser 'x509' registered
[ 1.318050] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[ 1.318087] io scheduler mq-deadline registered
[ 1.318114] io scheduler kyber registered
[ 1.321643] bcm2708_fb soc:fb: FB found 1 display(s)
[ 1.329854] Console: switching to colour frame buffer device 82x26
[ 1.334047] bcm2708_fb soc:fb: Registered framebuffer for display 0, size 656x416
[ 1.338508] raspberrypi-clk firmware-clocks: CPU frequency range: min 600000000, max 1400000000
[ 1.345135] bcm2835-rng 3f104000.rng: hwrng registered
[ 1.347123] vc-mem: phys_addr:0x00000000 mem_base=0x3f000000 mem_size:0x3f600000(1014 MiB)
[ 1.350917] vc-sm: Videocore shared memory driver
[ 1.352921] gpiomem-bcm2835 3f200000.gpiomem: Initialised: Registers at 0x3f200000
[ 1.367673] brd: module loaded
[ 1.381127] loop: module loaded
[ 1.384023] Loading iSCSI transport class v2.0-870.
[ 1.386423] libphy: Fixed MDIO Bus: probed
[ 1.388082] usbcore: registered new interface driver lan78xx
[ 1.389771] usbcore: registered new interface driver smsc95xx
[ 1.391270] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 2.121014] Core Release: 2.80a
[ 2.122517] Setting default values for core params
[ 2.124050] Finished setting default values for core params
[ 2.325936] Using Buffer DMA mode
[ 2.327407] Periodic Transfer Interrupt Enhancement - disabled
[ 2.328879] Multiprocessor Interrupt Enhancement - disabled
[ 2.330342] OTG VER PARAM: 0, OTG VER FLAG: 0
[ 2.331749] Dedicated Tx FIFOs mode
[ 2.333709] WARN::dwc_otg_hcd_init:1074: FIQ DMA bounce buffers: virt = ba514000 dma = 0xfa514000 len=9024
[ 2.336464] FIQ FSM acceleration enabled for :
Non-periodic Split Transactions
Periodic Split Transactions
High-Speed Isochronous Endpoints
Interrupt/Control Split Transaction hack enabled
[ 2.342936] dwc_otg: Microframe scheduler enabled
[ 2.343002] WARN::hcd_init_fiq:457: FIQ on core 1
[ 2.344312] WARN::hcd_init_fiq:458: FIQ ASM at 806bc564 length 36
[ 2.345627] WARN::hcd_init_fiq:497: MPHI regs_base at be810000
[ 2.346931] dwc_otg 3f980000.usb: DWC OTG Controller
[ 2.348288] dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
[ 2.349710] dwc_otg 3f980000.usb: irq 56, io mem 0x00000000
[ 2.351078] Init: Port Power? op_state=1
[ 2.352390] Init: Power Port (0)
[ 2.354034] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.04
[ 2.356664] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.358077] usb usb1: Product: DWC OTG Controller
[ 2.359463] usb usb1: Manufacturer: Linux 5.4.42-v7+ dwc_otg_hcd
[ 2.360902] usb usb1: SerialNumber: 3f980000.usb
[ 2.362925] hub 1-0:1.0: USB hub found
[ 2.364288] hub 1-0:1.0: 1 port detected
[ 2.366305] dwc_otg: FIQ enabled
[ 2.366315] dwc_otg: NAK holdoff enabled
[ 2.366324] dwc_otg: FIQ split-transaction FSM enabled
[ 2.366339] Module dwc_common_port init
[ 2.366666] usbcore: registered new interface driver usb-storage
[ 2.368191] mousedev: PS/2 mouse device common for all mice
[ 2.370912] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer
[ 2.373536] cpufreq: cpufreq_online: CPU0: Running at unlisted freq: 600000 KHz
[ 2.377567] cpufreq: cpufreq_online: CPU0: Unlisted initial frequency changed to: 700000 KHz
[ 2.381945] sdhci: Secure Digital Host Controller Interface driver
[ 2.383234] sdhci: Copyright(c) Pierre Ossman
[ 2.384963] mmc-bcm2835 3f300000.mmcnr: could not get clk, deferring probe
[ 2.386773] sdhost-bcm2835 3f202000.mmc: could not get clk, deferring probe
[ 2.388270] sdhci-pltfm: SDHCI platform and OF driver helper
[ 2.391289] ledtrig-cpu: registered to indicate activity on CPUs
[ 2.392783] hidraw: raw HID events driver (C) Jiri Kosina
[ 2.394277] usbcore: registered new interface driver usbhid
[ 2.395582] usbhid: USB HID core driver
[ 2.397667] vchiq: vchiq_init_state: slot_zero = (ptrval)
[ 2.400542] [vc_sm_connected_init]: start
[ 2.403957] vc_vchi_sm_init: failed to open VCHI service (-1)
[ 2.403964] [vc_sm_connected_init]: failed to initialize shared memory service
[ 2.406557] [vc_sm_connected_init]: end - returning -1
[ 2.409807] Initializing XFRM netlink socket
[ 2.411137] NET: Registered protocol family 17
[ 2.412534] Key type dns_resolver registered
[ 2.414493] Registering SWP/SWPB emulation handler
[ 2.415997] registered taskstats version 1
[ 2.417237] Loading compiled-in X.509 certificates
[ 2.418856] Key type ._fscrypt registered
[ 2.420053] Key type .fscrypt registered
[ 2.431626] uart-pl011 3f201000.serial: cts_event_workaround enabled
[ 2.432889] 3f201000.serial: ttyAMA0 at MMIO 0x3f201000 (irq = 81, base_baud = 0) is a PL011 rev2
[ 2.436714] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver
[ 2.439130] mmc-bcm2835 3f300000.mmcnr: mmc_debug:0 mmc_debug2:0
[ 2.440397] mmc-bcm2835 3f300000.mmcnr: DMA channel allocated
[ 2.467943] sdhost: log_buf @ (ptrval) (fa513000)
[ 2.506079] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[ 2.508899] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[ 2.511692] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[ 2.515692] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[ 2.518545] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[ 2.521812] of_cfs_init
[ 2.523046] of_cfs_init: OK
[ 2.526546] Waiting for root device PARTUUID=738a4d67-02...
[ 2.578679] random: fast init done
[ 2.579850] Indeed it is in host mode hprt0 = 00021501
[ 2.664465] mmc0: host does not support reading read-only switch, assuming write-enable
[ 2.669913] mmc1: new high speed SDIO card at address 0001
[ 2.671625] mmc0: new high speed SDXC card at address aaaa
[ 2.673539] mmcblk0: mmc0:aaaa SC64G 59.5 GiB
[ 2.677953] mmcblk0: p1 p2
[ 2.704956] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[ 2.706046] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[ 2.789747] usb 1-1: new high-speed USB device number 2 using dwc_otg
[ 2.789882] Indeed it is in host mode hprt0 = 00001101
[ 3.030038] usb 1-1: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
[ 3.032268] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.034175] hub 1-1:1.0: USB hub found
[ 3.035566] hub 1-1:1.0: 4 ports detected
[ 3.202759] EXT4-fs (mmcblk0p2): recovery complete
[ 3.205912] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
[ 3.208555] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[ 3.218325] devtmpfs: mounted
[ 3.225505] Freeing unused kernel memory: 1024K
[ 3.227188] Run /sbin/init as init process
[ 3.349752] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[ 3.480105] usb 1-1.1: New USB device found, idVendor=0424, idProduct=2514, bcdDevice= b.b3
[ 3.482841] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.484947] hub 1-1.1:1.0: USB hub found
[ 3.486464] hub 1-1.1:1.0: 3 ports detected
[ 3.840703] dwc_otg_handle_wakeup_detected_intr lxstate = 2
[ 3.914945] systemd[1]: System time before build time, advancing clock.
[ 4.039389] NET: Registered protocol family 10
[ 4.042168] Segment Routing with IPv6
[ 4.093499] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[ 4.098564] systemd[1]: Detected architecture arm.
[ 4.161890] systemd[1]: Set hostname to <K3sNode2>.
[ 4.339851] usb 1-1.1.1: new high-speed USB device number 4 using dwc_otg
[ 4.480278] usb 1-1.1.1: New USB device found, idVendor=0424, idProduct=7800, bcdDevice= 3.00
[ 4.483072] usb 1-1.1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.751419] lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): No External EEPROM. Setting MAC Speed
[ 4.755381] libphy: lan78xx-mdiobus: probed
[ 4.785845] lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): int urb period 64
[ 5.043269] random: systemd: uninitialized urandom read (16 bytes read)
[ 5.058717] random: systemd: uninitialized urandom read (16 bytes read)
[ 5.062605] systemd[1]: Created slice User and Session Slice.
[ 5.066736] random: systemd: uninitialized urandom read (16 bytes read)
[ 5.068800] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[ 5.074496] systemd[1]: Listening on udev Kernel Socket.
[ 5.078552] systemd[1]: Listening on fsck to fsckd communication Socket.
[ 5.085189] systemd[1]: Created slice system-getty.slice.
[ 5.089504] systemd[1]: Listening on Journal Socket (/dev/log).
[ 5.093273] systemd[1]: Reached target Swap.
[ 5.902337] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 5.996763] systemd-journald[133]: Received request to flush runtime journal from PID 1
[ 6.534610] vc_sm_cma: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.537791] bcm2835_vc_sm_cma_probe: Videocore shared memory driver
[ 6.537809] [vc_sm_connected_init]: start
[ 6.538483] vc_sm_cma_vchi_init: failed to open VCHI service (-1)
[ 6.538497] [vc_sm_connected_init]: failed to initialize shared memory service
[ 6.587552] mc: Linux media interface: v0.10
[ 6.644959] videodev: Linux video capture interface: v2.00
[ 6.728556] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.733251] bcm2835_audio bcm2835_audio: card created with 8 channels
[ 6.797017] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.799223] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.801281] bcm2835_mmal_vchiq: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.861711] bcm2835_v4l2: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.865328] bcm2835_isp: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.871428] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 6.871723] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 6.873646] bcm2835_codec: module is from the staging directory, the quality is unknown, you have been warned.
[ 6.879392] bcm2835_mmal_vchiq: Failed to open VCHI service connection (status=-1)
[ 7.156775] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[ 7.315554] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[ 7.421699] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 7.432541] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 7.432987] usbcore: registered new interface driver brcmfmac
[ 7.476853] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt failed with error -2
[ 7.662127] random: crng init done
[ 7.662141] random: 7 urandom warning(s) missed due to ratelimiting
[ 7.688048] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 7.720435] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2
[ 9.293635] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 9.294698] 8021q: 802.1Q VLAN Support v1.8
[ 9.298707] uart-pl011 3f201000.serial: no DMA platform data
[ 9.318958] Bridge firewalling registered
[ 9.639558] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[ 10.032600] 8021q: adding VLAN 0 to HW filter on device eth0
[ 16.101378] Bluetooth: Core ver 2.22
[ 16.101472] NET: Registered protocol family 31
[ 16.101481] Bluetooth: HCI device and connection manager initialized
[ 16.101507] Bluetooth: HCI socket layer initialized
[ 16.101523] Bluetooth: L2CAP socket layer initialized
[ 16.101548] Bluetooth: SCO socket layer initialized
[ 16.118093] Bluetooth: HCI UART driver ver 2.3
[ 16.118111] Bluetooth: HCI UART protocol H4 registered
[ 16.118184] Bluetooth: HCI UART protocol Three-wire (H5) registered
[ 16.118427] Bluetooth: HCI UART protocol Broadcom registered
[ 16.499671] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 16.499756] Bluetooth: BNEP filters: protocol multicast
[ 16.499782] Bluetooth: BNEP socket layer initialized
[ 44.848695] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
alemairebe commented
same behavior here with
5.4.45-v7+ #1321 SMP Wed Jun 10 17:33:27 BST 2020 armv7l GNU/Linux
cat /etc/debian_version
10.4
cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 console=ttyAMA0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory elevator=deadline fsck.repair=yes rootwait
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 2 1 1
cpu 3 24 1
cpuacct 3 24 1
devices 4 24 1
freezer 6 1 1
net_cls 5 1 1
pids 7 29 1
while a previous version of raspbian works as expected :
4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux
cat /etc/debian_version
10.3
cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=0 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:1F:7D:55 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=PARTUUID=18fd3f5e-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 2 3 1
cpu 6 35 1
cpuacct 6 35 1
blkio 7 35 1
memory 3 226 1
devices 9 35 1
freezer 4 3 1
net_cls 8 3 1
pids 5 39 1
idcrook commented
This just manifest itself in "release" Raspberry Pi OS on my 3B+
Is there a known good kernel I can revert to?
grep "status installed" /var/log/dpkg.log | grep -e raspberrypi -e libraspberrypi
2020-07-20 06:30:07 status installed raspberrypi-kernel:armhf 1.20200717-1
2020-07-20 06:30:17 status installed raspberrypi-bootloader:armhf 1.20200717-1
2020-07-20 06:30:18 status installed libraspberrypi0:armhf 1.20200717-1
2020-07-20 06:30:18 status installed libraspberrypi-doc:armhf 1.20200717-1
2020-07-20 06:30:18 status installed libraspberrypi-bin:armhf 1.20200717-1
2020-07-20 06:30:18 status installed libraspberrypi-dev:armhf 1.20200717-1
kernel state
uname -a
Linux rpihp2 5.4.51-v7+ #1326 SMP Fri Jul 17 10:45:25 BST 2020 armv7l GNU/Linux
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 5 1 1
cpu 7 28 1
cpuacct 7 28 1
devices 4 28 1
freezer 6 1 1
net_cls 2 1 1
pids 3 32 1
memory cgroup not listed, even though it is in /boot/cmdline.txt
--- /boot/cmdline.txt 2020-07-20 07:30:54.000000000 -0600
+++ /boot/cmdline_backup.txt 2020-07-20 07:11:22.000000000 -0600
@@ -1 +1 @@
-console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes cgroup_memory=1 cgroup_enable=memory rootwait
+console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwaitmnorrsken commented
Same problem here. Previously it worked without cmdline.txt parameters.
Have not tested with 32 bit kernel on rpi 4, but I had to add "cgroup_enable=memory" with 64 bit kernel.
Fix: use 64 bit kernel and add cgroup_enable=memory
(however this reduces IO performance on rpi3b+ with around 20%)
See my post at rpi forum: https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=280656
idcrook commented
Fix: use 64 bit kernel and add cgroup_enable=memory
(however this reduces IO performance on rpi3b+ with around 20%)
Your post at rpi form does explain or point to how to move to 64 bit kernel.
I'd rather not use this workaround on a 3B+ with 1GB RAM running Raspberry Pi OS as a k3s control plane node. 64 bit ubuntu for the Pi is my OS of choice on the 4GB/8GB RAM Pi 4s I have available for k3s until this raspberry pi kernel issue is fixed.
The "fix" would actually be to re-define the CONFIG_MEMCG and related CGROUP defines (is there a reason why they stopped being defined?) in the kernel released...
idcrook commented
I see that editing /boot/config.txt to include
arm_64bit=1
will choose the 64bit kernel, which includes the memory in cgroup-s, as @mnorrsken mentioned.
$ uname -a
Linux rpihp2 5.4.51-v8+ #1326 SMP PREEMPT Fri Jul 17 10:58:13 BST 2020 aarch64 GNU/Linux
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 9 14 1
cpu 3 41 1
cpuacct 3 41 1
blkio 5 41 1
memory 6 79 1
devices 8 41 1
freezer 2 14 1
net_cls 7 14 1
pids 4 45 1Found here: Pi4 64-bit raspbian kernel for testing - Focus on Pi4 - Raspberry Pi Forums
stevbev commented
This just manifest itself in "release" Raspberry Pi OS on my 3B+
Is there a known good kernel I can revert to?
grep "status installed" /var/log/dpkg.log | grep -e raspberrypi -e libraspberrypi 2020-07-20 06:30:07 status installed raspberrypi-kernel:armhf 1.20200717-1 2020-07-20 06:30:17 status installed raspberrypi-bootloader:armhf 1.20200717-1 2020-07-20 06:30:18 status installed libraspberrypi0:armhf 1.20200717-1 2020-07-20 06:30:18 status installed libraspberrypi-doc:armhf 1.20200717-1 2020-07-20 06:30:18 status installed libraspberrypi-bin:armhf 1.20200717-1 2020-07-20 06:30:18 status installed libraspberrypi-dev:armhf 1.20200717-1kernel state
uname -a Linux rpihp2 5.4.51-v7+ #1326 SMP Fri Jul 17 10:45:25 BST 2020 armv7l GNU/Linux cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 5 1 1 cpu 7 28 1 cpuacct 7 28 1 devices 4 28 1 freezer 6 1 1 net_cls 2 1 1 pids 3 32 1memory cgroup not listed, even though it is in
/boot/cmdline.txt--- /boot/cmdline.txt 2020-07-20 07:30:54.000000000 -0600 +++ /boot/cmdline_backup.txt 2020-07-20 07:11:22.000000000 -0600 @@ -1 +1 @@ -console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes cgroup_memory=1 cgroup_enable=memory rootwait +console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
I just ran into this issue when the new kernel hit stable repository. I installed the updates across all my devices and took down my k8s cluster. The 20200717 and 20201723 kernels appear to be missing blkio and memory cgroup support. I resolved by downgrading to the previous 20200601 kernel packages. Commands below should help quickly revert.
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-bootloader_1.20200601-1_armhf.deb
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20200601-1_armhf.deb
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-bin_1.20200601-1_armhf.deb
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-dev_1.20200601-1_armhf.deb
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi-doc_1.20200601-1_armhf.deb
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/libraspberrypi0_1.20200601-1_armhf.deb
sudo dpkg -i *deb
pelwell commented
The memory and block IO cgroups have been added to BCM2835-7, to match BCM2711 and previous kernels.
popcornmix commented
Can you test rpi-update kernel and check if it has the features you need.
idcrook commented
The "fix" would actually be to re-define the CONFIG_MEMCG and related CGROUP defines (is there a reason why they stopped being defined?) in the kernel released...
and
Can you test rpi-update kernel and check if it has the features you need.
before update
$ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Plus Rev 1.3
$ uname -a
Linux rpihp2 5.4.51-v7+ #1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 5 1 1
cpu 4 1 1
cpuacct 4 1 1
devices 2 25 1
freezer 7 1 1
net_cls 3 1 1
pids 6 31 1
$ sudo rpi-update
# ... reboot ...rpi-update output
*** Updating kernel modules
*** depmod 5.4.51-v8+
*** depmod 5.4.51-v7+
*** depmod 5.4.51-v7l+
*** depmod 5.4.51+
*** If no errors appeared, your firmware was successfully updated to 56f62dee2a2385fe030724f7b39344fbc8c8d7ab
after update
$ uname -a
Linux rpihp2 5.4.51-v7+ #1328 SMP Mon Jul 27 16:51:11 BST 2020 armv7l GNU/Linux
$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes cgroup_memory=1 cgroup_enable=memory rootwait
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 6 1 1
cpu 3 1 1
cpuacct 3 1 1
blkio 5 1 1
memory 8 56 1
devices 2 25 1
freezer 4 1 1
net_cls 7 1 1
pids 9 30 1
blkio 5 1 1
memory 8 56 1
So cgroups_memory is now present again.
idcrook commented
$ cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled blkio 5 1 1 memory 8 56 1
So
cgroups_memoryis now present again.
And I can confirm that k3s control plane node is now content running again on a Raspberry Pi 3 Model B Plus Rev 1.3 with Raspberry Pi OS using the kernel version from recent rpi-update
bderenzo commented
Same problem on RPi 4, I update to lastest kernel version using rpi-update but doesn't have the cgroups_memory.
hypi ~ $ sudo rpi-update 56f62dee2a2385fe030724f7b39344fbc8c8d7ab
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Performing self-update
*** Relaunching after update
*** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
*** Your firmware is already up to date
hypi ~ $ cat /boot/cmdline.txt
console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
hypi ~ $ uname -a
Linux hypi 5.4.51-v7l+ #1328 SMP Mon Jul 27 16:57:05 BST 2020 armv7l GNU/Linux
hypi ~ $ cat /sys/firmware/devicetree/base/model
Raspberry Pi 4 Model B Rev 1.1h
And cgroups
hypi ~ $ cat /proc/cgroups | column -t
#subsys_name hierarchy num_cgroups enabled
cpuset 3 2 1
cpu 2 26 1
cpuacct 2 26 1
blkio 4 26 1
memory 0 33 0
devices 6 26 1
freezer 7 2 1
net_cls 5 2 1
pids 8 31 1
idcrook commented
Your output shows that it indeed does have cgroups configured in the kernel
memory 0 33 0
enable in cmdline.txt by including : cgroup_memory=1 cgroup_enable=memory
pelwell commented
Hint: Adding significant information by editing a comment effectively hides it from anyone following by email.
bderenzo commented
mnorrsken commented
Afaik memory cgroup enabled has a memory overhad of 8MB on a 1GB system (source: internet). So its understandable if default setting is disabled.
Also: great news, I will test the new kernel later.
bderenzo commented
Afaik memory cgroup enabled has a memory overhad of 8MB on a 1GB system (source: internet). So its understandable if default setting is disabled.
Also: great news, I will test the new kernel later.
8MB on 1GB is less than 1%. The problem is not so much the fact that it is not enabled, but the fact that it is no longer enabled by default.
swarred commented
Hello, I was hesitant to do a firmware update as I did not want to have to deal with potential breaking changes for other configured software I had on one of my nodes. My issue was:
failed to find memory cgroup, you may need to add "cgroup_memory=1 cgroup_enable=memory" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)
when trying to join a node to my k3s cluster.
My fix was to edit the /boot/cmdline.txt, as others have already suggested. However, do ensure that you are manually typing out cgroup_memory=1 cgroup_enable=memory on the same line. I do think it resolves the issue without firmware update needed.
bhkamath commented
I have the same error; observe cgroups: memory related error even after a reboot
cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=b7e4e595-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles
cgroup_enable=cupset cgroup_memory=1 cgroup_enable=memory
CGROUPS_MEMORY: missing
CGROUPS_HUGETLB: missing
[WARNING SystemVerification]: missing optional cgroups: hugetlb
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR SystemVerification]: missing required cgroups: memory
Request your help.
bhkamath commented
Got resolved, as it appeared to be a line separation issue.