pcengines/apu2-documentation

APU4 Opnsense no back LEDs, only front LEDs

Opened this issue · 6 comments

stodec commented

On my APU4 with Opnsense 21.1.1 and BIOS v4.13.0.3 I implemented the workaround from @alexpro by setting debug.acpi.avoid to \_SB_.PCI0.GPIO in the Opnsense tunables, rebooted and did this as root:

$ kldload apuled
$ echo 1 > /dev/led/led3 # front LED 3 on APU is now on
$ echo 0 > /dev/led/led3 # LED off
$ ls /dev/led/           # expected to also see igb0, igb1, igb2, igb3
led1    led2    led3

I expected to also see igb0, igb1, igb2 and igb3 like users @ktk and @wschlich described in two comments on opnsense issue 2114 so that I could turn on/off the network LEDs. Am I missing something (or is this the wrong place to ask)? Thanks!

miczyg1 commented

Probably not related to apuled driver, but at least we can check where these Ethernet port LEDs originate from and point you in the correct direction.

  • 👍1
MarcFontaine commented

On Linux at least it is also possible to poke directly to the registers that control the LEDs.
I use the following command to set the LEDs of eth0 to inverted mode:

/bin/pcimem /sys/module/igb/drivers/pci\:igb/0000\:01\:00.0/resource0 0x0e00 w 0x000004246

The effect is that the LEDs are off if a link is detected or if there is traffic
and turn on if the up-link is dead or somebody has unplugged the cable.

stodec commented

Thanks, @MarcFontaine, inverted mode is even better. In Opnsense (21.1.2/FreeBSD 12.1) I tried pciconf. It displays igb0:

$ pciconf -lbcv igb0
igb0@pci0:1:0:0:	class=0x020000 card=0x00008086 chip=0x15398086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I211 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xf7800000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x1000, size 32, disabled
    bar   [1c] = type Memory, range 32, base 0xf7820000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 512(512) FLR RO NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 000db9ffff53f35c
    ecap 0017[1a0] = TPH Requester 1

It seems to be able to read values from it:

$ pciconf -r igb0 0x00:0xff
15398086 00100406 02000003 00000010
f7800000 00000000 00001001 f7820000 
00000000 00000000 00000000 00008086
00000000 00000040 00000000 00000118 
c8235001 00002108 00000000 00000000
01807005 00000000 00000000 00000000 
00000000 00000000 00000000 00000000
8004a011 00000003 00002003 00000000 
00000000 00000000 00000000 00000000
00000000 00000000 00000000 ffffffff 
00020010 10008cc2 00102850 00425c11
10110040 00000000 00000000 00000000 
00000000 0000001f 00000000 00000000
00000001 00000000 00000000 00000000 
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000

Writing does seems to have no effect (and I don't know which address ;)).

$ pciconf -r igb0 0x0e00 
00000000
$ pciconf -w igb0 0x0e00 0x00004246
$ pciconf -r igb0 0x0e00 
00000000

In the APU4D4 schematics on page 9 (titled Gigabit Ethernet) I can see LED connectors for the network adapter but I don't know if that would help. Any pointers are appreciated.

MarcFontaine commented

I think pciconf is not the right tool here. If I interpret the man page correctly it can only write to the PCI configuration of the device.
That changes how the device is mapped into system memory but does not write to the device registers itself.
The -D seems to dump/read device registers but cannot write.

The device maps three regions. The 128K size region is the correct one.
bar [10] = type Memory, range 32, base 0xf7800000, size 131072, enabled
Maybe you can dump that region with pciconf -D and at least read out the current LED configuration.

I have not used FreeBSD before. Probably it is possible to do the same trick as with Linux but I don't know how.
There is a generic FreeBSD proto kernel driver that can give access to an device (but only if the device is not used)
and devices can also be mapped to /dev/... .

Maybe there is something in /dev/igb0/.. ?

The setup of the LEDs itself is described in i211-ethernet-controller-datasheet.pdf (google).
The are lots of different modes that can be set for the LEDs.

The default setting for the LEDs is 0x0E00: 0x00078406 .
Reading it with pcimem looks like this:

[root@apu:/sys/module/igb/drivers/pci:igb/0000:02:00.0]# pcimem /sys/module/igb/drivers/pci\:igb/0000\:02\:00.0/resource0 0x0e00 w
/sys/module/igb/drivers/pci:igb/0000:02:00.0/resource0 opened.
Target offset is 0xe00, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xe00)
PCI Memory mapped to address 0x7f2bd0cac000.
0x0E00: 0x00078406
stodec commented

Thank you, @MarcFontaine, with your detailed explanation I could in fact read the existing LED settings in FreeBSD. Nice!

$ pciconf -D igb0 0x10 | hexdump -s 0x0e00 -n 4
0000e00 8406 0007
0000e04

I also never used FreeBSD before, only for Opnsense. I hoped for something like /dev/igb0, but only found /dev/led (with the three front leds).

$ ls /dev
acpi		ctty		geom.ctl	midistat	reroot		ttyu1.init	ttyvb
ada0		cuau0		gpt		modesw		sequencer0	ttyu1.lock	tun1
ada0p1		cuau0.init	gptid		msdosfs		sndstat		ttyv0		tun3
ada0p2		cuau0.lock	hpet0		music0		speaker		ttyv1		ufssuspend
ada0p3		cuau1		input		netdump		stderr		ttyv2		ugen0.1
apm		cuau1.init	io		netmap		stdin		ttyv3		ugen1.1
apmctl		cuau1.lock	kbd0		nfslock		stdout		ttyv4		ugen1.2
audit		devctl		kbdmux0		null		sysmouse	ttyv5		uinput
auditpipe	devctl2		klog		pass0		tcp_log		ttyv6		urandom
bpf		devstat		kmem		pci		ttyu0		ttyv7		usb
bpf0		fd		led		pf		ttyu0.init	ttyv8		usbctl
console		fido		mdctl		pts		ttyu0.lock	ttyv9		xpt0
consolectl	full		mem		random		ttyu1		ttyva		zero

My summary of the Intel Ethernet Controller I211 datasheet section 8.2.8 (revision 3.4, January 2021):

LED control (0x0E00, base address register 0x10)

0x00            0x07            0x84            0x06
31              23              15              7
0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0
    LED2 blink ─┘ │     ├─┴─┴─┘ │ │     ├─┴─┴─┘ │ │ │ │ └─┴─┴─┴─ LED0 mode
    LED2 invert ──┘     │       │ │     │       │ │ │ └─ LED_PCI_MODE
    LED2 mode ──────────┘       │ │     │       │ │ └─── Global Blink Mode
                    LED1 blink ─┘ │     │       │ └─ LED0 invert
                    LED1 invert ──┘     │       └─── LED0 blink
                    LED1 mode ──────────┘       

Other bits are reserved, write 0.

Blink:
0b = Do not blink asserted LED output.
1b = Blink asserted LED output.

Global Blink Mode:
0b = Blink at 200 ms on and 200 ms off.
1b = Blink at 83 ms on and 83 ms off.

Invert:
0b = Do not invert LED source (LED active low).
1b = Invert LED source (LED active high).
In mode 0100b (link/activity) this field must be 0 (LED active low).

LED_PCI_MODE:
0b = Use LEDs as defined in the other fields of this register.
1b = Use LEDs to indicate PCI3 lanes idle status in SDP mode (only when the led mode is set to 1000b - SDP mode)
LED0 indicates electrical idle status.

Mode Selected Mode Source Indication
0000b LINK_10/1000 Asserted when either 10 or 1000 Mb/s link is established and maintained.
0001b LINK_100/1000 Asserted when either 100 or 1000 Mb/s link is established and maintained.
0010b LINK_UP Asserted when any speed link is established and maintained.
0011b FILTER_ACTIVITY Asserted when link is established and packets are being transmitted or received that passed MAC filtering.
0100b LINK/ACTIVITY Asserted when link is established and when there is no transmit or receive activity. When BLINK, indicates LINK and activity (either receive or transmit)
0101b LINK_10 Asserted when a 10 Mb/s link is established and maintained.
0110b LINK_100 Asserted when a 100 Mb/s link is established and maintained.
0111b LINK_1000 Asserted when a 1000 Mb/s link is established and maintained.
1000b SDP_MODE LED activation is a reflection of the SDP signal. SDP0, SDP1, SDP2 are reflected to LED0, LED1, LED2 respectively.
1001b FULL_DUPLEX Asserted when the link is configured for full duplex operation (de-asserted in half-duplex).
1010b COLLISION Asserted when a collision is observed.
1011b ACTIVITY Asserted when link is established and packets are being transmitted or received.
1100b LINK_10/100 Asserted when either 10 or 100 Mb/s link is established and maintained.
1101b PAUSED Asserted when the I211’s transmitter is flow controlled.
1110b LED_ON Always high (asserted)
1111b LED_OFF Always low (de-asserted)

If the above is interesting for other users, I could create a pull request for the APU documentation.

MarcFontaine commented

It looks like the 'Generic prototyping and diagnostics driver'
https://www.freebsd.org/cgi/man.cgi?query=proto&apropos=0&sektion=4&manpath=FreeBSD+11-current&format=html
could give access to those settings.
But, as far as I know, the igb driver will block the proto driver.
On would have to do the LED settings first then load the igb driver and hope that that driver does not reset the LEDs back to default.
All of that is just wild speculation, I've not tried it.