RPi3b+ 5.4.51 - Cannot set Ethernet interface link to up state after it has been brought down
ThomasZeman opened this issue · 6 comments
When I boot Raspberry OS (20th of August 2020) on my Raspberry PI 3B+ the Ethernet link is working fine. However, once the link has been brought "down" it is not possible to bring it up again. This is also reproducible with Alpine 3.12 32bit as well as 64bit. This was working before with the 4.x.x kernels on the same device.
The following list of commands shows the problem:
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.51-v7+ #1333 SMP Mon Aug 10 16:45:19 BST 2020 armv7l GNU/Linux
pi@raspberrypi:~ $ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.183/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
valid_lft 472sec preferred_lft 397sec
inet6 fe80::54fc:8f3:8fb7:312b/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
pi@raspberrypi:~ $ sudo ip link set eth0 down
pi@raspberrypi:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
pi@raspberrypi:~ $ sudo ip link set eth0 up
pi@raspberrypi:~ $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
pi@raspberrypi:~ $
Further information:
cat /etc/rpi-issue
Raspberry Pi reference 2020-08-20
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 9a3a10bf1019ebb2d59053564dc6b90068bad27d, stage4
vcgencmd version
Aug 19 2020 17:40:15
Copyright (c) 2012 Broadcom
version e90cba19a98a0d1f2ef086b9cafcbca00778f094 (clean) (release) (start)
Maybe related: raspberrypi/firmware#1100
and: https://www.spinics.net/lists/netdev/msg693693.html
Try 'mii-tool -vv eth0' before bringing up the interface.
I just tried again to confirm: The problem is not reproducible with the Feburary 2020 Rasbian release and kernel 4.19.97-v7+
hence I don't think the two issues you mentioned are related to this. Calling ip link set eth0 up
brings up the interface immediately on the 4.19.97 kernel.
Meanwhile I made more observations with the 5.4.x kernel:
- The problem does not occur with an USB Ethernet interface (
If 0, Class=Vendor Specific Class, Driver=ax88179_178a, 480M
) neither on the 4.19 nor 5.4.x kernel - Once the lan78xx interface is down it actually can be brought up again at some point when executing down/up calls for a couple of times. This speaks for some sporadic, timer depended bug.
- In the state where
ip link set eth0 up
does not bring the interface up, mii-tool reports that eth0 is fully operationally and a link has been established.
(On a different note: Is there any documentation how this part of the kernel works and how to/where to enable kernel debug logs? I would be interested in fixing these kind of problems but the entry barrier seems quite high.)
Here a (rather long) reproduction of the observations above on 5.4.51-v7+:
root@raspberrypi:/home/pi# ip link set eth0 down
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
SIOCGMIIPHY on 'eth0' failed: Invalid argument
root@raspberrypi:/home/pi# ip link set eth0 up
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
eth0: negotiated 1000baseT-FD flow-control, link ok
registers for MII PHY 1:
1040 79ed 0007 c132 05e1 c5e1 000f 0000
0000 0200 0800 0000 0000 0000 0000 3000
0000 0000 0088 0000 0000 0000 3200 0004
0040 a000 a000 0000 8035 0000 0000 0000
product info: vendor 00:01:f0, model 19 rev 2
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip link set eth0 down
root@raspberrypi:/home/pi# mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
SIOCGMIIPHY on 'eth0' failed: Invalid argument
root@raspberrypi:/home/pi# ip link set eth0 up
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip link set eth0 down
root@raspberrypi:/home/pi# ip link set eth0 up
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip link set eth0 down
root@raspberrypi:/home/pi# ip link set eth0 up
root@raspberrypi:/home/pi# mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
eth0: negotiated 1000baseT-FD flow-control, link ok
registers for MII PHY 1:
1040 79ed 0007 c132 05e1 c5e1 000f 0000
0000 0200 0800 0000 0000 0000 0000 3000
0000 0000 0088 0000 0000 0000 3200 0004
0040 a000 a000 0000 a035 0000 0000 0000
product info: vendor 00:01:f0, model 19 rev 2
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi# ip link set eth0 down
root@raspberrypi:/home/pi# ip link set eth0 up
root@raspberrypi:/home/pi# mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
eth0: no link
registers for MII PHY 1:
1040 79c9 0007 c132 05e1 c5e1 000f 0000
0000 0200 4800 0000 0000 0000 0000 3000
0000 0000 0088 0000 0000 0000 3200 0004
0040 a000 a000 0000 2005 0000 0000 0000
product info: vendor 00:01:f0, model 19 rev 2
basic mode: autonegotiation enabled
basic status: no link
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
root@raspberrypi:/home/pi# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:0c:51:55 brd ff:ff:ff:ff:ff:ff
inet6 fe80::54fc:8f3:8fb7:312b/64 scope link tentative
valid_lft forever preferred_lft forever
4: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:59:04:00 brd ff:ff:ff:ff:ff:ff
root@raspberrypi:/home/pi#
Although 'mii-tool -vv eth0' doesn't 'fix' this, it could still be the same underlying problem as in #1100. Look at the output of mii-tool when the link doesn't come up, i.e.:
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether b8:27:eb:3e:ab:fb brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:6b:fe:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.99.51/24 brd 192.168.99.255 scope global dynamic noprefixroute wlan0
valid_lft 863028sec preferred_lft 755028sec
inet6 fe80::b9c5:38b1:ecb1:6140/64 scope link
valid_lft forever preferred_lft forever
+ sleep 2
+ mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
eth0: negotiated 1000baseT-FD flow-control, link ok
registers for MII PHY 1:
1040 79ed 0007 c132 05e1 cde1 000f 0000
0000 0200 3800 0000 0000 0000 0000 3000
0000 0000 0088 0000 0000 0000 3200 0004
0040 a000 a000 0000 8035 0000 0000 0000
product info: vendor 00:01:f0, model 19 rev 2
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
The 3rd column in the last row of the PHY register dump shows 'a000' which means a link-change interrupt is pending but doesn't get serviced for some reason. Now in the case where the interface gets an IP:
+ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:3e:ab:fb brd ff:ff:ff:ff:ff:ff
inet 192.168.99.89/24 brd 192.168.99.255 scope global dynamic noprefixroute eth0
valid_lft 863999sec preferred_lft 755999sec
inet6 fe80::cb34:40ac:c489:fed4/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:6b:fe:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.99.51/24 brd 192.168.99.255 scope global dynamic noprefixroute wlan0
valid_lft 863011sec preferred_lft 755011sec
inet6 fe80::b9c5:38b1:ecb1:6140/64 scope link
valid_lft forever preferred_lft forever
+ sleep 2
+ mii-tool -vv eth0
Using SIOCGMIIPHY=0x8947
eth0: negotiated 1000baseT-FD flow-control, link ok
registers for MII PHY 1:
1040 79ed 0007 c132 05e1 cde1 000f 0000
0000 0200 3800 0000 0000 0000 0000 3000
0000 0000 0088 0000 0000 0000 3200 0004
0040 a000 0000 0000 a035 0000 0000 0000
product info: vendor 00:01:f0, model 19 rev 2
basic mode: autonegotiation enabled
basic status: autonegotiation complete, link ok
capabilities: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
advertising: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
link partner: 1000baseT-FD 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
That register value is now '0000' (which is the expected value), so no interrupt pending. Not sure if that unserviced interrupt is the cause for the interface not getting an IP or just a side-effect of some other problem but it sure looks fishy.
I just realized that register 28 is different as well. In the 'bad' case its value is '8035' compared to 'a035' in the good case. That differing bit is 'MDI/MDI-X Crossover Indication'. Huh?
A possible fix for this is in the current rpi-update kernel, which has just switched to rpi-5.10.y.
I cannot reproduce this anymore with Linux -- 5.15.4-0-rpi4 #1-Alpine. So, fine for me to close.