The dependency on libusbsio vendors prebuilt shared libraries
Closed this issue · 14 comments
Hi! When trying to package this project for Arch Linux I evaluated the list of dependencies.
The dependency on libusbsio is very problematic for downstream distributions in the context of supply chain security and transparency, as it provides prebuilt shared libraries. It is not clear how these shared libraries have been built and they also can not be adapted to the best practices of a given downstream distribution in regards to build flags etc.
If these shared libraries can not be built from source and then be used in the context of the libusbsio Python package, I will remove the dependency on libusbsio from the sources of this project when packaging it, which seems rather excessive given that the library is apparently BSD-3-clause licensed and sources could be made available.
It seems that the NXP website provides source code downloads of the libusbsio library in use by the Python package libusbsio only to customers. Source code downloads (including prebuilt shared libraries) of the Python package libusbsio are only made availabe via sdist tarballs on pypi.org.
Given that this makes packaging defacto impossible, is it possible to provide the sources of the libusbsio shared library to users without them having to share their private details with NXP (as that is what creating an account entails)?
The changes required for the removal of libusbsio are unfortunately quite invasive:
diff -ruN a/spsdk/mboot/interfaces/__init__.py b/spsdk/mboot/interfaces/__init__.py
--- a/spsdk/mboot/interfaces/__init__.py 2022-02-04 14:27:29.000000000 +0100
+++ b/spsdk/mboot/interfaces/__init__.py 2022-02-14 00:15:14.754121893 +0100
@@ -10,4 +10,3 @@
from .base import Interface
from .uart import Uart, scan_uart
from .usb import RawHid, scan_usb
-from .usbsio import UsbSioI2C, UsbSioSPI, scan_usbsio
diff -ruN a/spsdk/mboot/interfaces/usb.py b/spsdk/mboot/interfaces/usb.py
--- a/spsdk/mboot/interfaces/usb.py 2022-02-04 14:27:29.000000000 +0100
+++ b/spsdk/mboot/interfaces/usb.py 2022-02-14 00:12:32.227394246 +0100
@@ -11,8 +11,6 @@
from struct import pack, unpack_from
from typing import Sequence, Union
-import libusbsio
-
from spsdk.utils.usbfilter import NXPUSBDeviceFilter, USBDeviceFilter
from ..commands import CmdPacket, CmdResponse, parse_cmd_response
@@ -249,23 +247,4 @@
:param usb_device_filter: USBDeviceFilter object
:return: List of interfaces found
"""
- devices = []
-
- # use HID_API of LIBUSBSIO library to enumerate all USB HID devices
- sio = libusbsio.usbsio()
- all_hid_devices = sio.HIDAPI_Enumerate()
-
- # iterate on all devices found
- for dev in all_hid_devices:
- if usb_device_filter.compare(dev) is True:
- new_device = RawHid()
- new_device.device = sio.HIDAPI_DeviceCreate()
- new_device.vid = dev["vendor_id"]
- new_device.pid = dev["product_id"]
- new_device.vendor_name = dev["manufacturer_string"]
- new_device.product_name = dev["product_string"]
- new_device.interface_number = dev["interface_number"]
- new_device.path = dev["path"]
- devices.append(new_device)
-
- return devices
+ return []
diff -ruN a/spsdk/sdp/interfaces/usb.py b/spsdk/sdp/interfaces/usb.py
--- a/spsdk/sdp/interfaces/usb.py 2022-02-04 14:27:29.000000000 +0100
+++ b/spsdk/sdp/interfaces/usb.py 2022-02-14 00:16:40.930819269 +0100
@@ -8,11 +8,8 @@
"""Module for USB communication with a terget using SDP protocol."""
import logging
-import platform
from typing import Sequence, Tuple, Union
-import libusbsio
-
from spsdk import SPSDKError
from spsdk.utils.usbfilter import NXPUSBDeviceFilter, USBDeviceFilter
@@ -240,21 +237,4 @@
:param usb_device_filter: USBDeviceFilter object
:return: List of interfaces found
"""
- devices = []
- sio = libusbsio.usbsio()
- all_hid_devices = sio.HIDAPI_Enumerate()
-
- # iterate on all devices found
- for dev in all_hid_devices:
- if usb_device_filter.compare(dev) is True:
- new_device = RawHid()
- new_device.device = sio.HIDAPI_DeviceCreate()
- new_device.vid = dev["vendor_id"]
- new_device.pid = dev["product_id"]
- new_device.vendor_name = dev["manufacturer_string"]
- new_device.product_name = dev["product_string"]
- new_device.interface_number = dev["interface_number"]
- new_device.path = dev["path"]
- devices.append(new_device)
-
- return devices
+ return []
diff -ruN a/spsdk/utils/nxpdevscan.py b/spsdk/utils/nxpdevscan.py
--- a/spsdk/utils/nxpdevscan.py 2022-02-04 14:27:29.000000000 +0100
+++ b/spsdk/utils/nxpdevscan.py 2022-02-14 00:18:41.700860824 +0100
@@ -11,7 +11,6 @@
import logging
from typing import Sequence
-import libusbsio
from serial.tools.list_ports import comports
from spsdk.mboot.interfaces.uart import scan_uart as mb_scan_uart
@@ -41,33 +40,7 @@
:extend_vid_list: list of VIDs, to extend the default NXP VID list (int)
:return: list of dicts corresponding to NXP devices
"""
- sio = libusbsio.usbsio()
- all_usb_devices = sio.HIDAPI_Enumerate()
- nxp_usb_devices = []
-
- search_vids = NXP_USB_DEVICE_VIDS
-
- if extend_vid_list is not None:
- search_vids = list(set(search_vids) | set(extend_vid_list))
-
- for usb_device in all_usb_devices:
- for nxp_vid in search_vids:
- if nxp_vid == usb_device["vendor_id"]:
- # We found our device, let's create container for it
- vid = usb_device["vendor_id"]
- pid = usb_device["product_id"]
- path = convert_usb_path(usb_device["path"])
- product_string = usb_device["product_string"]
- manufacturer_string = usb_device["manufacturer_string"]
- name = ", ".join(get_usb_device_name(vid, pid, None))
- usb_dev = USBDeviceDescription(
- vid, pid, path, product_string, manufacturer_string, name
- )
-
- nxp_usb_devices.append(usb_dev)
- break
-
- return nxp_usb_devices
+ return []
def search_nxp_uart_devices() -> Sequence[DeviceDescription]:
I would be happy if instead of using the closed source libusbsio this project could rely on an open library (e.g. libusb and pyusb) for the detection and enumeration of USB devices.
Unfortunately, we can't use available open USB libraries. In the past, we used them but we had to steer away. Mainly because some of our devices were not working correctly with them.
On top of that, the LIBUSBSIO is an integral part of the functionality we're working on right now (and will be presented in version 1.7) Open-source libraries don't provide the functionality we need.
So for now, LIBUSBSIO is a mandatory dependency of SPSDK
@mstarecek thanks for the reply!
Unfortunately, we can't use available open USB libraries. In the past, we used them but we had to steer away. Mainly because some of our devices were not working correctly with them.
Have you tried upstreaming fixes for your devices into the respective free USB libraries? Open source is not a one-way street :)
On top of that, the LIBUSBSIO is an integral part of the functionality we're working on right now (and will be presented in version 1.7) Open-source libraries don't provide the functionality we need. So for now, LIBUSBSIO is a mandatory dependency of SPSDK
Unfortunately that prevents downstreams to adopt your software, as they can not build an integral part of it from source and renders spsdk either broken (if libusbsio is removed) or non open-source as well (if libusbsio is used).
I have had a chance to look at the sources for the shared library and I find it highly concerning that the libusbsio package on pypi.org bundles the shared library, which falls under the terms of the LA_OPT_NXP_Software_License (a full-on proprietary license, which is so loosely phrased that it could be used to sue me just for mentioning its name) while the package claims to be licensed under the terms of the BSD-3-clause license. As a byproduct users of that Python package may unknowingly render themselves liable for prosecution under the terms of the former license (so does pypi.org currently and so would I have if I had just distributed it for Arch Linux).
As it stands this renders spsdk proprietary software as well and I will abstain from further packaging it under these circumstances as I am not keen to be sued over the use of libusbsio. :-/
Additionally I hereby advise anyone looking into redistributing the libusbsio Python package to not do that (unless complying with the LA_OPT_NXP_Software_License which can be obtained by creating an account with https://www.nxp.com and downloading the sources of libusbsio).
Open source is not a one-way street :) I totally agree!
I was told that the patch we needed was rejected. Unfortunately, this happed way before I joined the team, thus I don't have much info on why it was rejected, or even who was the original submitter :(
I'll ask my colleague, who understands licensing way better than I do, to answer the LA_OPT stuff.
@dvzrv
The libusbsio REDISTRIBUTABLE as is on pypi.org is licensed under BSD license. The binaries are build from sources, which are, however, licensed under LA_OPT_NXP_Software_License at the moment. From this perspective we are talking about dual-licensing model. You are free to use the pypi.org bundle, observing only demands of BSD-3-Clause license. If you decide to use sources, you are obliged to follow the LA_OPT_NXP_Software_License licensing terms.
Both the sources as well as the libusbio REDISTRIBUTABLE on pypi.org are owned by NXP.
I was told that the patch we needed was rejected. Unfortunately, this happed way before I joined the team, thus I don't have much info on why it was rejected, or even who was the original submitter :(
@mstarecek It would be very awesome to get that sorted out with the libusb upstream. Looking at their merge requests I can at least not find any mentioning "nxp". Maybe you can find more information on this. That and a successive move towards using libusb would be very much appreciated by anyone taking supply chain security seriously! <3
The libusbsio REDISTRIBUTABLE as is on pypi.org is licensed under BSD license. The binaries are build from sources, which are, however, licensed under LA_OPT_NXP_Software_License at the moment. From this perspective we are talking about dual-licensing model. You are free to use the pypi.org bundle, observing only demands of BSD-3-Clause license. If you decide to use sources, you are obliged to follow the LA_OPT_NXP_Software_License licensing terms.
@Renegade85 If by "dual-licensing" you mean "relicensing", then yes, the sources are relicensed from LA_OPT_NXP_Software_License (source) to BSD-3-clause (binary). Dual-licensing in this context would mean to apply two separate licenses to the same thing (e.g. the source code), which is not the case here.
Both the sources as well as the libusbio REDISTRIBUTABLE on pypi.org are owned by NXP.
If by "owned by" you mean "distributed by" and "copyright held by", then yes that is the case and this would indeed allow downstreams to re-distribute these binaries under the terms of the BSD-3-clause. Previously I was not really sure whether that is what NXP as a company envisioned doing with this and even in this ticket it only becomes clear vaguely as its participants are not clearly marked and identified as NXP employees that are allowed to make a decision on the licensing and relicensing of its software. A clarification on this topic would be much appreciated!
However, on the topic of supply chain security and access to source code, I would like to point out the following:
- not freely providing the sources for an integral part of the communications layer of spsdk diminishes the trust in this tooling, as it allows for supply chain attacks to go completely unnoticed. This is a security concern for all downstreams making use of this software, as the weak link in this scenario is the security setup of a single person taking care of binary releases at NXP (presumably Michal Hanak). There is no cryptographic signature for any release and all anyone has is the hope that the libraries they have just installed actually originated from the person who claimed to have uploaded them and were not created by a malicious thirdparty.
edit: Case in point: The sources for the latest release (2.1.10 at the time of writing) are not available as download from the NXP website (2.1.8 at the time of writing), which may mean a supply chain attack in motion, but no user is able to verify this and would blindly install a malicious shared library that may compromise their system or attached device.
- not freely providing the sources is a problem for anyone trying to audit the code before using the binaries that were supposedly build from them. This is particularly problematic for security related hardware such as the nitrokey, which currently relies on libusbsio via spsdk for provisioning firmware updates to its newest device. Given that these devices are supposed to store the private key material of thousands of users, which are themselves in key positions of critical infrastructure all over the world, you may understand how little anyone one of them would like to leave to chance when it comes to security.
- not freely providing the sources is a problem for all downstreams with specific build requirements (e.g. CFLAGS, LDFLAGS, general hardening), which can not be applied, as they would have to rely on prebuilt binaries built with unknown build flags.
- not freely providing the sources does not allow other operating systems but the ones you explicitly provide binaries for to even use this tooling (e.g. FreeBSD)
- linking against specific versions of shared libraries that were present during build time will lead to problems for everyone using these libraries as soon as their respective system library introduces a soname change or if the required shared libraries are not available. For
libusbsio.soon Linux this is currentlyldd bin/linux_x86_64/libusbsio.so linux-vdso.so.1 (0x00006a9e379be000) libudev.so.1 => /usr/lib/libudev.so.1 (0x00006a9e37941000) libc.so.6 => /usr/lib/libc.so.6 (0x00006a9e37737000) librt.so.1 => /usr/lib/librt.so.1 (0x00006a9e37732000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00006a9e37717000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00006a9e37712000) /usr/lib64/ld-linux-x86-64.so.2 (0x00006a9e379c0000)
Given the above major problems I hereby urge you to reevaluate the use of proprietary code in the context of spsdk and to move towards using libusb directly and/or providing the libusbsio sources under the terms of an open-source or free software license.
Many direct and indirect users will appreciate this very much and it will additionally help identify problems with the library more easily, while establishing trust in using spsdk.
@dvzrv
So we had a discussion internally in the team and the sources will be released under BSD license. These will be part of the libusbsio zip file on our nxp.com web pages.
So we had a discussion internally in the team and the sources will be released under BSD license. These will be part of the libusbsio zip file on our nxp.com web pages.
That is some awesome news and is very much appreciated! Would the .zip files then be free to download (as in without a user account)? This would be necessary to be able to use them in a packaging context, as otherwise downstreams need to keep a copy of those sources somewhere else.
Would the .zip files then be free to download (as in without a user account)?
Currently, you still need to have an account on nxp.com to be able to download the zip files. Not sure, whether this is going to change in the near future, but that's the current state.
Currently, you still need to have an account on nxp.com to be able to download the zip files. Not sure, whether this is going to change in the near future, but that's the current state.
Okay. Let me know once you do have the BSD licensed sources available. I can look into hosting them on our distribution's gitlab instance for packaging purposes in the meantime then.
Slowly but surely things are progressing :) There's a plan to add the libusbsio source code into the PyPI package. The C code will be compiled as an extension in the target system.
Whether there will be a dedicated GitHub repo is still an open question.
@dvzrv Hey there, as promised, new release has been put on our web pages with BSD license:
https://www.nxp.com/downloads/en/libraries/libusbsio-2.1.11-src.zip
Have a nice day
@mstarecek thank you! A dedicated github repo could be good for further questions and improvements (as such a discussion would not fit to this issue tracker).
@Renegade85 thank you! I have just created a package: https://github.com/archlinux/svntogit-community/blob/packages/libusbsio/trunk/PKGBUILD
Generally it would be great to only have the sources in the download and to have the python build files (e.g. pyproject.toml and setup.{py,cfg}) in the sources and not in the sdist tarball if possible.