No results when scanning on MacOS 12
peterloron opened this issue · 55 comments
- bleak version: 0.12.1
- Python version: 3.9.6
- Operating System: MacOS 12 Beta (Monterey)
Description
I installed bleak via pip, and used the scanning example from the docs. When the code is run, no devices are found. I can run a different BLE scanner and see many devices. Running as root doesn't change anything.
Happy to help investigate, but I'm not sure where to go from here.
Happy to help investigate, but I'm not sure where to go from here.
Hello @dlech. Apologies, I should have mentioned I did read the troubleshooting section, open issues, etc. I enabled the debug env var, but it did not show anything interesting:
2021-09-08 16:00:37,577 bleak.backends.corebluetooth.CentralManagerDelegate DEBUG: centralManagerDidUpdateState_
DEBUG:bleak.backends.corebluetooth.CentralManagerDelegate:centralManagerDidUpdateState_
2021-09-08 16:00:37,577 bleak.backends.corebluetooth.CentralManagerDelegate DEBUG: Bluetooth powered on
DEBUG:bleak.backends.corebluetooth.CentralManagerDelegate:Bluetooth powered on
2021-09-08 16:00:37,578 bleak.backends.corebluetooth.CentralManagerDelegate DEBUG: 'isScanning' changed
DEBUG:bleak.backends.corebluetooth.CentralManagerDelegate:'isScanning' changed
2021-09-08 16:00:42,581 bleak.backends.corebluetooth.CentralManagerDelegate DEBUG: 'isScanning' changed
DEBUG:bleak.backends.corebluetooth.CentralManagerDelegate:'isScanning' changed
2021-09-08 16:00:42,581 bleak.backends.corebluetooth.CentralManagerDelegate DEBUG: 'isScanning' changed
DEBUG:bleak.backends.corebluetooth.CentralManagerDelegate:'isScanning' changed
...
The code I am running is exactly the detection_callback.py code you pointed to.
I did a packet capture, but I'm unable to determine if there is any useful information in there. I can post it somewhere if it would be useful.
I can look at the packet capture if you attach it to the issue (just put it in a .zip file).
bleakMacOS.pklg.zip
Running into the same issue attached packet logger output if it helps. When enabling debug logging it's the same results reported above.
Bleak 0.13
Python 3.10
MacOS 12
Thanks for the info. I'm able to reproduce the issue on my Mac as well now. I also found https://stackoverflow.com/q/69661809/1976323, so I don't think it is just Bleak. I would suggest filing a bug with Apple.
This thread hints that there is a bug in macOS and they were able to work around it, but I don't think it is open source, so I don't know what the fix is. https://forums.zwift.com/t/macos-monterey-cannot-establish-bluetooth-connection/554193/27
I tried this workaround but it doesn't seem to make a difference.
Confirming this is still an issue in macOS 13
Just want to echo that I'm encountering the same issue in monterey.
I see there are changes on https://github.com/ronaldoussoren/pyobjc, but not a new version since Jun 7, 2021. Probably this could help to solve the issue.
What specific changes might be related to Bluetooth? I have already tried https://pypi.org/project/pyobjc/8.0b1/.
PyObjC v8.0 is out now and it doesn't fix the problem
PyObjC v8.0 is out now and it doesn't fix the problem
I can conform that.
This really needs to be fixed! It's a showstopper for me
Please report the issue to Apple using the Feedback Assistant app. I'm not sure what else we can do since the problem seems to come from an undocumented change in the OS. If enough people report the issue, maybe they will prioritize it.
Can someone who understands this help me wrap my head around why I can still scan for devices using Chrome's Web Bluetooth, but can't with bleak? Is there somehow a different set of library functions exposed in c++? Is this a bug at the pyobjc level? I've been trying to read the chromium source code but must admit I'm struggling.
Can someone who understands this help me wrap my head around why I can still scan for devices using Chrome's Web Bluetooth, but can't with bleak? Is there somehow a different set of library functions exposed in c++? Is this a bug at the pyobjc level? I've been trying to read the chromium source code but must admit I'm struggling.
I am unfamiliar with the framework that Web Bluetooth uses but bleak is built on top of Core Bluetooth which is currently broken in macOS Monterey...
Since tools such as "LightBlue", which I like to use to develop in the BLE environment, and others work perfectly, the fault - in my humble opinion - is more likely to be with the PyObjc Framework CoreBlueooth or even within Bleak.
Stupid question: I digged into the code "behind" BleakScanner.discover().
I see:
@classmethod
async def discover(cls, timeout=5.0, **kwargs) -> List[BLEDevice]:
async with cls(**kwargs) as scanner:
await asyncio.sleep(timeout)
devices = scanner.discovered_devices
return devices
and scanner.discovered_devicesis:
@property
@abc.abstractmethod
def discovered_devices(self) -> List[BLEDevice]:
raise NotImplementedError()
See
bleak/bleak/backends/scanner.py
Line 166 in fc07589
How can this method ever return a list of devices? It has no return. However, no exception is raised for me either.
Ah, sorry. Stupid me. It's abstract. I need to look in the respective implementation which is here:
bleak/bleak/backends/corebluetooth/scanner.py
Line 115 in e492699
I have been digging through the code on two Mac systems.
One system is an Intel Mac running Mac OS 11.6 and the other - where it doesn't work - is an M1 Mac running Mac OS 12.0.1.
In the end, it's all about the here documented function retrievePeripheralsWithIdentifiers from Apple's coreBluetooth package.
On Mac OS 11.6 this returns a list of devices and on 12.0.1 nothing. retrieveConnectedPeripheralsWithServicesdoesn't work either on 12.0.1.
What I noticed is that when I called the sample code with a BleakScanner.discover() on Mac OS 11.6, I was asked via a dialogue box if I wanted to authorise the application to use Bluetooth.
On Mac OS 12.0.1 nothing of the sort came up - maybe we have just a permission problem here.......
I submitted feedback to Apple
Hello, just want to share some findings because I encountered a similar problem on macOS 12 for a proprietary software I'm developing involving a C++ executable using Apple's CoreBluetooth
My executable is simply scanning Bluetooth devices and it asks the Bluetooth permissions via a dialog box (just by linking to CoreBluetooth - no code involved) on macOS 11/12.
Permissions if granted are correctly reflected in the Bluetooth manager authorization status
My problem is that on macOS 12, didDiscoverPeripheral is never called which seems similar to this issue
But if I create the executable as a bundle that contains in the Info.plist manifest file the NSBluetoothAlwaysUsageDescription key, scanning does work.
(If I embed the manifest in the binary file bluetooth scanning does not work which seems a regression from Apple's side)
So if my findings are applicable to your project maybe it could work by adding this key in the Info.plist embedded in a python installation that comes as a bundle.
Creating a bundle executable using for instance py2app (not convenient of course) could also work but maybe the correct solution would be to operate directly on the pyobjc library and creating a bundle library containing a manifest instead of .so file if Apple does not reply
Anyway this is a bug from Apple imho because even if the permission system hardens, the error should be clear (a crash for instance for Privacy violation like usual) but not silently fail
Potential work-around: sonsongithub/SBMeter@aabf32c#diff-8e1d3fc4aff84e0803e9a73332cdf232fed0292d4ef493c00ad3a482d7b4baf9R45-R47
It appears that maybe filtering by specific services is another work-around for some use-cases (issue was referenced above but making it clear that is a work-around might help some folks)
Also, fwiw, the app I'm trying to get working (which uses a different btle library -- it uses deviceplug/btleplug#224 which is already referenced in this thread -- but I'm tracking this issue to see about potential fixes) has that NSBluetoothAlwaysUsageDescription field in the Info.plist and it doesn't work still on Monterey so there might be multiple bugs here.
EDIT: I have actually been able to get it to work setting service_uuids. I haven't been able to get the workaround to work for me, but like I mentioned before
Also, I can connect to my same device via Web Bluetooth [edit: and scan for devices] (which is also built on top of Core Bluetooth because as far as I understand everything on mac has to be). If anyone else also wants to look at the source of that, the Chromium implementation is here https://chromium.googlesource.com/chromium/src/+/refs/heads/main/device/bluetooth
I can also confirm that bleak scanning does not work on Mac OS 12.0.1. The Web Bluetooth still works.
EDIT: I have actually been able to get it to work setting
service_uuids.
What does that mean? Can you please explain this to me in more detail!
This is indeed problematic but, as has been stated above, this is most proabably something Bleak cannot address without Apple fixing this and pushing an update for macOS 12. If there is a problem in macOS, I do not believe that there are any reliable workarounds, given that it seems like the detection events that Bleak are using are not sent by Core Bluetooth.
I do not have any access to any macOS 12 device, so I will not be able to do any troubleshooting here. But please, keep escalating the findings you make to Apple. They will have little effect in this issue I am afraid.
I would re-iterate the point made above that Chrome still works fine for discovering devices via the Web Bluetooth API without a service filter. Clearly there is some path that works just fine on MacOS 12 even if the current APIs used by bleak are not functioning in the same way.
I still believe, that this is a permission problem combined with the bug, that the Apple API doesn't give an error message if the caller does not have permission.
Is still do not get what @challiwill meant with his comment #635 (comment). Does some one know?
Is still do not get what @challiwill meant with his comment #635 (comment). Does some one know?
While I was researching this issue, I came across for forum post that suggested this as well. The suggestion is to provide a non-empty list of service uuids to scanForPeripheralsWithServices:options:. Bleak currently passes None to that method. However, I'm pretty sure I tried providing a list that matched my device without results.
Chromium also passes a non-emtpy list here, so that is, in fact, one of the differences between Chromium and Bleak. The other difference is that Chromium is an app and therefor could have the suggested permissions in it's pinfo.list.
When I digged into the bleak code, I think I found that bleak uses retrievePeripheralsWithIdentifiersbut not scanForPeripheralsWithServices
See here:
bleak/bleak/backends/corebluetooth/scanner.py
Line 117 in e492699
Maybe it should use scanForPeripheralsWithServices instead? (Besides that is actually something different....)
It is here:
bleak/bleak/backends/corebluetooth/CentralManagerDelegate.py
Lines 117 to 119 in fc07589
@dlech Are you sure, that this is the method that is called, when doing BleakScanner.discover()?
My backtrace leads to
bleak/bleak/backends/corebluetooth/scanner.py
Line 117 in e492699
because this class implements the abstract class
given in class
BaseBleakScanner:
as
@property
@abc.abstractmethod
def discovered_devices(self) -> List[BLEDevice]:
raise NotImplementedError()
I don't know what
is good for, but I don't find it in the path if I simply trace backBleakScanner.discover() - which is the method that is not using in my caseAre you sure, that this is the method that is called, when doing
BleakScanner.discover()?
100% sure. The aysnc with statement indiscover() implicitly calls __aenter__() which calls start() which calls start_scan().
While I was researching this issue, I came across for forum post that suggested this as well. The suggestion is to provide a non-empty list of service uuids to scanForPeripheralsWithServices:options:. Bleak currently passes
Noneto that method. However, I'm pretty sure I tried providing a list that matched my device without results.
@dlech I only monkey patched this into bleak at this line, like this:
scan_options["service_uuids"] = ["6E400001-B5A3-F393-E0A9-E50E24DCCA9E".lower()]
Awesome, I must have done something wrong when I tried it. I've made a PR to make it easier to pass a list of service uuids as a service_uuids kwarg, e.g. BleakScanner(service_uuids=[...]) or BleakScanner.discover(service_uuids=[...]).
Try it with:
pip install https://github.com/hbldh/bleak/archive/refs/heads/macos12-scanning.zip
You can also pass UUIDs to the detection callback example:
python examples/detection_callback.py 6E400001-B5A3-F393-E0A9-E50E24DCCA9E
The other difference is that Chromium is an app and therefor could have the suggested permissions in it's pinfo.list.
If I'm reading the docs correctly, the NSBluetoothAlwaysUsageDescription plist key is not available on macOS, so unless that changed but wasn't updated in the docs it doesn't seem likely to me to be the culprit.
I'm wondering about this last paragraph here because it matches the behavior we're seeing, but there's nowhere that bleak sets bluetooth scanning to background mode as far as I can tell, unless somehow it happens implicitly:
Your app can scan for Bluetooth devices in the background by specifying the bluetooth-central
background mode. To do this, your app must explicitly scan for one or more services by specifying
them in the serviceUUIDs parameter. The CBCentralManager scan option has no effect while
scanning in the background.
@dlech Then again, the NSBluetoothAlwaysUsageDescription is indeed present in a chromium/content/shell/app/app-Info.plist.
I've now tried building my app using py2app and setting NSBluetoothAlwaysUsageDescription in the plist and I cannot get it to work still.
I have an Info.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDisplayName</key>
<string>My App</string>
<key>CFBundleIdentifier</key>
<string>app</string>
<key>CFBundleVersion</key>
<string>1.0</string>
<key>NSBluetoothAlwaysUsageDescription</key>
<string>reasons</string>
</dict>
</plist>
and my setup.py is:
from setuptools import setup
setup(
app=["app.py"],
data_files=[],
options=dict(py2app=dict(
plist="Info.plist",
)),
setup_requires=["py2app"],
)
@arthurbiancarelli If you're able to provide any more information about how you got this to work that would be helpful.
I have created a minimal test application using pure Objective-C that scans Bluetooth devices. This application only works when compiling in the Apple Bundle format and with NSBluetoothAlwaysUsageDescription in the manifest (and also for Intel architecture, see below) Since the standard output is not simply accessible when executing a bundle application, I've made a simple GUI that displays the scanning results.
You can check out the code here
I then created a minimal GUI application using bleak, tkinter and py2app -> https://github.com/arthurbiancarelli/bleak/commits/ab/macOs12/1
Same results. The application works when built as a bundle for x86 architecture. Does not work when launched from the Python interpreter
I tried using a custom recompiled terminal (iTerm2) / Python Interpreter that has the amended manifest but no luck so far.
I also tried the service filter trick that some reported but it does not work on my system. However it seems related to the iOS restrictions on background scanning [edit] as you already stated @challiwill
Maybe it does work with BLE devices that advertise their services without the need for an active scan as explained in this article (https://medium.com/@cbartel/ios-scan-and-connect-to-a-ble-peripheral-in-the-background-731f960d520d) ?
Note for Apple M1 the application must be signed with codesign if built natively (for arm architecture). Compiling for x86_64 and executing within Rosetta emulator does not require this signing stage
From the release notes
New in macOS 11 on Macs with Apple silicon, and starting in macOS Big Sur 11 beta 6, the operating system enforces that any executable must be signed before it’s allowed to run.
I also tried the service filter trick that some reported but it does not work on my system
Can you share your raw advertising data (you can get this, for example, with the nRF Connect app for iOS or Android, or by logging Bluetooth packets on macOS). So far I've only tested with 128-bit UUIDs, so I wonder if we may need to make a tweak for 16-bit and 32-bit UUIDs.
The device I'm working on has an advertising packet comprised strictly of the advertisement type, the manufacturer name and the shortened device name.
I have performed some more tests and if I add a service UUID into the advertisement packet I can scan the device both with bleak #692 and my executable (without the bundle + manifest and by manually specifying the expected Bluetooth service).
So as I understand it would seem that macOS Monterey only performs a passive Bluetooth scan if the application is not executing in the "Apple context" (meaning a GUI compiled as a Bundle) similarly to the iOS restrictions previously discussed
This is kind of a good news but not completely satisfactory because it requires some firmware modifications for Bluetooth devices that do not advertise their services if one wants to scan it via a CLI
Can you please start a new issue for the connection problem?
I tried https://github.com/hbldh/bleak/tree/macos12-scanning but I'm not sure where to get the correct service UUIDs.
I'm very inexperienced with this kind of bluetooth things, but I couldn't find any documentation on how to find service UUIDS. If I understood correctly you need to have already established a connection to that device to have a service UUID?
Thanks!
I tried https://github.com/hbldh/bleak/tree/macos12-scanning but I'm not sure where to get the correct service UUIDs.
I'm very inexperienced with this kind of bluetooth things, but I couldn't find any documentation on how to find service UUIDS. If I understood correctly you need to have already established a connection to that device to have a service UUID?
Thanks!
You can use the nRF Connect mobile app fo find UUIDs around you. Start scanning and make the BLE device advertise itself.
Either I don't understand something, or this is a chicken-and-egg problem:
I want to use discover() to display the devices in the vicinity to see what services they offer.... If I have to use nRF Connect for this, I don't need discover().... in bleak.
Can someone clarify that!?
As of right now, it is no longer possible to discover arbitrary devices using Bleak on macOS 12, so you will have to go with an alternate solution.
Okay, thx!
I also tried the service filter trick that some reported but it does not work on my system
Can you share your raw advertising data (you can get this, for example, with the nRF Connect app for iOS or Android, or by logging Bluetooth packets on macOS). So far I've only tested with 128-bit UUIDs, so I wonder if we may need to make a tweak for 16-bit and 32-bit UUIDs.
The current implementation works also with 16-bit. For 16-bit the mac implementation will automatically extend it by the default 128-bit mask:
> python examples/detection_callback.py 0xFE07
2021-12-30 08:14:49,474 __main__ INFO: ... service_uuids=['0000fe07-0000-1000-8000-00805f9b34fb
Thanks for sharing. I'm surprised it works even works from the command line. 👍
Not sure if this is already known here but Bluetility UI works on my mac without any specific service uuids.
I have no idea what's actually the difference since the code looks quite similar:
I think the difference is that it is an app (and possibly in addition to that it requests permission) and is not a command line tool like Python. There are several other comments in this issue that also noticed this that may be of interest.
FYI, I've added a summary of the findings of this issue with additional insights at #720 (comment).
Any updates on this issue? I'm currently experiencing problems trying to connect to a BLE device using a script with Bleak in Python.
Apple fixed the bug in macOS 12.2. If you still have problems the best way to get help is to start a new discussion and include you code, error messages and logs.