Transparently forwarding messages without any filtering
sterago opened this issue · 17 comments
Hi @ruundii and thanks for the awesome project.
I am trying to use this to switch my Apple Magic Keyboard and Apple Magic Mouse 2 between two MacBooks.
I got the keyboard to partially work using this filtering logic:
class HIDMessageFilter:
def filter_message_to_host(self, msg):
if len(msg) == 10:
return b'\xa1' + msg[:-1]
else:
return b'\xa1' + msg
return msg
def filter_message_from_host(self, msg):
return msg[1:]
However, I can't get the mouse to work.
I tried recording HID reports using hid-tools but apparently that's not supported on Macs.
Do you have any recommendation on how to find out what messages are normally sent by this mouse model and how to reproduce them using a filter?
And as a more general question, since these Apple devices are ultimately targeting 2 MacBooks is a filter even needed in the first place? Can bthidhub forward the bluetooth messages it receives from HID devices over to the bluetooth hosts exactly as they are received and would this work?
See #23 for a probable solution to this.
For me, I was able to get my mouse working by basically removing the mouse part in /etc/bluetooth/sdp_record.xml (run hid-recorder on the host to figure out which part that is) and replacing it with the descriptor for the mouse (run hid-recorder on RPi to find that). I just needed to readd 8503 (Report ID (3)) at the same location as the original sdp_record.xml in order for it to work correctly.
You'll also need to remove these lines which filter out the events for some unknown reason:
https://github.com/ruundii/bthidhub/blob/master/mouse_message_filter.py#L10,L11
thanks @Dreamsorcerer I will try that out!
If my vague instructions were not clear enough, post both outputs from hid-recorder and I'll show you what to change.
Let's track this feature in #23. Feel free to come back with questions on getting it working in the meantime though.
If my vague instructions were not clear enough, post both outputs from
hid-recorderand I'll show you what to change.
I'm so close to getting this working, got through #59, then #55, and it's finally up and running.
I can sync devices, but nothing gets through.
Here's the output for my Magic Trackpad when running sudo hid-recorder:
Available devices:
/dev/hidraw0: lucas' Trackpad
Select the device event number [0-0]: 0
# lucas' Trackpad
# 0x05, 0x01, // Usage Page (Generic Desktop) 0
# 0x09, 0x02, // Usage (Mouse) 2
# 0xa1, 0x01, // Collection (Application) 4
# 0x85, 0x02, // Report ID (2) 6
# 0x05, 0x09, // Usage Page (Button) 8
# 0x19, 0x01, // Usage Minimum (1) 10
# 0x29, 0x02, // Usage Maximum (2) 12
# 0x15, 0x00, // Logical Minimum (0) 14
# 0x25, 0x01, // Logical Maximum (1) 16
# 0x95, 0x02, // Report Count (2) 18
# 0x75, 0x01, // Report Size (1) 20
# 0x81, 0x02, // Input (Data,Var,Abs) 22
# 0x95, 0x01, // Report Count (1) 24
# 0x75, 0x06, // Report Size (6) 26
# 0x81, 0x03, // Input (Cnst,Var,Abs) 28
# 0x05, 0x01, // Usage Page (Generic Desktop) 30
# 0x09, 0x01, // Usage (Pointer) 32
# 0xa1, 0x00, // Collection (Physical) 34
# 0x16, 0x81, 0xff, // Logical Minimum (-127) 36
# 0x26, 0x7f, 0x00, // Logical Maximum (127) 39
# 0x36, 0xc3, 0xfe, // Physical Minimum (-317) 42
# 0x46, 0x3d, 0x01, // Physical Maximum (317) 45
# 0x65, 0x13, // Unit (Inch,EngLinear) 48
# 0x55, 0x0d, // Unit Exponent (-3) 50
# 0x09, 0x30, // Usage (X) 52
# 0x09, 0x31, // Usage (Y) 54
# 0x75, 0x08, // Report Size (8) 56
# 0x95, 0x02, // Report Count (2) 58
# 0x81, 0x06, // Input (Data,Var,Rel) 60
# 0x75, 0x08, // Report Size (8) 62
# 0x95, 0x04, // Report Count (4) 64
# 0x81, 0x01, // Input (Cnst,Arr,Abs) 66
# 0xc0, // End Collection 68
# 0x06, 0x02, 0xff, // Usage Page (Vendor Usage Page 0xff02) 69
# 0x09, 0x55, // Usage (Vendor Usage 0x55) 72
# 0x85, 0x55, // Report ID (85) 74
# 0x15, 0x00, // Logical Minimum (0) 76
# 0x26, 0xff, 0x00, // Logical Maximum (255) 78
# 0x75, 0x08, // Report Size (8) 81
# 0x95, 0x40, // Report Count (64) 83
# 0xb1, 0xa2, // Feature (Data,Var,Abs,NoPref,Vol) 85
# 0xc0, // End Collection 87
# 0x06, 0x00, 0xff, // Usage Page (Vendor Defined Page 1) 88
# 0x09, 0x14, // Usage (Vendor Usage 0x14) 91
# 0xa1, 0x01, // Collection (Application) 93
# 0x85, 0x90, // Report ID (144) 95
# 0x05, 0x84, // Usage Page (Power Device) 97
# 0x75, 0x01, // Report Size (1) 99
# 0x95, 0x03, // Report Count (3) 101
# 0x15, 0x00, // Logical Minimum (0) 103
# 0x25, 0x01, // Logical Maximum (1) 105
# 0x09, 0x61, // Usage (Vendor Usage 0x61) 107
# 0x05, 0x85, // Usage Page (Battery System) 109
# 0x09, 0x44, // Usage (Vendor Usage 0x44) 111
# 0x09, 0x46, // Usage (Vendor Usage 0x46) 113
# 0x81, 0x02, // Input (Data,Var,Abs) 115
# 0x95, 0x05, // Report Count (5) 117
# 0x81, 0x01, // Input (Cnst,Arr,Abs) 119
# 0x75, 0x08, // Report Size (8) 121
# 0x95, 0x01, // Report Count (1) 123
# 0x15, 0x00, // Logical Minimum (0) 125
# 0x26, 0xff, 0x00, // Logical Maximum (255) 127
# 0x09, 0x65, // Usage (Vendor Usage 0x65) 130
# 0x81, 0x02, // Input (Data,Var,Abs) 132
# 0xc0, // End Collection 134
#
R: 135 05 01 09 02 a1 01 85 02 05 09 19 01 29 02 15 00 25 01 95 02 75 01 81 02 95 01 75 06 81 03 05 01 09 01 a1 00 16 81 ff 26 7f 00 36 c3 fe 46 3d 01 65 13 55 0d 09 30 09 31 75 08 95 02 81 06 75 08 95 04 81 01 c0 06 02 ff 09 55 85 55 15 00 26 ff 00 75 08 95 40 b1 a2 c0 06 00 ff 09 14 a1 01 85 90 05 84 75 01 95 03 15 00 25 01 09 61 05 85 09 44 09 46 81 02 95 05 81 01 75 08 95 01 15 00 26 ff 00 09 65 81 02 c0
N: lucas' Trackpad
I: 5 004c 0265
E: 000000.000000 13 31 60 54 48 98 80 44 23 3a 4c 11 0b 05
E: 000000.011133 13 31 b8 54 48 fd a0 48 6f 38 4c 11 14 85
E: 000000.022391 13 31 10 55 48 5f c1 4c 8f 3e 4b 11 1d 85
E: 000000.033633 13 31 68 55 48 7a c1 4e 8f 41 4a 10 22 e5
E: 000000.034868 13 31 c8 55 48 8b 61 50 8f 43 47 11 24 e5
Here's the output from my keyboard:
Available devices:
/dev/hidraw1: lucas’s Magic Keyboard
Select the device event number [1-1]: 1
# lucas’s Magic Keyboard
# 0x05, 0x01, // Usage Page (Generic Desktop) 0
# 0x09, 0x06, // Usage (Keyboard) 2
# 0xa1, 0x01, // Collection (Application) 4
# 0x85, 0x01, // Report ID (1) 6
# 0x05, 0x07, // Usage Page (Keyboard) 8
# 0x15, 0x00, // Logical Minimum (0) 10
# 0x25, 0x01, // Logical Maximum (1) 12
# 0x19, 0xe0, // Usage Minimum (224) 14
# 0x29, 0xe7, // Usage Maximum (231) 16
# 0x75, 0x01, // Report Size (1) 18
# 0x95, 0x08, // Report Count (8) 20
# 0x81, 0x02, // Input (Data,Var,Abs) 22
# 0x95, 0x05, // Report Count (5) 24
# 0x75, 0x01, // Report Size (1) 26
# 0x05, 0x08, // Usage Page (LEDs) 28
# 0x19, 0x01, // Usage Minimum (1) 30
# 0x29, 0x05, // Usage Maximum (5) 32
# 0x91, 0x02, // Output (Data,Var,Abs) 34
# 0x95, 0x01, // Report Count (1) 36
# 0x75, 0x03, // Report Size (3) 38
# 0x91, 0x03, // Output (Cnst,Var,Abs) 40
# 0x95, 0x08, // Report Count (8) 42
# 0x75, 0x01, // Report Size (1) 44
# 0x15, 0x00, // Logical Minimum (0) 46
# 0x25, 0x01, // Logical Maximum (1) 48
# 0x06, 0x00, 0xff, // Usage Page (Vendor Defined Page 1) 50
# 0x09, 0x03, // Usage (Vendor Usage 0x03) 53
# 0x81, 0x03, // Input (Cnst,Var,Abs) 55
# 0x95, 0x06, // Report Count (6) 57
# 0x75, 0x08, // Report Size (8) 59
# 0x15, 0x00, // Logical Minimum (0) 61
# 0x25, 0xe7, // Logical Maximum (231) 63
# 0x05, 0x07, // Usage Page (Keyboard) 65
# 0x19, 0x00, // Usage Minimum (0) 67
# 0x29, 0xe7, // Usage Maximum (231) 69
# 0x81, 0x00, // Input (Data,Arr,Abs) 71
# 0x95, 0x01, // Report Count (1) 73
# 0x75, 0x01, // Report Size (1) 75
# 0x15, 0x00, // Logical Minimum (0) 77
# 0x25, 0x01, // Logical Maximum (1) 79
# 0x05, 0x0c, // Usage Page (Consumer Devices) 81
# 0x09, 0x00, // Usage (Vendor Usage 0x00) 83
# 0x81, 0x01, // Input (Cnst,Arr,Abs) 85
# 0x95, 0x01, // Report Count (1) 87
# 0x75, 0x01, // Report Size (1) 89
# 0x06, 0x01, 0xff, // Usage Page (Vendor Usage Page 0xff01) 91
# 0x09, 0x03, // Usage (Vendor Usage 0x03) 94
# 0x81, 0x02, // Input (Data,Var,Abs) 96
# 0x05, 0x0c, // Usage Page (Consumer Devices) 98
# 0x09, 0x40, // Usage (Menu) 100
# 0x95, 0x01, // Report Count (1) 102
# 0x75, 0x01, // Report Size (1) 104
# 0x81, 0x02, // Input (Data,Var,Abs) 106
# 0x95, 0x01, // Report Count (1) 108
# 0x75, 0x05, // Report Size (5) 110
# 0x81, 0x03, // Input (Cnst,Var,Abs) 112
# 0x06, 0x02, 0xff, // Usage Page (Vendor Usage Page 0xff02) 114
# 0x09, 0x55, // Usage (Vendor Usage 0x55) 117
# 0x85, 0x55, // Report ID (85) 119
# 0x15, 0x00, // Logical Minimum (0) 121
# 0x26, 0xff, 0x00, // Logical Maximum (255) 123
# 0x75, 0x08, // Report Size (8) 126
# 0x95, 0x40, // Report Count (64) 128
# 0xb1, 0xa2, // Feature (Data,Var,Abs,NoPref,Vol) 130
# 0xc0, // End Collection 132
# 0x06, 0x00, 0xff, // Usage Page (Vendor Defined Page 1) 133
# 0x09, 0x14, // Usage (Vendor Usage 0x14) 136
# 0xa1, 0x01, // Collection (Application) 138
# 0x85, 0x90, // Report ID (144) 140
# 0x05, 0x84, // Usage Page (Power Device) 142
# 0x75, 0x01, // Report Size (1) 144
# 0x95, 0x03, // Report Count (3) 146
# 0x15, 0x00, // Logical Minimum (0) 148
# 0x25, 0x01, // Logical Maximum (1) 150
# 0x09, 0x61, // Usage (Vendor Usage 0x61) 152
# 0x05, 0x85, // Usage Page (Battery System) 154
# 0x09, 0x44, // Usage (Vendor Usage 0x44) 156
# 0x09, 0x46, // Usage (Vendor Usage 0x46) 158
# 0x81, 0x02, // Input (Data,Var,Abs) 160
# 0x95, 0x05, // Report Count (5) 162
# 0x81, 0x01, // Input (Cnst,Arr,Abs) 164
# 0x75, 0x08, // Report Size (8) 166
# 0x95, 0x01, // Report Count (1) 168
# 0x15, 0x00, // Logical Minimum (0) 170
# 0x26, 0xff, 0x00, // Logical Maximum (255) 172
# 0x09, 0x65, // Usage (Vendor Usage 0x65) 175
# 0x81, 0x02, // Input (Data,Var,Abs) 177
# 0xc0, // End Collection 179
# 0x06, 0x00, 0xff, // Usage Page (Vendor Defined Page 1) 180
# 0x09, 0x4b, // Usage (Vendor Usage 0x4b) 183
# 0xa1, 0x01, // Collection (Application) 185
# 0x06, 0x00, 0xff, // Usage Page (Vendor Defined Page 1) 187
# 0x09, 0x4b, // Usage (Vendor Usage 0x4b) 190
# 0x15, 0x00, // Logical Minimum (0) 192
# 0x26, 0xff, 0x00, // Logical Maximum (255) 194
# 0x85, 0x20, // Report ID (32) 197
# 0x95, 0x6b, // Report Count (107) 199
# 0x75, 0x08, // Report Size (8) 201
# 0x81, 0x02, // Input (Data,Var,Abs) 203
# 0x09, 0x4b, // Usage (Vendor Usage 0x4b) 205
# 0x85, 0x21, // Report ID (33) 207
# 0x96, 0x89, 0x02, // Report Count (649) 209
# 0x75, 0x08, // Report Size (8) 212
# 0x81, 0x02, // Input (Data,Var,Abs) 214
# 0x09, 0x4b, // Usage (Vendor Usage 0x4b) 216
# 0x85, 0x22, // Report ID (34) 218
# 0x95, 0x3e, // Report Count (62) 220
# 0x75, 0x08, // Report Size (8) 222
# 0x81, 0x02, // Input (Data,Var,Abs) 224
# 0xc0, // End Collection 226
#
R: 227 05 01 09 06 a1 01 85 01 05 07 15 00 25 01 19 e0 29 e7 75 01 95 08 81 02 95 05 75 01 05 08 19 01 29 05 91 02 95 01 75 03 91 03 95 08 75 01 15 00 25 01 06 00 ff 09 03 81 03 95 06 75 08 15 00 25 e7 05 07 19 00 29 e7 81 00 95 01 75 01 15 00 25 01 05 0c 09 00 81 01 95 01 75 01 06 01 ff 09 03 81 02 05 0c 09 40 95 01 75 01 81 02 95 01 75 05 81 03 06 02 ff 09 55 85 55 15 00 26 ff 00 75 08 95 40 b1 a2 c0 06 00 ff 09 14 a1 01 85 90 05 84 75 01 95 03 15 00 25 01 09 61 05 85 09 44 09 46 81 02 95 05 81 01 75 08 95 01 15 00 26 ff 00 09 65 81 02 c0 06 00 ff 09 4b a1 01 06 00 ff 09 4b 15 00 26 ff 00 85 20 95 6b 75 08 81 02 09 4b 85 21 96 89 02 75 08 81 02 09 4b 85 22 95 3e 75 08 81 02 c0
N: lucas’s Magic Keyboard
I: 5 004c 029a
Please can you guide me on what to do?
It's been some time since I did this, let's see if I can remember how this works...
I'll need the output of hid-recorder from the host machine for the RPi as well, so I can compare what needs to be replaced (I don't have the project running currently, as I need someone with bluez knowledge to help fix my issues).
Thanks for getting back!
I appreciate it's been a while, I hope we can figure it out together.
What I've shared above is the output on the RPi when running hid-recorder, so we can see that it's pairing correctly and receiving input, it's just the forwarding on via bthidhub that's not currently working.
I'm trying to work figure out this:
For me, I was able to get my mouse working by basically removing the mouse part in /etc/bluetooth/sdp_record.xml (run hid-recorder on the host to figure out which part that is) and replacing it with the descriptor for the mouse (run hid-recorder on RPi to find that). I just needed to readd 8503 (Report ID (3)) at the same location as the original sdp_record.xml in order for it to work correctly.
You'll also need to remove these lines which filter out the events for some unknown reason:
https://github.com/ruundii/bthidhub/blob/master/mouse_message_filter.py#L10,L11
This is what's in sdp_record.xml for me:
<record>
<attribute id="0x0001">
<sequence>
<uuid value="0x1124" />
</sequence>
</attribute>
<attribute id="0x0004">
<sequence>
<sequence>
<uuid value="0x0100" />
<uint16 value="0x0011" />
</sequence>
<sequence>
<uuid value="0x0011" />
</sequence>
</sequence>
</attribute>
<attribute id="0x0005">
<sequence>
<uuid value="0x1002" />
</sequence>
</attribute>
<attribute id="0x0006">
<sequence>
<uint16 value="0x656e" />
<uint16 value="0x006a" />
<uint16 value="0x0100" />
</sequence>
</attribute>
<attribute id="0x0009">
<sequence>
<sequence>
<uuid value="0x1124" />
<uint16 value="0x0100" />
</sequence>
</sequence>
</attribute>
<attribute id="0x000d">
<sequence>
<sequence>
<sequence>
<uuid value="0x0100" />
<uint16 value="0x0013" />
</sequence>
<sequence>
<uuid value="0x0011" />
</sequence>
</sequence>
</sequence>
</attribute>
<attribute id="0x0100">
<text value="BT HID Hub" />
</attribute>
<attribute id="0x0101">
<text value="BT HID Hub" />
</attribute>
<attribute id="0x0102">
<text value="Raspberry Pi" />
</attribute>
<attribute id="0x0200">
<uint16 value="0x0100" />
</attribute>
<attribute id="0x0201">
<uint16 value="0x0111" />
</attribute>
<attribute id="0x0202">
<uint8 value="0xC0" />
</attribute>
<attribute id="0x0203">
<uint8 value="0x00" />
</attribute>
<attribute id="0x0204">
<boolean value="false" />
</attribute>
<attribute id="0x0205">
<boolean value="true" />
</attribute>
<attribute id="0x0206">
<sequence>
<sequence>
<uint8 value="0x22" />
<text encoding="hex" value="13505010902a10185020509190129021500250195027501810295017506810305010901a1001681ff267f0036c3fe463d016513550d09300931750895028106750895048101c00602ff09558555150026ff0075089540b1a2c00600ff0914a101859005847501950315002501096105850944094681029505810175089501150026ff0009658102c0"/>
</sequence>
</sequence>
</attribute>
<attribute id="0x0207">
<sequence>
<sequence>
<uint16 value="0x0409" />
<uint16 value="0x0100" />
</sequence>
</sequence>
</attribute>
<attribute id="0x0209">
<boolean value="false" />
</attribute>
<attribute id="0x020a">
<boolean value="true" />
</attribute>
<attribute id="0x020b">
<uint16 value="0x0100" />
</attribute>
<attribute id="0x020c">
<uint16 value="0x0c80" />
</attribute>
<attribute id="0x020d">
<boolean value="true" />
</attribute>
<attribute id="0x020e">
<boolean value="false" />
</attribute>
<attribute id="0x020f">
<uint16 value="0x0640" />
</attribute>
<attribute id="0x0210">
<uint16 value="0x0320" />
</attribute>
</record>
The only bit I changed was the hex for which I replaced with the hid-recorder output from my trackpad under R:, but I don't think this was the correct thing to do. What do you think was meant by "replacing it with the descriptor for the mouse"?
I'm most confused about the "re-adding 8503 (Report ID (3))" bit you mentioned, I'm not sure what this refers to.
Finally, you recommended removing lines 10 and 11 from here:
https://github.com/ruundii/bthidhub/blob/master/mouse_message_filter.py#L10,L11
However, I have a feeling the file has changed because it currently points to a comment and a blank line. Which are the two lines which filter out events?
It feels like you figured this out and hopefully we can reverse engineer this to get it working again :)
What I've shared above is the output on the RPi when running hid-recorder, so we can see that it's pairing correctly and receiving input, it's just the forwarding on via bthidhub that's not currently working.
Yes, I need the same output from your host machine, so that I can understand what the current sdp_record is doing (make sure you revert to the original file first).
Specifically, the R part is the actual record and all the comments above that give a decoding of the values. So, in the original sdp_record, we'll need to find the block that refers to the mouse and replace it (likewise with the keyboard). I think, because your mouse already has Report IDs in it (as it has multiple functions), you'll need to tweak those IDs (whereas mine didn't have any, so I just needed to insert 8503). (I mentioned this as a requirement in #23 (comment))
So, the sdp_record has multiple functions in it (mouse, keyboard), and we are going to replace those functions with the functions from your mouse and keyboard, so we end up with a mega sdp_record that contains all the functions of all your devices.
Thanks for the explanation!
I reverted to the original file.
hid-recorder shows:
Available devices:
/dev/hidraw0: lucas’s Magic Keyboard
/dev/hidraw1: lucas' Trackpad
Select the device event number [0-1]:
So, although my MacBook is shown as connected in the HiD Hub interface, it's not showing in hid-recorder.
HiD showing "connected": https://ibb.co/TrwG0K8
macOS showing "connected": https://ibb.co/z6g3h3X
HiD showing options for both devices: https://ibb.co/9qVkWDs
Should hid-recorder also list output devices?
Other
I've found 'Connected HID devices' (where you select capture, report filter etc.) to be unreliable, sometimes the input devices don't show even though they're "connected" in the Pair screen and are reliably receiving in hid-recorder.
Also, selecting compatibility mode gives a 'Could not set compatibility mode. Internal Server Error'.
Not trying to create more problems, but hoping a full picture will highlight the underlying issue.
In your first picture, the Mac is not paired as a host. Which is the same problem I have now with every host device, so unless you want to try working on the bluez code, you're screwed (#21). I'd happily update everything in this repo except bluez, so if someone can figure out the bluez parts for me, I'll get the rest of the project working properly.
Thanks for your patience @Dreamsorcerer and getting me up-to-speed.
I can try to work through this (or at least help move things forward). Where's the relevant part in this repository?
Or are we simply relying on https://github.com/bluez/bluez? Is there an open issue for what we're seeing?
No, it's a custom fork with a couple of changes to make it work for our application.
The repo is https://github.com/ruundii/bluez (installed at https://github.com/ruundii/bthidhub/blob/master/install/on_rpi/on_pi_setup.sh#L41)
You'll probably need to look through the ruundii commits, which it looks like is summarised under:
bluez/bluez@master...ruundii:bluez:master
And figure out how those changes work. I'm not sure how it calculates whether a device is a host or not, but (as mentioned in that other issue) it does correctly choose the icon as 'computer'. So, if the logic can be changed to set a device as host simply if the icon is 'computer' or 'phone', then I think we'd be sorted.