New Linea Micra API
ebfio opened this issue · 55 comments
Hi there,
While researching why my Linea Micra wouldn't connect to the integration, I just discovered there's a new API in place for it. It's using HTTP on port 8081.
GET /api/v1/config HTTP/1.1
Host: 10.53.50.70:8081
Connection: keep-alive
Accept: */*
User-Agent: La Marzocco/2.1.3 (iPhone; iOS 16.0.2; Scale/3.00)
Accept-Language: en-US;q=1
Authorization: Bearer XXXXXXXXXXX
Accept-Encoding: gzip, deflate
I'll be mapping all the API calls today - @rccoleman
Response to the above call:
{
"version": "v1",
"preinfusionModesAvailable": [
"ByDoseType"
],
"machineCapabilities": [
{
"family": "MICRA",
"groupsNumber": 1,
"coffeeBoilersNumber": 1,
"hasCupWarmer": false,
"hasWhaterProbe": false,
"steamBoilersNumber": 1,
"teaDosesNumber": 1,
"machineModes": [
"BrewingMode",
"StandBy"
],
"schedulingType": "weeklyScheduling"
}
],
"machine_sn": "Sn220XXXXXXXX",
"isPlumbedIn": false,
"isBackFlushEnabled": false,
"standByTime": 0,
"tankStatus": true,
"groupCapabilities": [
{
"capabilities": {
"groupType": "EP_Group",
"groupNumber": "Group1",
"boilerId": "CoffeeBoiler1",
"hasScale": false,
"hasFlowmeter": false,
"numberOfDoses": 0
},
"doses": [],
"doseMode": {
"groupNumber": "Group1",
"brewingType": "Time"
}
}
],
"machineMode": "StandBy",
"teaDoses": {
"DoseA": {
"doseIndex": "DoseA",
"stopTarget": 0
}
},
"boilers": [
{
"id": "SteamBoiler",
"isEnabled": true,
"target": 131
},
{
"id": "CoffeeBoiler1",
"isEnabled": true,
"target": 93.3000030517578
}
],
"boilerTargetTemperature": {
"SteamBoiler": 131,
"CoffeeBoiler1": 93.3000030517578
},
"preinfusionMode": {
"Group1": {
"groupNumber": "Group1",
"preinfusionStyle": "PreinfusionByDoseType"
}
},
"preinfusionSettings": {
"mode": "Disabled",
"Group1": [
{
"groupNumber": "Group1",
"doseType": "Continuous",
"preWetTime": 5,
"preWetHoldTime": 5
}
]
},
"weeklySchedulingConfig": {
"enabled": false,
"monday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"tuesday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"wednesday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"thursday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"friday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"saturday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"sunday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
}
},
"clock": "2022-11-25T12:31:51",
"firmwareVersions": [
{
"name": "machine_firmware",
"fw_version": "1.11"
},
{
"name": "gateway_firmware",
"fw_version": "v2.2-rc0"
}
]
}
Interesting! So the machine now provides a proper local REST API rather than the primitive, proprietary peek/poke interface? I just checked my GS/3 and it does not appear to accept connections on 8081, so it looks like it's just the Micra. I wonder if that's implemented in the gateway firmware - mine is "v1.1RC8", and was part of the recent firmware update. I see that yours is "v2.2-rc0". In any case, having the integration communicate with the machine via a REST interface could certainly be done, but it would be quite a different implementation. And without a local example to play with, it's would most likely be an exercise for someone with access to such a machine.
Working on it @rccoleman... the weird part is... from what I'm seeing (and I can definitely be wrong), it connects to the usual interface but via the 8081 port.
I'll report back!
Hi, since Friday I'm also part of the Micra club. Any news on the integration? I would be happy to test or help with the implementation.
@rccoleman I'm trying to work on the Micra functionality, but am a bit stuck... I'm currently playing around with lmdirect and interestingly enough getting the status from test.py (2) appears to be working just fine (with new client creds), however any update calls yield "Connection refused" exceptions.
Any hints on how to debug what's going on here? How did you find the raw message codes?
If I do any update in the app (like turning the machine on/off) nothing is recorded in mitmproxy, was that different for your machine(s)?
mitmproxy is only necessary and useful for communication with the La Marzocco gateway and doesn't play a part in the local communication with the machine. I use the client credentials to connect to the gateway to grab information about the machine and, most importantly, the AES key used to encrypt the local communication. Once I get that, I close the gateway connection and do everything locally via port 1774 encrypting and decrypting as necessary. So it's no surprise that there's no mitmproxy communication while you're using lmdirect to control the machine.
To identify and decode the commands/responses when communicating locally, I used Wireshark to do a remote pcap capture from the AP that my machine was connected to (I use Unifi APs) while using the mobile app on my phone. I then filtered for packets going to and from the machine's IP address, did "Export packet dissections->as JSON", and used my "parse" app to decrypt/decode them (described here). That'll give you a view of what was sent to and from the machine while you control the machine in the mobile app. None of this is straightforward and I had the joy of relearning how to do it with the recent firmware release, but it's not too hard (especially with the "parse" tool I wrote).
"Connection refused" makes it sound like it wasn't able to connect locally at all, which is consistent with the REST data shown above. It has a lot more information than I get from the gateway, and points to more of a cloud-centric model vs local control (boo!). An easy way to see if the machine is listening on the local 1774 port is to try to go to <machine_ip_address>:1774 in your browser. If it says something like this, it's listening:
192.168.1.215 sent an invalid response.
ERR_INVALID_HTTP_RESPONSE
If it says this, it's not:
192.168.1.215 refused to connect.
My guess is that La Marzocco is moving away from the proprietary local connection and toward a public REST API. To confirm and start to map endpoints, use mitmproxy while using the mobile app to control the machine and see what you get. Assuming you can figure out how to use the REST API, a new package (maybe called "lmremote") could be written to use the implement the same API based on the new REST API and plugged into the integration.
thanks for the detailed response.
Based on the first post in the issue I was also expecting a more REST based approach, but could not find any calls except the /api/v1/config call and a /api/v1/streaming call (which is just answered with a 101). I then Wiresharked the connections to the machine from my iPhone (and also to anything containing lamarzocco in the URL), but there were only a lot of ACK packages. Maybe listening to Unifi is worth a shot.
Curling 1774 just returns a connection refused, but then I'm really wondering where the packages are going...
I didn't see the local communication with the machine until I set up an SSH remote capture in Wireshark with this:
Fill in the server address of the Unifi AP, port (22), & authentication (username/password that you set up in the Unifi controller for the devices) as appropriate. I also used this filter to focus on what mattered (change IP addresses as needed):
((ip.dst == 192.168.1.215 && tcp.port == 1774) or ip.src == 192.168.1.215) && ip.src != 192.168.1.159 && ip.dst != 192.168.1.159 && tcp.flags.push == 1
The last two bits with 192.168.1.159 just filter out commands and responses directed to the HA integration if I leave it running.
You can also use nmap:
rcoleman@rcoleman-linux ~ % sudo nmap -sT -p- -O 192.168.1.215
Starting Nmap 7.80 ( https://nmap.org ) at 2023-01-23 14:55 PST
Nmap scan report for LaMarzoccoGS3.xxx (192.168.1.215)
Host is up (0.031s latency).
Not shown: 65534 closed ports
PORT STATE SERVICE
1774/tcp open global-dtserv
Yep, the remote tcpdump is exactly what I've been trying in the meantime. I discovered that the machine seems to get some data from cloud-mqtt.relayr.io but I can't read it because of TLS. nmap is worth a shot.
What I find strange is, if they really switched to a more cloud based REST approach I would expect to see that traffic in mitmproxy but that's not the case (that would afaik be the only possible place to read the TLS traffic as well).
What would be interesting: Do you have an example how a local package with a command to your machine looks like in Wireshark?
I found that MQTT FQDN a while ago just sitting in the machine's memory space along with a username and password that allowed me to connect via MQTT Explorer, but it then immediately disconnected. Seemed interesting, but I couldn't make anything out of it. I'll attach a packet trace in a bit, but you're not really going to get much out of it. It's just an encrypted payload starting with @ and ending with %:
40:4a:73:78:52:49:53:6c:30:34:51:72:6e:39:5a:61:49:71:6d:64:72:73:52:4b:68:4f:79:6d:5a:50:30:6d:31:5a:6f:78:45:58:4e:35:6e:2f:2f:73:3d:25
I don't have that specific packet broken down, but when a packet is decrypted and run through my "parse" tool, it looks like this (for enabling preinfusion):
App: W 00 0B 00 01 02 4C
Machine: W 00 0B 00 01 OK 84
Here's an early pcap file for turning the machine on and off.
pcap.tgz
thanks for the pcap that helped identifying some relevant packages between app and machine.
The following findings:
- communication is indeed only happening on port 8081
- payload size is no longer 40 bytes but only 6 bytes, need to figure out how to decode that now
- how many bytes does you AES key have? I get a 64 byte string for the key, which leads to a ValueError with the AES package...
The size of the transaction isn't fixed - it's based on the command and response. The interface is just simple peek/poke, with the command and response starting with [R|W|Z]AAAALLLL, where the first character is the type of transaction, then 16 bits of address, then 16 bits for the length. I wrote a bunch of this up in the main forum thread when I discovered it, and there's a brief paragraph on the lmdirect github. Getting the AES key out of the initial JSON response was really annoying without just doing it in the Python code as here. String encoding is still a bit of a mystery to me, but Python tells me that my key is a 32-byte string.
What I meant that in your pcap it was mostly 40 Bytes.
I used breakpoints in your code to extract my key, weird that it's 64 Byte compared to your 32 now. Not sure how that is used, it can't be AES with that key length...
Getting the key in the right format was quite an ordeal with a lot of code stealing. I don't know why yours would be different... I felt like I had broken Enigma when I finally got useful data out of the decryption :)
I thought about it maybe being the IV concatenated, but the IV for AES is only 16 bytes.
I'm wondering as well why it might be different, but it's what your code (and Python's length calculation) tells me... Also, if I try to run test.py with port 8081 I get a "Incorrect AES key length (64 bytes)" exception from AES. I didn't understand why that was yesterday, but considering the different key sizes it makes sense...
Maybe check with Postman on your desktop machine. It can handle the OAUTH flow and provide the token directly to the REST request, and then you can find the key in there.
Postman returns the same key your code does. However I discovered something interesting: That 64 Byte "key" is used as Bearer Token for the local API (/api/v1/config). If only I could find more of those local endpoints (especially POSTs/PUTs). That would a) make things considerably easier and b) just be good design for the machine.
But I can't find anything of that sort and the 6 byte packages I found earlier don't look like actual commands either on closer inspection...
Ok, made some progress: If I cut my machine's power (really pulling the plug) the app seems to resort to remote calls to gw.lamarzocco.io. Not the local API I was hoping for, but at least something. I will try to start working on a lmremote package which we can then plug into this integration.
Ha, I think I went through a similar process. If you just want the app to only go through the cloud, you can just turn off wifi on your phone :)
But if I just turn off WiFi I don't have mitmproxy in between to sniff the TLS traffic, that's why I took the power cycle route. (Also tried to block the local traffic in the Unifi firewall but somehow couldn't get that working)
Ah, okay. I think I discovered that, too :)
@rccoleman could you come up with some kind of interface class we can both inherit from? It should define properties and abstractmethods we need for the HASS integration so we can switch the packages easily. And you know best what we need for this integration. Or do you have a better idea?
btw. I started work over here https://github.com/zweckj/lmcloud
I sure hope that a clear, well-structured REST interface is easier than working with their proprietary interface. I'll think about what an interface layer would look like. Is what you're starting with in lmcloud actually working? I guess you'll still need to poll the rest endponts.
Sure, definitely a lot easier. Yes, the code there is working so far. Only some properties are untested.
I'm pretty much done writing the interface methods for the API calls and some getters. If we agree on an interface layer, I can start building that in. Did you have the chance to think about that?
I thought about doing something like this https://github.com/zweckj/lamarzocco/blob/micra_support/custom_components/lamarzocco/interface.py but I'm not really sure if that's the best way moving forward... @rccoleman any opinions?
I am also having the issue that connection is not possible. I get the message Failed to connect. Even though I successfully intercepted all the credentials needed. I also can not find the Linea Micra by auto-discovery.
Is there any progress on the above to get the Linea Micra working with this integration?
The same here, current gateway version: V2.2-RC0, FW1.11
Failed to connect
The library to support the Micra is done, but I didn't find the time yet to work on the interface layer to add it to this integration.
Hey @rccoleman I could use your help, if you got some time:
- I implemented an interface in my fork which is (partially) running for me, but of course I have no idea if the other machines are still working. I tried to only wrap your existing code, but still...
I'm struggling to understand how the current_status dictionary looks like fully filled. Could you send me one (filled) sample, so I can remodel that in my lmcloud package
For those waiting: My fork https://github.com/zweckj/lamarzocco is already working for the Micra. Once Rob finds the time to check with the old machines we can merge, but until then you can install it directly from my fork.
This can be done by going to your HACS integrations and selecting the 3 dots, then add a custom repository
@zweckj For those waiting: My fork https://github.com/zweckj/lamarzocco is already working for the Micra. Once Rob finds the time to check with the old machines we can merge, but until then you can install it from my fork.
I did add your fork to HACS, but I can not find the Micra through auto-discovery, same for the integration when I try to add it manually.
Edit: The Micra is also correctly connected through Wifi, since I can connect with it through the app, and router.
I did add your fork to HACS, but I can not find the Micra through auto-discovery, same for the integration when I try to add it manually.
Hard to say if you can't add the integration at all.
Make sure you really have the correct repo (it should have 0 stars)
Also make sure you actually downloaded the repo after adding it to HACS and did a restart after downloading.
I can confirm it works correctly!
Just pushed another update. You should now get drink counter, water reservoir status and correct current temperatures for steam and coffee.
I was able to get this working though I had to get my own own client ID and key (is that expected? do all Micra's have the same credentials?)
Thanks for working on this @zweckj!
(is that expected? do all Micra's have the same credentials?)
I only know what you discovered as well: the keys on the wiki page are not working for the Micra. I don't know if there is one id/secret pair or multiple, as nobody shared theirs with me.
That 64 Byte "key" is used as Bearer Token
@zweckj where is the key found? I've tried both keys in the README and I get Not Authorized response.
@mrvautin what are you trying to do? Initialize this integration or the lmcloud package? That key you're talking about is
- not exposed to the user
- individual for each user
you'd have to find out your client ID & client secret like described in the README and then get the machineInfo from LM's API and in there you'd find the communicationKey aka key
Since upgrading my Linea Mini, the HA integration doesn't work. I was poking around to see if the update added the Micra API and it may well have considering /api/v1/config returns Not Authorized and when turning the machine off I get no response/time out.
then get the machineInfo from LM's API
How is this done?
UPDATE: I see your repo here explains it. I will give it a go.
I actually tried using your repo and it's not working for my Linea Mini too. So I'm just having a poke around to see what I can do/help for the Linea Mini owners.
@mrvautin yeah my repo is completely untested with the "old" machines as I got none and I haven't heard back from @rccoleman in months. Also, even if my code worked for the old machines untested (which would be a miracle 😅) it would still have the same problems as this integration, as I just wrapped the existing code for those machines. However, if you're willing to do some development I'd love to work with you to develop a version which works for everyone.
Put a breakpoint in line 148 in lmcloud.py and inspect the _machine_info, you will find the key in there
Put a breakpoint in line 148 in lmcloud.py and inspect the _machine _info, you will find the key in there
Thanks. I've found the token and indeed there is an API. When I make a GET call to /api/v1/config I get:
{
"version": "v1",
"preinfusionModesAvailable": [
"ByDoseType"
],
"machineCapabilities": [
{
"family": "LINEA",
"groupsNumber": 1,
"coffeeBoilersNumber": 1,
"hasCupWarmer": false,
"steamBoilersNumber": 1,
"teaDosesNumber": 1,
"machineModes": [
"BrewingMode",
"StandBy"
],
"schedulingType": "weeklyScheduling"
}
],
"machine_sn": "",
"machine_hw": "0",
"isPlumbedIn": false,
"isBackFlushEnabled": false,
"standByTime": 30,
"tankStatus": true,
"groupCapabilities": [
{
"capabilities": {
"groupType": "AV_Group",
"groupNumber": "Group1",
"boilerId": "CoffeeBoiler1",
"hasScale": false,
"hasFlowmeter": true,
"numberOfDoses": 1
},
"doses": [
{
"groupNumber": "Group1",
"doseIndex": "DoseA",
"doseType": "PulsesType",
"stopTarget": 0
}
],
"doseMode": {
"groupNumber": "Group1",
"brewingType": "PulsesType"
}
}
],
"machineMode": "BrewingMode",
"teaDoses": {
"DoseA": {
"doseIndex": "DoseA",
"stopTarget": 0
}
},
"boilers": [
{
"id": "SteamBoiler",
"isEnabled": true,
"target": 0,
"current": 0
},
{
"id": "CoffeeBoiler1",
"isEnabled": true,
"target": 104,
"current": 105
}
],
"boilerTargetTemperature": {
"SteamBoiler": 0,
"CoffeeBoiler1": 104
},
"preinfusionMode": {
"Group1": {
"groupNumber": "Group1",
"preinfusionStyle": "PreinfusionByDoseType"
}
},
"preinfusionSettings": {
"mode": "Enabled",
"Group1": [
{
"groupNumber": "Group1",
"doseType": "DoseA",
"preWetTime": 2,
"preWetHoldTime": 5
}
]
},
"weeklySchedulingConfig": {
"enabled": false,
"monday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"tuesday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"wednesday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"thursday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"friday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"saturday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
},
"sunday": {
"enabled": false,
"h_on": 24,
"h_off": 24,
"m_on": 0,
"m_off": 0
}
},
"clock": "2023-07-06T17:11:23",
"firmwareVersions": [
{
"name": "machine_firmware",
"fw_version": "2.10"
},
{
"name": "gateway_firmware",
"fw_version": "v3.1-rc4"
}
]
}That looks exactly what I get from the Micra. What you could try next is add the MODEL_LM to LM_CLOUD_MODELS in a fork of my fork in const.py (https://github.com/zweckj/lamarzocco/blob/master/custom_components/lamarzocco/const.py) and see what that does
Or not... the gateway update also changed the API of my machine quite significantly... need to update my lmcloud library. The current API still works, but god knows how long and if it does as well for the Mini.
Hmm ok. Was thinking it was just me. Your repo installed (the official one didn't) with me adding the LM to the const.py but none of the entities work for me. If you get it working, I'd be happy to test and help on my Linea Mini.
@mrvautin if you turn the machine on/off via app or physical switch does that already reflect in HA (might take a couple of seconds).
@zweckj yes. If I turn the machine off in the app, then move the paddle on the machine, the app updates after a second. All the app changes reflect on the machine quickly too.
sorry, yes that's of course expected. What I meant is if you turn the machine on/off in the official LM app, will you see those states in Home Assistant?
@zweckj right. None of the entities in Home Assistant work at all. I'm trying to debug the lmclient to see why.
@zweckj Legend. That worked. Was the bluetooth code you commented out. All the entities seem to work for me now.
@zweckj Legend. That worked. Was the bluetooth code you commented out. All the entities seem to work for me now.
Not only that, had to adapt the downstream library, too 😉 However, I'm not 100% sure that everything is working properly again, so handle with care
I'm archiving this repo because this integration no longer works with current machine firmware. Please move to https://github.com/zweckj/lamarzocco.