Authentication failed
WayneManion opened this issue · 67 comments
I have been using this Sensi integration for a few weeks. Today I had a notification that the authorization for the Sensi integration was expired. I went to the Integrations page in Home Assistant and clicked "Re-configure" for the Sensi integeration.
A modal box with no text and just a "sleepy eyeball" appeared. I entered my Sensi account password. Authentication failed. I went to the mobile app on my phone and changed the password. I was able to log into the mobile app with the new password. I used the new password in the Sensi integration and again, authentication failed.
This started for me as well. Removed and reinstalled but cannot get past the credentials screen.
Authentication failed.
+1 for having this issue pop up today.
Same here, I got the auth request, tried my password, and then removed and reinstalled but cannot get past the credentials screen.
Same for me...
Issue started today for me as well.
Same here.....
Same issue for me too
same.
There is a new header required:
rct: xxxxxxx
It changes with every login based on MITMPROXY and Android app.
The code to create the value seems to be at least partially in com.emerson.sensi.util.recaptcha
So far I don't know yet how exactly it is calculated....
I had filtered to traffic with Sensi only, haha.
There are calls to:
https://www.recaptcha.net/recaptcha/api3/mri
https://www.recaptcha.net/recaptcha/api3/mrr
https://www.recaptcha.net/recaptcha/api3/mlg
It seems like the first one gets static data (mostly) and the response to mrr contains the header we need for rct.
Need some more digging but should be fairly straight forward.
Maybe someone finds docu for that API?
SEEMS to be Google Reccaptcha V3 (maybe)
Add me to the pile 😁
I am having this issue as well
I am also having this issue.
+1
+1 same issue
I really don't think any more +1 comments are really helping the issue. This is going to happen for EVERYONE!
I am pretty sure the repo owner got that point and maybe we should limit comments to things that are helpful to fixing the issue.
The way I understand reCAPTCHA is the app talks to captcha server using an app specific key. The server returns a token based on bot identification and tells the backend server. This token is then passed to the backend server during authentication. I am suspecting the rtc header is this token.
I have been looking at the decompiled app and have not been able to find any key so far. I will keep digging.
https://cloud.google.com/recaptcha-enterprise/docs/instrument-android-apps
https://www.geeksforgeeks.org/how-to-integrate-google-recaptcha-in-android/
Key needed:
6LdS6SYpAAAAAFCti9uo4Yg47fSIGygzTGl_6lEV
This is from Android app...
I can create a fake POST to mri endpoint. It contains a few values needed for mrr endpoint.
mrr takes a total of 7 strings protobuf encoded.
String 1 and 2 are in response from mri, String 3 is the key and String 4 is "login"
String 5-7, so far no clue what they are or where to get them....
Once we have those we can post to mrr and the response will contain the value needed for the auth header that we are missing now.
+1 - but don't understand how to fix it.
The integration through homekit still works, but doesn't have as many options.
+1 how can I fix the issue?
@troyboy27 @omarquis This cannot be addressed without code changes. The authentication at Sensi end has changed.
I am also investigating the {"grant_type":"refresh_token","refresh_token": "some_random_string"} , if the refresh_token is long lived and how to acquire it. More than likely it will rely on the aforementioned recaptcha.
Will report when I know more...
data = {
"client_id": client_id,
"client_secret": client_secret,
"grant_type": "refresh_token",
"refresh_token": refresh_token
}
headers = {
"accept": "*/*",
"accept-language": "en-US,en;q=0.9",
"content-type": "application/x-www-form-urlencoded; charset=utf-8"
}
More than likely it will rely on the aforementioned recaptcha.
You get that as part of the auth process. You generally get the token and a refresh token. Once the token expires you exchange the refresh token for a new pair. That step requires the recaptcha as well.
BUT, Google has libraries that take care of that on Android, not sure if Sensi had to implement sth for iOS or if there is a library or even a different process...
Since it does not matter for us we might get lucky with iOS and the way it's done there.
Can you log traffice between the app and sensi/google? Using sth like MITM proxy or so.
That is how I found out about the recaptcha process...
Ok thanks,
I got the "refresh_token" by logging the traffic via MITM proxy and have been using that in the sensi service for HA as a workaround.
I did see the recaptcha as well., but I could not follow the output. I will have another try at it and report back later. Thanks for the tips!
I'll take some of what I said back... once you have the refresh tokoen you do not need recaptcha anymore!
have been using that in the sensi service for HA as a workaround.
I tried that and it is failing... how did you tell HA to use it? I tried to enter it into the box that asks for auth....
but not working.
have been using that in the sensi service for HA as a workaround.
I tried that and it is failing... how did you tell HA to use it? I tried to enter it into the box that asks for auth.... but not working.
I tested the methodology outside HA and then provided it via configurable variable inside the auth.py. Maybe @iprak can expose this variable in the configuration.
post_data = {
# "username": config.username,
# "password": config.password,
"client_id": client_id,
"client_secret": client_secret,
"refresh_token": refresh_token,
"grant_type": "refresh_token",
}
headers = {
"accept": "*/*",
"accept-language": "en-US,en;q=0.9",
"content-type": "application/x-www-form-urlencoded; charset=utf-8",
}
Sensi has removed previous authentication mechanism. Even the previous Android app 8.6.2 fails to login. My initial hope was that there was a fall back mechanism tied to the appVersion passed as header. Even faking "ios" did not help.
I am not seeing how would one get the refresh_token ?
All my work is based on the latest android version 8.6.3
You get the refresh_token as part of the auth flow. 'All' we need is the recaptcha value...
Oh I misread that comment .. I thought that recaptcha value was not needed.
The Android client uses a WebView to host reCaptcha content. I will play with some Python libraries which offer reCaptcha implementation.
Sensi has removed previous authentication mechanism. Even the previous Android app 8.6.2 fails to login. My initial hope was that there was a fall back mechanism tied to the appVersion passed as header. Even faking "ios" did not help.
I am not seeing how would one get the
refresh_token?
It does not appear that the 'grant_type':'refresh_token' is working for me anymore.
The app is now using the {'grant_type': 'password'} with the rpt: header.
Not sure if this helps, but I went to there website and signed in with inspect open. Did a search for token on the network and saw this for a device path listing. The ***** after FL33T is my accounts username.
Headers Tab
Request URL: https://oauth.sensiapi.io/token?device=FL33T-*****-581025838
:path: /token?device=FL33T-*****-581025838
Payload Tab
client_id: fleet
https://manager.sensicomfort.com/
It's the same site someone would use if they paid for there subscription to access there thermostats from there site.
username/password is the same on both app and website.
nope, not paying, so no functionality. But i figured it was worth a shot to see if it would help at all. Tho atm I am using the Homekit method in HA to control my Sensi Touch. But would be nice to see this up and running again.
site says first month is free, cancel any time. so I guess it could be tested without having to pay for it. Just remember to cancel before the first month ends.
Actually, I don't think that is even needed. Just looked at the traffic....
The response looks VERY similar to the app and contains the required tokens.
@macwriter you already have modified sources... I think it would be super easy for you to check if the response tokens work.
Simply log into https://manager.sensicomfort.com/ and look at the traffic to grab the token from the oauth call.
I have a good feeling about this!
@Pheelix you might have solved the mystery for us...
@iprak The website also uses recaptcha but it is fully browser based. I ASSUME that python can emulate a webbrowser in a way that you load the login page and let it do it's thing to take care of the recaptcha?
I can confirm, the tokens work!
Changed auth.py around line 67
refresh_token = "copied from browser response"
client_id = "fleet" # instead of android
client_secret = "JLFjJmketRhj>M9uoDhusYKyi?zUyNqhGB)H2XiwLEF#KcGKrRD2JZsDQ7ufNven" # different secret
post_data = {
#"username": config.username,
#"password": config.password,
#"client_id": CLIENT_ID,
#"client_secret": CLIENT_SECRET,
#"grant_type": "password",
"client_id": client_id,
"client_secret": client_secret,
"refresh_token": refresh_token,
"grant_type": "refresh_token",
}
Error in HA goes away after a restart...
I guess if @iprak can simulate the browser login then we are back in action!
I guess worst case solution would be to have the user login to the website and copy the refresh token and have that as an external config value that we have to paste. I guess other add-ons do the same where the user needs to create an account with company X and provide some keys manually.
I guess my manual fix will stop working as soon as my current token expires since I hard-coded the refresh token and it only works once. But if we can feed the initial refresh_token and than the addon keeps managing it like in the past it should work long term.
@Pheelix amazing catch!
@iprak I just looked at the code and noticed that it seems like you never really use the refresh_token. Is that right?
If I understood it right you are sending user/pwd again when the original token expires using "grant_type": "password".
That is not the intention of OAuth... Ideally your addon would never know my user/pwd and only save the 2 tokens.
Once the access token expires you ask for a fresh one using "grant_type": "refresh_token" and the saved refresh_token value.
That will give you a new pair and you repeat.
You only ever need user/pwd ONCE at the very beginning.... Then never again.
With this changed flow the user could provide a single valid refresh_token and no user/pwd and you don't have to worry about recaptcha since that seems to be needed only for the user/pwd exchange!
Python cannot emulate reCaptcha, some library attempt to do that but they either implement v2 or are paid. reCaptcha involves a rotating sort of token which is generated by reCaptcha library.
Sensi uses the latest google reCaptcha. In web, this
e.g. https://www.google.com/recaptcha/enterprise.js?render=6LebYk0pAAAAAIUlMrqCdXOq83pY2O4u-CxuTh28
returns an executing block which returns the token.
The ideal way to address would be to change the initial authentication itself. This is how authentication is implemented for 3rd party verification e.g. https://sensiapi.io/authorize
In my experimentation, extracting token from web and passing to Python did not work. There might be additional headers/cookies.
Yes, you are right about refresh_token. I just login in again; I could not confirm when refreshing would be adequate.
I see two possible solutions here:
A: Change the setup flow to ask the user for a refresh_token. Explain that the user can get it by logging into https://manager.sensicomfort.com/ and then copying the value using the browser developer tools. This step is needed ONCE to setup. After that the refresh_token can be exchanged for an auth_token and a new refresh_token. Whenever sensi complains about expired credentials simply exchange again. This is the simplest impementation and would not require much change I think?!
B: Have python emulate a webwroser and login to https://manager.sensicomfort.com/ using username/pwd and then grab the refresh_token from the web traffic. Would be easy in and Android WebView, not sure if Python can do that.
Once you have that token the flow should be like in A. Don't store user/pwd but only refresh_token and renew when needed.
This hardcoded snippet DOES work until the token expires (hard coded refresh_token works only once.
But by replacing the hard-coded token with one initially provided by the user and then updated on every refresh this really works.
I see all my thermostats in HA as we speak!
auth.py around line 68ish
refresh_token = "copied from browser response"
client_id = "fleet" # instead of android
client_secret = "JLFjJmketRhj>M9uoDhusYKyi?zUyNqhGB)H2XiwLEF#KcGKrRD2JZsDQ7ufNven" # different secret
post_data = {
#"username": config.username,
#"password": config.password,
#"client_id": CLIENT_ID,
#"client_secret": CLIENT_SECRET,
#"grant_type": "password",
"client_id": client_id,
"client_secret": client_secret,
"refresh_token": refresh_token,
"grant_type": "refresh_token",
}
I was just about to suggest using selenium and chrome webdriver for the browser interactions. I know the workload can be containerized, as well. But honestly, I feel like @stleusc's first suggested solution is far straighter forward.
The tokens are JWT. Paste them at jwt.io to see their contents. The expiration date for my refresh_token is way out into 2034, but it does report as an Invalid Signature. Stleusc, how does yours report?
@akseidel learned something new today about JWT :-)
Yes, the refresh_token is valid for 10 years. Which is the idea behind it. The access_token is short lived, the refresh_token is long lived. HOWEVER, you generally can only use it ONCE to convert to a new access_token. You have 10 years to do that, but can do it only one time.
I guess what I stated above depends on the implementation. In other words, if we get a refresh_token in the reponse, I would update to that one (also gives 10 more years). If there is none in the response, keep the old one.
Other sites that I worked with in the past that used OAuth allowed only one exchange...
From https://www.oauth.com/oauth2-servers/access-tokens/refreshing-access-tokens/
If everything checks out, the service can generate an access token and respond. The server may issue a new refresh token in the response, but if the response does not include a new refresh token, the client assumes the existing refresh token will still be valid.
Regading signature: Signature Verified
The tokens are JWT. Paste them at jwt.io to see their contents. The expiration date for my refresh_token is way out into 2034, but it does report as an Invalid Signature. Stleusc, how does yours report?
The access token reports invalid sig, the refresh token valid sig.
I think the page needs to know the secret for checking the sig and we did not enter that!
@macwriter you already have modified sources... I think it would be super easy for you to check if the response tokens work. Simply log into https://manager.sensicomfort.com/ and look at the traffic to grab the token from the oauth call. I have a good feeling about this!
Yes, works for me with the updated information (client_id, client_secret,refresh_token). Great job finding this!
def test(client_id,client_secret,refresh_token):
params=()
data = {
"client_id": client_id,
"client_secret": client_secret,
"grant_type": "refresh_token",
"refresh_token": refresh_token
}
headers = {
"accept": "*/*",
"accept-language": "en-US,en;q=0.9",
"content-type": "application/x-www-form-urlencoded; charset=utf-8",
}
_response = requests.post("https://oauth.sensiapi.io/token", params=params, data=data, headers=headers)
print(_response)
print(_response.json().keys())
test(**options_from_Sensi_Manager)
<Response [200]>
dict_keys(['access_token', 'refresh_token', 'token_type', 'expires_in', 'password_reset', 'eula_accepted', 'offers', 'alerts', 'user_id'])
I think we have all we need to get this working again....
My option A from above should be pretty easy to implement and we already tested all the elements that are involved.
Now we just need to hope that @iprak finds the time to work on it.
B: Have python emulate a webwroser and login to https://manager.sensicomfort.com/ using username/pwd and then grab the
refresh_tokenfrom the web traffic.
Ironically, this could be foiled by recaptcha! lol
But maybe not.
@bperry11 using something like an Android WebView would work since that is basically the component used to build browser apps. Of course Python would be different...
I personally prefer A ;-)
Sorry for late response, this integration usually only gets my attention on weekends.
When I wrote this extension, my aim was to provide an easy to setup integration with the ultimate goal to have it be integrated in Core release.
While solution "A" makes it work, I do not think it is the right approach. It certainly puts the extension out of non-developers or folks just using mobile device.
I do not think browser emulation is possible in Python without using something like Selenium which makes it a quite heavy integration. This also might not work well not work in the constraints of HomeAssistant.
There are 3rd party subscription based libraries for solving reCaptcha but that again might nor work since reCaptcha is tied to the site.
Sensi does have external integration support e.g. Whoosh Air, SmartThings and electricity providers. I personally have Sensi integrated via the last 2 paths. My hope is that Sensi's external integration could be used to pass back the authentication information via webhook. But I don't know enough yet about the implementation. https://sensiapi.io/authorize however is the site for establishing external integration.
I understand that this issue is a headache for many. I do not think this integration is subtainable due to the additional restrictions imposed at Sensi end. I will try to get refresh_token based authentication out soon but I won't be making any more updates to this extension unless I can establish the authentication to be more standard.
While 'A' is not ideal, there are certainly many other integrations that are more complicated. Eg https://www.home-assistant.io/integrations/smartthings/ or https://www.home-assistant.io/integrations/alexa.smart_home/
And I also don't think that there have been any changes in recent years that would have broken that approach.
I assume you could contact Sensi and see if they allow you to become a partner, but I have a feeling they would not.
That being said, thank you for your continued support. I hope you change your mind and keep supporting this great integration!
Those two do have some setup but those are using established public facing workflows. Launching DevTools is at a different level.
One can link Sensi from SmartThings app which establishes authentication via https://sensiapi.io/authorize and this gives me hope that such a possibility might be there. The initial call does pass in a different token and a callback, if that is not locked down then the callback could be HomeAssistant webhook (same as SmartThings) to receive the token.
It was Homebridge, but I had to do very similar process to integrate Google Nest. Login, open devtool, grab the cookie / or token, and paste into homebridge. At least for the time being, it wouldn't be too big of a deal IMO. Either way, thanks for your work on this!
I tried to do what's shown in this comment but it still wasn't working. I'm guessing the refresh_token I was using was wrong. What exactly does it look like?
I tried to do what's shown in this comment but it still wasn't working. I'm guessing the refresh_token I was using was wrong. What exactly does it look like?
Mine's a 333 characters of upper and lowercase letters, digits, and symbols. Did you also get the client_secret? You'll need to put that in the auth.py file as well.
Find and change the following in the asnyc def login() function around line 48. (I commented out the old and put the new underneath:
post_data = {
# "username": config.username,
# "password": config.password,
# "client_id": CLIENT_ID,
# "client_secret": CLIENT_SECRET,
# "grant_type": "password",
"client_id": "fleet",
"client_secret": "<INSERT_YOUR_ NEW_CLIENT_SECRET>",
"grant_type": "refresh_token",
"refresh_token": "<INSERT_YOUR_RESFRESH_TOKEN>",
}
headers = {
# "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
# "x-platform":"android",
# "accept": "*/*",
"accept": "*/*",
"accept-language": "en-US,en;q=0.9",
"content-type": "application/x-www-form-urlencoded; charset=utf-8",
}