Oauth v3
fkhera opened this issue · 63 comments
Tesla changed to v3 oauth api about last Friday so all their auth basically broke I have a work around in my python code 👍
https://github.com/fkhera/powerwallCloud/blob/abd0daa069888e68b346201464ec5dfcda771144/powerwallBackup.py#L132
Does local Powerwall access need the updated v3 oauth code? My local access is still working although I'm only reading SOC so I'm not logged in. I suppose I thought I could log in locally and implement some on the set functions?
Also - thanks for the work on the api especially the SSL certification - worked perfectly on Node Red
I am no longer able to access aggregates. I assume oauth is the reason? Sorry for my ignorance.
https://XXX,XXX.XX.X/api/meters/aggregates
Error:
{"code":403,"error":"Unable to GET to resource","message":"User does not have adequate access rights"}
I have also tried doing basic authentication, received a token and tried using it in the header to access.
Error:
{"code":401,"error":"token contains an invalid number of segments","message":"Invalid m2m/bearer token"}
I also confirm that local access to the gateway no longer works for me, e.g. to get aggregates.
My Powerwall has recently been updated to version 20.49.0. As a result, I can no longer see power data on the gateway web page. Attempts to retrieve data using curl fail with "User does not have adequate access rights". Only if I log into the gateway do I see data on its web page.
I tried basic authentication with curl by adding:
-u UUU:PPP
but that didn't help. I also tried providing credentials a different way with curl:
-H 'Content-Type: application/json' -X POST
-d '{"username":"UUU","password":"PPP"}'
but that results in "Method Not Allowed" (the POST?).
I can understand Tesla wishing to tighten up on security, but unfortunately the recent change seems to have broken easy programmatic access.
I'm still on 20.40.3 and can still see aggregates.
In the version history it is reporting:
Jan 2021 version 20.49:
"Improved security for local gateway interface for Gateway 1 and Gateway 2" -
so it looks like I will be loosing it soon!
OK, I'm not sure how to solve this, but the original local API was documented by people who worked out what the local website (http(s)://ip.of.your.powerwall) was displaying. If I open up the network tab of developer view with my local display showing the moving dots showing energy flow, I can see calls to soe, grid_status and aggregates every 2.5 seconds. Aggregates is called from GetMeterAggregates, which is, in turn, called from loopSitemasterInfo.
Assuming that those like @ISO8807 who have been upgraded to 20.49 can still see the animated energy flow and charge state on the main website, it should be possible to understand how that is authenticating to call meters aggregates. It might be possible to then work out how to re-gain access to the high speed interface that is available locally for logging SOC and flows.
Hopefully once one of the original (@vloschiavo) documenters is updated to 20.49, they'll be able to see the calls and spot the solution. I'm happy to help, but a little out of my depth, I suspect.
So far, I've had success copying the cookie from the local web UI and using that for monitoring, but I have no idea how long that cookie will last, or if will will work through reboots of the unit.
I got updated at 6am this morning, and local monitoring failed. If I browse to https://ip.of.my.powerwall, the local display is frozen, but it also shows registration incomplete. I'm going to call Tesla this morning out of interest to see what they have to say for themselves!
just got 20.49 and found myself In the same boat. Can confirm @tekkamanendless findings, authenticate in web ui, results in cookie, using that cookie lets me pull the api paths as i did before, I too do not know how long this will work for, going on 3 hours so far without an issue
Just logged in as customer with a previously set customer password and it took me straight to the original registration process - name, email, address, opt in for monitoring, opt in for grid services, opt in for marketing. Should have taken a screen shot before completing it.
Interestingly, if I log out, it still shows as unregistered, but if I click on register (which prompts for log in), and log in as customer, it no longer takes me to the registration page, but takes me to the monitoring page.
What is that register button doing ?
Interesting - seemed to be able to get it to run the registration process again by deleting cookies:
When you run through the full registration process as installer, you get to set up a customer login - which I'd previously done and saved to my password manager. It is that locally generated (and locally stored) userid and password that I used to log in.
I've confirmed a couple more things - firstly, that I can re-use an old AuthCookie login, even after creating a new one! Secondly, I can log in posting to https://powerwall/api/login/Basic with {"username":"customer", "password":"P@ssw0rd", "email":"customer@customer.domain", "force_sm_off":true}, which sets a new AuthHeader cookie. So, from the logging point of view, it is log in as customer, grab the cookie, and return it in subsequent calls for things like SOC and energy flows.
@BJReplay Your instructions worked perfectly, only thing missing was username: customer. I can now get the cookies as you described and leverage them within my poller.
only thing missing was username: customer
It's there if you look carefully :)
This has broken HomeAssistant integration as well.
...Secondly, I can log in posting to https://powerwall/api/login/Basic with {"username":"customer", "password":"P@ssw0rd", "email":"customer@customer.domain", "force_sm_off":true}, ...
Any chance you can detail to a more novice how to do that? I've been trying to use cURL to see if I ca get it to get anywhere using something like the below with real email and password that works at the GUI
curl -k -d '{"username":"customer", "password":"P@ssw0rd", "email":"customer@customer.domain", "force_sm_off":true"}' -H "Content-Type: application/json" -X POST https://%powerwall_IP%/api/login/Basic
but just get > at the terminal.
PS: I know enough to be dangerous
EDIT:
So this doesn't fail at least:
curl -s -k -i -X POST -H "Content-Type: application/json" -d '{"username":"customer","password":"P@ssw0rd", "email":"customer@customer.domain","force_sm_off":false}' "https://%powerwall_IP%/api/login/Basic"
This works for me on Windows, which requires escaping of quotes on the command line:
curl "https://<powerwallip>/api/login/Basic" -H "content-type: application/json" --data-raw "{\"username\":\"customer\",\"password\":\"<password>\",\"email\":\"<email>\",\"force_sm_off\":false}" --insecure
FWIW the tokens I generated earlier today are now returning 401: Token Expired. Guess a periodic refresh will have to happen.
This pair is working for me to login, save the cookie, and then request the aggregates using that cookie
curl -s -k -i -c /home/pi/cookie.txt -X POST -H "Content-Type: application/json" -d '{"username":"customer","password":"P@ssw0rd", "email":"customer@customer.domain","force_sm_off":false}' "https://%powerwall_IP%/api/login/Basic"
curl -k -b /home/pi/cookie.txt https://%powerwall_IP%/api/meters/aggregates
If anyone is good at python and can convert these curl commands into python code, now that would be awesome :)
Any chance you can detail to a more novice how to do that? I've been trying to use cURL to see if I ca get it to get anywhere using something like the below with real email and password that works at the GUI
I've only just got it working (VB.net), but had been testing using PostMan - highly recommended.
It was a little non-obvious - I had to add a cookie container to the request for the response cookies to be populated (Grr, took a while to find that), but now back to logging SOC and energy flow.
I've also hit the 401 as well as 403. Now back to working out how to add oauth v3 to the cloud login in VB.Net.
Here you go for anyone that is interested, here is some python code that:
- Creates a session to store the login cookie
- Logs in using your customer credentials storing the cookie in a session
- Accesses the api/meters/aggregates and prints them to screen
- Accesses the api/system_status/soe and prints them to screen
import requests
headers = {
'Content-Type': 'application/json',
}
data = '{"username":"customer","password":"P@ssw0rd", "email":"customer@customer.domain","force_sm_off":false}'
s = requests.Session()
s.post('https://%testlaIP%/api/login/Basic', headers=headers, data=data, verify=False)
print("")
r=s.get('https://%testlaIP%/api/meters/aggregates', verify=False)
print(r.text)
print("")
r=s.get('https://%testlaIP%/api/system_status/soe', verify=False)
print(r.text)
print("")
s.close()
Thanks to everyone for their investigations - particularly @GBDownUnder.
I can confirm that the curl recipe from @GBDownUnder works for me. I'd wondered if "username":"customer" should refer to my login name, but literal "customer" is correct.
Like others, I'm wondering how long the cookie is valid for. I see that the AuthCookie is regenerated on every login. It would be nice if this was a long-lasting cookie - ideally indefinitely.
As it happens I'm using Java to access the gateway locally, but I should be able to take it from here :-)
I believe the "username":"customer" relates to the option on the GUI login page that you select between customer and installer.
I have my token refreshing every hour and it worked well overnight, never had an auth or connection issue. May look to turn it down to 2 hours but not sure it matters.
Just as a quick note the email address is unneeded for customer logins, I don't believe it's checked. I've been using a made up email address with the correct password and username as "customer" and it's been working fine.
My ideal solution would involve an Authorization token of some kind, since telegraf's http plugin has support for reading the Authorization header value from a file. It doesn't have support for getting cookies or any other kind of authentication workflow.
Accessing the endpoints with the login/Basic token (Bearer) results in Invalid m2m/bearer token, and accessing the endpoints with the UserRecord JWT from the cookie (Bearer) results in Invalid bearer token (User session not found from [...]).
For those struggling to do this with PowerShell... here you go:
Invoke-WebRequest -Method POST -ContentType 'application/json' -SessionVariable 'session' -Body '{"username":"customer","password":"*************","email":"user@domain.com","force_sm_off":true}' -Uri https://powerwall/api/login/Basic
$aggregates = Invoke-WebRequest -Method GET -ContentType 'application/json' -Uri https://powerwall/api/meters/aggregates -WebSession $session
I noticed in the verbose output from curl when logging in:
Added cookie AuthCookie="..." for domain powerwall, path /, expire 0
I'm hoping that "expire 0" means the cookie doesn't expire (though a timer-based repeat login would be possible).
Like @gpez I noticed that the email address doesn't seem to be significant when logging in, though the password has to be correct. I guess that the password is really associated with the "customer" login and not the customer's email address.
In case it might help anyone, I can provide key bits of Java code for accessing the gateway.
I have my token refreshing every hour and it worked well overnight, never had an auth or connection issue. May look to turn it down to 2 hours but not sure it matters.
I coded my system to close the session and try to log in again if it has any issues reading the soe or aggregate (I put it in the error handling) however I think the token/cookie lasts quite a while. I have a tablet displaying the GUI on it 24/7, I logged into the PW on that about 20 hours ago and it hasn't lost its connection yet.
Interesting about the email not needing to actually be 'valid'.
Has anyone figured out how to resolve authentication within telegraf? This is my input:
[[inputs.http]]
interval = "3s"
urls = [
"https://192.168.0.51/api/system_status/grid_status", "https://192.168.0.51/api/meters/aggregates"
]
method = "GET"
insecure_skip_verify = true
timeout = "1s"
data_format = "json"
json_string_fields = ["grid_status"]
name_override = "aggregates"
#to have grid_status and aggregates in the same measurements we need to exclude url tag
tagexclude = ["url"]
# Keep all measurements that end with "instant_power"
fieldpass = ["*instant_power", "*energy_*", "grid_status"]
#drop empty fields
fielddrop = ["load_energy_exported", "solar_energy_imported"]
#tag for routing
[inputs.http.tags]
influxdb_database = "electricity"`
For telegraf, for right now, you'll need to sign in and get a cookie (either through /api/login/Basic or by logging in the web UI and grabbing the cookie from your browser's web console).
Once you have that, you can set the Cookie header:
headers = {"Cookie" = "AuthCookie=DUnlqjVfQpc[...]; UserRecord=eyJlbWFpbCI[...]"}
Mine has been running successfully for over a day now, but I have no idea how long the cookie will be good for 🤷♀️
@jplivingston08 Sorry mate, no idea. My HomeAssistant PW component is just reporting "Unavailable" for all components and the web panel display of the GUI powerflow won't even let me log into it to display (I used to be able to accept the certificate in another tab and then refresh LoveLace and it would display power flow but that doesn't work at all now).
I have lodged a support ticket with Tesla about what they have broken with this Firmware changes. I wouldn't hold my breath on them caring about us and restoring it on future updates somewhere, in the mean time a lot of things have broken because of this.
@GBDownUnder - thanks for posting both the curl and python examples - both worked for me. I was only caught out by the password which is the PW password not my Tesla App password - obvious after the event! I had a go at converting the scripts to node red but have ended up simply running the python script as a cron job and reading a log file back to node red. At the moment I'm running the full script every 5 minutes including refreshing the cookie, which doesn't feel right to me. Would you by any chance have a split version of the python script (as per your curl example) that I could use to log soc every ~5 mins and then refresh cookie every ~24hrs. From a point of zero knowledge about such things it just feels like a better way of running it!!
What do other people think about the maximum data rate? If getting the cookie is all happening locally then maybe it doesn't make any difference - but if getting a cookie means hitting the Tesla servers - then every 5mins is perhaps too fast?
@oldironUK hey there. I am not sure what you mean by 'a split version' of the script, however if you go to my page here https://barwood.net.au/teslapanel (which is what I have been making and the reason I landed in this site in the first place), in there I have a download for a python script I am using to get the data to display on the LED panel.
What I have done is to add a generic level error catch, which first tries to close the existing session and try to log in again. It means there may be a flicker outage for a second but if it is just a cookie time out, it should automatically create a new session.
I can't say this approach has had the chance to be tested yet as I don't think the existing cookie has expired yet.
Cheers
Guy
Well my auto re-login code didn't work, I found my display showing no output this morning, about 2 and a half days after starting it. I'll work out why in time :) A reboot restarted my script and it came back to life. As I said from the start, I know enough to be dangerous :)
It's not perfectly clean but I simply cache my token and keep track of when it was generated. I'll then internally expire it (right now its lifetime is 1 hour) before the next aggregates call and regenerate it first then make the aggregates call.
My code is C#:
if(DateTime.Now - _LastAuthRefresh > TimeSpan.FromHours(1))
{
RefreshAuthToken();
}
if (!string.IsNullOrEmpty(_AuthToken))
{
var aggregatesTask = GetAggregates();
var chargeTask = GetCharge();
var gridStatusTask = GetGridStatus();
Task.WaitAll(aggregatesTask, chargeTask, gridStatusTask);
}
Edit: Oh no, I just listened to a voicemail from Tesla from a missed call yesterday. They have DOWNGRADED my firmware to fix this.
The question is now what 'good' did I loose from the downgrade, how long till I get upgraded again and it breaks again etc...
I didn't even think about the numbers, the last firmware was 20.49.0.
Can anyone confirm if there is NO LONGER a need to log into the powerwall to get the aggregates and soe???
My Gateway firmware is now reporting 20.40.3 and I no longer need to log in to see the powerflow or read the aggregates or soe
I had lodged a ticket with Tesla when this happened, I missed a call from them yesterday, I guess it might have been because of this but it seems like they have RESTORED previous capabilities.
My home assistant is also now displaying the output again!
Looks like Tesla rolled back this change
@GBDownUnder Did I get you right, they downgraded your PW from 20.49.0 to 20.40.3?
@emeins 100% correct
Hmm, my one is still running with 20.49.0 and the old certificate based auth is still responding with an error... as expected.
{"code":403,"error":"Unable to GET to resource","message":"User does not have adequate access rights"}
@emeins yeah sorry I got excited as mine stopped needing auth to get the readings, I assumed they had upgraded me, only to realise they had downgraded me.
I had lodged a technical support issue with Tesla about this issue, this was their solution. I have asked if they did this until they have a new firmware without the issue or am I stuck on 20.40.3, It will take some time for a response of course.
Yep understood. For me your cookie based logon is working well since days with an 2-hour refresh of the cookie. So I am fine for now.
Yup it’s broken for almost a month but I opened the issue to share code incase anyone had issues
Any chance you can detail to a more novice how to do that? I've been trying to use cURL to see if I ca get it to get anywhere using something like the below with real email and password that works at the GUI
I've only just got it working (VB.net), but had been testing using PostMan - highly recommended.
It was a little non-obvious - I had to add a cookie container to the request for the response cookies to be populated (Grr, took a while to find that), but now back to logging SOC and energy flow.
I've also hit the 401 as well as 403. Now back to working out how to add oauth v3 to the cloud login in VB.Net.
Would you be so kind as to share your vb.net code? I've managed to get the token but I don't know how to use that with regards to a cookie and subsequently using it to get the SOC. I'd really appreciate some help. Thanks!
The colleages here seemed to get a working version - maybe this helps someone
https://openwb.de/forum/viewtopic.php?f=9&t=2491&p=25291&hilit=tesla+powerwall#p25291
This commit seems to have fixed the login: snaptec/openWB@ca90b3b
Would you be so kind as to share your vb.net code? I've managed to get the token but I don't know how to use that with regards to a cookie and subsequently using it to get the SOC. I'd really appreciate some help. Thanks!
Sure - see here https://github.com/BJReplay/PowerwallService/blob/master/PowerwallService/TeslaAuth.vb based on excellent C# example provided here https://github.com/tomhollander/TeslaAuth with call demonstrated here https://github.com/BJReplay/PowerwallService/blob/master/PowerwallService/PowerwallService.vb#L1027
I just got hit by this authentication issue, so I created a dummy proxy in Go that will log into the gateway for me. I just changed telegraf to point to this proxy instead of the real gateway, and I have all my statistics again.
For those struggling to do this with PowerShell... here you go:
Invoke-WebRequest -Method POST -ContentType 'application/json' -SessionVariable 'session' -Body '{"username":"customer","password":"*************","email":"user@domain.com","force_sm_off":true}' -Uri https://powerwall/api/login/Basic $aggregates = Invoke-WebRequest -Method GET -ContentType 'application/json' -Uri https://powerwall/api/meters/aggregates -WebSession $session
After seeing everyone's notes on this issue come through for the last fortnight, my own update finally happened overnight - Thanks for the heads-up, everyone!
Thankfully, it was a very minor adjustment to my existing PowerShell scripts to get things going again, making appropriate use of the SessionVariable and WebSession parameters of Invoke-WebRequest, as @BlindTrevor mentioned above.
So, expanding on that post, for anyone running their PowerShell scripts in a continuous For loop and interested in auto-token renewal but only as often as necessary, wrap just your first Invoke-WebRequest (mine is the request to SiteMaster) in a Try/Catch to renew the token only when it's required, and put your renewal request followed by that first Invoke-WebRequest in the Catch statement.
Works like a charm!
In general, I'm also not the biggest fan of any passwords being stored in plaintext at any time, no matter how innocuous the purpose, so earlier in my script, there's also a bunch of SecureString logic which decrypts the password from a pre-existing encrypted text file, and stores it in the $Password variable for use in the token request.
This requires the creation and conversion of a small custom JSON object in order to pass a variable value into the body of Invoke-WebRequest, but well worth it from a best-practice security point of view.
Also I was able to use force_sm_off as false, instead of true for this particular purpose.
I also added a command to output a datestamp to text whenever the token renewal was invoked, so I can determine how often that's actually occurring, will report back.
Try {
$Data_SiteMaster = (Invoke-WebRequest -Uri $($RootURL+$URI_SiteMaster) -Method Get -UseBasicParsing -WebSession $session)
} Catch {
Get-Date -UFormat "%Y%m%d_%H%M%S" | Out-File "$RootPath\tokenexpiry.txt" -Append
$Body_OAuth = @{
"username" = "customer"
"password" = "$Password"
"email" = "$Email"
"force_sm_off" = $false
}
$JSONBody_OAuth = ConvertTo-Json $Body_OAuth
$Data_OAuth = (Invoke-WebRequest -Uri $($RootURL+$URI_OAuth) -Method Post -UseBasicParsing -ContentType 'application/json' -SessionVariable 'session' -Body $JSONBody_OAuth)
$Data_SiteMaster = (Invoke-WebRequest -Uri $($RootURL+$URI_SiteMaster) -Method Get -UseBasicParsing -WebSession $session)
}
For
telegraf, for right now, you'll need to sign in and get a cookie (either through/api/login/Basicor by logging in the web UI and grabbing the cookie from your browser's web console).Once you have that, you can set the
Cookieheader:headers = {"Cookie" = "AuthCookie=DUnlqjVfQpc[...]; UserRecord=eyJlbWFpbCI[...]"}Mine has been running successfully for over a day now, but I have no idea how long the cookie will be good for 🤷♀️
Please could you be a bit more specific on which section this line needs to be added to? telegraf.conf is very big 😯 and as a bit of a novice I'm not sure where to put it!
For
telegraf, for right now, you'll need to sign in and get a cookie (either through/api/login/Basicor by logging in the web UI and grabbing the cookie from your browser's web console).
Once you have that, you can set theCookieheader:headers = {"Cookie" = "AuthCookie=DUnlqjVfQpc[...]; UserRecord=eyJlbWFpbCI[...]"}Mine has been running successfully for over a day now, but I have no idea how long the cookie will be good for woman_shrugging
Please could you be a bit more specific on which section this line needs to be added to? telegraf.conf is very big hushed and as a bit of a novice I'm not sure where to put it!
You'll need to add that cookie to the telegraf.conf where you are pulling from the [http] for your tesla gateway, mine is right below this block for me -
[[inputs.http]]
urls = [
"https://192.168.x.x/api/meters/aggregates",
"https://192.168.x.x/api/system_status/soe",
]
method = "GET"
insecure_skip_verify = true
timeout = "5s"
data_format = "json"
Thanks everyone for figuring this out. I've added a quick and dirty bash script in the samples directory that demonstrates logging in, saving a cookie, and using that cookie to pull stats from the powerwall based on all of your feedback here.
Check it out here: https://github.com/vloschiavo/powerwall2/blob/master/samples/powerwallstats.sh
-V
I know that this is closed, but I wanted to follow up with what Tesla Support said when I inquired about any documentation related to the change:
Tesla does not support third party or self made APIs. We are unable to assist with this inquiry.
lol, its their API, not ours!
I am still on downgraded 20.40.3.
lol, its their API, not ours!
I am still on downgraded 20.40.3.
Their position is that their API is working perfectly fine as designed and any use of that API is reserved for Tesla. Any other uses, such as us pulling data for our own dashboards, is not supported and they won't even provide any documentation around it
we have worked out how to log in to keep access anyway, so its not a huge biggie anymore.
Having to log in to see the https GUI power flow revealing the STOP SYSTEM is a major security problem though
What's annoying about Tesla is, that it is mostly served as is. Thats why for my home i chose a Fronius Hybrid inverter
But at my friends house i need the data to control an stepless immersion heater.
My current fix in node red is to read the cookie of the browser and to always deliver them in a request to the gateway.
Has someone made a node-red script that handles the cookie and data acquisition automatically?
So far i failed to save the cookie of the previous request for the repetitive data acquisition. My problem is that the powerwall is in my friends house, so i can't work from home and try out. Has anyone a working copy with automatic cookie acquisition for node red so far?