Secure web interface becomes unresponsive, have to reboot to get working again
Closed this issue · 33 comments
Use the template below if you have an issue or want to report a bug. If you have a question or a feature request you can ignore the questions below.
NOTE: If you are having issues with your serial connection, please read this page before posting:
https://github.com/jgyates/genmon/wiki/3.6---Serial-Troubleshooting
If you are having other issues, please see the following page:
https://github.com/jgyates/genmon/wiki/3.5---General-Troubleshooting
If you need to send you logs and registers to the developer, if you email is setup and working properly you can click send your logs on the About page in the web interface.
Expected Behavior
{Please write here}
I purchased the pre-built solution. I have tried to set up the secure webserver so I can do port forwarding on my router so I can access remotely. I am told a VPN would help, but I don’t know how to set it up, and don’t know the pros and cons of doing so.
Actual Behavior
{Please write here}
After successful signon, the web interface becomes unresponsive. For example if I was on the “status” screen display and clicked on the “Monitor” menu item, nothing would happen. Then if I close the window, I am then unable to sign in again to the secure web server.
Steps to Reproduce (including precondition)
{Please write here}
Logged into the secure webserver. Web interface becomes unresponsive. I closed the browser window, then tried to log in again. I didn’t get the “connection not private” message again. It simply hung up and eventually gave the error “Server stopped responding”.
I closed windows and tried repeately. Once the interface started coming up and hung on the screen “loading generator monitor” The circular icon kept spinning around and I finally closed the window after 2 minutes.
Rebooting the pi allows me to sign in securely immediately, but it hangs up again.
So I rebooted the pi for the second or third time, then again was able to sign on securely immediately. I changed the settings to turn off the “webserver security settings” last night. After I did that, the system has worked flawlessly today. But of course I can’t access the system remotely now with the setup I have.
I will attempt to send logs, etc.
Screenshot or Pictures relating to the problem (if possible)
{Please write here}
Your Environment
- Generator Model: {Please write here}. Generac 005882, 8kw
- Generator Registers: Submit via the About page
- Genmon Version: {Please write here}
I also forgot to mention that I’ve run the “update software” twice.
In the test you describe above, you are doing this on your local network or are you going thru port forwarding?
VPN is most definitely safer and more secure. The setup really depends on your router. Most routers have this functionality build in so you can probably do a google search and find an article on how to setup your specific router with either an iPhone or android phone, then you can access genmon remotely.
Regarding your problem with secure web access, I received your logs. Nothing looks out of the ordinary. Can you duplicate this on your local network with another browser. This would tell us if there is a browser security setting or plug in that is causing the problem.
I noticed in your picture above you are addressing your pi with the ip address of 192.168.1.170.
When not using secure settings the address would be :
The address for secure settings would be:
Note the non secure address is different in 2 ways.
- http is used for non secure, https is used for secure
- the non secure uses port 8000, the secure uses the standard port.
If you do not specify https your browser is using the non secure port and it will not find anything. Most production web browsers will redirect non secure traffic to the secure site, but genmon does not do that so if you don't use the secure address it will be looking in the wrong place.
Try the above address (https://192.168.1.170) with the secure settings and let me know if it works for you.
Have you tried Chrome? Might try to get another data point.
I will try the https link in your note when I change the settings back to secure.
The link I’m using for non-secure access is:
http://192.168.1.170:7000/
I don’t know why it appears truncated in the address bar.
I changed the “Port of webserver” setting from 8000 to 7000 because I have another device using port 8000 that already has port forwarding, and my router documentation said I can only have a port number used once for port forwarding.
Is the 7000 port causing a problem?
@DC072012 I never caught that. When you turn on secure it defaults to 443 or https:etc. You can change the 443 but that's in the hidden settings page. If you click the upper right wrench icon twice, you will be on an advanced settings page and you can use a different port than 443. Good catch @jgyates.
BTW, the logs print the port used, you are using 443 (default for secure) when secure setting is enabled. Most web apps use 443 for secure and 80 for non secure. Genmon used 8000 for non secure to stay away from other conflicts. If you change the port from 8000 to something else, the secure setting will still use 443 unless you set the advanced setting as @liltux mentioned above.
You really should not have to change the ports unless something on the raspberry pi is using that port. If another computer is using port 8000 it is associated with another IP address so it should not be a conflict. Your router typically can forward a given port on your outside IP address to an IP and port on an address inside your network.
short answer is no, 7000 is not a problem, but I question if that is necessary.
I forgot the last step - I had to reboot the genmon in order to send the logs, etc.
I would recommend testing with the standard port 443, validate that this works, then, if you need to use another port then try that. Between browser security features and antivirus software blocking ports, it is hard to say what your issue is. Your logs look great, no reported errors.
One question that I have is this: when the pi becomes unresponsive, can you ping or log into the pi via ssh. This would let us know if the issue is browser related. If you are using a Mac you can open a terminal window and type:
ssh pi@192.168.1.170
you will be prompted for a password, The default pi password is raspberry, unless liltux changed it in his setup, the you should have some info on that if that is the case.
If you can log in, then the pi is still responsive. To log you type:
exit
you can also ping the pi from the consol.
ping 192.168.1.170
it should output something like this:
64 bytes from 192.168.11.170: icmp_seq=2 ttl=64 time=1.41 ms
repeated over and over. To stop the ping test type Ctrl+C
also, if you are responding via email instead of posting from the github.com forum you can not attach photos. the photos will be stripped off by github. Photos are only allowed when posting from the web site.
How is the NodeRed UI, is it ever unresponsive?
Under normal http using port 8000, does it mess up?
another thing you can do is to use the ClientInterface.py program. This is a text based python program that will show the status of the generator. this page has more info:
https://github.com/jgyates/genmon/wiki/1----Software-Overview#clientinterfacepy
I’m about 99% sure I found the problem, but I don’t know WHY it was a problem. I was going to wait a few more hours to make sure the unit didn’t become unresponsive again, then post an update.
After you had reviewed 2 sets of logs and said everything looked normal, I set the webserver port back to 8000 and the ssl port back to 443 like you suggested.
Pinging the pi worked fine. Signing onto the pi with SSH worked fine. But the problem persisted.
I rebooted my router, modem, etc and that didn’t help either.
Don’t know why, but I started going through the menus on router to see if there might be something, anything that might cause a problem. I ran across a section called “AiProtection” which was turned on. Parental controls were not on. AiProtection had a bunch of hits on “Malicious site blocking”, and more on “two-way IPS”. I don’t know what these things mean but it sounded like the router was doing something good.
I turned the AiProtection off, rebooted the router and modem again. Ever since then the genmon has worked consistently in the secure mode.
I’m a little concerned having turned off the AiProtection but that has solved the problem I was having.
Any idea why? I have screen shots, etc from the troubleshooting I did that I could post in case it might help. I had some weird things in the browser address bar with flashing ip addresses and what appeared to be error messages, etc.
The AiProtection did not identity the genmon as an error or a problem source once. But clearly it is what “broke” my secure connection to the genmon.
Any thoughts or insights will be welcome, including the risk of having the Ai thing turned off.
Thx
I have not used AiProtection, but a quick search found that it appears to be part of the asus router firmware that dynamically collects info about potential threats on your network and blocks them. This is what many anti-virus programs do. The thinking is that they collect a database for threats that is collected from their users. The downside is that they can make bad decisions when they run into scenarios they do not have data on. Also, they can sell your data they collect on your network.
Unless you have had issues with a lot of PC virus infections, you would likely be OK, but that is just my opinion. If it is right for you, It really depends on you and your network usage, etc.
There may be a setting you can can make that would make it by pass anything on a local (non internet routable) address (like 191.168.1.x).
Thanks for the info. Regarding risk, I’m unclear if the AiProtection is really necessary. My previous routers didn’t have it and we never had a problem with infections. My sense is Apple blocks a lot of stuff using something called XProtect.
Of course nothing is completely effective.
You had mentioned that a VPN is more secure for remote access. I’m unclear if a VPN would also help prevent PC infections, etc.
Thx again. Now that it’s working Genmon is a great program.
VPN is more about securing your network and preventing hackers from coming in your network. Something like AiProtect is more about trying to identify weaknesses you try to access from your network. they are 2 different approaches. If you open up ports on your router to access genmon, you run the risk of outsiders trying to access your network because you are opening a door to your network that the outside world can access. A VPN would make it much harder for them to access your network as it uses encryption keys to "lock the door", only allowing users with keys access. Just opening a port and using a password on the web interface is less secure than a VPN.
I’m hoping we can open this again as I’m having the problem again. Last Sunday the unit would intermittently stop responding when I tried to bring up the web UI. Same thing happened Tuesday. Today, pretty much all day, I’ve been unable ot access it either locally or using my port forward address. I just had to reboot the unit so I can send the logs and registers. Up until now I think the last time I was able to access the web UI might have been early this morning. When trying to access remotely, when unable to access the genmon UI, I was able to access other items on my network that I have port forwarding set up for. So port forwarding was working on the router. No idea why this is happening. The AI protection that I found earlier is off, and I only have basic firewall rules turned on in the router.
As a reminder, this is one of the pre-built units. Thx
Is AiProtection enabled in your router?
If it is, the problem is likely that AiProtetion is blocking the web app because it uses a self signed certificate.
To create a secure web app you need a certificate. You can either get a certificate from a known certificate supplier or create one yourself (self signed). You have the option of doing either with genmon. If you are using the self signed certificate option, every time the software is restarted it creates a new certificate. My guess is that AiProtection is blocking the self signed certificate. If this were a web app hosted on the internet (instead of your local network) then a self signed certificate could be considered a potential weakness from the perspective if AiProtection since the identity of the certificate provider is unknown with self signed certificates. With full certificates, security software has the added security of knowing the identity of the entity that created the certificate.
I would suggest trying these things:
- Test this with AiProtetion disabled.
- Look at the options within AiProtetion and see if disabling specific features within AiProptetion all you do access your pi
According to this site:
https://www.routeria.com/aiprotection-what-it-is-and-how-it-works/
There are way to disable some of the checking that AiProtection does. I don't use Asus routers so I can test this. There may be an option within AiProtection that allows you to whitelist a specific site. if so then you could add the pi to the whitelist and it should bypass the security checks for the pi. I don't know that a whitelist exist within AiProtection, but with many intrusion protection apps this is a common feature.
Also, when you send screenshots, you must post them in the web interface. If you send a screenshot as an attachment to an email github will strip off the attachment.
I’m sorry, my message must have not come through completely. I’m using the github web interface to post the screen shot and I see it on the last message I sent.
Also, AIprotection is completely turned off and has been for the past 19 days.
I just read the info at the routeria.com link. Given Aiprotection is completely off, I don’t think it applies.
If Aiprotection was on, there is apparently no way to whitelist anything. So I’ve had it off for the past 19 days and am still having problems.
How do I know if the problem is something within the pi operating system, and maybe nothing to do with genmon? Thx
I don't definitively know there is not an issue, however the things that make the thing this is likely an issue specific to your network are:
- The code in question (https and web interface) has been running unchanged for many months.
- I have not heard of any reports like this with anyone using genmon.
- The fact that you logs do not contain any errors make me thing that genmon is functioning properly
Let's try this:
On your mac, open firefox then pull up the web interface, then type option + command + I (option key + command key + the letter i key). This will bring up the inspector window. Hit the same key sequence to close the inspector. Once the inspecteor is open click the netowork tab. then refresh the browser wait a few secodns and take a screen shot and save it.
Then click on the inspector console tab, refresh your browser, take a screen shot.
then attached the 2 secreenshots to this thread. This may give more insight into what is happening.
Another thing we can do that would let us rule out the browser is this. On your mac, open a terminal window and type this:
curl -I https://192.168.1.170
the output should look something like this:
HTTP/1.0 200 OK
Content-Length: 2245
Content-Type: text/html; charset=utf-8
Last-Modified: Wed, 26 Jun 2019 20:29:04 GMT
Cache-Control: no-cache, no-store, must-revalidate, public, max-age=0
Expires: 0
ETag: "1561580944.49-2245-3473673333"
Date: Fri, 28 Jun 2019 03:29:48 GMT
Accept-Ranges: bytes
Pragma: no-cache
Server: Werkzeug/0.14.1 Python/2.7.13
Then post your output here. If this fails, the we can likely rule out a browser issue.
Another test from the terminal is to type:
openssl s_client -connect 192.168.1.170:443
this should display info about the HTTPS requests and the response.
I don't definitively know there is not an issue, however the things that make the thing this is likely an issue specific to your network are:
- The code in question (https and web interface) has been running unchanged for many months.
- I have not heard of any reports like this with anyone using genmon.
- The fact that you logs do not contain any errors make me thing that genmon is functioning properly
Let's try this:
On your mac, open firefox then pull up the web interface, then type option + command + I (option key + command key + the letter i key). This will bring up the inspector window. Hit the same key sequence to close the inspector. Once the inspecteor is open click the netowork tab. then refresh the browser wait a few secodns and take a screen shot and save it.
Then click on the inspector console tab, refresh your browser, take a screen shot.then attached the 2 secreenshots to this thread. This may give more insight into what is happening.
Thank you very much for the continued help. I'm attaching the inspector and network screen shots via the github web interface, so hopefully you can see them:
Another thing we can do that would let us rule out the browser is this. On your mac, open a terminal window and type this:
curl -I https://192.168.1.170the output should look something like this:
HTTP/1.0 200 OK Content-Length: 2245 Content-Type: text/html; charset=utf-8 Last-Modified: Wed, 26 Jun 2019 20:29:04 GMT Cache-Control: no-cache, no-store, must-revalidate, public, max-age=0 Expires: 0 ETag: "1561580944.49-2245-3473673333" Date: Fri, 28 Jun 2019 03:29:48 GMT Accept-Ranges: bytes Pragma: no-cache Server: Werkzeug/0.14.1 Python/2.7.13Then post your output here. If this fails, the we can likely rule out a browser issue.
Another test from the terminal is to type:
openssl s_client -connect 192.168.1.170:443this should display info about the HTTPS requests and the response.
Keeping in mind I'm currently able to access the genmon web interface (not timing out). I'm not sure why some of the output is in bold type when I posted it. It isn't bold in the Terminal. This is the output from step 1, curl -I https://192.168.1.170:
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
Next, output from openssl s_client -connect 192.168.1.170:443:
CONNECTED(00000003)
depth=0 CN = *, O = Dummy Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *, O = Dummy Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:/CN=*/O=Dummy Certificate
i:/CN=Untrusted Authority/O=Self-Signed
Server certificate
-----BEGIN CERTIFICATE-----
MIIC0zCCAbsCBAP1mm8wDQYJKoZIhvcNAQELBQAwNDEcMBoGA1UEAwwTVW50cnVz
dGVkIEF1dGhvcml0eTEUMBIGA1UECgwLU2VsZi1TaWduZWQwHhcNMTkwNjI4MDEy
MDM3WhcNMjAwNjI3MDEyMDM3WjAoMQowCAYDVQQDDAEqMRowGAYDVQQKDBFEdW1t
eSBDZXJ0aWZpY2F0ZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKmt
1kfP71WLihqiGkA4bRuOq5D0CaX8hCXM4orhKbssJXOa1j0OMEVHCup9rEUWyYt8
HfapX5kLuGriP7eK+QBo16Fo5hcisKLXj7efP4ZjJSGPTzAGjeOX6RcgQboel3VG
OYjtdDA77fvAIrt8ffbDLUHDr1Ghlb+EX6H60zrIxAHSBQhG+HEbylhLe1q6xjDg
zInjx19pUq+ElSL8M2u8wdhQdzGajv6NwNcNQwtXXKVx9dvjeVZkEgbNSVr3xrbX
anw/pcENVFNOBiWoTPlgxB7eYQGMlwxCd+z4CKPctU29xkfWPI+qRabREUCSeik+
9oXj0dMlGv/V471dUPsCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAMdcSKkqXrhxy
Af1mfRaDbkn3V+dF/dX9UgNd56GAQYG/Kk0dJQtbr3L6otzlh220/3nLYLbgv9eV
sHmY7M/9Mz4xweZYGbxvW+L9aJPMhiile9ZoocWJXVt9vzWVYOToc/DtyzNUhzEz
5FQJ1YTH9AV0RcDYzNKodyWeVGBkBDDkvTepYAGqHG8AtrJOUn5hIOApqNdyvm6b
TULy7gZDla23CJHVquUdgPO41S5jybPM0msqwE53MWeyf2TADQhk/+M3DVeZLTEy
+NC6YNmW4slIRCcLS4jj1LG/brt12eHh0BOxlhoSe38ZGLR9m6bblrSb3EwE5h18
0dqOlJAoZQ==
-----END CERTIFICATE-----
subject=/CN=*/O=Dummy Certificate
issuer=/CN=Untrusted Authority/O=Self-Signed
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
SSL handshake has read 1348 bytes and written 293 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 08B54D11810AA9F54D5F2C3E4CF4A759F4E27FFDFB79F5C6CF0BF58AB8FAD10D
Session-ID-ctx:
Master-Key: 5989546B24A42929818C2DD6FFBC72E071C547AD1AA8709F5E0F2AE7388C25B7AD8C6F3FD10A8E715287D9A9555422AC
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - a3 be cd e5 08 90 e6 d7-76 91 d8 13 b7 d2 54 0a ........v.....T.
0010 - 06 a2 23 0f d6 b4 76 37-e4 46 32 5d 35 22 43 81 ..#...v7.F2]5"C.
0020 - 48 5c 52 89 70 29 4c ec-3e 46 82 40 3d f1 f5 2f H\R.p)L.>F.@=../
0030 - 36 b9 c7 73 03 bb f6 c6-b3 88 c2 1e ce d1 2d 8c 6..s..........-.
0040 - 22 d5 de 44 15 66 3d b4-af 6b 6d 8e c5 76 57 be "..D.f=..km..vW.
0050 - b9 ac 30 4e ca 6b a7 a3-52 6b 67 95 dd be 1d 44 ..0N.k..Rkg....D
0060 - b4 9d f3 3a 3b 3e 15 e7-4d 70 51 17 37 6a 19 42 ...:;>..MpQ.7j.B
0070 - 9c ac 62 e6 25 91 96 df-c7 1f e1 14 26 46 50 31 ..b.%.......&FP1
0080 - 00 90 b5 8a c9 85 47 18-77 38 37 01 0a d7 13 69 ......G.w87....i
0090 - a0 ef d2 5f b8 ef 16 54-48 b1 cf b0 dc 24 70 a9 ..._...TH....$p.
Start Time: 1561731829
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
All of this look normal. I would need to see the same info when it is not working for you. Also, instead of the "Inspector" screenshot, send the "Console" screen shot.
OK, next time it becomes unresponsive I will repeat the process except sending console instead of inspector. thx
Probably would not hurt to make sure you router firmware is up to date. What router model and firmware version do you have?
Router is an Asus RT-AC66U B1 (which has the same CPU and hardware as the RT-AC68U). Dual core 1Ghz cpu. I have Merlin firmware on it version 384.11_2. I see there is an update but the version I have is relatively current.
I can try updating to the most recent firmware in the next day or so.
I don't know if it would help to do a factory reset and set everything up from scratch, but that means our network here will have to be down for about an hour while I'm working on it.
OK, I’m back......
Liltux and I have been working on this problem offline for quite a while. I think it’s finally solved.
But I have no idea why it was a problem.
I was using port forwarding as noted in earlier posts, using a DDNS service provided by ASUS (the maker of my router). Grasping at straws I looked for anything in my network setup that could, might, cause a problem. The only thing I hadn’t done is try a different DDNS provider.
I changed from ASUS to NO-IP.com and for 2 1/2 weeks had no problems at all.
So, why would changing the DDNS provider make a difference? Is anyone with an ASUS router using their DDNS service going to have a problem?
Right now I’m test driving a Synology 2600AC router to see how their onboard VPN servers work. While I’m testing I still am using DDNS to port forward to the genmon pi, but I’m now using the Synology DDNS service. I’ve had no problem using their service.
Not sure if this information might help others in the future, but it was an extremely frustrating process to finally get this working.
Thanks for the info. That does sound odd. I guess ASUS is trying to pack too much functionality and skimping on the testing / QA.