InsecureRequestWarning: Unverified HTTPS request is being made.
Closed this issue · 47 comments
Hi
Thanks for your awesome script. But I get this warning in my log when i run your script "InsecureRequestWarning: Unverified HTTPS request is being made." and of course your script continue to work. How can i fix this warning. i try the link for urllib3 but i couldn't get anywhere.
PS : why some site open then closed suddenly, I can get the site ip through dig and nslookup but when i open it it shows then my censorship page come up. this doesn't happened on youtube or few other sites.
Hey @arminmacx,
Sure, the reason you get this warning message is because we need to contact Google DNS via it's IP not it's hostname because of course, you have no DNS to resolve the hostname... Essentially this causes a hostname mismatch on the SSL certificate. Maybe you can work around this by adding dns.google.com in your host file and then editing the script to contact https://dns.google.com rather than the IP it currently tries to use. I am not sure if this will work, it just seems to make sense in my head. Please let me know if this works out for you.
Best Regards,
Rob
Hi Rob,
Thanks for your fast response. I did what you told me to do but still i get the warning. I don't think its need our host to be change, because the problem is with something with certificate and coonectionpool.py in urllib3. I read the link the warning gave me which its all about urllib3 and how to solve it but as far as I saw i have all preq files. but I didn't get where should i put those code in my poolmanager.py or connectionpool.py. maybe if you looking at those you figure it out.
https://urllib3.readthedocs.io/en/latest/security.html
I found this link and it says maybe your script use verify=false. i did change it but something weird happend and all complain about ssl error.
https://stackoverflow.com/questions/27981545/suppress-insecurerequestwarning-unverified-https-request-is-being-made-in-pytho
BTW why some sites for example like tunein.com don't pass my censorship system and youtube can?
Hi @arminmacx,
Yes you would need to add dns.google.com to your host file e.g.
216.58.201.46 dns.google.com
Then you would need to change the URL here https://github.com/robputt796/Py-DNS-over-HTTPS-Proxy/blob/master/https_dns_proxy/__init__.py#L11 to https://dns.google.com/resolve?
And then change the request by removing verify=False, this will then check the certificate of dns.google.com during the DNS lookups and as the hostname matches should stop the InsecureRequestWarning, in theory anyway, I haven't tested it.
I am not sure what you are going on about in terms of your censorship system. I expect you'd need more than just DNS spoofing to get around any censorship stuff, you'd probably need some proxy of some kind for all the web traffic not just the DNS.
Best Regards,
Rob
Ok, so I tried doing this and instead I get an InsecurePlatformError, this is easy to fix...
pip install pyopenssl ndg-httpsclient pyasn1
Restart the script and wellah, no more SSL warnings, seems to work great, but I did not try this using it as the system resolver... I will try this next.
Hi Rob,
Ok I test what you said and yes its stop showing that warning but now its not open lots of normal site too and its really slow. I've changed it back to your original one and its work like charm and fast.
Ok, in this case it sounds like the Python script doesn't use the host file for lookups but instead recursivley queries itself. I need to read about DNS resolution in urllib before I can think of a fix for this.
@arminmacx is InsecureRequestWarning really an issue, I mean it makes sense for the direct to IP request it is performing as the IP is not a hostname in the SSL cert, so it is understandable why it happens?
Rob,
I think of something I'm not sure if its work or not but anyway, Is there anyway to hash or encrypt the response from dns so nobody knows what we request.
PS:
By censorship i mean i can open youtube.com but in mean while i can't open tunein.com or bbc.co.uk . they are not censored by ip but with dns because i can dig and it returns its ip fine and not my country filtering system ip.
@robputt796,
No its not problem but i want to be sure if its doing something wrong with some request.
@robputt796
your script works out of box fine without any problem :)
Ok, so you want it to stop DNS lookup if the server at the other end stops looking like Google DNS and starts behaving like some other server... Interesting maybe there is a way we can check the certificate, let me look at the requests docs. Just because the certificate is not valid because of hostname mismatch doesn't mean we cannot check the validty of the certificate by comparing it to the known certificate ;-)
@robputt796
I have something else in my mind im not sure if its work or even possible.
Can we make server and client and when user send zxc.com to server through client and in server we implement zxc.com as youtube.com and server check and return youtube.com in zxc.com ?
Im not sure if i can explain it truly
The point is, if they block via DNS fine, maybe you can go direct to the IP if you can resolve the DNS, but if they block by IP, regardless of if you have true DNS the result will still be a block page... Anyway you can use a proxy, but lets explore the DNS option first, if you think they only use DNS lookup for blocking not the actual IP.
BTW this is a disclaimer - I am not helping you to circumvent your country's filtering here, I am just trying to make your DNS more secure... I cannot be held liable if you have any issue if you get caught using this in a way your country's authorities do not like. But yes I'd like to implement SSL certificate pinning here as it's probably a good thing for any user of this script.
@robputt796 dont worry about my country, And if its disclaimer and make you in bad position im not ask you to go further because i dont want anything happens to you or your account
@robputt796
but i think it would better if the response and request to google dns hash-able or encrypted because its in clear text and easly readable by man-in-middle
@robputt796
So is there anyway you can encrypt request and response?
One important thing, Is it possible for you to make cache for dns so it more faster.
Sure, so this is encrypted because it is HTTPS (SSL) but we all know SSL is susceptable to man in the middle attacks if you want to, particularly in this case where we do not validate the certificate. The problem is requests library only validates the certificate how it wants to, so this does not suit our use case, what we need todo is certificate pinning using our own mechanism, I think this can be done by subclassing some object in requests to check the SSL fingerprints or potentially maybe there is a way to get the SSL certificate information from requests without subclassing some object inside of it. Either way I am reading the docs as SSL certificate pinning would be a good feature to add to this...
Nice, I wish you to find best way and I wait for testing it. be sure to implement caching into your code so it makes it faster.
This was really a proof of concept to see if it would work, I do not have the time to dedicate to caching etc... at the moment. Maybe if I get certificate pinning to work, you could PR with caching?
Im not familiar with python, Im programming with swift for ios and mac some times.
but itry find few dns proxy which working with google https dns and have cache implemeted in them and send them to you OK
Ok....
So I added this https://github.com/robputt796/Py-DNS-over-HTTPS-Proxy/blob/master/https_dns_proxy/__init__.py#L97-L101 which basically checks if the certificate returned by dns.google.com is the certificate which is expected or not... If the certificate is not as expected it kills the DNS proxy to stop any further requests and also prints out that maybe there is a MITM attack... Anyway, if the certificate ever changes at Google DNS we must replace the Base64 encoded representation of the certificate here... https://github.com/robputt796/Py-DNS-over-HTTPS-Proxy/blob/master/https_dns_proxy/__init__.py#L16-L59. This must be done on a connection which is trusted, as if the certificate is performed on a MITM'd connection it will of course use the MITM certificate not the official one.
@arminmacx I also implemented very basic in process memory caching for 30 minutes 👍
Check it out here...
Cache Miss: live.github.com
2017-02-06 20:41:37 [DNSHandler:HTTPSResolver] Reply: [127.0.0.1:56787] (udp) / 'live.github.com.' (A) / RRs: A,A
2017-02-06 20:41:38 [DNSHandler:HTTPSResolver] Request: [127.0.0.1:49980] (udp) / 'api.github.com.' (AAAA)
Cache Hit: api.github.com
2017-02-06 20:41:38 [DNSHandler:HTTPSResolver] Request: [127.0.0.1:58722] (udp) / 'api.github.com.' (A)
Cache Hit: api.github.com
Let me know if this is what you are after.
Regards,
Rob
@robputt796 man nice work, I check it now and will let you know
BTW @arminmacx does Tor work in your country, maybe if you need a proxy this would be a method you could use...?
@robputt796 Ok im done few thing :
1 - i've change my hosts and add ip to dns address and remove verify and change the ip to dns.google.com in py file and test my connection. it works like before fine and in some site i get this error about dns cannot be reached
2 - i redo what i did and start py normally as usual and i get cache hit and miss msg and faster browsing and still some site not open
What happens if you do not edit the file and do not use hosts file and just have the warnings from the SSL cert? Do all the sites work then?
ok now after few second bbc.co.uk opened i left that page open and now i see its opened :).
what happened i don't know
Maybe it is just slow to do the lookup, remember
Normal DNS = 47ms
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$ dig @8.8.8.8 news.bbc.co.uk
; <<>> DiG 9.8.3-P1 <<>> @8.8.8.8 news.bbc.co.uk
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54721
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;news.bbc.co.uk. IN A
;; ANSWER SECTION:
news.bbc.co.uk. 427 IN CNAME newswww.bbc.net.uk.
newswww.bbc.net.uk. 73 IN A 212.58.244.57
newswww.bbc.net.uk. 73 IN A 212.58.246.81
;; Query time: 47 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Feb 6 21:23:27 2017
;; MSG SIZE rcvd: 94
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$
HTTPS DNS = 137ms
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$ dig @localhost news.bbc.co.uk
; <<>> DiG 9.8.3-P1 <<>> @localhost news.bbc.co.uk
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43616
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;news.bbc.co.uk. IN A
;; ANSWER SECTION:
news.bbc.co.uk. 465 IN CNAME newswww.bbc.net.uk.
newswww.bbc.net.uk. 186 IN A 212.58.244.56
newswww.bbc.net.uk. 186 IN A 212.58.246.80
;; Query time: 137 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 6 21:24:19 2017
;; MSG SIZE rcvd: 94
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$
I cannot do much about this as HTTPS is so much heavier than either TCP or UDP DNS.
@robputt796 nothing happens if i change my file or host its works same way but if i change file and host i didnt receive cache hit and miss, its show some hit or miss but not always and i dont get that warning error and when i change back to normal it gives me warning error and show me lots of cache hit and miss msg and both way it work as same
I know, HTTPS put a layer of security on package and header, but its interesting why it does not happens to youtube or facebook ?
the caching not works properly because i open chrome and go for site for first time and when i open safari and go for same site in few min later it says cache missing. as i see you put limitation for 30 min
I think the caching works ok no?
BTW do you know anything about dns over GRPC I found it on someone's github maybe you like to see it : https://github.com/albertito/dnss
if you saw pdnsd or bind9 they are kept dns for that amount of time you want and each time you dig that address its response back in 1 ms. i think its not working yet.
I believe the caching is working... Check these digs...
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$ dig robertputt.co.uk
; <<>> DiG 9.8.3-P1 <<>> robertputt.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37180
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;robertputt.co.uk. IN A
;; ANSWER SECTION:
robertputt.co.uk. 5439 IN A 149.202.161.86
;; Query time: 309 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 6 21:49:28 2017
;; MSG SIZE rcvd: 50
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$ dig robertputt.co.uk
; <<>> DiG 9.8.3-P1 <<>> robertputt.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34912
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;robertputt.co.uk. IN A
;; ANSWER SECTION:
robertputt.co.uk. 5439 IN A 149.202.161.86
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 6 21:49:30 2017
;; MSG SIZE rcvd: 50
MRF28PG8WN:Py-DNS-over-HTTPS-Proxy robe8437$
First one is a long time, second one returns in 1msec.
Having said that I may convert the caching to use TTL... poisoning TTL to 30 minutes is probably not a good idea as some people use it for HA failovers.
I see that and yes its work but why opening same site on 2 different browser it missed?
Or maybe im mistaken this, but for what ever it is your script is the best between all this script that i work with them and you are awesome and fast to make difference in you code and helped me
Maybe the query was not exactly the same, i mean for example
www.google.com A
is different from google.com A
and different from google.com AAAA
and so on...
But yeah I should probably make this into something more substantial than this simple script, like a proper installable and configurable DNS server...
maybe you could make all preq and package it into something that people install using pip and run without writing sudo python .....
Yep maybe, I only really wrote this to test if it actually worked, I didn't really think anyone would actually want to use it :-), but yeah i think authbind to port 53 on it's own user with a proper systemd script and setuptools stuff... would be reasonable.
actually dont make it too much hard because lots of people don't know the stuff and just want to run command and go :D.
BTW its time to update your README.md to what you did in few min ago so people knows what happens to the code and its developer still working to improve it.
PS: why you think nobody gonna use it :), I told you your script by far is the best in this job. believe me i test all others nothing compare to yours in working and simplicity
accept my apology because i ask you lots of question and not helping in anything.
Is there any way to use your script on vps. i used it and my request go to vps and i see it responding but unlike my local version its not working as local mode. I change script to listen on 0.0.0.0 and i point my iphone to my mac ip and everthings fine i can open youtube but when i run script in my vps i cant open youtube.
Hey @arminmacx
I suspect that maybe somewhere between your PC and the VPS someone is doing a tcpdump on port 53 and detecting what you are trying to visit via the plain text DNS request... This should always be ran as close to the endpoint as possible to prevent MITM.
Best Regards,
Rob
Hi @arminmacx,
Unless you have any other query I will close this issue I think I answer any of your questions.
Let me know if you need anything else, I will close this issue tommorow.
Regards,
Rob
hey @robputt796,
Thanks for your work and thank you for answering me.
Regards
Armin