api.ipify.org shows wrong IP
jonashaag opened this issue · 21 comments
On ipify.org:
$ curl 'https://api.ipify.org?format=json'
{"ip":"46.x.x.17"}
46.x.x.17 is my correct external IP. This is also what https://ident.me/ shows.
On api.ipify.org:
172.16.x.x
This is the internal/local IP.
Is it possible that your computer/server is located in the same data center where api.ipify.org is located?
Paste a traceroute to api.ipify.org here.
Are you using any floating IPs or similar technologies?
Hm, this is going to be difficult to debug – I was on a train when this happened. I'll see if I can reproduce this next time.
To be specific, a German ICE train with Wifi operated by http://www.icomera.com/
This is really odd. This means that your connection was somehow getting the X-Forwarded-For header set to an internal IP (very weird).
The way ipify works is by reading the incoming IP address from the load balancer. So, it's possible this was forged in the TCP request.
Good news: I can reproduce this all the time.
Bad news: There's probably not a lot you can do.
With HTTPS:
* Trying 23.23.102.58...
* TCP_NODELAY set
* Connected to api.ipify.org (23.23.102.58) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: api.ipify.org
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
> GET /?format=json HTTP/1.1
> Host: api.ipify.org
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Cowboy
< Connection: keep-alive
< Content-Type: application/json
< Date: Thu, 01 Jun 2017 18:17:05 GMT
< Content-Length: 21
< Via: 1.1 vegur
<
* Curl_http_done: called premature == 0
* Connection #0 to host api.ipify.org left intact
{"ip":"46.x.x.8"}
With HTTP:
* Trying 54.235.148.27...
* TCP_NODELAY set
* Connected to api.ipify.org (54.235.148.27) port 80 (#0)
> GET / HTTP/1.1
> Host: api.ipify.org
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: Cowboy
< Content-Type: text/plain
< Date: Thu, 01 Jun 2017 18:19:13 GMT
< Content-Length: 13
< X-Cache: MISS from IMP-cache
< X-Cache-Lookup: MISS from IMP-cache:3128
< Via: 1.1 vegur, 1.0 IMP-cache (squid/3.1.20)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
<
* Curl_http_done: called premature == 0
* Connection #0 to host api.ipify.org left intact
172.x.x.32
So there's some internal IP rewriting going on here -- no caching though:
curl 'http://api.ipify.org/?random=ooTh7aighaeB2ziez5uy'
172.x.x.32
Interestingly, 54.235.148.27 == herokucdn.com
your http calls gets tunneled huh? 😢 Is is a MITM or similar?
Yeah, probably some internal caching going on, understandably. Though the caching itself doesn't seem the be the problem here, but the side effect of changed headers
I have the same. http shows the Forwarded IP, https shows the public IP.
I can check the difference with http://www.watismijnip.nl/ (in Dutch).
It shows the difference in IPadress and ForwardIP.
Cheers, Airell.
This is happening due to the way the headers are being sent. If those headers are modified (X-Forwarded-For) ipify won't get you the right results.
In the long-term, something we could do is modify ipify to NOT sit behind a load balancer, and instead grab the direct connection IP. That's a bit of a tricky thing to do, however, due to the way ipify is scaled/architected currently.
OK, having debugged this a bit further, here's my best guess on what's the problem and how to fix this.
The cache MITM sets the X-Forwarded-For header to 172.x.x.x for non-HTTPS requests. Your internal load balancer seems to consider this the client's external IP (which is 46.x.x.x). Solution: In your frontmost load balancer, if a request has X-Forwarded-For set, replace its value with the actual external IP.
Basically, what you've already said. I wonder why you can't modify the load balancer settings, though?
The reason this isn't doable right now is the architecture of the deployment. ipify is deployed to Heroku, which uses a custom load balancer that proxies through the value of X-Forward-For. If a client OVERWRITES that field, it invalidates what the load balancer does to detect the source IP, because it assumes the client WANTS to pass that new value through (which it does).
This is good for proxy use cases in general, but not ideal for a service like ipify.
The only solution is to migrate off Heroku to handle something like this. I did some experimentation a while ago with:
- Using digital ocean to run ipify servers
- Running HAProxy on DO and configuring it to grab source connection IP, and pass that through exclusively to the app servers
- Then the ipify servers receive the x-Forward-For header properly, which cannot be spoofed
Overall, it's a big architectural change, hence the issue.
Perhaps a silly question: couldn't you just use HTTPS to get around this issue?
Perhaps a silly question: couldn't you just use HTTPS to get around this issue?
Yeah, there are many ways to get around the issue. I have simply switched to ident.me. Which also has the advantage of being much faster (warm DNS cache):
$ time curl http://api.ipify.org
0.23 real 0.00 user 0.00 sys
$ time curl https://api.ipify.org
0.62 real 0.01 user 0.00 sys
$ time curl http://ident.me
0.04 real 0.00 user 0.00 sys
$ time curl https://ident.me
0.10 real 0.01 user 0.00 sys
Re: @benyanke HTTPs doesn't solve this. The service already allows HTTPs access (and insecure, because some legacy devices that use this service don't have libssl), but that won't help if a client is getting their X-Forward-For header modified.
Let's say you make this curl request:
$ curl -H "X-Forwarded-For: 127.0.0.1" https://api.ipify.orgYou will get back an IP of 127.0.0.1 from ipify. This is because of the way the load balancers handle the incoming request header (as I mentioned above).
Sure: SSL helps with MITM which is what you're likely referring to: but if you're connecting TO A PROXY they will almost definitely be overwriting the X-Forwarded-For header which is the problem.
And re: @jonashaag I've explained above the only way to solve this. What it basically boils down to is that we'll consider moving to new infrastructure when possible. Right now there's just not enough time to do so.
And... Warm DNS cache has nothing to do with that performance difference. The difference in performance is simply because the ident.me service isn't designed for scale. Let me explain:
Check out the dig record for ident.me:
$ curl https://ident.me
; <<>> DiG 9.10.3-P4-Ubuntu <<>> ident.me
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46044
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;ident.me. IN A
;; ANSWER SECTION:
ident.me. 3 IN A 176.58.123.25
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Oct 30 10:30:20 PDT 2017
;; MSG SIZE rcvd: 53See how there's a single A record with a single public IP address? That means the service is running on a single server directly (with no redundancy).
This is fine for small services that don't need to deal with high volumes of requests, but ipify doesn't have that luxury. ipify was designed from day 1 to support billions of requests per day, and to be able to seamlessly scale as much as necessary.
Take a look at the dig info for ipify:
$ dig api.ipify.org
; <<>> DiG 9.10.3-P4-Ubuntu <<>> api.ipify.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4320
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;api.ipify.org. IN A
;; ANSWER SECTION:
api.ipify.org. 243 IN CNAME nagano-19599.herokussl.com.
nagano-19599.herokussl.com. 1541 IN CNAME elb097307-934924932.us-east-1.elb.amazonaws.com.
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 184.73.220.206
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 23.23.170.235
elb097307-934924932.us-east-1.elb.amazonaws.com. 14 IN A 174.129.241.106
;; Query time: 40 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Oct 30 10:32:59 PDT 2017
;; MSG SIZE rcvd: 188See the CNAMEs? We're currently running on 3 separate load balancers. And that's JUST the load balancer redundancy. These are massive load balancers that are not only highly redundant (as you can see), but are also distributed to multiple different availability zones in Amazon. This means in the event of massive outages, the service will still be running without problems.
When you're only accepting requests to a single server, you can get away with a small amount of lower latency because the request is only hitting one server.
When you're talking to scalable services like ipify, you are actually talking to multiple servers.
When a request goes to ipify, it first hits one of those load balancers, then is forwarded to a localized web server in the same availability zone. This is done on a round-robin basis to evenly distribute load.
The web server (running ipify's source) is then going to process the request in parallel with millions of others, and return a response to the load balancer, which, in turn, forward the response to the user.
Finally: one additional thing to note: latency depends on where you're connecting from in the world.
ipify is hosted on Amazon in the US-East region to be globally accommodating as much as possible. That location is split between EU/US West coast, and is statistically the shortest length to most parts of the world by cable (if we're talking about JUST us-based zones.
If you perform a geoip lookup on ident.me's service, you'll see it's running on a single Linode instance in their London region. This means that users who are closer to London will obviously have less latency than reaching the US east coast.
I'm on a fiber optic connection currently in San Francisco, CA, and here's my results:
$ time curl https://ident.me
64.85.227.231curl https://ident.me 0.04s user 0.01s system 5% cpu 0.808 total
$ time curl https://api.ipify.org
64.85.227.231curl https://api.ipify.org 0.04s user 0.00s system 11% cpu 0.410 totalAs you can see, there is no difference.
This is sort of a lengthy response, but I wanted to share it to explain the actual info behind all this, so hopefully you understand what's happening: I want to be totally transparent.
Finally: if you want to host ipify yourself on your own servers (for instance, if you want to spin up a Linode instance), you can easily do this (that's why it is open source!) -- this way, you won't need to worry about any load balacner issues.
-R
NOTE: ipify currently does nearly 30 billion requests per month. Moving infrastructure isn't a super simple / quick task.
@rdegges Fair enough. ident.me is faster for me, and it's more reliable for me, so I'll go with it. As a user I don't care about the underlying architecture. Interesting read though :)
Just wanted to clarify things for the sake of transparency. <3
I work with reverse-proxies (and outgoing proxies) on a daily basis. This is simply a case of following X-Forwarded-For too much.
Typically, when chasing headers such as X-Forwarded-For, you would list your trusted proxies, and it would only chase up to those proxies, and then whatever is left over is your answer. In other words, you want what your outermost trusted reverse proxy saw as the client IP.
From the code:
// We'll always grab the first IP address in the X-Forwarded-For header
// list. We do this because this is always the *origin* IP address, which
// is the *true* IP of the user. For more information on this, see the
// Wikipedia page: https://en.wikipedia.org/wiki/X-Forwarded-For
ip := net.ParseIP(strings.Split(r.Header.Get("X-Forwarded-For"), ",")[0]).String()
This is unfortunately incorrect behaviour.... more precisely, it is prone to giving unexpected results when a user has to go out via a proxy.
Let's say I'm a server in a datacentre, and to get out to the internet, I must go through a proxy. Assume that my server's IP is 10.1.2.3. I want to know what IP services on the internet will see me coming from (so I know which IP needs to be added to their whitelist). Assume that my public ip is 123.1.2.3
Currently, I would get this:
http_proxy=http://proxy.example.com:3128 curl http://api.ipify.org
10.1.2.3
In this case, Squid (the outgoing proxy), can inject my IP (10.1.2.3) into the X-Forwarded-For header, because it is forwarding HTTP.
In in HTTPS case, squid just gets a CONNECT request and relays the TCP byte-stream (which is HTTPS) once the connection has been made by the proxy. Thus, Squid cannot inject X-Forwarded-For. This is equivalent to the case of NAT (or where an outgoing proxy has set some different header).
https_proxy=http://proxy.example.com:3128 curl https://api.ipify.org
123.1.2.3
Server-side
Let's take a look at how we would enable a web-server running behind a reverse-proxy (or multiple reverse-proxies), so that it gets the 'correct' IP ('correct' meaning 'Internet-routeable IP the connection was seen to come from).
In nginx, we might have the following configuration, assuming that the connections from the reverse proxy to our server was seen to come from 10.10.11.12 or 10.10.11.13 (at the TCP layer).
set_real_ip_from 10.10.11.12/31;
real_ip_header X-Forwarded-For;
real_ip_recursive off;
If Apache httpd (2.4) is more familiar, we might use mod_remoteip:
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.10.11.12/31
(and changing any LogFormat mentioning %h to %a)
In golang, there are multiple libraries (eg. https://github.com/sebest/xff, https://github.com/stanvit/go-forwarded and others -- no idea which if any are good, those were just the first two hits)
If you're looking for a relevant standard, then check RFC7239 and its Errata:
Thanks for providing api.ipify.org!
Cheers,
Cameron
I have a similar issue - we use an expensive corporate proxy at work and ipify reports our gateway IP address and not the actual public IP, ident.me get's it right, so I will gamble on that service staying up till this issue gets closed.
I have the same issue but in a different context.
When I use my smartphone internet connection, ipify and ident.me doesn't get correct public address, not sure this this the same root cause, maybe the provider is using a proxy.
$ curl -s https://api.ipify.org
106.133.33.29
$ curl -s https://ident.me
106.133.33.29
But if I lookup for my ip through a web browser to ident.me I have a different IP address:
Same for ipify.org
Also my web server receive this ip address: 106.133.33.29
Still a bit confuse how all this works :)
I am getting an issue:
- If I am running API on the browser directly then getting IP: 157.42.119.213
- If running API on the website and getting IP from below code then getting different IP: 208.91.198.24
$ip4 = file_get_contents('https://api.ipify.org')