HttpClient specify interface to bind to / make requests from
narciero opened this issue · 70 comments
On a machine with multiple NICs, how can i specify (if possible) which interface IP to use for making requests?
I am sometimes on a server where the default management interface is not connected to the internet, however there are multiple other NICs that are internet-facing that I would like to be able to use.
With .NET Framework + ServicePointManager I believe you could use BindIPEndPoint.
BindIPEndPoint delegate was never part of HttpClient. But it was part of HttpWebRequest on .NET Framework.
This is not supported on .NET Core.
Are there any plans to support a multi network card scenario in .NET Core?
On a windows machine with multiple NICs how does HttpClient choose which interface to use?
cc: @stephentoub @karelz
That sounds like reasonable API request for HttpClient. Did we have any plans/idea how to eventually expose such functionality on HttpClient?
cc @geoffkizer
Can I ask why you can't just let the system pick the correct network card? What's your network topology such that the wrong NIC might ever be picked?
If the system is confused enough to send regular traffic on the management NIC, having your app pick the management NIC is the least of your worries :-)
@PeterSmithRedmond It is common (and expected) to be able to control which network interface to bind to at the application level.
A machine may have multiple NICs all configured for standard traffic in addition to the management NIC. For example, consider a scenario where there are 3 NICs labeled A - C (+ mgmt card). Lets say I would like to make HTTP requests on interface "C", subscribe to multicast traffic on interface "B", and setup TCP listener on interface "A".
This functionality is already available for TcpClient/TcpListener and UdpClient which handles the 2 non-HTTP cases above, making it easy if I decide to rearrange my setup and move my tcp listener to interface B, etc. It would be great if HttpClient could support this type of configuration which is quite common in enterprise scenarios.
as a follow up to @karelz post, are there any plans to eventually implement this? i am happy to help out, thanks.
It looks like ConnectHelper is where the necessary code should be to make this happen. In the try-catch after creating the socket (right before the call to ConnectAsync you would just need socket.Bind(localEP):
public static async ValueTask<Stream> ConnectAsync(string host, int port, EndPoint localEP = null)
{
var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
try
{
// **optionally bind to local endpoint**
if (localEP != null)
socket.Bind(localEP);
// TODO dotnet/runtime#23148: cancellation support?
await (IPAddress.TryParse(host, out IPAddress address) ?
socket.ConnectAsync(address, port) :
socket.ConnectAsync(host, port)).ConfigureAwait(false);
}
catch (SocketException se)
{
socket.Dispose();
throw new HttpRequestException(se.Message, se);
}
return new NetworkStream(socket, ownsSocket: true);
}I think the main question is how do we want the localEP argument to bubble up the stack so it can easily be configured on a per client basis?
Happy to open a pull request for this if you guys are open to including this in the near future as its a pretty small change (and already supported by Socket).
When you document this, please remind people that there's a bunch of traffic that might happen that won't go over that interface. For example, DNS lookups and getting certificates and revocation lists won't be compelled to use the IP address.
Can you go into more detail on the rationale? you say that people can set up different cards for different traffic -- is this a best practice for data centers? Normally, the best practice is to let the OS decide. What is the benefit of deliberately splitting up the traffic this way?
What is your use case @narciero aside from desire to control? In many cases you can set up routing table to have desired behavior. It is IP's job to decide packet routing, not HTTP. BTW I think the comparison to Listener does not make much sense to me. In that case you trying to create service which will wait for requests. Responses will be routed according to routing table AFAIK.
Many APIs, especially with high security requirements like payments, require IP white-listing. More common scenario is Facebook recommending to maintain Server IP Whitelist to protect Graph API calls. Routing cannot solve this issue because the target IPs aren't constant nor completely known in advance.
I'd like to voice my support for this as well. In our scenario, we have a set of dynamic VMs (one active, a few on standby), and one of the VMs is set (elected) as a current work node with a Global IP bound to it. The Global IP can be re-assigned to another VM if the current is unavailable (a fail-over event).
This creates a situation when a VM has 2 public IPs. We need to connect to a remote web server (say, WebAPI 3rd party server) which has our Global IP whitelisted. If we can bind our Global IP as an outgoing IP for the HttpClient, it will solve the issue.
Another voice of support: On our IOT use cases, we often need to select the network (e.g. WLAN vs. Broadband) based on specific criteria that may hold per request (such as expected download volume). This is crucial to our ability to connect.
The Application is on a mobile device, we often do not even know much about the target network(s) and windows doesn't either.
We don't have concrete plans for .net core in this setting but this is a blocker.
@simonthum that is interesting scenario, thanks for bringing it to our attention.
Ask for everyone: Please upvote top comment if it is a feature you need - it will help us with prioritization.
Personally I'd prefer to see this functionality supported via allowing a callback to either create or configure the Socket, e.g. along the lines of
public Func<Socket> CreateSocketFunction { get; set; }or
public Action<Socket> ConfigureSocketFunction { get; set; }Then a client could handle this scenario by calling Bind on the Socket, but other configuration would be supported as well, via either what Socket provides or even platform-specific via P/Invokes.
@stephentoub Yes Func<Socket> is better than Func<IPAddress>. In our IOT setting we sometimes connect over several networks to check which network is able to connect to the target. The first answer wins, the rest disconnects.
It is a fall-back, but even then in a Func<IPAddress> I would be forced to turn down an established connection and hope the caller will re-establish it. This makes an already ugly fallback strategy considerably more ugly.
I think it would need to be something like:
public Func<Socket, string, string, int> ConnectFunction { get; set; }The arguments here being scheme, hostname, and port.
Or even:
public Func<Stream, string, string, int> ConnectFunction { get; set; }Which would allow you to return an arbitrary stream.
There are a few details that would need to be worked out:
(1) How does this interact with proxy lookup?
(2) If we allow you to return an arbitrary stream, then how does this interact with SSL handling? I'm guessing that the specified SSL options are ignored and you can just use whatever options you want. Also, we need to retrieve stuff like ALPN result from the SslStream in order to do HTTP2, but I suppose we could just test if the resulting stream is SslStream and if so, retrieve that info.
Is there any progress with this issue? We're stuck by not being able to explicitly specify an ip address for a query. We're considering using a wrapper for a C++ library as a temporary solution but it doesn't sound super exciting...
It sounds like a 100% solution would be to factor out the connection setup such that individual steps can be performed by the user selectively. I could open the socket, a stream, decide on reuse (pipelining, http/2), specify a binding address, a proxy, TLS certs etc. but not in every combination. Like a connection management interface with several default implementations to support the desired degree of control.
That would be complex, though certainly possible and useful beyond HttpClient.
I don't like the (existing) Func<IPAddress, xxx> based approach much, but it delivers 80% of the value for 10% of complexity.
Sorry guys for a likely stupid question but I'm a little lost here. On the one hand, this issue says HttpClient doesn't support specifying an explicit ip address but then I see issues like this one (restsharp/RestSharp#1162) that shows seemingly that you can change an ip address
@siberianguy You can using HttpWebRequest which (I presume) is not in .net core and quite ancient.
I think it would need to be something like
Sure, I was simply highlighting using a Func that returns a Socket or an Action that configures one. We could figure out what arguments to pass in. I could imagine you might want more than just the scheme/host/port in some situations, as well.
Which would allow you to return an arbitrary stream.
That's a whole different ballgame. Just providing a delegate to a configure a socket, or even to create one, leaves the whole connect mechanism up to the SocketsHttpHandler implementation, and as it implemented the connect, it can make assumptions based on it. Elevating that to a delegate that creates the Stream is potentially more powerful, yes, but also is more intrusive, forces our hands on various SocketsHttpHandler implementation details (we may no longer have access to the underlying NetworkStream, we can no longer assume SslStream in cases where we cast to it, etc.), etc., and causes the questions you answer. I've also not seen a use case that would demand this level of control.
In contrast, there are known uses for being able to at least configure the socket before the connection attempt is made, and even creating the socket itself (e.g. whether to create it only for IPv4 or IPv6). I'm sure we could come up with use cases for a custom stream, I'm just not yet convinced its worth it.
Whatever we do, though, we should probably do it in conjunction with https://github.com/dotnet/corefx/issues/27949, or at least factoring that in, as that could also impact how the socket is created.
It may also help with cases like dotnet/corefx#31951 when someone may want more aggressive retries.
@stephentoub Yes, I agree with your points. I was mostly just tossing out some ideas above.
there are known uses for being able to at least configure the socket before the connection attempt is made, and even creating the socket itself
Agreed. To answer my own questions above, I don't think this should affect proxy behavior or SSL. It should just affect how connections are created.
Re configure socket vs create socket, it seems to me there are compelling cases for creating the socket, so I lean toward that model. Basically, something like a Func<Socket, DnsEndpoint>.
I'm sure we could come up with use cases for a custom stream, I'm just not yet convinced its worth it.
I basically agree. We should lean toward being conservative here, i.e. Socket. That said, most of the code internally is Stream based anyway because it needs to run over SslStream as well as NetworkStream. But I could go either way on this.
Is there any hope with this issue? Honestly, it's beyond my understanding how we've got so far with .net core without having a key feature for many enterprise solutions
Why do you think it impacts "many" enterprise solutions? I see only 8 upvotes on the top issue, also we rarely hear about it from other channels.
The bottom line is that it is on our backlog, but not too high up. We have larger things to finish first: HTTP/2, proxy support for enterprise scenarios, etc.
I hope we will be able to get to issues like this post 3.0.
There's a long list of use cases which require choosing an ip address for a request. And it's not only complicated enterprise solutions. Here's our scenario. We're working with crypto exchanges and we have to be very careful with the amount of requests we're making from one ip address, so it's vital to be able to send requests under all the ip addresses attached to a server. We're struggling with the lack of this feature to say the least. May be I'm missing something but in my opinion this feature is something fundamental, so I'm confused why it's considered to be optional.
@siberianguy almost every bug/feature can be viewed as fundamental/critical in certain scenarios. The key is "How many customers does it impact?"
This is a feature which impacts narrower set of customers. We will eventually get to it, it is just not priority right now as we're working on features/bugs which impact larger number of customers. Does it make sense?
@karelz I definitely see where you're coming from but I believe the analysis should be more in-depth than "how many people are influenced by this". I'm pretty sure if we sit together and go through the changes list for 3.0 we will find a bunch of really optional things. Yes, they likely cover wider audiences but they're minor improvements that don't influence those wider audiences' life that much. Here we're talking about a fundamental network-related feature which makes it impossible to implement a significant number of use cases using .net core
I have another example:
If the 'default' network IP address is NULL routed due to a DDoS mitigation.
A web site can still be accessed via other another IP address by clients, but if the web service needs to make a server side call using HttpClient to an external resource, the call will fail to connect, since its source IP will be the NULL routed default IP on the server.
@siberianguy the analysis should be more in-depth than "how many people are influenced by this"
What alternative do you suggest?
I'm pretty sure if we sit together and go through the changes list for 3.0 we will find a bunch of really optional things
You can look at all our issues in milestone 3.0. We will need to do another round of triage in January, but I am fairly sure we marked only: Test failures, regressions and compat bugs with .NET Framework, couple of features (like the proxy for enterprise I mentioned) and HTTP/2.
@seagulledge If the 'default' network IP address is NULL routed due to a DDoS mitigation.
Can you share more details about this? I never heard of something like this - this is the first time someone brings it up. How common is it?
There was discussion while back about ability to pass socket to SocketHttpClient. Maybe we should look at proposal from @geoffkizer. That should not be that difficult to add and it solves some other use cases as well.
There was discussion while back about ability to pass socket to SocketHttpClient. Maybe we should look at proposal from @geoffkizer. That should not be that difficult to add and it solves some other use cases as well.
I don't think having callers manage individual sockets is the right approach.
.NET Framework already had a design for this to allow HttpWebRequest users the ability to control the creation of a socket when a new connection is required. It involved the use of a callback BindIPEndPointDelegate
I think we need a similar approach. Or perhaps a mechanism to allow for priority ordering of interfaces and pass those settings to HttpClient (via the handlers).
An ISP may elect to NULL route an IP address on a server when there is a severe enough attack.
Secondary assigned IP addresses can still be used on the same network card when that happens, but outbound HTTPClient calls will fail. Hard to say how common the problem is, but we ran into this issue today.
@seagulledge thanks for context. If we find out it is more common scenario and not just rare one off, we might use it to raise priority.
Such situation means that the SW has to be adapted to it as well and configured. Which is unlikely if they didn't run into it before. So it will be about how common it is.
+1
+1
I think we should just add a LocalIPEndPoint property to SocketsHttpHandler. This is straightforward to implement.
That sounds like reasonable approach @geoffkizer
I assume the caller would be responsible to monitor address changes if for example on DHCP, right?
I like the simplicity even if BindDelegate may provide more flexibility.
I am certainly adding in my vote towards wanting this functionality, it has many uses, such as one I am trying to use it for which is to make requests from varying IP addresses, to the system all the IPs are legit choices, however I have my own set of progamatically dictated rules to follow ontop of that, which eliminates IPs.
Also the NULL routing comment is spot on, I have used this many times, I am surprised this wasn't already considered before, this is a basic level ground rule for hosting sites and apps and protecting them from attacks of any kind etc.
Is there any kind of update as to how or when this functionality can be seen or used, as this now means I am going to likely strip out all the code I've written for HttpClient & replace it with something else which is rather disappointing to be honest.
@karelz Appreciate the heads up, I'll go back to look for another way to achieve this or stop using .net core until a later date :)
voting for this.
I'm voting for this too. I'm working on a application that needs to make many api requests (as an automated proxy for users), the api is rate limited by IP, I'd like to split request over multiple nics if needed, depending on recent api usage by the application
Was anyone able to achieve this in dotnet core? Some pointers would be much appreciated if so, otherwise I'll need to stop using dotnet core for this project :(
edit: meant dotnet core
@tundeanderson It not available for .NET Core only. In the classic framework you can use BindIPEndPoint
Or you can switch to a lower level and use other primitives instead.
Needed to connect to a system where the connection needed to be whitelisted but the it could only be configured with one IP.
so any chance we'll see this soon?
@tundeanderson same issue here, were you able to find a solution?
Another vote here. And please backport to full framework HttpClient to ward off reflection based solutions like: https://stackoverflow.com/questions/39689858/how-to-use-httpclient-to-send-a-request-from-a-specific-ip-address-c-sharp
@tundeanderson same issue here, were you able to find a solution?
Unfortunately not
here's an important security-related scenario: every application that allows user-provided webhooks is exposed to the risk of a hacker trying to access local ressources as in http://localhost/... - and that's harder to prevent that it looks at a first glance: see http://blog.fanout.io/2014/01/27/how-to-safely-invoke-webhooks/ and https://news.ycombinator.com/item?id=7139176
using dedicated NICs that are only routed to the internet would be the easiest way to prevent this. if only I could specify a binding.
+1 for the notion that this is just a must-have for any HTTP client. we should not have to argue our case.
+1
Triage: We plan to address this as part of #28721 ... keeping it open due to high popularity of this issue.
@tundeanderson It not available for
.NET Coreonly. In the classic framework you can use BindIPEndPointOr you can switch to a lower level and use other primitives instead.
Would you be able to specify which "lower level" primitives you were thinking of? I'm not sure where to start looking for alternatives to HttpClient
EDIT:
I ended up using the Socket class and it worked.
Amazing that in two years, a feature with all these comments and legit requests hasn't been actioned on.
Is this even going to be in 5.0 @karelz ?
@georgiosd also the request is about a common functionality where you expect it have already in by default in any framework / language.
@bonesoul it's a long thread already with arguments for either side so given that hasn't worked out too well I don't think there's a point in continuing.
I get @karelz need to prioritize based on what the majority of users "need" but frankly I think it's bad for business to piss any one group of people off for such a long time, no matter how small the group.
I'm also a bit pissed personally because I realized that even though HttpWebRequest exposes this in 3.1, it doesn't work, it seems to ignore the setting. Same code in net47 works. So basically I introduced a bug in my client's software by upgrading and I didn't even know about it until now (months later), when I was checking an unrelated issue.
Things often are not as simple as they seem. This is part of larger proposal (#28721) which morphed into more general solution in #1793 (which is in active development since November - experiments, architecture review rounds, things to consider and unify with ASP.NET models, compromises on the way). So, yes, it will be part of 5.0 unless something super horrible happens (the API is scheduled for larger API review next week).
I am sorry that you've ran into HttpWebRequest limitations -- those APIs are compat-only to ease migration from .NET Framework, quite a few of their properties are no-ops. Starting with 5.0 it seems we will have finally built-in mechanism to convey this information to developers in a reasonable way via Roslyn analyzers (tracked in #33125).
So, yes, it will be part of 5.0 unless something super horrible happens (the API is scheduled for larger API review next week).
By chance is this review going to be recorded like Immo usually does?
BTW... looking forward to your ZBB as 5.0 release nears. 😄
Yes, it should be part of normal streamed/recorded API review (cc @scalablecory).
Is this feature scheduled?
#1793 is planned for .NET 5 and should address this.
Closing this -- resolved by #1793.
This has been resolved via the API added here in .NET 5: #41949
I have simple task - detect remote IP address to which http client is connected. I don't want to DNS resolve, but need exactly remote IP.
I didn't understand how to detect remote IP in net core 3.1.
Also I have no time for reading all of topics regarding breaking changes and new API.
I have found solution for .net - https://stackoverflow.com/questions/6655713/how-to-get-ip-address-of-the-server-that-httpwebrequest-connected-to but it doesn't work as it is for dot net, but not core.
Answer please, my simple question.
I can use WebRequest or HttpClient or anything you suggest.
@scalablecory is there a sample for usage?
is there a sample for usage?
runtime/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs
Lines 2394 to 2401 in 7b93881
for anyone that may be interested a created a sample repo as an example: https://github.com/bonesoul/dotnet_5_httpclient_rest_bind
@bonesoul Nice, thanks. BTW you can just create SocketsHttpHandler directly, instead of creating HttpClientHandler and then doing the reflection stuff. The tests do the latter because that's how some of the underlying test infra code is set up.