For use with SignalR Hub Client
junkomatic opened this issue · 47 comments
I am unable to get this to work with my signalR hub client, trying to bypass the cloudflare recently implemented over bittrex.com websocket endpoints. I attach the _cfduid and cf_clearance cookies, and also use the same User-Agent header on my hubConnection obj, but i still get blocked with 503 status:
`
var handler = new ClearanceHandler();
var client = new HttpClient(handler);
var hubConnection = new HubConnection("https://www.bittrex.com/");
try
{
HttpRequestMessage msg = new HttpRequestMessage()
{
RequestUri = new Uri("https://www.bittrex.com/"),
Method = HttpMethod.Get
};
HttpResponseMessage response = await client.SendAsync(msg);
if (response.IsSuccessStatusCode)
{
Console.WriteLine("SUCCESS");
}
IEnumerable<string> heads;
if (msg.Headers.TryGetValues("User-Agent", out heads))
{
foreach (string s in heads)
{
Console.WriteLine(s);
//Outputs: "Client/1.0"
//Add User-Agent header to hubConnection
hubConnection.Headers.Add("User-Agent", s);
}
}
hubConnection.CookieContainer = new CookieContainer();
IEnumerable<string> cookies;
//set the "_cfduid" and "cf_clearance" cookies we recieved on the hubConnection
if (response.Headers.TryGetValues("set-cookie", out cookies))
{
foreach (var c in cookies)
{
Uri target = new Uri("https://www.bittrex.com/");
hubConnection.CookieContainer.SetCookies(target, c);
}
}
}
catch (AggregateException ex) when (ex.InnerException is CloudFlareClearanceException)
{
Console.WriteLine(ex.InnerException.Message);
}
Console.WriteLine("CONNECTING");
//try to connect to the hub with attached cookies and user-agent header:
await hubConnection.Start();
btrexHubProxy.On<MarketDataUpdate>("updateExchangeState", update =>
BtrexRobot.UpdateEnqueue(update));
//btrexHubProxy.On<SummariesUpdate>("updateSummaryState", update => Console.WriteLine("FULL SUMMARY: "));
btrexHubProxy = hubConnection.CreateHubProxy("coreHub");`
Hi!
I'm also stuck with the same problem. Any luck with that?
I'm spend two hours for resolve this issue -_- . To fix that, you must see this post for getting cookies #14
Get it and add it to your HubConnection before calling CreateHubProxy
Sample:
_hubConnection.CookieContainer = cookies;
_hubProxy = _hubConnection.CreateHubProxy("coreHub");
// Start Connect
await _hubConnection.Start().ContinueWith(task =>
{
if (task.IsFaulted)
{
Debug.WriteLine($"We have error when connecting to Bittrex Service Hub: {task.Exception?.GetBaseException()}");
return;
}
if (task.IsCompleted)
{
Debug.WriteLine("Connect to Bittrex Service Hub successfully.");
// Handle receive message from SignalR
_hubProxy.On("updateExchangeState", marketJSon => HandleUpdatedExchangeState(marketJSon));
}
});Hi @raymondle Could you please provide full example, where you connect to the WS? :)
@raymondle, would like to know as well. This example does not work (setting the agent header and cookie container):
string agent = "";
string cookie = "";
CookieContainer cookies = new CookieContainer();
try
{
var handler = new ClearanceHandler();
var client = new HttpClient(handler);
HttpRequestMessage msg = new HttpRequestMessage()
{
RequestUri = new Uri("https://www.bittrex.com/"),
Method = HttpMethod.Get
};
// First client instance
var client1 = new HttpClient(new ClearanceHandler(new HttpClientHandler
{
UseCookies = true,
CookieContainer = cookies // setting the shared CookieContainer
}));
var response = client1.SendAsync(msg).Result;
IEnumerable<string> heads;
if (msg.Headers.TryGetValues("User-Agent", out heads))
{
foreach (string s in heads)
{
Console.WriteLine(s);
agent = s;
}
}
}
catch (AggregateException ex) when (ex.InnerException is CloudFlareClearanceException)
{
Console.WriteLine(ex.InnerException.Message);
}
//set connection
HubConnection conn = new HubConnection("https://socket.bittrex.com/signalr");
conn.Headers.Add("User-Agent", agent);
conn.CookieContainer = cookies;
//Set events
conn.Received += (data) =>
{
//Get json
//JObject messages = JObject.Parse(data.ToString());
//Console.WriteLine(messages["M"]);
Log.Info(GetCurrentUnixTimestampMillis() + "|" + Regex.Replace(data, "(\"(?:[^\"\\\\]|\\\\.)*\")|\\s+", "$1"));
};
conn.Closed += () => Console.WriteLine("SignalR: Connection Closed...");
conn.CookieContainer = cookies;
hub = conn.CreateHubProxy("coreHub");
conn.Start().Wait();
@mhamburg I'm connect successfully because i'm clone this project and modified set User-Agent in file ClearanceHandler.
As you see, why we add User-Agent to Headers of HubConnection and use same cookies but it not success? Because when you Start HubConnection, this SignalR Client will append SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0) to your User-Agent, so it let you can't connect to Bittrex Socket because not same Browers. So to fix this, you can clone this project and edit EnsureClientHeader method in ClearanceHandler.cs to
private static void EnsureClientHeader(HttpRequestMessage request)
{
if (!request.Headers.UserAgent.Any())
request.Headers.TryAddWithoutValidation("User-Agent", "SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0)");
}And remember dont try add User-Agent to your HubConnection.
P/s: My English not good, so please forgive me if i have a mistake.
@raymondle Thx! (it is now working properly)
You do not need to make changes to this library:
HttpRequestMessage msg = new HttpRequestMessage()
{
RequestUri = new Uri("https://www.bittrex.com/"),
Method = HttpMethod.Get
};
msg.Headers.TryAddWithoutValidation("User-Agent", "SignalR.Client.NetStandard/2.2.2.0 (Unknown OS)");
Just add the header before using the ClearanceHandler.
(I used the .net standard header since it is different from the .net45 header)
Thanks for the help @raymondle, i got it working as well.
The User-Agent string the SignalR client uses gets generated here, so you can derive what you need:
https://github.com/SignalR/SignalR/blob/ed4a08d989bf32a1abf06cd71a95571a8790df82/src/Microsoft.AspNet.SignalR.Client/Connection.cs#L915-L952
@mhamburg @raymondle I'm still getting 503. Could you please share your code, where you connect to SignalR server?
@tompetko Could you please post your code snippet? Only part that was wrong in my code snippet was setting the wrong User-Agent header
Thanks @mhamburg but I finally managed to get it working.
static void Main(string[] args)
{
try
{
const string feedUrl = "https://socket.bittrex.com";
const string userAgent = "SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0)";
var feedUri = new Uri(feedUrl);
var requestMessage = new HttpRequestMessage(HttpMethod.Get, "https://bittrex.com");
//requestMessage.Headers.TryAddWithoutValidation("User-Agent", userAgent);
var cookieContainer = new CookieContainer();
var client = new HttpClient(new ClearanceHandler(new HttpClientHandler
{
CookieContainer = cookieContainer
}));
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", userAgent);
var response = client.SendAsync(requestMessage).Result;
var hubConnection = new HubConnection(feedUrl, true);
hubConnection.CookieContainer = cookieContainer;
hubConnection.Start().Wait();
}
catch (AggregateException ex) when (ex.InnerException is CloudFlareClearanceException)
{
// After all retries, clearance still failed.
}
catch (AggregateException ex) when (ex.InnerException is TaskCanceledException)
{
// Looks like we ran into a timeout. Too many clearance attempts?
// Maybe you should increase client.Timeout as each attempt will take about five seconds.
}
}
@mhamburg I realized, that this works only for connecting to websocket server. When I try to subscribe, 503 is returned. Have to try subscribing?
@tompetko yep, i'm having same issue with you. Because when subscribe to market, it call request without User-Agent so that why we can't subscribe :(. I'm trying clone source and modify SignalR.Client Library https://github.com/SignalR/SignalR
@raymondle I don't believe it's caused by absence of User-Agent header value. You can wrap DefaultHttpClient, pass it as argument to Start method of HubConnection and you can see, that in prepareRequest callback of Post method is User-Agent set.
I've been trying to debug this exact problem. Haven't found a solution yet, but I know whats different between the SignalR .NET client and what Chrome is doing.
When using SignalR from .NET it seems to send the SubscribeToExchangeDeltas command using a regular HTTP request. This means SignalR fell back to using Server Sent Events. I've had this issue before when using .NET core instead of the regular .NET Framework. Something goes wrong when SignalR negotiates the connection and it falls back. What happens then is the Connect part does work, but sending the commands doesn't.
I'm not sure what part of the ClearanceHandler is causing this behavior, or if this is just another consequence of the introduction of CloudFlare on websockets.
Here's my packet dump that shows SSE is being used: WSS.pdf
Seems like it's going wrong in the WSS upgrade request:
SignalR from .NET
GET /signalr/connect?clientProtocol=1.4&transport=webSockets&connectionData=[%7B%22Name%22:%22corehub%22%7D]&connectionToken=HIZWRIqEQZGF7BwQ6IQlel9NPRAsNpxGbcOkIWxz6WY9HBekGplLSckie7Mo9jF9ZXY3cx9MVMB5tYjulT0knUwo1FliTaPyOgvkgtKso3wz3hCH HTTP/1.1
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: Glg5CpMPLQ4B5/5Zsl7tYg==
Sec-WebSocket-Version: 13
Host: socket.bittrex.com
Cookie: __cfduid=dd99c1bf6193752e41551572f5b330b621510612521; cf_clearance=c54d2ebf1bea6fd3117a9afa8c1be2bc1e219697-1510612526-10800Response:
HTTP/1.1 503 Service Temporarily UnavailableBrowser
GET wss://socket.bittrex.com/signalr/connect?transport=webSockets&clientProtocol=1.5&connectionToken=EkVK%2FdDXCiHFqxMv4KnKEhNkw%2FSaGO86LtT8ax26NZq7Ez23E6s%2FAS5sQAce0uSUmw%2BNXkQg95%2FYzaW3pz7e%2BYHjVUjfrsOO3%2FCeaIXkn05gfwu%2F&connectionData=%5B%7B%22name%22%3A%22corehub%22%7D%5D&tid=2 HTTP/1.1
Host: socket.bittrex.com
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: https://bittrex.com
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,nl;q=0.6,af;q=0.4
Cookie: __cfduid=d7053452b970221dda141e228815ff6171507369717; .AspNet.ApplicationCookie=UzgOhOvot5wbNZOcSq68pZzVbUcIw0EwzirWT0a9HWifSht1RZ1sc7dFFXi5KIpN7d2DxAD7Lwe5wKlZdhdyE1lVPEHiF-ukrtxEshe0U0MuEKgpqOCLWMZj2KSerPBmX7WptREs2tWku2DW1jJJPhL96KNrYdPuJ3oTRqn0KgLwOmfZsIt8x9m-hzbEQXMEo2LVjHFpnUgzvDhlZTzr24BHIE_ItAeanO7w4XhkltKQxp-HyBRMahpx-xmaVuRaismjdN3NI8hcjWQXM918fMmz0mspF0KJJyn8BoOQF0EIVYWdflpCPOkv5W2VJ6P8VwkdN1fmOWT10eNFWqacm5OY87pg9S9DwW1KO3Gea1zEljei3l7WaWN91-1oOY-_2zYjOdAb_6mfKeWBzPvoBbkcpd-hZIfhz12Wckw9SFYZeMuX-1N6GBavC8nXVkeNjRNgIFxnt2Jya_ciuaXVFBmag90; cf_clearance=4ff47da9e02631b7272dff29983d42a19b2eb1b9-1510612725-10800
Sec-WebSocket-Key: Ik7JJDVOM2J7jPyqvkEmdQ==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bitsResponse
HTTP/1.1 101 Switching ProtocolsMight be the User-Agent?
Related issue: SignalR/SignalR#2058
I could never get the default client to work using .net core. So I found a way to do it using a package forked from Coinigy. I used the purewebsocket and modified it to use headers after I got blocked from yesterday. Then I ran my wpf code and the user agent is unable to be set. So I then used runtime directives to do what I could. It is fully functional, however the nuget reference is in the folder as debug. Getting a strange problem with one of 3 wpf apps not finding .net HTTP. I have tested it in both .net framework and .net core. Net core minimum is 2.0 and framework is 4.7 . The repositories are located at pcartwright81/PureWebSockets and pcartwright81/PureSignalRClient.
.NET core support got me spending some time hunting for solutions too. But I believe it won't work because of this: https://github.com/SignalR/SignalR/blob/7d4e196653821ef5f26ef4e8b97f6a6eea3fe256/src/Microsoft.AspNet.SignalR.Client/Transports/AutoTransport.cs#L31-L33
I'll give the PureWebSocket a try later. Thanks!
@Grepsy did you try passing in your own new WebSocketTransport to force the hubconnection to use WebSockets and never use SSE? I've tried the following but I get the same result as you're seeing:
class TestClient : DefaultHttpClient
{
protected override HttpMessageHandler CreateHandler()
{
return new TestHandler();
}
}
class TestHandler : System.Net.Http.DelegatingHandler
{
public TestHandler() : base(new HttpClientHandler())
{ }
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
if (!request.Headers.UserAgent.Any())
{
request.Headers.TryAddWithoutValidation("User-Agent", "SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0)");
}
return base.SendAsync(request, cancellationToken);
}
}
...
hubConnection.Start(new WebSocketTransport(new TestClient())).Wait();
This problem at ClientWebSocket, they are not support add User-Agent to Headers so that why we can't subscribe or using WebSocketTransport.
@raymondle You're right. I've overriden a lot of WebSocket related classes of SignalR library just to allow setting of User-Agent header key. It's working. I'll upload working proof-of-concept solution later today, hopefully.
I have a (temporary) fix using the reverse-proxy Fiddler script to supply the User-Agent. It's pretty simple, you just add these lines using the 'Edit rules...' dialog.
if (oSession.url.Contains("/signalr/connect?")) {
oSession.oRequest.headers.Add("User-Agent", "SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0)");
}I've anyone finds a more elegant solution I'd be happy to hear about it.
Try using this websocket-sharp support custom headers at https://github.com/felixhao28/websocket-sharp. I had successfully for SubscribeToExchangeDeltas to Bittrex 😄
Hi @raymondle can I ask you for piece of code? :) I'm getting Not a WebSocket handshake response.
@tompetko i would love to see an example of your altered signalR client library, as it would be the desired solution (connect & SUBSCRIBE) for my existing .NET project which utilizes that lib (see OP). Thanks guys!
Hi @junkomatic @raymondle ! This is my promised solution. https://github.com/tompetko/SignalR.WebSocketSharpTransport. Feel free to create pull requests.
works/runs for me! it looks like the signalR client lib is intact and you added a wrapper from WebSockeSharp-customheaders lib, in addition to the Cloudflare scrape utility. nice work!
I can close this issue tonight when I integrate this workaround into my project. @tompetko if i had any money right now id tip you. Thanks very much.
@junkomatic You're really welcome! Anyway, I didn't manage to create unit tests for this solution, so any contribution would be appreciated :) So don't hesitiate to create pull requests guys!
I just finished integrating this into my project, and it totally works. Man, I was sad before, but now I am so HAPPY!
Thanks. It seems Bittrex killed "updateSummaryState" 👎
ericsomdahl/python-bittrex#57
Edit: looking at the source log from the referenced issue, "updateSummaryState" works if you're using "https://socket-stage.bittrex.com" as socket feed Uri.
that does not work for me either.
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://socket-stage.bittrex.com/negotiate?connectionData=%5B%7B%22name%22%3A+%22coreHub%22%7D%5D&clientProtocol=1.5
In the demo, leave the bittrexUri as it is. Just change the bittrexFeedUri.
using python, so probably different?
https://paste.xsnews.nl/view/vdO1nNggk73kBk
Ah. Yes, I was talking about this repo, which is c#.
https://github.com/tompetko/SignalR.WebSocketSharpTransport
To mimic what this does in python, probably you need to make a simple HTTP get request to "https://www.bittrex.com". Get the returned headers and cookies from that request and open the feed to the "socket-stage" uri, using those headers / cookies. By simple I mean using the cloudflare package, of course.
Hi guys. I made an update to https://github.com/tompetko/SignalR.WebSocketSharpTransport so updateSummaryState can be received again after subscribing to it via hub invocation of SubscribeToSummaryDeltas :)
Great, thanks.
gah cant get it to work for python yet
@p0nt can you updateSummaryState with this code?
https://github.com/slazarov/python-bittrex-websocket/blob/master/bittrex_websocket/summary_state.py
I created an issue at SignalR .NET Core repository, and they confirmed a new version (still in alpha) does support custom headers. So it seems like the simplest solution would be just to use a new SignalR.
I'm trying to build a simple proof-of-concept with a .NET Core version of SignalR. Here's what I've got so far:
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using CloudFlareUtilities;
using Microsoft.AspNetCore.SignalR.Client;
using Microsoft.AspNetCore.Sockets;
using Microsoft.Extensions.Logging;
namespace ConsoleApplication
{
class Program
{
static async Task Main(string[] args)
{
const string feedUrl = "https://socket.bittrex.com/signalr/CoreHub";
const string userAgent = "SignalR.Client.NET45/2.2.2.0 (Microsoft Windows NT 6.2.9200.0)";
var requestMessage = new HttpRequestMessage(HttpMethod.Get, "https://bittrex.com");
requestMessage.Headers.TryAddWithoutValidation("User-Agent", userAgent);
var cookieContainer = new CookieContainer();
var client = new HttpClient(new ClearanceHandler(new HttpClientHandler
{
CookieContainer = cookieContainer
}));
var response = await client.SendAsync(requestMessage);
var hubConnection = new HubConnectionBuilder()
.WithUrl(feedUrl)
.WithConsoleLogger(LogLevel.Trace)
.WithWebSocketOptions(options =>
{
options.Cookies = cookieContainer;
options.SetRequestHeader("User-Agent", userAgent);
})
.Build();
await hubConnection.StartAsync();
}
}
}
Theoretically it looks like it should work (at least you can specify custom headers) but currently I get an error "System.FormatException: No transports returned in negotiation response." I might've configured it wrong (i. e. I'm not sure if I've specified a correct feedUrl). Would love if anyone joined me on this one.
P. S. To make it build you need a nightly build of Microsoft.AspNetCore.SignalR.Client from https://dotnet.myget.org/F/aspnetcore-dev/api/v3/index.json
It turned out .NET Core SignalR client dues not support old SignalR server... I expected them to implement the same SignalR protocol but it doesn't seem to be the case.
Old SignalR itself does support .NET Core, so WebSocket-Sharp is the only problem. There project seems to be dead, but there's a PR with .NET Core support over here: sta/websocket-sharp#299
I gave up implementing .NET Core support for now. Will report my conclusions in case anyone needs it.
The old SignalR itself does support .NET Core but it doesn't support custom headers. To send custom headers we integrated SignalR with websocket-sharp. Unfortunately this library's maintainer is a little weird and he did not merge custom headers support for websocket-sharp. So it exists only in a fork. What makes it worse, websocket-sharp is not actively maintained anymore, so it doesn't support .NET Core. There is a pull-request that implements .NET Core support for websocket-sharp. So basically, to make the solution above work on .NET Core we need to take a custom-header's fork, merge .NET Core support pull-request to it and then hope there're no critical bugs as both the fork and PR are quite outdated...
I hacked together a .NET Core client using CloudFlareUtilities to pass Cloudflare clearance. I, however, noticed (and am trying to figure out why) that even after obtaining the clearance cookie any subsequent requests fail clearance again. My process is basically as follows:
- Obtain clearance cookie by navigating to https://bittrex.com:
this.cookieContainer = new CookieContainer();
HttpClientHandler httpHandler = new HttpClientHandler()
{
UseCookies = true,
CookieContainer = cookieContainer
};
this.clearanceHandler = new ClearanceHandler(httpHandler);
this.client = new HttpClient(this.clearanceHandler);
var pageContent = await client.GetStringAsync(“https://bittrex.com/“);
- Negotiate transport protocol by sending a GET request using the class scoped HttpClient from step above to "https://socket.bittrex.com/signalr/negotiate?[some querystring params]".
Now, I'd expect that in step 2 the clearance cookie obtained in step 1 would still be valid but that isn't so. The request fails clearance and has to take a new challenge. I have spent days on this now but cannot figure it out. Anyone else has any input on this? I looked into node.bittrex.api and there any subsequent http requests pass clearance after excuting step 1 without any problems.
Right - I can finally answer my own question thanks to this thread. Basically I have set a different User-Agent on requests subsequent to Step 1 above. As you can see in Step 1 no User-Agent is being specified so it defaults to the one set by the ClearanceHandler in EnsureClientHeader.
In subsequent requests I set a User-Agent which is different from the one in the very first request. I suppose that CloudFlare will always validate the cookies against the User-Agent also (maybe it is being used as a salt when generating the cfuid or clearance cookies) and hence my requests where always rechallenged. Hope this helps someone else, too!
Hi @tompetko I hacked together my own client as the .NET Core one only supports the latest server version as @siberianguy noted as well.