Are all quantities and prices transmitted and received as integers with implicit scale factors?
dforgeas opened this issue · 17 comments
Just trying a ticker request in my browser gets me:
{"last":"7813.00","high":"8000.00","low":"7630.00","vwap":"7826.60","volume":"233.6178","bid":"7794.00","ask":"7819.00"}
These look like floating point to me. Can you clarify?
The SCALE.md document apples to Coinfloor's native WebSocket API. The BIST API that you are using is an emulation of Bitstamp's API and thus uses floating-point numbers. They are handled internally as decimal floating-point, so there is no loss of precision; this is why they are transmitted as strings. Please note, we recommend that all new applications use our native API for superior performance and better features. BIST is provided as a low-friction migration path from old software written to work with Bitstamp's API.
I use the websocket API for some things but some applications are built around HTTP. I notice the WebAPI that wraps the WebSocket API is deprecated - should it not be used? It seems a better option than BIST.
The old "web API" is VERY MUCH deprecated, and our tech team is eager to decommission it. The server-side implementation is very inefficient, as it creates and authenticates a WebSocket connection per request, whereas BIST keeps a long-lived WebSocket connection per user (and the new implementation of BIST doesn't even go through the WebSocket layer anymore, although this new version isn't deployed to production yet).
Ah interesting. Does the new implementation tackle any of the limitations of the BIST API (like all requests being one market at a time, not being able to fetch closed orders, or even an order by it's ID). These could be done with extra endpoints retaining compatibility with the old ones.
The BIST API won't be changing, as it's an emulation of Bitstamp's v1 API. BIST itself is only intended as a transitional compatibility layer for existing software that was written for the Bitstamp API. New software should be using the WebSocket API, which is Coinfloor's native API (implemented closer to the core of the trade engine and in a lower-level language for higher throughput and lower latency).
Regarding your specific concerns, you can fetch all of your open orders across all markets or fetch a single order by ID via the WebSocket API. Closed orders no longer exist anywhere in the system, so there is no facility to fetch information about them. If you want to monitor when your orders fill, you should be using the WebSocket API. You can specify "persist":false on your orders so that they don't remain open if you get disconnected.
Thanks for the info.
I do use the websocket API but the use cases are different (and much more complex to implement a transactional style application, and of course impossible for a stateless one). For example a mobile app would be impossible to implement as it's not possible to know if a closed order was cancelled or filled. Even for my realtime apps I worry about network interruptions.
It is evidently possible to access this info as the BIST API does provide it in a roundabout way via transactions that include the order ID - hugely inefficient but the info is still there. I think not being able to support clients other live monitoring is a surprising omission compared to other exchanges. It's a shame as coinfloor is one of my favourites for many other aspects.
Anyway hope you see this constructively.
I do see this constructively! I understand your desire for a clean method of determining which previously submitted orders were filled. The WebSocket API interfaces only to the trade engine, and the trade engine does not have access to historical records. (Coinfloor's internal architecture is very compartmentalized.) This is why there is no method in the WebSocket API to return historical records.
I do believe that there ought to be an API method for accessing historical trade records across all markets and for returning all trade records that resulted from a specified order. My concern is that such a method would invite abuse. (Clients love to hammer on poll-style methods rather than subscribing to an event mechanism and sitting back and relaxing.) Do you have any ideas for how we could address this?
I would simply address it with rate limiting. For performance replicating the old stuff to a read optimised horizontally scalable document store would handle any number of requests you want.
This functionality is useful not only in the http API but also in websocket. I find that the state of the order book drifts over time due to missed messages, which is rectifiable by re subscribing. For one's own orders you have to keep asking for all the open orders if you want to check for missing trades, which is inefficient for coinfloor, and for cancelled orders again impossible to know about. This check operation is by its nature transactional - I need an update on an order (or set of orders) and need to wait for it before making any further decisions. I don't mind building that logic client side but I do need the tools available to do it. Another way to look at it were to make it possible from the client side to work out when a message has been missed (eg sequential messages from the order book) combined with a solution for closed orders.
If what you are saying is true, it is due to faulty logic on your side. The WebSocket API never drops messages from the trade engine; it's not possible. If a WebSocket API server ever gets disconnected from the trade engine core, the process exits, which means you would lose your connection to it. As long as your connection remains open, you are guaranteed to receive all events to which you are subscribed. We have internal processes connected to the public WebSocket API for durations on the scale of months, and they suffer no "drift."
I am willing to help you ascertain where you have gone wrong. My first guess is that you may be unaware that you can receive OrdersMatched notices on your orders even before you have received the reply to your PlaceOrder command. You would not yet know the order ID of your order at this time, so you may not be recognizing the OrdersMatched notices as being relevant to you. Note that the bid_tonce or ask_tonce field is present in all OrdersMatched notices pertaining to your own orders. (Its value will be null if you did not specify a tonce on your order.)
Not sure I follow, it sounds like you are saying you guarantee that each message is received and processed by the client before sending the next over the huge decentralised network that lies in between the coinfloor server and mine. How does this work?
I am talking about the order book by the way. Not observed this yet in placing orders (and I use the tonces to identify own orders anyway) so it can't be what you mention exactly, but it has given me an idea. If I received a cancellation before a match then it'd be possible I might re add the order to the book (I remove closed ones to avoid a memory leak, but i could flush them after some delay to avoid that). Not sure exactly, off the top of my head I can't recall what is in the ordermatched message, if it has the remaining values it'd be plausible (though pretty sure I am logging errors instead). But it does come back to what I was saying earlier: sequential identifiers can save a lot of complex logic about what order of messages is valid.
Hmm, is this the best place to be discussing this? To be honest I have not completed development so I wasn't intending to raise it as an issue with the API, the main point I was making is that clients disconnect for all sorts of reasons (some by design) and the websocket API doesn't fill that 'what happened to this order while I was gone' case is all.
Not sure I follow, it sounds like you are saying you guarantee that each message is received and processed by the client before sending the next over the huge decentralised network that lies in between the coinfloor server and mine. How does this work?
Sorry, I was trying to say that the WebSocket server never omits notifications from the stream of notifications that it's transmitting to your client, and the WebSocket server does reliably receive all events from the trade engine core. So as long your client remains connected, you will not experience any missing notifications. If your client ever does get disconnected, then you must throw away any and all state that you may have cached and start over, as there is no provision for fetching notifications that you would have received had you remained connected.
If I received a cancellation before a match
This is not possible. Once you've received an OrderClosed notice, then you will not receive any more OrdersMatched notices referencing the closed order.
sequential identifiers can save a lot of complex logic about what order of messages is valid.
The order in which you receive notifications is the order in which the events occurred in the trade engine. Notifications are never reordered. If you like, you can imagine the TCP sequence numbers in your WebSocket connection as the "sequential identifiers," although you need not concern yourself with them, as TCP handles ordered delivery internally.
- You will never receive an
OrderOpenedorOrdersMatchedfor any particular order after you have already received anOrderClosedfor that order. - You will never receive an
OrderOpenedfor any particular order after you have already received anOrdersMatchedreferencing that order.
Hmm, is this the best place to be discussing this?
We could take it to email if you prefer, although I think the issues we're discussing here would be of general interest to others implementing clients of Coinfloor's APIs.
the websocket API doesn't fill that 'what happened to this order while I was gone' case is all.
That's true. So I reiterate: “I do believe that there ought to be an API method for accessing historical trade records across all markets and for returning all trade records that resulted from a specified order.”
So you can get an ordermatched before a response to a placeorder but the open, match and close subscription events will always be in the right order? I have never noticed anything different I just thought it might be an explanation for what I have observed but so long as the send order is guaranteed for those then it's probably just the way network interruptions are disconnected in the client or something.
In any case like I say it was not meant to mean I think there is an issue, only that an application needs to deal with interruptions and therefore retrieving state, which means it applies to both to websocket and HTTP. I only mention that because we started off talking about BIST and that you said the websocket API is only concerned with the trade engine, I'm just saying it'd be cool to also have this method via socket.
Anyway back to that method. Some exchanges have a separate closedorders method, that and a getclosedorder would be enough, I see no reason to poll that method any more than what you can already do for open orders, if you are (sadly) using the BIST API to check state on an order today then you would have to be doing it on the open orders endpoint anyway. Getorder might be more efficient for the client (returning regardless of open or closed) compared to repeatedly calling open orders to figure out when an order is filled l, but I don't know if clients really do that.
So you can get an ordermatched before a response to a placeorder but the open, match and close subscription events will always be in the right order?
Correct. You can think of your command/reply dialogue with the server as a separate stream that is independent from the stream of asynchronous notifications being delivered to you. That both streams are multiplexed over a single WebSocket connection is a performance optimization.
it was not meant to mean I think there is an issue, only that an application needs to deal with interruptions and therefore retrieving state
Ah, I understand now.
I'm just saying it'd be cool to also have this method via socket.
I agree that it would be really handy to have historical records available via the WebSocket. Although it is against Coinfloor's internal security policy for the trade engine to have access to the record-keeping system, and it would be impractical for the trade engine to duplicate the record-keeping system, a possible future enhancement might be for the trade engine to remember information about the last several trades for each user. You wouldn't be able to fetch a complete trade history via the WebSocket API, but at least you could check what happened to the last batch of orders you had open. I'll have to analyze the feasibility of implementing this.
compared to repeatedly calling open orders to figure out when an order is filled
Why would you do this? You should only need to call GetOrders once per connection. Once your connection is authenticated, you will be notified of all changes in the set of your open orders. If by "repeatedly," you only mean "upon (re)establishing a connection," then there is no problem there, and the efficiency isn't a big concern, as authenticating a connection has a much higher overhead than requesting your open orders.
Sorry it's late here and I was switching contexts. I was referring to the issue of clients using the HTTP API to poll frequently, and how to avoid that for a getorderbyid endpoint. I was just speculating that any client today using BIST to poll for changes to orders would have to do it by polling getorders. I myself would only need to retrieve order state infrequently.
To be honest I think the main impediment to using websocket is it is harder to implement. Fewer client libraries like CCXT, more varied implementations across exchanges, and a different mental model. Also typically there are a mix of transactional and notification paradigms, and transactional ones take a lot more code to do reliably than an HTTP call. It's just more effort basically.
Wouldn't polling a GetOrders HTTP resource generate less overhead than polling a GetOrderByID HTTP resource, given that you'd have to do the latter for each order you have open?
Generally, polling is evil, and no one should do it ever. 😁
To be honest I think the main impediment to using websocket is it is harder to implement.
Interesting. It's a W3C standard, so I'd expect plenty of library support.
a different mental model
I think this might be more the issue. It's a different way of doing things than the old HTTP request/response model, and even though the WebSocket model is more appropriate for an exchange, many clientele remain more comfortable in the old model.
It might interest you to know that Coinfloor has had an Event Stream API publicly available for quite a long time, but it can't be publicly documented yet because there is a breaking change that is stalled in the deployment pipeline. As soon as that change hits production, I will be publishing documentation of this Event Stream API, which would give you an alternative to the WebSocket for receiving notifications of events pertaining to the order books and your orders and balances. You might find that Event Stream is easier to implement than WebSocket.
That's interesting thanks for the heads up.
APIs is my day job so have witnessed the evolution with interest. I guess not all W3C standards are equal, and at this point in its lifecycle HTTP is hard to match for implementation support! Low level websocket libraries are plentiful enough though, it's the higher level functional libraries (ie crypto exchange libraries) that provide some extra leverage. Implementing it in the ccxt async lib could be helpful there. Anyone who's doing anything serious would gravitate towards the websocket API anyway, but there are fewer conventions shared between exchanges so it can be really helpful to not have to implement them one by one and code around their specific behaviours.
@whitslack appreciate this is an old conversation, but I do think that the web socket feed is no replacement for a REST API. I appreciate some customers may try to abuse it, but you can guard against that. The web socket feed has no mechanism for getting historical data. So if I lose my connection, say I'm doing maintenance or something, then I have no way to recover data on orders that were executed while I was offline.
Can we please get some much needed improvements to the REST API?