Unable to get historical data from CEX.io

Question

Unable to get historical data from CEX.io

askmike opened this issue 11 years ago · 67 comments

Since a couple of days the CEX.io API does not give back old trades anymore (using the since parameter). After contacting support they stated:

Hi Mike,

We have limited our trade_history call to 1000 trades now, We will be implementing a daily trade static file in a future update I apologize for the inconvenence and we will try to get this out ASAP.

Which means that until the static file is offered we need to redesign CEX.io implementation to:

on gekko start, start gathering trades and save them in the database (this functionality is already there). But don't calculate EMA's or give any advice yet.
Monitor how many trades we have stored in the database, as soon as the oldest data we have is old enough to calculate the start EMA's (config.EMA.candles * config.EMA.interval) start giving advice and trade.

Answer 1 · 2013-12-24T05:33:27.000Z

looking forward for the CEX.io historical data fix. I truly congratulate you for an outstanding dev, this project is awesome!
Kudos,

Nico

Answer 2 · 2013-12-24T09:42:49.000Z

Thanks!

I am working on a new version of Gekko that is able to store history locally, this means it is able to monitor:

CEX.io
BTC-e (including altcoin markets)
Bitstamp (was broken for a long time)
All other API's that don't offer historical data.

So far I have written the functionality that can monitor the exchange and store candles. If people could test it that would be awesome!

Test new Gekko

Go to the dev branch and download it.
Cd to it & npm install
Set the config to either cexio, bitstamp or btce. (Mt. Gox not implemented yet, just as: real trading / mailer / advice). [edit: and pick a new market here]
Run it
History is now saved in the history folder.
Give me feedback!

Answer 3 · 2013-12-24T09:59:58.000Z

Is this okay ?
I dont know what to expect, so you just got the output here :

gekko-localDB : node gekko.js 
2013-12-24 10:48:25 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-24 10:48:25 (INFO): Let me show you some Exponential Moving Averages.
2013-12-24 10:48:25 (INFO): Using normal settings to monitor the live market
2013-12-24 10:48:25 (INFO): NOT trading with real money
2013-12-24 10:48:25 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-24 10:48:25 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:48:25 (DEBUG):    Fetched 1001 new trades, from 09:29:18 (UTC) to 09:48:23 (UTC)
2013-12-24 10:48:25 (DEBUG):    Creating a new daily database for day 2013-12-24
2013-12-24 10:48:25 (DEBUG):    inserting candle 569 (09:29:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 570 (09:30:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 571 (09:31:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 572 (09:32:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 573 (09:33:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 574 (09:34:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 575 (09:35:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 576 (09:36:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 577 (09:37:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 578 (09:38:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 579 (09:39:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 580 (09:40:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 581 (09:41:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 582 (09:42:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 583 (09:43:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 584 (09:44:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 585 (09:45:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 586 (09:46:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 587 (09:47:00 UTC)
2013-12-24 10:48:25 (DEBUG):    Leftovers: 588
2013-12-24 10:49:26 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:49:26 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:49:25 (UTC)
2013-12-24 10:49:26 (DEBUG):    inserting candle 588 (09:48:00 UTC)
2013-12-24 10:49:26 (DEBUG):    Leftovers: 589
2013-12-24 10:50:27 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:50:27 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:50:26 (UTC)
2013-12-24 10:50:27 (DEBUG):    inserting candle 589 (09:49:00 UTC)
2013-12-24 10:50:27 (DEBUG):    Leftovers: 590
2013-12-24 10:51:27 (DEBUG):    Scheduling next fetch: in 1 minutes

Answer 4 · 2013-12-24T10:08:08.000Z

Thanks for testing :)

Looks great, I'll explain some things:

Why so much logging? I am still developing it and I want all feedback I can get.
You now have a new history folder (inside Gekko) that has files like cexio-BTC-GHS-2013-12-24.db, this is all the trade information Gekko was able to store!
The times are off: no they are not, they're in in UTC. The reason for this is that everyone from over the world can share historical candles and the timezones doesn't mess everything up.

One small thing I am going to look into though: It shouldn't fetch once every minute (doesn't cause any trouble, but a lot of overhead).

Answer 5 · 2013-12-24T10:16:29.000Z

I configured it to fetch every minute, and run in debug mode to get some activity on stdout ;-)
The time fits my timezone.
Im running on OSX btw...

Answer 6 · 2013-12-24T12:30:24.000Z

Ah, that explains it. And the answers were also for clarifying the new logic for everyone.

On CEX.io [and BTC-e] you only get the last 1000 [or 150] trades, whether they happened in a minute or over the last hour. This makes it quite hard to determine when to fetch. But I quite like how the current setup deals with it: after every fetch it calculates a new fetchtime based on trades/min from the previous fetch. It appears to do well but it needs to prove itself over langer periods of time.

Here is a log where you can see this in action:

Fetched 1001 new trades, from 12:06:49 (UTC) to 12:07:19 (UTC)
inserting candle 726 (12:06:00 UTC)
Leftovers: 727
Scheduling next fetch: in 1 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:08:17 (UTC)
inserting candle 727 (12:07:00 UTC)
Leftovers: 728
Scheduling next fetch: in 2 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:09:22 (UTC)
inserting candle 728 (12:08:00 UTC)
Leftovers: 729
Scheduling next fetch: in 3 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:11:00 (UTC)
inserting candle 729 (12:09:00 UTC)
inserting candle 730 (12:10:00 UTC)
Leftovers: 731
Scheduling next fetch: in 5 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:13:52 (UTC) 
inserting candle 731 (12:11:00 UTC)
inserting candle 732 (12:12:00 UTC)
Leftovers: 733
Scheduling next fetch: in 8 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:18:34 (UTC)
inserting candle 733 (12:13:00 UTC)
inserting candle 734 (12:14:00 UTC)
inserting candle 735 (12:15:00 UTC)
inserting candle 736 (12:16:00 UTC)
inserting candle 737 (12:17:00 UTC)
Leftovers: 738
Scheduling next fetch: in 13 minutes

I do want to make it more aggressive, missing trades is not acceptable.

Answer 7 · 2013-12-24T13:55:37.000Z

I like it !
But... i miss the ADVICE in stdout - and can't figure out if it can be enabled somehow?!

Answer 8 · 2013-12-24T14:14:32.000Z

Well this new change means that I need to rewrite how Gekko deals with data under the hood. Both the historicalCandleFetcher.js and the realtimeCandleFetcher.js are going to be obsolete and they provided basically all internal communication (from fetching to passing to trading method, etc).

I am working on it, but it's a lot of work.

Answer 9 · 2013-12-24T15:23:44.000Z

I downloaded the localDB version and ran some test, but it only seems to monitor candle behaviour, is that right?
Should I send you the logs?
if you need a volunteer to test the trading feature, count me in.

Answer 10 · 2013-12-26T12:58:53.000Z

Is the test version able to trade yet or is it just at the candle/advice section for now ?

Answer 11 · 2013-12-26T15:41:31.000Z

Hey - CEX seems to be running fine, but I get this error when I try to launch with Mt. Gox now (this is with debug turned on):

2013-12-26 08:37:43 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-26 08:37:43 (INFO): Let me show you some Exponential Moving Averages.


2013-12-26 08:37:43 (INFO): Using normal settings to monitor the live market
2013-12-26 08:37:43 (INFO): NOT trading with real money
2013-12-26 08:37:43 (INFO): Starting to watch the market: MtGox USD/BTC
either start looping right away (`since`) or first determine starting point dynamically

Answer 12 · 2013-12-26T16:39:11.000Z

@bppsp1 nope, not yet. I'll gladly accept pull requests though :)

@streetlogics, the code works in that it takes different paths for different situations, the situations are:

No historical data available. (BTC-e, bitstamp, CEX.io)
Historical data available request-able via tid, while we need trades based on a point in time. (BTCChina I think)
Historical data available request-able via timestamp (MtGox)

I am still finishing up the first scenario. Though because you can always request historical data from Mt. Gox, you can do anything I'm trying to achieve already (except store the historical data) using the master branch.

Answer 13 · 2013-12-26T16:45:41.000Z

Cool cool - no worries if you're aware of it then, just wanted to let you know just in case. Happy to test again on the branch once it's implemented.

Answer 14 · 2013-12-28T10:46:57.000Z

Mike - looking forward to the trade engine implementation...
Have been running this version and just noticed that it didn't put any data in the new day file - it created them but they just had the "{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}" line in them. Restarting gekko gave me a rash of 'creating new realtime candle' and todays file got populated... Having been running since 26th, I have an empty file for the 27th and todays only got populated when I restarted.

Just another suggestion while looking at candle generation - would it be possible to also store volume weighted price (VWP) as well and also to give the option to use this in calcs rather than the centre price? This would mean iterating the trade data set but this could be calculated when assigning them to candles (and wouldn't it be as easy to pick up the OHCL figures at this point rather than using first/last/min/max later?)

Thanks

Answer 15 · 2013-12-28T12:40:23.000Z

kwest$ node gekko
2013-12-28 16:37:15 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-28 16:37:15 (INFO): Let me show you some Exponential Moving Averages.

2013-12-28 16:37:15 (INFO): Using normal settings to monitor the live market
2013-12-28 16:37:15 (INFO): NOT trading

/Users/kwest/gekko/gekko.js:60
throw invalid;
^
At this moment Gekko can't monitor CEX.io, find out more info here:

#90

Answer 16 · 2013-12-28T12:45:01.000Z

@kkwestt did you read the issue? The rest of the discussion is not about a stable version. Read more here.

@djmuk Thanks. I pushed a new version a couple of hours ago that should fix the daily DB creation. I like the idea for the VWP, will add it to the candles.

Answer 17 · 2013-12-28T13:03:55.000Z

ok
i run new gekko

172-0-0-7:gekko2 kwest$ node gekko
2013-12-28 17:03:02 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-28 17:03:02 (INFO): Let me show you some Exponential Moving Averages.


2013-12-28 17:03:02 (INFO): Using normal settings to monitor the live market
2013-12-28 17:03:02 (INFO): NOT trading with real money
2013-12-28 17:03:02 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-28 17:03:03 (DEBUG):    Scheduling next fetch: in 5 minutes
2013-12-28 17:03:03 (DEBUG):    Fetched 1001 new trades, from 2013-12-28 12:17:56 (UTC) to 2013-12-28 13:02:45 (UTC)
TRADES 1001
TRADES 12:17:56
TRADES 13:02:45
MINIMUM: 2013-12-28 13:00:00
FILTERED TRADES 16
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:05
FILTERED TRADES 2013-12-28 13:00:23
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:43
FILTERED TRADES 2013-12-28 13:01:03
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:20
FILTERED TRADES 2013-12-28 13:02:03
FILTERED TRADES 2013-12-28 13:02:04
FILTERED TRADES 2013-12-28 13:02:13
FILTERED TRADES 2013-12-28 13:02:45
FILTERED TRADES 2013-12-28 13:02:45
2013-12-28 17:03:03 (DEBUG):    inserting candle 780 (13:00:00 UTC) vol: 3.43184006
2013-12-28 17:03:03 (DEBUG):    inserting candle 781 (13:01:00 UTC) vol: 0.99916237
2013-12-28 17:03:03 (DEBUG):    Leftovers: 782

Answer 18 · 2013-12-28T16:32:38.000Z

@kkwestt I consider it stable when it runs a while on busy markets as well as empty ones. Could you let me know if it crashes or anything like that?

I think I got it almost stable now:

It now also adds empty candles, it does this to differentiate between times when gekko was off and times when there were no trades at the exchange. The result is that all candles generated before (unless there was a trade in every minute) are now considered corrupt.
fixed lots of bugs around filling gaps with empty candles (between two fetches, around midnight, etc.)
added VWP
lots of other bug fixes.

Answer 19 · 2013-12-28T17:30:14.000Z

The latest tip of localdb is borking out with JS Allocation failed - process out of memory - trying to track down what & where currently..

Answer 20 · 2013-12-28T17:48:29.000Z

@dasher very strange? Could you post (or link to) the output with debug enabled? Haven't seen that behaviour yet. Also: on what market is this happening and do you have a history build up with previous versions?

Answer 21 · 2013-12-28T18:30:26.000Z

Yep - I have debug on at all times at the mo - https://gist.github.com/dasher/b564fcb61008dbcb192a
Market: cex.io GHS/BTC

I have 2 days worth of histories from other runs - I didn't clean them out when I updated.

Answer 22 · 2013-12-28T18:33:45.000Z

Just renamed the history/cex* files to ensure it has a clean run this time around (I noticed you'd changed it a bit in the last day) and running it again now

Answer 23 · 2013-12-28T18:40:16.000Z

Renaming the old history files & rerunning appears to keep gekko happy - no crashes yet :)

Answer 24 · 2013-12-29T06:53:47.000Z

Is the bot supposed to create the history files as soon as you start it?
I'm asking because i've been running it for some hours but after telling me that "No history found, starting to build one now" it just started fetching as it should without errors but it still hasn't created any database file.

Answer 25 · 2013-12-29T09:00:29.000Z

I've been running the localDB version roughly 12 hours now and it seems to be collecting the cex.io data just fine. However, when it passed midnight it crashed (and will no longer run). Here's the stacktrace:

2013-12-29 00:18:17 (DEBUG):    Fetched 1001 new trades, from 2013-12-28 23:34:16 (UTC) to 2013-12-29 00:18:09 (UTC)
TRADES 1001
TRADES 23:34:16
TRADES 00:18:09
2013-12-29 00:18:17 (DEBUG):    MINIMUM TRADE TRESHOLD: 2013-12-28 00:00:00
2013-12-29 00:18:18 (DEBUG):    This batch includes trades for a new day.

/home/ubuntu/gekko/databaseManager.js:457
      var max = mom.minutes + 1;
                ^
ReferenceError: mom is not defined
    at /home/ubuntu/gekko/databaseManager.js:457:17
    at Function.forEach (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:3297:15)
    at Manager.addEmtpyCandles (/home/ubuntu/gekko/databaseManager.js:447:5)
    at bound [as addEmtpyCandles] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Manager.processTrades (/home/ubuntu/gekko/databaseManager.js:351:23)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Fetcher.processTrades (/home/ubuntu/gekko/tradeFetcher.js:163:8)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)

Sounds like it fails to create the new history file?

Answer 26 · 2013-12-29T09:54:49.000Z

It's still not creating any database for me.
I've tried to run it on both windows and linux and i've tried with different versions of node.js but the result it's still the same: the bot just continues to fetch and schedule without ever creating the database and without throwing any error.

Answer 27 · 2013-12-29T09:57:23.000Z

@bizio The history directory is created in the current working directory by default. Check your config.js and put an explicit path in there?
Askmike - is it worth printing / logging the full path of the history file when it is created?

Answer 28 · 2013-12-29T10:16:05.000Z

The path for the history directory is the default one and the folder gets created automatically if it doesn't exist or i delete it manually, the problem is that the bot just doesn't create the database files.

Answer 29 · 2013-12-29T10:56:11.000Z

@bizio55 There was a bug in the latest version I pushed 10 hours ago, should be good now.

@iceydee That also has been fixed, thanks for the feedback!

@djmuk Since you can now configure the location of the history directory I think this is a great idea. But note that the massive logging we have right now is just to make it easier to debug. As soon as this runs stable I'll remove most of the logging (but creating a new DB is something that happens once a day top)

Answer 30 · 2013-12-29T11:12:28.000Z

I'm using the main gekko branch for trading at btc-e and downloaded this one for testing. Running both on a rasbperry pi (node 10.2).

Dev branch seems to have started fine:

2013-12-29 13:02:06 (INFO): Using normal settings to monitor the live market
2013-12-29 13:02:06 (INFO): NOT trading with real money
2013-12-29 13:02:08 (INFO): Starting to watch the market: BTC-e USD/BTC
2013-12-29 13:02:10 (DEBUG):    Scheduling next fetch: in 2 minutes
2013-12-29 13:02:10 (DEBUG):    Fetched 150 new trades, from 2013-12-29 10:51:04 (UTC) to 2013-12-29 11:02:06 (UTC)
2013-12-29 13:02:10 (INFO): No history found, starting to build one now
2013-12-29 13:02:10 (DEBUG):    Creating a new daily database for day 2013-12-29
2013-12-29 13:02:11 (DEBUG):    minimum trade treshold: 2013-12-29 00:00:00
2013-12-29 13:02:11 (DEBUG):    processing 150 trades
2013-12-29 13:02:11 (DEBUG):    inserting candle 651 (10:51:00 UTC) vol: 0.4267344 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 652 (10:52:00 UTC) vol: 0.7105197 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 653 (10:53:00 UTC) vol: 1.0400067 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 654 (10:54:00 UTC) vol: 0.0920233 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 655 (10:55:00 UTC) vol: 0.458507 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 656 (10:56:00 UTC) vol: 0.1845 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 657 (10:57:00 UTC) vol: 7.978032 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 658 (10:58:00 UTC) vol: 0.56077001 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 659 (10:59:00 UTC) vol: 6.07177633 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 660 (11:00:00 UTC) vol: 6.92122311 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 661 (11:01:00 UTC) vol: 1.6634326 2013-12-29
2013-12-29 13:02:11 (DEBUG):    Leftovers: 662

I will let it run for some time and report any bugs.

I have a question on how exactly does gekko calculates the buy and sell volumes? Right now it seems that when the advise is to BUY it will buy BTC for all available money and if the advise is to SELL it will sell all BTC, is my understanding correct?

Answer 31 · 2013-12-29T11:39:56.000Z

@gabbello Thanks, let me know if Gekko starts acting strange.

If anyone wants to discuss some details, has some specific questions or wants to be involved get on IRC: #gekkobot on freenode.

EDIT:

@gabbello yes it will buy/sell using all available funds. The EMA method tries to detect trends, upon detecting an uptrend the advice is to buy the asset so that you can ride the trend. The hard part lies in detecting trends (that stay a trend in the future).

Answer 32 · 2013-12-29T15:03:45.000Z

The localDB version doesn't appear to be currently provide advice - I've had a quick look around to try and find out why without luck - any suggestions on what needs changing to get that to start happening?
I'd like to tinker about with some trading algos

Answer 33 · 2013-12-29T15:29:12.000Z

Ah I probably wasn't clear enough:

This new version is a rewrite of the whole core of Gekko, nothing works except for storing candles at CEX.io, Bitstamp and BTC-e.

So at this moment the localDB branch:

doesn't give any advice (mail or log).
doesn't do any real trading.
doesn't work on BTCChina or Mt. Gox.

But if you are already storing data right now (and we are lucky that I don't have to change anything anymore on the datastorage level which would invalidate current data) and this functionality comes out you can use the history you are building now for calculating advice.

EDIT: I could use some help to get this working ASAP. If you want to help implement these features come to #gekkobot (freenode) on IRC!

Answer 34 · 2013-12-29T15:42:34.000Z

Yep - I'm on track with the rewrite - it looks cleaner and more flexible in ways that should make adding/changing stuff later. It looks good.

I'm just wondering about how best to get things moving with algos so we can test out approaches that might work better on the exchanges with unusual markets - like GHS on cex.io

Answer 35 · 2013-12-29T16:12:32.000Z

We still need some things to get this working, it is a lot of work to do this on my own. I was also thinking for CEX.io:

Instead of watching for trends happening in the market, calculate how much profit one GHS is going to make you (using one of the online profit calculators for example) and buy if price < profit, sell if price > profit.

To get Gekko spitting out advice we need to have the following implemented:

Glue trading methods to the candleManager. (tradingMethods were glued to realtimeCandleFetcher before, this isn't used anymore).
If the history is complete (there are checks for this on startup, though if it isn't Gekko will currently never check this again), pass the history from candleManager to tradingMethod using the new glue.
sync real candles (not the 1m based internal ones) with the available history (right now it calcs real candles based on 1m candles inserted since startup) and pass them from DBmanager -> candleManager -> trading method.

Answer 36 · 2013-12-29T17:58:10.000Z

WRT to trading methods - how about writing a skeleton trading method that just fetches the candle data but can act as a template for adding our own methods?

For CEX I reckon a simple algo - if the trend is up buy some (%age of available funds or fixed amount), if it is down then sell some (again %age or fixed amount). In both cases until we run out of cash/asset. At present the profit per GHS doesn't support the price....!

Someone else suggested a 'grid' method - setting matched buy/sell pairs either side of the market but I don't think the spreads on CEX would let this work

Answer 37 · 2013-12-29T18:27:13.000Z

There was a simple script that did something similar but since the API does not provide historical data anymore (see openingspost) you either:

need something that doesn't require an history (everything related to trends does).
or keep everything in memory and build up a history there (really simple & straightforward), start with giving advice once the history has been build.

Last one does mean that every time you want to make a change you need to rebuild the history again, and you always need to wait for the history the be built. At default EMA settings Gekko needs ~3 days of history (so it can give advice after it has been on for 3 days).

My idea is that after the data store is done I am going to implement very simple trading methods (like buy if price < 100, sell if price > 200) and write documentation like the exchange documentation that will explain the API all methods can use to interact with. So it becomes extremely trivial to add your own methods.

But since almost all APIs hammered down on historical data, we first need a proper way to save it ourselves. Which is exactly what I'm trying to do right now.

Answer 38 · 2013-12-30T00:12:40.000Z

Just did a re-sync (as it crashed when I restarted after midnight) and now crashes:-

"2013-12-30 00:10:34 (INFO): Using normal settings to monitor the live market

2013-12-30 00:10:34 (INFO): NOT trading with real money
2013-12-30 00:10:34 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-30 00:10:34 (DEBUG): Requested trade data from CEX.io ...
2013-12-30 00:10:35 (DEBUG): Fetched 1001 new trades, from 2013-12-29 23:47:5
2 (UTC) to 2013-12-30 00:10:39 (UTC)
2013-12-30 00:10:35 (INFO): Full history available
GOT FULL HISTORY, SIZE: 839
2013-12-30 00:10:35 (DEBUG): minimum trade treshold: 2013-12-29 23:58:00 UTC
2013-12-30 00:10:35 (DEBUG): processing 551 trade(s)
2013-12-30 00:10:35 (DEBUG): This batch includes trades for a new day.
2013-12-30 00:10:35 (DEBUG): TODO: test if this is working correctly..
2013-12-30 00:10:35 (DEBUG): Creating a new daily database for day 2013-12-30

D:\Users\David\Documents\GitHub\gekko\databaseManager.js:497
throw 'a';
^
a

So I guess the answer to the TODO is that it isn't... The file is created but is empty... I don't program in JS but do you need to include a test for an empty set of candles?

Answer 39 · 2013-12-30T00:53:59.000Z

Thanks for letting me know! Also noticed that at midnight, should be fixed now. Could you let me know if it still bothers you?

Answer 40 · 2013-12-30T11:43:00.000Z

We are discussing issues related to memory usage and leaks that the localDB branch is causing. If anybody wants to chip in, now is the time to do so give your input: either comment in the issue or get on IRC: #gekkobot (freenode)!

Answer 41 · 2013-12-30T13:04:49.000Z

Is it not possible that we accumulate all history on our own server space and Gekko get the history from one place instead of everyone storing it on their individual machine? ... If required ... I can give you some space on my vps where may be you can install a script which will keep pulling live data from CEX ... then all machines can pull data from that one place?

Answer 42 · 2013-12-30T13:19:01.000Z

I was just thinking about that and the main reason I want everybody to build their own history is that you have to trust the dataprovider (I got a lot of flaming about people not wanting to trust bitcoincharts before). If gekko is open source and only uses the APIs of the exchanges everybody can verify that nobody is mocking with anything. Trusting others is really hard when there is real money at stake, you could use these points to basically manipulate the market (in theory, it would probably take a while to manipulate bitstamp BTC/USD though if you can figure out that some altcoin almost no one is trading is being traded by a Gekko...).

Though in the far, far future we might do:

I offer the history through wizbit. Unless you read all the code you are putting your trust on me already. (I hold the keys to stuff something bad in here and I am sure it won't get noticed for at least a little while, I also hold the keys for this library which has access to people's funds on Bitstamp that's getting downloaded over 1200 times a month).
If the data is getting to big: I can offer checksums on wizbit for each day and others can setup free or paid dataproviders. Gekko could download the candles from each dataprovider (maybe for a fee) and Gekko would verify that those are valid using checksums stored on Wizbit.

Answer 43 · 2013-12-30T14:24:23.000Z

I would prefer to also have the ability to use my local data, since any solution of storing the data remotely will induce some costs and will be prone to service unavailability. As a remote storage solution maybe Amazon S3 services could be of interested?

Answer 44 · 2013-12-31T01:03:51.000Z

Hi has there been a fix for issue #90 yet?

Answer 45 · 2013-12-31T02:29:16.000Z

Exciting news guys:

I glued together the EMA method to the localDB, which means you now start seeing EMA advice!

Please be aware that there will probably be bugs and I highly advice to ~~double~~ triple check the advice before even considering it.

What should work:

Fetch trades at: CEX.io, Bitstamp & BTC-e (altcoin markets included!)
Store the history locally (inside the history folder).
If on start your local history is big enough, start giving EMA advice right away.
If after X time your local history is big enough, start giving EMA advice (this is untested, if it doesn't work let me know and restart Gekko).

What is known to not work:

Auto trading
Mailing advice
Fetching at Mt. Gox

Answer 46 · 2013-12-31T02:43:12.000Z

Nice, was wondering if it would be possible to add cryptsy support (and their various altcoins) in this new release.

Answer 47 · 2013-12-31T08:22:56.000Z

It gets into "should work" to often. Looks we need automated test, but
first let's settle on an interface. I can volunteer writing tests.

Exciting news guys:

I glued together the EMA method to the localDB, which means you now start
seeing EMA advice!

Please be aware that there will probably be bugs and I highly advice to
~~double~~ triple check the advice before even considering it.

What should work:

Fetch trades at: CEX.io, Bitstamp & BTC-e (altcoin markets included!)
Store the history locally (inside the history folder).
If on start your local history is big enough, start giving EMA advice right
away.
If after X time your local history is big enough, start giving EMA advice
(this is untested, if it doesn't work let me know and restart Gekko).

What is known to not work:

Auto trading
Mailing advice
Fetching at Mt. Gox

—
Reply to this email directly or view it on GitHub.

Answer 48 · 2013-12-31T13:55:10.000Z

I started running localDB branch yesterday on two instances. Both of them was crached on last night.

---------------------------------------- First instance ----------------------------------------
Config:
exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
currency: 'USD',
asset: 'BTC',

Last lines from log:

2013-12-31 02:00:35 (DEBUG):    Creating a new daily database for day 2013-12-31
2013-12-31 02:00:35 (DEBUG):    inserting candle 1439 (2013-12-30 23:59:00 UTC) volume: 7.035317699999999
2013-12-31 02:00:35 (DEBUG):    Leftovers: 0
2013-12-31 02:00:35 (INFO):     Processed trades, sleeping for a minute ...
2013-12-31 02:01:47 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:47 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:55:13 (UTC) to 2013-12-31 00:01:27 (UTC)
2013-12-31 02:01:47 (DEBUG):    minimum trade treshold: 2013-12-31 00:00:28 UTC
2013-12-31 02:01:47 (DEBUG):    processing 44 trade(s)

Weird error

When I started it again:

2013-12-31 15:44:00 (INFO):     Using normal settings to monitor the live market
2013-12-31 15:44:00 (INFO):     NOT trading with real money
2013-12-31 15:44:00 (INFO):     Starting to watch the market: BTC-e USD/BTC
2013-12-31 15:44:00 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 15:44:01 (DEBUG):    Fetched 150 new trades, from 2013-12-31 13:43:41 (UTC) to 2013-12-31 13:43:54 (UTC)
2013-12-31 15:44:01 (WARN):     Found a corrupted database ( 2013-12-31 ), going to clean it up
2013-12-31 15:44:01 (DEBUG):    This should not happen, please post details here: https://github.com/askmike/gekko/issues/90
BAIL
2013-12-31 15:44:01 (INFO):     No history found, starting to build one now

---------------------------------------- Second instance ----------------------------------------

Config:
  exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'LTC',

Last lines from log:

2013-12-31 01:59:22 (DEBUG):    NEW REAL CANDLE
2013-12-31 01:59:22 (DEBUG):    inserting candle 1437 (2013-12-30 23:57:00 UTC) volume: 0
2013-12-31 01:59:22 (DEBUG):    Leftovers: 1438
2013-12-31 01:59:22 (INFO):     Processed trades, sleeping for 2 minutes ...
2013-12-31 02:01:22 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:23 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:30:34 (UTC) to 2013-12-31 00:01:20 (UTC)
2013-12-31 02:01:23 (DEBUG):    minimum trade treshold: 2013-12-30 23:58:12 UTC
2013-12-31 02:01:23 (DEBUG):    processing 6 trade(s)
2013-12-31 02:01:23 (DEBUG):    This batch includes trades for a new day.
2013-12-31 02:01:23 (DEBUG):    Creating a new daily database for day 2013-12-31

TypeError: Cannot set property 's' of undefined.  at Manager.processTrades (/home/user11/gekko/databaseManager.js:372:19)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Fetcher.processTrades (/home/user11/gekko/tradeFetcher.js:163:8)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at process (/home/user11/gekko/exchanges/btce.js:135:7)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Request._callback (/home/user11/gekko/node_modules/btc-e/btc-e.js:76:7)
    at Request.self.callback (/home/user11/gekko/node_modules/btc-e/node_modules/request/index.js:142:22)

Works after restart but probably will crash on next night again.

Anyway. I will now update latest version and try again.

Answer 49 · 2013-12-31T14:28:48.000Z

@GoodDragon that should have been resolved by this commit. Let me know what happens next time.

@yin Totally agree, though because of the huge todo I have not written tests yet. Also because of the hard to test edgecases around having parted history / receiving weird fetch data it's hard to write bullet proof testing. Though I'm very interested in building a test suite asap. Pull requests more than welcome!

Answer 50 · 2013-12-31T14:46:55.000Z

Already working on tests, well one is already passing - Store.read(). I
mean the CSV store from your gist. Correct me, if I am on wrong track.

I used nodeunit, so we can have test framework and examples at hand. Have a
look if you're interested.

https://github.com/yin/gekko/commits/localDB

I'd also copy in my timing code from yesterday as a nice module.

Regards / S pozdravom

Matej Gagyi

On Tue, Dec 31, 2013 at 3:28 PM, Mike van Rossum
notifications@github.comwrote:

@GoodDragon https://github.com/GoodDragon that should have been
resolved by this commithttps://github.com/askmike/gekko/commit/80c3c51e921443925d8c979c6f9d510e5923ec48.
Let me know what happens next time.

@yin https://github.com/yin Totally agree, though because of the huge
today I have not written tests yet. Also because of the hard to test
edgecases around having parted history / receiving weird fetch data it's
hard to write bullet proof testing. Though I'm very interested in building
a test suite asap.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/90#issuecomment-31396383
.

Answer 51 · 2013-12-31T17:10:14.000Z

I've got the local database version up and running, but it won't let me change out of BTC/TRC....

// Monitor the live market
config.normal = {
  enabled: true,
  exchange: 'btce', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'TRC',
  tradingEnabled: true,

when I change to

  currency: 'USD',
  asset: 'BTC',

I get this:

/Users/dave/Downloads/gekko-localDB/config.js:39
  currency: ‘USD’,
            ^
SyntaxError: Unexpected token ILLEGAL
    at Module._compile (module.js:439:25)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.util.getConfig (/Users/dave/Downloads/gekko-localDB/core/util.js:14:15)
    at Object.<anonymous> (/Users/dave/Downloads/gekko-localDB/core/util.js:105:19)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)

am I missing something obvious? haven't seen anyone else mention this...

Answer 52 · 2013-12-31T17:16:13.000Z

@ArgyleEl1te it seems you are using the wrong quotes, they should not be curly but normal ones. (if unsure copy quotes from somewhere else and change the USD inside).

Note that Gekko watches per market, so you do need to rebuild a history for this other one.

Answer 53 · 2013-12-31T17:21:13.000Z

@askmike wow... not sure how that happened. Even copied and pasted out of the old config.js from the previous version... but yes, you are right. Wiped out databases, it fired up, excited to see how it goes. Thanks for the 5 minute help turnaround!

Answer 54 · 2014-01-01T01:39:02.000Z

Ran for about 4 hours, then threw this error and crashed:

2013-12-31 17:59:53 (INFO): Processed trades, sleeping for 2 minutes ...
2013-12-31 18:01:28 (DEBUG): Requested trade data from BTC-e ...
2013-12-31 18:01:28 (DEBUG): Fetched 150 new trades, from 2013-12-31 23:57:32 (UTC) to 2014-01-01 00:01:18 (UTC)
2013-12-31 18:01:28 (DEBUG): minimum trade treshold: 2013-12-31 23:59:51 UTC
2013-12-31 18:01:28 (DEBUG): processing 47 trade(s)
2013-12-31 18:01:28 (DEBUG): This batch includes trades for a new day.
2013-12-31 18:01:28 (DEBUG): Creating a new daily database for day 2014-01-01

TypeError: Cannot set property 's' of undefined
at Manager.processTrades (/Users/dave/Downloads/gekko-localDB/core/databaseManager.js:373:19)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at EventEmitter.emit (events.js:117:20)
at bound as emit
at Fetcher.processTrades (/Users/dave/Downloads/gekko-localDB/core/tradeFetcher.js:163:8)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at process (/Users/dave/Downloads/gekko-localDB/exchanges/btce.js:135:7)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at Request._callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/btc-e.js:76:7)
at Request.self.callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/node_modules/request/index.js:142:22)

Loaded it back up and it gave me this:
2013-12-31 19:16:42 (INFO): I'm gonna make you rich, Bud Fox.

2013-12-31 19:16:43 (INFO): Starting to watch the market: BTC-e USD/BTC
2013-12-31 19:16:43 (DEBUG): Requested trade data from BTC-e ...

2013-12-31 19:16:43 (INFO): Seting up a new price actor:
2013-12-31 19:16:43 (INFO): Trade advisor
2013-12-31 19:16:43 (INFO): Calculate trading advice based on the Exponential Moving Average Crossovers

2013-12-31 19:16:48 (DEBUG): Fetched 150 new trades, from 2014-01-01 00:48:42 (UTC) to 2014-01-01 01:16:24 (UTC)
2013-12-31 19:16:48 (WARN): Found a corrupted database ( 2014-01-01 ), going to clean it up
2013-12-31 19:16:48 (DEBUG): This should not happen, please post details here: #90
2013-12-31 19:16:48 (INFO): No history found, starting to build one now
2013-12-31 19:16:48 (INFO): Expected to start giving advice in 3 hours (2014-01-01 04:36:48 UTC)
2013-12-31 19:16:48 (DEBUG): Creating a new daily database for day 2014-01-01
2013-12-31 19:16:48 (DEBUG): minimum trade treshold: 2014-01-01 00:00:00 UTC
2013-12-31 19:16:48 (DEBUG): processing 150 trade(s)
2013-12-31 19:16:48 (DEBUG): inserting candle 48 (2014-01-01 00:48:00 UTC) volume: 0.207574
2013-12-31 19:16:48 (DEBUG): inserting candle 49 (2014-01-01 00:49:00 UTC) volume: 0.128544
2013-12-31 19:16:48 (DEBUG): inserting candle 50 (2014-01-01 00:50:00 UTC) volume: 0
2013-12-31 19:16:48 (DEBUG): inserting candle 51 (2014-01-01 00:51:00 UTC) volume: 1.4048787200000001
2013-12-31 19:16:48 (DEBUG): inserting candle 52 (2014-01-01 00:52:00 UTC) volume: 0
2013-12-31 19:16:48 (DEBUG): inserting candle 53 (2014-01-01 00:53:00 UTC) volume: 0

So, corrupted database somehow.... FYI...

Answer 55 · 2014-01-03T15:43:39.000Z

@ArgyleEl1te hmm the first error is weird, let me know if it still happens...

For the second one I am going to write a document explaining the issue, but it boils down to: it's not really corrupt but just incomplete:

You started it 75 minutes after the crash, but BTC-e most likely won't tell you what happened before a couple of minutes.

you should think of it like this: you are trying to count the cars on the road for 24 hours
if you are watching for 23 hours and counted how many cars passed by you know a lot. But if you take a break for the last hour you don't know the total
and you can never make assumptions about what happened during breaks if you want to make financial decisions.
(or give advice)

So I created a new issue that hosts the TODO before I consider all the localDB stuff stable (and will merge this into master):

#114

Answer 56 · 2014-01-04T19:11:44.000Z

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you 😕

If you see this please let me know the following:

~Sat Jan 04 2014 18:38:56 GMT+0000
~Sat Jan 04 2014 19:08:00 GMT+0000

Btw: Local time was 20:08 (MEZ/GMT+1)

Answer 57 · 2014-01-04T19:24:20.000Z

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you

Awesome thanks! So my question is: did it crash it did it work?

Answer 58 · 2014-01-04T19:28:09.000Z

@askmike Actually it looks like it works. At least no crash. 😉

Answer 59 · 2014-01-06T07:17:00.000Z

May I recommend removing the "If you see this please let me know the following:"
I've been running this in two different instances (mtgox btc-e)since the midnight commit. No issues yet and the history looks good. Gotta say I'm impressed at this from looking over localDB and look forward to working on this project in the future.

Answer 60 · 2014-01-06T13:26:09.000Z

@trader-cheat-codes I'm still working on that code and there still is one situation where it crashes after that. Gonna be out soon!

Answer 61 · 2014-01-18T01:51:25.000Z

i have this error :

user@host:/gekko$ node gekko
2014-01-18 03:39:06 (INFO): I'm gonna make you rich, Bud Fox.
2014-01-18 03:39:06 (INFO): Let me show you some Exponential Moving Averages.
2014-01-18 03:39:06 (INFO): Using normal settings to monitor the live market
2014-01-18 03:39:06 (INFO): NOT trading with real money
/home/user/gekko/gekko.js:60
throw invalid;
^
At this moment Gekko can't monitor CEX.io, find out more info here:
#90
user@host:/gekko$

do I have a mistake or a problem with the source?

Answer 62 · 2014-01-18T01:56:32.000Z

@ejko it depends a little on what you want:

There is a stable Gekko (the one you downloaded), this one does not have support for CEX.io but can trade based on EMA on Mt. Gox or BTC-e.
There is an unstable Gekko which does have support for CEX.io and advice using EMA, MACD or PPO. However you can't trade with this version yet and it is not stable.

Answer 63 · 2014-01-18T16:51:12.000Z

Hey,

Ran the Gekko-LocalDB for 48 hours to monitor LTC-USD with no isseus.
A few Trade propositions were made.

Not sure if you need a log from a working one?

Cant wait to test the Auto Trade future for Altcoin! :)

With friendly Regards,
R

Answer 64 · 2014-01-18T20:37:14.000Z

I apologize again, I'm new in this system.
how to install unstable Gekko ?

Answer 65 · 2014-01-18T22:48:46.000Z

Because the unstable and unverified state of the new version I've disabled real trading for now. All code is located in another branch (like a nightly build), to get it download and install git and run:

git clone -b localDB --single-branch git://github.com/askmike/gekko.git
cd gekko
npm install

and follow instructions in the readme.

Answer 66 · 2014-01-28T00:08:58.000Z

hope this helps

2014-01-27 04:53:45 (INFO): I'm gonna make you rich, Bud Fox.

2014-01-27 04:53:45 (INFO): Setting up Gekko in realtime mode

2014-01-27 04:53:45 (INFO): Setting up:
2014-01-27 04:53:45 (INFO): Trading Advisor
2014-01-27 04:53:45 (INFO): Calculate trading advice
2014-01-27 04:53:45 (INFO): Using the trading method: MACD

2014-01-27 04:53:45 (INFO): Setting up:
2014-01-27 04:53:45 (INFO): Profit Simulator
2014-01-27 04:53:45 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:46 (INFO): Setting up:
2014-01-27 04:53:46 (INFO): Profit Simulator
2014-01-27 04:53:46 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:48 (INFO): Starting to watch the market: BTC-e BTC/USD
2014-01-27 04:53:49 (INFO): We don't have enough history yet to start giving advice, I'll need to gather more data first.
2014-01-27 04:53:49 (INFO): I will start giving advice around 2014-02-19 17:54:00 (local time, that's in 24 days).
2014-01-27 20:21:10 (WARN): DOUBLE UNIQUE INSERT, this should never happen. Please post details here: #90 2014-01-27 undefined { [Error: ENOSPC, write] errno: 54, code: 'ENOSPC' }

/home/pi/gekko-localDB/core/candleManager.js:898
throw err;
^
Error: ENOSPC, write

Answer 67 · 2014-02-11T23:36:47.000Z

Excited to announce that the new localDB branch is out. Trading is CEX.io is working and stable.