F1LT3R/bitcoin-scraper

url or parameters changed

bukosabino opened this issue · 6 comments

bukosabino commented

Hi Alistair,

Your job is really good but I think bitcoincharts.com have changed the url or parameters...

In this moment, if you execute your code every data file get the same content (last day datas).

I have tried with differents urls:

joernroeder commented

@bukosabino i've tried to use your url but this just gives me data for the last two days. did you manage to get more data out of the api?

bukosabino commented

@joernroeder bitcoincharts api has a data limit. You need a api call per day or per 2 days if you want datas with 1 minute period.
This repo is a good example with javascript or you can see this implementation with python.

F1LT3R commented

@bukosabino - I don't see any limits here on 1 min API. Perhaps it helps if you have a session open in your browser at the same IP? (But I would doubt they are throttling that way). I am able to get the entire history after making a small update. Cool Python implementation, I should probably have a way of combining the data into a single file. Perhaps I will use a separate script for this.

@joernroeder - I was having a similar issue with the script. I would get data for some files but not others. Some of the requests are getting throttled. I updated the code to work on streams. I've set me maxstreams to 8, and it works fine. If something goes wrong, you can pick up where you left of and not have to re-fetch data for days you already have.

F1LT3R commented

@bukosabino you were right. The url params did need updating. When charting the data I got back, it was a mess.

I've updated the params and pushed changes.

I would be interested to see if it works for you now.

joernroeder commented

@F1LT3R thanks for updating the script! I'll give it a try on the weekend.