The behavior of bn.binancedownload(freq=:daily) is strange.
Closed this issue · 3 comments
janckerchen commented
The behavior of freq=:daily is strange in three ways:
- It overwrites existing data instead of appending to it.
- It downloads the oldest data instead of the most recent data.
- It can only download just over a year of data, rather than the entire history.
expect to do
# Download monthly data up until the previous month.
bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
# Download daily data to complete the data for this month.
bn.binancedownload("eth", market=:data, freq=:daily, kind=:klines)
# After the two steps, we should get the most complete historical data possible.actually happen
freq==:monthly # Mission accomplished successfully, from 2017-08-17 to 2024-05-31
freq==:daily #overwrite above with data from 2017-08-17 to 2018-12-29btw, the parameter freq=:monthly is unnecessary for bn.binanceload, because data from both parameter are saved to tf_1m.
janckerchen commented
A new issue was discovered during testing.
julia> ] activatie Scrapers
julia> using Scrapers: Scrapers as scr, BinanceData as bn
julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
ERROR: MethodError: no method matching haskey(::Base.Generator{Base.Iterators.Filter{Lang.var"#7#9"{…}, Base.Pairs{…}}, Lang.var"#6#8"}, ::Symbol)
Closest candidates are:
haskey(::Base.TTY, ::Symbol)
@ Base ttyhascolor.jl:25
haskey(::Pkg.Types.Manifest, ::Any)
@ Pkg ~/.julia/juliaup/julia-1.10.4+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Types.jl:319
haskey(::StructTypes.SubTypeClosure, ::Symbol)
@ StructTypes ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:589
...
Stacktrace:
[1] binancedownload(syms::Vector{…}; zi::Data.ZarrInstance{…}, quote_currency::String, reset::Bool, kwargs::@Kwargs{…})
@ Scrapers.BinanceData /opt/opensource/jl/PingPong.jl/Scrapers/src/binance.jl:259
[2] binancedownload(args::String; kwargs::@Kwargs{market::Symbol, freq::Symbol, kind::Symbol})
@ Scrapers.BinanceData /opt/opensource/jl/PingPong.jl/Lang/src/module.jl:284
[3] top-level scope
@ REPL[12]:1
Some type information was truncated. Use `show(err)` to see complete types.
janckerchen commented
pull 837dd and test again, everything's working as expected, great!
# Download data up to last month
julia> bn.binancedownload("eth", market=:data, freq=:monthly, kind=:klines)
# Download last data in this month and append to above
julia> bn.binancedownload("eth", market=:data, freq=:daily, kind=:klines)
# We got the full data from the beginning to yesterday
julia> bn.binanceload("eth", market=:data, kind=:klines)
Data.PairData("ETHUSDT_data_klines", "1m", 3605519×6 DataFrame
Row │ timestamp open high low close volume
│ DateTime Float64 Float64 Float64 Float64 Float64
─────────┼─────────────────────────────────────────────────────────────────────
1 │ 2017-08-17T04:01:00 301.13 301.13 301.13 301.13 2.75787
2 │ 2017-08-17T04:02:00 300.0 300.0 300.0 300.0 0.0993
3 │ 2017-08-17T04:03:00 300.0 300.0 300.0 300.0 0.31389
4 │ 2017-08-17T04:04:00 301.13 301.13 301.13 301.13 0.23202
5 │ 2017-08-17T04:05:00 300.0 301.13 300.0 301.13 0.75705
6 │ 2017-08-17T04:06:00 300.1 300.1 300.1 300.1 0.90018
7 │ 2017-08-17T04:07:00 300.1 300.1 300.1 300.1 0.0
8 │ 2017-08-17T04:08:00 300.1 300.1 298.0 298.0 0.31493
9 │ 2017-08-17T04:09:00 298.0 298.0 298.0 298.0 0.0
10 │ 2017-08-17T04:10:00 298.0 298.0 298.0 298.0 0.0
11 │ 2017-08-17T04:11:00 298.0 298.0 298.0 298.0 0.0
12 │ 2017-08-17T04:12:00 298.0 298.0 298.0 298.0 0.0
13 │ 2017-08-17T04:13:00 298.0 298.0 298.0 298.0 0.0
⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
3605507 │ 2024-06-24T23:47:00 3355.79 3356.37 3355.21 3355.62 435.85
3605508 │ 2024-06-24T23:48:00 3355.61 3355.62 3355.55 3355.56 807.622
3605509 │ 2024-06-24T23:49:00 3355.56 3357.76 3355.55 3356.84 1078.05
3605510 │ 2024-06-24T23:50:00 3356.88 3357.33 3355.55 3355.55 233.154
3605511 │ 2024-06-24T23:51:00 3355.56 3362.41 3354.13 3362.21 1494.04
3605512 │ 2024-06-24T23:52:00 3362.21 3363.18 3358.8 3362.54 936.479
3605513 │ 2024-06-24T23:53:00 3362.55 3362.55 3357.26 3357.26 452.129
3605514 │ 2024-06-24T23:54:00 3357.26 3359.68 3357.2 3358.79 315.136
3605515 │ 2024-06-24T23:55:00 3358.79 3358.8 3355.21 3355.21 351.497
3605516 │ 2024-06-24T23:56:00 3355.21 3356.8 3354.26 3354.27 275.629
3605517 │ 2024-06-24T23:57:00 3354.27 3354.27 3352.87 3352.88 179.285
3605518 │ 2024-06-24T23:58:00 3352.88 3353.39 3351.51 3351.72 151.901
3605519 │ 2024-06-24T23:59:00 3351.71 3353.24 3351.3 3352.73 209.344
3605493 rows omitted, ZArray{Float64} of size 3605519 x 6)