Error using Yahoo connector

[Iscalc]

First, I would like to thank you very much for this package.

Today started getting the following error message. After

using Temporal
a = yahoo("AAPL", from="2010-06-09", thru=string(Dates.today()), freq='w')

I get

connect: connection timed out (ETIMEDOUT)
in yieldto(::Task, ::ANY) at .\event.jl:136
in wait() at .\event.jl:169
in wait(::Condition) at .\event.jl:27
in stream_wait(::TCPSocket, ::Condition, ::Vararg{Condition,N}) at .\stream.jl:44
in wait_connected(::TCPSocket) at .\stream.jl:265
in connect at .\stream.jl:960 [inlined]
in connect(::IPv4, ::Int64) at .\socket.jl:677
in open_stream(::HttpCommon.Request, ::MbedTLS.SSLConfig, ::Float64, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\streaming.jl:186
in #do_stream_request#23(::Dict{AbstractString,AbstractString}, ::Void, ::Void, ::Void, ::Array{Requests.FileParam,1}, ::Void, ::Dict{Any,Any}, ::Bool, ::Int64, ::Array{HttpCommon.Response,1}, ::MbedTLS.SSLConfig, ::Bool, ::Bool, ::Bool, ::Nullable{URIParser.URI}, ::Nullable{URIParser.URI}, ::Requests.#do_stream_request, ::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:381
in do_stream_request(::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:344
in #do_request#22(::Array{Any,1}, ::Function, ::URIParser.URI, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:311
in #get#29(::Array{Any,1}, ::Function, ::URIParser.URI) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:444
in #get#28(::Array{Any,1}, ::Function, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Requests\src\Requests.jl:443
in #yahoo#64(::String, ::String, ::Char, ::Function, ::String) at C:\JuliaPro-0.5.0.5\pkgs-0.5.0.5\v0.5\Temporal\src\io.jl:291
in (::Temporal.#kw##yahoo)(::Array{Any,1}, ::Temporal.#yahoo, ::String) at .\<missing>:0

[haavardhvarnes]

I have the same problem. Copying the yahoo function and made some changes and it works. But parsing csv sometimes fails. Hope more can take a look at this problem. My adjusted functions below:

function getyahoocrumb()
ustr = "https://ca.finance.yahoo.com/quote/^GSPC/history?p=^GSPC";
response = Requests.get(ustr);
str = readstring(response);
m = match(r""user":{"crumb":"(.*?)"", str) ;
crumb = m[1]
mcookies = cookies(response)
return crumb, mcookies
end

function yahoo2(symb::String;
from::String="1900-01-01",
thru::String=string(Dates.today()),
freq::Char='d', crumb = getyahoocrumb())
@Assert freq in ['d','w','m','v'] "Argument freq must be in ['d','w','m','v']"
@Assert from[5] == '-' && from[8] == '-' "Argument from has invalid date format."
@Assert thru[5] == '-' && thru[8] == '-' "Argument thru has invalid date format."
dtFrom = Dates.DateTime(from)
dtThru = Dates.DateTime(thru)

# symb = "XOM"
# dtFrom = Dates.DateTime("2016-01-01")
# dtThru = Dates.DateTime("2017-05-18")
per1 = Int(floor(Dates.datetime2unix(dtFrom)))
per2 = Int(floor(Dates.datetime2unix(dtThru)))

crumb = "Df1WvGdhjEg"
crumb, mcookies = getyahoocrumb()
qrystr= string("https://query1.finance.yahoo.com/v7/finance/download/",symb,"?period1=", per1, "&period2=", per2,"&interval=1d&events=history&crumb=", crumb)
# println(qrystr)

# indata = Temporal.csvresp(get(qrystr))
indata = Temporal.csvresp(get(qrystr, cookies = mcookies))

sleep(0.8)
return ts(indata[1], indata[2], indata[3][2:end])

end

[AntonioSaragga]

It seems that yahoo is discontinuing its service.

[dysonance]

@AntonioSaragga Could you provide a source suggesting this is the case? Best I've found is an answer on this forum. I know R's quantmod package has been experiencing this issue lately, but not sure if it's resolved (and given that Joshua Ulrich is a very proactive package maintainer, that would tell me this is a non-trivial issue).

[dysonance]

@haavardhvarnes Thanks for taking a crack at getting it to work again, that's much appreciated! I'll try it out see if it's possible to get it worked out. If Yahoo hasn't discontinued the service and we can get this solved I'll try to merge that into the package ASAP.

[dysonance]

Seeing some workarounds for packages in other languages out there. This project implements a "temporary fix" for the function in the Pandas package in Python that downloads data from the Yahoo Finance API. See this Stack Overflow thread for some insights around the work done on R's quantmod package to solve the issue. If anyone has experience with either of these please provide thoughts/feedback on how they've been working.

[dysonance]

Alright all, with the great help of @haavardhvarnes's initial workaround above, I have managed to get a fix put in place for the Yahoo Finance data downloading functionality. There are some changes to the arguments required to fit the new URL formatting needs:

• freq argument now a String (not Char) that can be "d", "wk", or "mo"
• new event argument specifies what kind of data you want, can be "history", "div", or "split"
• new crumb_tuple argument in case you don't want to re-generate cookies every time you run it

I'm about to send through an update / PR to Julia's METADATA repo, so hopefully this fix should be available to the wider community with a Pkg.update() soon. Until then, feel free to pull the changes from git to your local package directory and you should be all set.

Thanks again everyone for the input on this issue and for helping out the legwork on a workaround. Hopefully this issue will be solved for the time being. However, it did also raise an interesting note in my mind that it would be good to have more options for market data sources going forward (Google, etc.). If anyone would like to contribute with any functionality for sources other than Yahoo and Quandl, I'd be happy to work with you on it. Cheers.