YFinance changed their api's (or their security)

Question

YFinance changed their api's (or their security)

TobiasFP opened this issue 2 years ago · 32 comments

Hi.
There seams to be an issue with the finance-go, since yFinance changed its encryption keys.
Apparantly they change their encryption keys every day or even sometimes more than once a day.
This has lead to finance-go not working for a while.
yFinance has the same issue:
ranaroussi/yfinance#1407

I just wanted to bring it to your attention. I have been very happy with this library, so thank you very much for your work.

Answer 1 · 2023-05-05T11:02:45.000Z

v6 of the API seems to work, not 7.

Answer 2 · 2023-05-07T10:53:23.000Z

@ktpx You saved me! :)

Answer 3 · 2023-05-24T16:26:57.000Z

I think they pulled the plug on v6 now. :(

Answer 4 · 2023-05-25T17:17:56.000Z

All projects I see using the Yahoo API are facing this. There is this nugget I got from the ticker issue report on the subject:

https://stackoverflow.com/questions/76065035/yahoo-finance-v7-api-now-requiring-cookies-python

Not sure how/if that helps...

Answer 5 · 2023-05-26T18:56:20.000Z

Saw somewhere that for instance, an url like this:
https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData
is working, but I'm not sure if the data structures are the same.
Is someone working on a patch?

Answer 6 · 2023-05-26T19:41:06.000Z

achannarasappa/ticker#250 (comment)
Looks like we're out of luck. 😢

Answer 7 · 2023-05-28T16:02:46.000Z

Saw somewhere that for instance, an url like this: https://query2.finance.yahoo.com/v6/finance/quoteSummary/AAPL?modules=financialData is working, but I'm not sure if the data structures are the same. Is someone working on a patch?

Totally different structure than the quotes. This show financial data. But atleast one can grab the currentprices.

Answer 8 · 2023-06-05T13:00:19.000Z

Crumb is a param that is easy to gain, not sure how it will behave in long term.

For now, it is possible to generate crumb via https://query2.finance.yahoo.com/v1/test/getcrumb and then pass crumb value as param https://query2.finance.yahoo.com/v7/finance/quote?symbols=AAPL&crumb=abcdef.g. (Works with v7 only)

I have had the same crumb for a few days now, so it's possible to cache it locally on disk to avoid this call.

Answer 9 · 2023-06-06T16:54:57.000Z

If you inspect the cookie header, it says one year. "Expires=Wed, 5 Jun 2024 22:54:09 GMT"

Answer 10 · 2023-06-12T11:13:23.000Z

Hi there, is there any workaround I could use to test ? This is broken for a while.

Answer 11 · 2023-06-29T16:04:05.000Z

So I have a fix for this and I have opened a PR for it. Not sure who should be reviewing it.
I also have "modernized" the finance-mock project used to test finance-go. PR can be found here: piquette/finance-mock#1

Answer 12 · 2023-06-29T16:26:48.000Z

I would love to review it, but i don't think we can get it in as i think the project is dead here. Maybe fork?

Answer 13 · 2023-06-29T16:50:52.000Z

Is it, though? There were a couple of PR that were merged a month ago by @ackleymi .

Answer 14 · 2023-06-29T17:04:29.000Z

My bad. Just add me to the pr if you want

Answer 15 · 2023-06-29T17:12:40.000Z

Doesn't look like I can add you, but feel free to head over the the PR and comment on it: #28

Answer 16 · 2023-07-11T05:22:58.000Z

Doesn't look like I can add you, but feel free to head over the the PR and comment on it: #28

Done. Good job!

Answer 17 · 2023-07-20T19:31:53.000Z

Did they add some kind of rate limit? I don't get a crumb anymore, just "Too many requests".

Answer 18 · 2023-07-20T20:03:39.000Z

I don't have access to a machine with the library on it atm. I'll check about this later today.

…

On Thu, Jul 20, 2023 at 3:32 PM ktpx ***@***.***> wrote: Did they add some kind of rate limit? I don't get a crumb anymore, just "Too many requests". — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATT2NOHUSYZVJRSGVCHALXRGBTHANCNFSM6AAAAAAVGDRPD4> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 19 · 2023-07-20T20:19:48.000Z

I don't have access to a machine with the library on it atm. I'll check about this later today.
…
On Thu, Jul 20, 2023 at 3:32 PM ktpx @.> wrote: Did they add some kind of rate limit? I don't get a crumb anymore, just "Too many requests". — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATT2NOHUSYZVJRSGVCHALXRGBTHANCNFSM6AAAAAAVGDRPD4 . You are receiving this because you commented.Message ID: @.>

Not a software issue really, API.

curl  https://query2.finance.yahoo.com/v1/test/getcrumb
Too Many Requests

Thats atleast the URL i used in my progs

Answer 20 · 2023-07-21T14:40:14.000Z

You need to have a cookie set when you call getcrumb. Just calling to it from curl won't work.

…

On Thu, Jul 20, 2023 at 4:19 PM ktpx ***@***.***> wrote: I don't have access to a machine with the library on it atm. I'll check about this later today. … <#m_9094901114993369745_> On Thu, Jul 20, 2023 at 3:32 PM ktpx *@*.*> wrote: Did they add some kind of rate limit? I don't get a crumb anymore, just "Too many requests". — Reply to this email directly, view it on GitHub <#25 (comment) <#25 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATT2NOHUSYZVJRSGVCHALXRGBTHANCNFSM6AAAAAAVGDRPD4 <https://github.com/notifications/unsubscribe-auth/AAATT2NOHUSYZVJRSGVCHALXRGBTHANCNFSM6AAAAAAVGDRPD4> . You are receiving this because you commented.Message ID: @.*> Not a software issue really, API. curl https://query2.finance.yahoo.com/v1/test/getcrumb Too Many Requests — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATT2M2VWMKEK3GUPQ4VOTXRGHG7ANCNFSM6AAAAAAVGDRPD4> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 21 · 2023-07-21T14:54:04.000Z

I do set cookie in the code, curl was bad example. But I still get same error. It worked, and literary didnt work half hour later, no code changed, so gotta be something yahoo changed.

Answer 22 · 2023-07-21T14:57:35.000Z

Hum. I don't know what to say. I just ran the "examples" program (examples/main.go) from my branch, and everything works nicely.

…

On Fri, Jul 21, 2023 at 10:54 AM ktpx ***@***.***> wrote: I do set cookie in the code, curl was bad example. But I still get same error. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATT2KW72BSDMITUYRSZTLXRKJZNANCNFSM6AAAAAAVGDRPD4> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 23 · 2023-07-21T14:58:31.000Z

Which site do you query to get the cookie? Seems might be the problem. Ive used fc.yahoo.com, as its the only one that worked for me. For regular api calls, query2.

Answer 24 · 2023-07-21T15:00:37.000Z

cookieURL = "https://login.yahoo.com" userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/113.0" func fetchCookies() (string, time.Time, error) { client := http.Client{} request, err := http.NewRequest("GET", cookieURL, nil) if err != nil { return "", time.Time{}, err } request.Header = http.Header{ "Accept": {"*/*"}, "Accept-Encoding": {"gzip, deflate, br"}, "Accept-Language": {"en-US,en;q=0.5"}, "Connection": {"keep-alive"}, "Host": {"login.yahoo.com"}, "Sec-Fetch-Dest": {"document"}, "Sec-Fetch-Mode": {"navigate"}, "Sec-Fetch-Site": {"none"}, "Sec-Fetch-User": {"?1"}, "TE": {"trailers"}, "Update-Insecure-Requests": {"1"}, "User-Agent": {userAgent}, } func fetchCrumb(cookies string) (string, error) { client := http.Client{} request, err := http.NewRequest("GET", crumbURL, nil) if err != nil { return "", err } request.Header = http.Header{ "Accept": {"*/*"}, "Accept-Encoding": {"gzip, deflate, br"}, "Accept-Language": {"en-US,en;q=0.5"}, "Connection": {"keep-alive"}, "Content-Type": {"text/plain"}, "Cookie": {cookies}, "Host": {"query1.finance.yahoo.com"}, "Sec-Fetch-Dest": {"empty"}, "Sec-Fetch-Mode": {"cors"}, "Sec-Fetch-Site": {"same-site"}, "TE": {"trailers"}, "User-Agent": {userAgent}, }

…

On Fri, Jul 21, 2023 at 10:58 AM ktpx ***@***.***> wrote: Which site do you query to get the cookie? Seems might be the problem. Ive used fc.yahoo.com, as its the only one that worked for me. For regular api calls, query2. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATT2ON5ODH36WILOHXD6DXRKKKFANCNFSM6AAAAAAVGDRPD4> . You are receiving this because you commented.Message ID: ***@***.***>

Answer 25 · 2023-07-21T15:10:09.000Z

Ok using login subdomain again got me the cookie, but still no crum. Mind showing your crumb URL.....? I didnt find it in the source

Answer 26 · 2023-07-21T15:13:01.000Z

I will refer you to my pull request:

#28

You should be able to get everything you need there.

Answer 27 · 2023-07-21T15:18:51.000Z

Thanks, its same url. Im guessing they are doing some IP based limiting, or blocking, im just getting the too many requests from here (EU).

Answer 28 · 2023-07-21T15:20:08.000Z

Have you sync'd my branch and run the examples program?

Answer 29 · 2023-07-21T20:40:43.000Z

Ok, confirmed is a location issue. So ill assume people using this tool wil also have
problem if aren't in US.

Answer 30 · 2023-08-08T13:58:00.000Z

So, that means this solution is only working if the request is made in the US?

Answer 31 · 2023-08-08T16:33:40.000Z

So, that means this solution is only working if the request is made in the US?

For the quotes API, it seems so yes. But I can't speak for other countries. But ive tried in 3 different EU countries, and no dice. It worked for a while, but no more.

Answer 32 · 2023-11-20T18:02:52.000Z

Hi @ackleymi , I saw that #28 was merged in, but there's no new tag or release with this change. Are there any plans to formally release this change? Thanks!