I couldn't download stocks

Question

I couldn't download stocks

Closed this issue 6 years ago · 8 comments

root@bad:~# YahooTickerDownloader.py stocks
Checking if we can resume a old download session
Downloader found on disk, resuming
Downloading stocks

req https://finance.yahoo.com/_finance_doubledown/api/resource/finance.yfinlist.symbol_lookup;p=1;s=a;b=0;m=all;t=S?device=console&returnMeta=true
Retry attempt: 1. Sleep period: 5 seconds.
req https://finance.yahoo.com/_finance_doubledown/api/resource/finance.yfinlist.symbol_lookup;p=1;s=a;b=0;m=all;t=S?device=console&returnMeta=true
Retry attempt: 2. Sleep period: 25 seconds.
req https://finance.yahoo.com/_finance_doubledown/api/resource/finance.yfinlist.symbol_lookup;p=1;s=a;b=0;m=all;t=S?device=console&returnMeta=true
Retry attempt: 3. Sleep period: 125 seconds.

I think that url does not work.
That response is below.
{"output":{"message":"Not Found"},"meta":{}}

Thank you

Answer 1 · 2018-04-21T19:32:03.000Z

Yes! You are right. The API probably changed.

Answer 2 · 2018-04-29T10:52:46.000Z

I looked a bit around on google finance and could not find a GUI to list symbols using a search string. It could be they (re)moved it. If it is removed this utility can't do it's job and is rendered useless. If anyone known of a api to find a list of stocks please say so.

Answer 3 · 2018-04-29T10:55:55.000Z

So basically the unofficial Yahoo API broke, and I'm not yet sure if there is a replacement.

Answer 4 · 2018-05-06T03:10:34.000Z

@Benny-
This seems to work to some extent:

https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=aa?bkt=fin-guce-ga&device=desktop&feature=canvassOffnet,ccOnMute,enableTradeit,newContentAttribution,relatedVideoFeature,videoNativePlaylist,enableCrypto,enableESG,enablePrivacyUpdate,enableGuceJs,enableGuceJsOverlay,enableCMP,livecoverage,enableSingleRail,enableChartiqFeedback,enableSKTVLrec&intl=us&lang=en-US&partner=none&prid=cddtv01desr5j®ion=US&site=finance&tz=America/New_York&ver=0.102.1312&returnMeta=true

In fact it can be pared down to simply:

https://finance.yahoo.com/_finance_doubledown/api/resource/searchassist;searchTerm=aa

...and it will still work. However, this only returns the first 10 results. I'm not clear on whether there are parameters to increase the page size or even choose a page other than 1. Any ideas?

Answer 5 · 2018-05-07T19:34:17.000Z

This is great! Do you know where this api is used, @jharker ? If it supports pagination this will be a suitable replacement.

Answer 6 · 2018-05-20T16:48:31.000Z

Hi, @Benny- Yup, the API is used by the "Symbol Lookup" tool on this page:

https://finance.yahoo.com/lookup/

But unfortunately, from what I can tell it doesn't support pagination. Still, it seems like it should be possible to use this to pull symbols in groups of ten or less using a few modifications to the search algorithm you're already using. I'll see if I have time to give it a shot in the next week or so.

Answer 7 · 2018-05-26T14:49:48.000Z

Hi, @Benny- I created a pull request that implements the new Yahoo API. You may want to modify this to be more seamlessly integrated into the old code -- I didn't want to rip too much out of the existing code so I created some parallel functionality instead.

I've tested this over the past few days and it works pretty well. I wish there was a way to make it more efficient in terms of the number of API calls, but with the API results set apparently limited to 10 and not paginated, I can't figure out a way to do it more efficiently than this.

https://github.com/Benny-/Yahoo-ticker-symbol-downloader/pull/44

Answer 8 · 2018-05-26T22:34:12.000Z

I ran it and I'm am quite impressed with the results so far. It got 84464 unique results so far.

Exporting still works and considering the other API did not work anymore I've published this right now as 3.0.0. This new version gives a lot of random 404 error from Yahoo, this is logged as #45.