/Yahoo-ticker-symbol-downloader

A web scraper for ticker symbols from yahoo finance

Primary LanguagePythonOtherNOASSERTION

Yahoo ticker downloader

Produces .csv, .xlsx, .json and .yaml files (All files contain same data but in a different format) for stocks, ETF, futures, indexes, mutual funds, currency, warrants and bonds. The ticker symbol, company name and exchange are saved for all symbols. The stock symbols also have a category name.

It gets its data from https://finance.yahoo.com/lookup/. Please note: it is not possible to get all the symbols due to limitations set by Yahoo. About 75%-90% of all symbols are gathered using this script depending on type.

Requirements

Python 2.7 or Python 3.5+

Install

From python package manager (preferred):

pip install Yahoo-ticker-downloader

From source:

python setup.py install

Example Usage

usage: YahooTickerDownloader.py [-h] [-i] [-e] [-E EXCHANGE] [-s SLEEP] [-p]
                                [-m MARKET]
                                type

positional arguments:
  type                  The type to download, this can be: future mutualfund
                    etf stocks index currency

optional arguments:
  -h, --help            show this help message and exit
  -i, --insecure        use HTTP instead of HTTPS
  -e, --export          export immediately without downloading (Only useful if
                        you already downloaded something to the .pickle file)
  -E EXCHANGE, --Exchange EXCHANGE
                        Only export ticker symbols from this exchange (the
                        filtering is done during the export phase)
  -s SLEEP, --sleep SLEEP
                        The time to sleep in seconds between requests
  -p, --pandantic       Stop and warn the user if some rare assertion fails
  -m MARKET, --market MARKET
                        Specify the Region of queried exchanges (us =
                        USA+Canada, dr=Germany, fr=France, hk=Hongkong,
                        gb=United Kingdom, default= all)

The first positional argument must be one of the following: stocks etf future index mutualfund currency warrant bond

For example to download all stock symbols you run it like:

YahooTickerDownloader.py stocks

The program takes several hours before it is finished so if you don't need all symbols it might be a good idea to specify the Region of the exchanges you're interested in. The program supports suspending and resuming a download. Press CTRL+C to suspend download. Restart the program in the same working directory to resume downloading. It is possible to export a partially downloaded results using the -e flag.

Example of CSV output:

Ticker,Name,Exchange,categoryName,categoryNr
ENZ,"Enzo Biochem Inc.",NYQ,"Medical Laboratories & Research",525
ENZN,"Enzon Pharmaceuticals Inc.",NMS,Biotechnology,515
ENZR,"Energizer Resources Inc.",PNK,,0
EOAA.DE,"E.ON AG",GER,"Diversified Utilities",913
KMX,"CarMax Inc.",NYQ,"Auto Dealerships",744
KMY.MU,KIMBERLY-CLARK,MUN,"Personal Products",323
KN1.DU,ECOUNION,DUS,"Business Software & Services",826
KNCAY,"Konica Minolta Holdings Inc.",PNK,,0
KND,"Kindred Healthcare Inc.",NYQ,"Long-Term Care Facilities",523
KNDI,"Kandi Technologies, Corp",NGM,"Auto Manufacturers - Major",330
...ect

Depending on the type you are downloading, you will get between 3.000 and 100.000+ entries.

Further resources

Download history for symbols: ystockquote

Changelog

  • Version 2.1.1 (2017-08-02)
    • A too old requests dependency was listed ( #35 )
  • Version 2.1.0 (2017-05-10)
    • Added market parameter ( pull request #33 )
  • Version 2.0.1 (2017-05-07)
    • Fixed issue where all downloads except stock and currency stopped working.
  • Version 2.0.0 (2017-05-05)
    • Switched over to JSON api
  • Version 1.0.0 (2017-04-04)
    • Reverted some changes from 0.10.0. Bond is back. Reverted back to English site instead of German.
    • Resolved CSV issue again. Closes #23 and #16.
    • Merged #26 Workaround Y! b>2000 limit
    • Scraper now scrapes a lot more at the expense of runtime.
    • Support for python2 is back. Latest python 2 & 3 are supported.
    • Removed xls support
    • Added xlsx support (#29)
  • Version 0.10.1 (2017-02-04)
    • More descriptive help message
  • Version 0.10.0 (2017-02-02)
    • Removed bond downloading option.
    • Uses different yahoo source. Fixes #18
    • Removed python2 from classifiers. Related to #16
  • Version 0.9.0 (unreleased)
    • Added a flag to restrict output to specific stock exchanges.
  • Version 0.8.1 (2016-08-17)
    • Workaround for #7 : downloading interruption
    • Solution for #9 : UnicodeEncodeError
  • Version 0.7.0 (2016-03-20)
    • Added --export option. It will transcode the .pickle file immediately to the desired output formats.
  • Version 0.6.0 (unreleased)
    • Add 3 retries with an exponential back-off if HTTPError or ChunkedEncodingError is raised when processing _fetchHtml.
  • Version 0.5.0 (2015-08-16)
    • Allows downloading using a insecure connection.
    • The temporarily download file-names now include the ticker type.
  • Version 0.4.0 (2014-10-28)
    • Warrant symbols can now be downloaded.
    • Bond symbols can now be downloaded.
  • Version 0.3.0 (2014-08-14)
    • Use HTTPS instead of HTTP
    • Retry to fetch a page if it contains no symbols (A "fix" for issue #4)
    • Renamed all 'Curreny' to 'Currency'
    • Relative imports are used
    • Fix: .csv file it outputs is encoded in UTF-8 when using python2
    • Performance: Considerable reduced memory consumption
    • It now outputs .json, .yaml and .xls files in addition to .csv