Iteration through the Screener object fails
Closed this issue · 4 comments
To start with, this is the filter I apply when screening for stocks: filters = ['sh_curvol_o300', 'ta_highlow52w_b0to10h', 'ind_stocksonly']
. When I print the length of the Screener object, the result is 515 (as I write this). However, when I try to iterate through the object and print the ticker symbols only, the iteration stops before it's completed. As a result, I can see only first tickers.
Any idea why is this happening? Or even better: is there a working solution?
Thank you very much.
To start with, this is the filter I apply when screening for stocks:
filters = ['sh_curvol_o300', 'ta_highlow52w_b0to10h', 'ind_stocksonly']
. When I print the length of the Screener object, the result is 515 (as I write this). However, when I try to iterate through the object and print the ticker symbols only, the iteration stops before it's completed. As a result, I can see only first tickers.
Any idea why is this happening? Or even better: is there a working solution?Thank you very much.
Can you post a code sample to reproduce the problem?
Hopefully, the following code illustrates what seems to be the issue. Although I am able to retrieve the Screener object, I am not able to iterate through it to have tickers saved in my own list.
Any idea what am I missing?
from finviz.screener import Screener
filters = ['sh_curvol_o300', 'ta_highlow52w_b0to10h', 'ind_stocksonly']
_stock_list = Screener(filters=filters)
print(f'Number of stocks found: {len(_stock_list)}')
tickers = []
for stock in _stock_list:
tickers.append(stock['Ticker'])
print(f'Number of tickers available: {len(tickers)}')
Here's what I've got:
from finviz.screener import Screener
filters = ['sh_curvol_o300', 'ta_highlow52w_b0to10h', 'ind_stocksonly']
_stock_list = Screener(filters=filters)
print(_stock_list)
No. | Ticker | Company | Sector | Industry | Country | Market Cap | P/E | Price | Change | Volume
--- | ------ | ---------------------------------------- | ---------------------- | ----------------------------------- | -------------- | ---------- | -------- | ------- | ------ | ----------
21 | ALB | Albemarle Corporation | Basic Materials | Specialty Chemicals | USA | 10.36B | 23.42 | 96.48 | 1.60% | 637,660
22 | ALC | Alcon Inc. | Healthcare | Medical Instruments & Supplies | Switzerland | 30.49B | - | 62.34 | 1.91% | 1,161,419
23 | ALV | Autoliv, Inc. | Consumer Cyclical | Auto Parts | Sweden | 7.34B | 52.66 | 84.94 | 0.88% | 730,671
24 | ALXN | Alexion Pharmaceuticals, Inc. | Healthcare | Biotechnology | USA | 26.20B | 32.58 | 121.61 | 1.62% | 994,100
25 | AMAG | AMAG Pharmaceuticals, Inc. | Healthcare | Biotechnology | USA | 472.03M | - | 13.67 | 0.07% | 713,047
26 | AMCR | Amcor plc | Consumer Cyclical | Packaging & Containers | Switzerland | 17.16B | 28.27 | 10.97 | -0.63% | 3,024,979
27 | AME | AMETEK, Inc. | Industrials | Specialty Industrial Machinery | USA | 24.56B | 27.93 | 107.76 | 0.70% | 880,223
28 | AMH | American Homes 4 Rent | Real Estate | REIT - Residential | USA | 9.11B | 109.34 | 29.96 | -0.83% | 1,299,784
29 | AMP | Ameriprise Financial, Inc. | Financial | Asset Management | USA | 20.62B | 8.91 | 170.39 | 0.04% | 492,842
30 | AMZN | Amazon.com, Inc. | Consumer Cyclical | Internet Retail | USA | 1637.38B | 123.17 | 3204.40 | 0.88% | 3,427,388
31 | ANSS | ANSYS, Inc. | Technology | Software - Application | USA | 28.49B | 71.97 | 330.22 | 1.05% | 349,885
32 | AOS | A. O. Smith Corporation | Industrials | Specialty Industrial Machinery | USA | 8.94B | 30.20 | 54.93 | 0.04% | 773,937
33 | APAM | Artisan Partners Asset Management Inc. | Financial | Asset Management | USA | 3.38B | 15.87 | 43.25 | -0.35% | 404,358
34 | APD | Air Products and Chemicals, Inc. | Basic Materials | Chemicals | USA | 64.47B | 34.08 | 294.00 | 0.60% | 477,477
35 | APG | APi Group Corporation | Industrials | Engineering & Construction | USA | 2.59B | - | 15.32 | 0.52% | 551,259
36 | APPN | Appian Corporation | Technology | Software - Infrastructure | USA | 5.35B | - | 78.28 | -1.17% | 466,328
37 | APTV | Aptiv PLC | Consumer Cyclical | Auto Parts | Ireland | 26.93B | 15.27 | 100.29 | 0.14% | 1,886,188
38 | AQN | Algonquin Power & Utilities Corp. | Utilities | Utilities - Renewable | Canada | 9.31B | 16.39 | 15.72 | 0.58% | 410,354
39 | AQUA | Evoqua Water Technologies Corp. | Industrials | Pollution & Treatment Controls | USA | 2.74B | 33.74 | 23.65 | 0.42% | 352,531
40 | ARE | Alexandria Real Estate Equities, Inc. | Real Estate | REIT - Office | USA | 20.23B | 49.11 | 160.54 | 0.24% | 385,475
41 | ARES | Ares Management Corporation | Financial | Asset Management | USA | 11.44B | 99.18 | 44.83 | 1.01% | 306,696
42 | ARNA | Arena Pharmaceuticals, Inc. | Healthcare | Biotechnology | USA | 4.73B | - | 81.34 | 1.23% | 336,929
43 | ARNC | Arconic Corporation | Industrials | Metal Fabrication | USA | 2.60B | 17.72 | 23.89 | 0.59% | 577,068
44 | ARRY | Array Technologies, Inc. | Technology | Solar | USA | 4.99B | - | 42.00 | -4.42% | 3,296,685
45 | ARW | Arrow Electronics, Inc. | Technology | Electronics & Computer Distribution | USA | 6.48B | 17.63 | 83.49 | -0.67% | 456,394
46 | ASML | ASML Holding N.V. | Technology | Semiconductor Equipment & Materials | Netherlands | 161.00B | 40.68 | 378.56 | 0.81% | 467,477
47 | ASO | Academy Sports and Outdoors, Inc. | Consumer Cyclical | Specialty Retail | USA | 1.26B | - | 14.58 | -1.22% | 496,672
48 | ATHM | Autohome Inc. | Communication Services | Internet Content & Information | China | 11.25B | 24.62 | 96.66 | -3.08% | 364,362
49 | ATSG | Air Transport Services Group, Inc. | Industrials | Integrated Freight & Logistics | USA | 1.79B | 29.27 | 29.88 | 1.98% | 719,648
50 | ATVI | Activision Blizzard, Inc. | Communication Services | Electronic Gaming & Multimedia | USA | 62.34B | 34.59 | 81.04 | 0.33% | 4,299,932
51 | AVGO | Broadcom Inc. | Technology | Semiconductors | USA | 151.28B | 69.86 | 372.72 | -0.17% | 885,677
52 | AVID | Avid Technology, Inc. | Communication Services | Electronic Gaming & Multimedia | USA | 412.34M | 30.16 | 9.35 | 5.06% | 1,355,458
53 | AVTR | Avantor, Inc. | Basic Materials | Specialty Chemicals | USA | 14.10B | 106.41 | 24.90 | -2.16% | 11,183,890
54 | AVXL | Anavex Life Sciences Corp. | Healthcare | Biotechnology | USA | 338.18M | - | 5.72 | 8.94% | 2,092,509
55 | AVY | Avery Dennison Corporation | Industrials | Business Equipment & Supplies | USA | 11.73B | 22.97 | 141.74 | 0.62% | 680,662
56 | AVYA | Avaya Holdings Corp. | Technology | Software - Application | USA | 1.53B | - | 17.99 | -3.85% | 1,013,464
57 | AWK | American Water Works Company, Inc. | Utilities | Utilities - Regulated Water | USA | 27.95B | 43.71 | 154.08 | 0.31% | 422,702
58 | AXNX | Axonics Modulation Technologies, Inc. | Healthcare | Medical Devices | USA | 1.97B | - | 49.94 | 2.93% | 367,271
59 | AY | Atlantica Sustainable Infrastructure plc | Utilities | Utilities - Renewable | United Kingdom | 3.06B | 181.92 | 30.38 | 1.17% | 376,463
60 | BABA | Alibaba Group Holding Limited | Consumer Cyclical | Internet Retail | China | 817.11B | 31.76 | 309.92 | 1.19% | 6,220,632
81 | BRO | Brown & Brown, Inc. | Financial | Insurance Brokers | USA | 13.18B | 30.12 | 46.72 | 0.17% | 791,337
82 | BSIG | BrightSphere Investment Group Inc. | Financial | Asset Management | USA | 1.22B | 6.76 | 15.19 | 0.33% | 318,998
83 | BYD | Boyd Gaming Corporation | Consumer Cyclical | Resorts & Casinos | USA | 3.76B | - | 34.70 | 0.61% | 1,503,009
84 | CAG | Conagra Brands, Inc. | Consumer Defensive | Packaged Foods | USA | 18.50B | 18.67 | 38.00 | 0.93% | 2,127,897
85 | CALX | Calix, Inc. | Technology | Software - Application | USA | 1.62B | - | 26.05 | 1.36% | 935,098
86 | CAT | Caterpillar Inc. | Industrials | Farm & Heavy Construction Machinery | USA | 89.29B | 22.71 | 168.59 | -0.63% | 2,848,034
87 | CC | The Chemours Company | Basic Materials | Specialty Chemicals | USA | 3.58B | - | 21.88 | 0.88% | 556,543
88 | CCCC | C4 Therapeutics, Inc. | Healthcare | Biotechnology | USA | 1.09B | - | 26.09 | -2.83% | 428,537
89 | CCIV | Churchill Capital Corp IV | Financial | Shell Companies | USA | 1.99B | - | 9.63 | -0.21% | 2,067,550
90 | CCK | Crown Holdings, Inc. | Consumer Cyclical | Packaging & Containers | USA | 12.06B | 25.16 | 90.38 | -1.16% | 1,043,833
91 | CCS | Century Communities, Inc. | Consumer Cyclical | Residential Construction | USA | 1.49B | 9.94 | 44.01 | 4.54% | 332,115
92 | CDAY | Ceridian HCM Holding Inc. | Technology | Software - Application | USA | 13.55B | 183.32 | 92.21 | 2.96% | 766,497
93 | CDNA | CareDx, Inc | Healthcare | Diagnostics & Research | USA | 2.49B | - | 50.18 | -1.12% | 328,344
94 | CDNS | Cadence Design Systems, Inc. | Technology | Software - Application | USA | 31.31B | 30.85 | 112.05 | 2.50% | 1,065,967
95 | CE | Celanese Corporation | Basic Materials | Chemicals | USA | 14.15B | 22.53 | 120.18 | 1.79% | 1,083,653
96 | CERN | Cerner Corporation | Healthcare | Health Information Services | USA | 22.25B | 44.46 | 73.72 | 1.56% | 847,895
97 | CGNX | Cognex Corporation | Technology | Scientific & Technical Instruments | USA | 11.81B | 86.76 | 69.93 | 0.45% | 430,757
98 | CHD | Church & Dwight Co., Inc. | Consumer Defensive | Household & Personal Products | USA | 23.20B | 32.05 | 91.91 | 1.01% | 957,878
99 | CHGG | Chegg, Inc. | Consumer Defensive | Education & Training Services | USA | 10.74B | 17352.00 | 86.76 | 0.24% | 1,897,805
100 | CHKP | Check Point Software Technologies Ltd. | Technology | Software - Application | Israel | 17.03B | 21.16 | 120.47 | -0.44% | 953,816
381 | RCII | Rent-A-Center, Inc. | Industrials | Rental & Leasing Services | USA | 1.77B | 11.50 | 32.76 | 1.42% | 413,848
382 | RCKT | Rocket Pharmaceuticals, Inc. | Healthcare | Biotechnology | USA | 1.56B | - | 28.10 | -2.45% | 304,440
383 | RCM | R1 RCM Inc. | Healthcare | Health Information Services | USA | 2.04B | 171.27 | 17.47 | 0.46% | 600,684
384 | REPL | Replimune Group, Inc. | Healthcare | Biotechnology | USA | 1.82B | - | 45.20 | 4.17% | 1,400,835
385 | RESI | Front Yard Residential Corporation | Real Estate | REIT - Residential | USA | 778.05M | - | 13.38 | 0.45% | 918,355
386 | REXR | Rexford Industrial Realty, Inc. | Real Estate | REIT - Industrial | USA | 6.02B | 105.72 | 48.21 | 0.33% | 493,836
387 | RH | RH | Consumer Cyclical | Specialty Retail | USA | 7.78B | 47.12 | 394.08 | 4.05% | 639,683
388 | ROK | Rockwell Automation, Inc. | Industrials | Specialty Industrial Machinery | USA | 28.01B | 36.87 | 243.10 | 1.58% | 562,222
389 | ROKU | Roku, Inc. | Communication Services | Entertainment | USA | 28.45B | - | 223.96 | 0.72% | 2,136,579
390 | ROL | Rollins, Inc. | Consumer Cyclical | Personal Services | USA | 19.63B | 92.76 | 60.20 | 1.26% | 758,540
391 | RPD | Rapid7, Inc. | Technology | Software - Application | USA | 3.34B | - | 65.71 | 2.40% | 381,492
392 | RPM | RPM International Inc. | Basic Materials | Specialty Chemicals | USA | 11.61B | 30.89 | 89.70 | 0.63% | 471,450
393 | RRC | Range Resources Corporation | Energy | Oil & Gas E&P | USA | 2.43B | - | 9.09 | -3.19% | 3,667,773
394 | RS | Reliance Steel & Aluminum Co. | Basic Materials | Steel | USA | 7.09B | 15.88 | 111.24 | 0.45% | 374,862
395 | RSG | Republic Services, Inc. | Industrials | Waste Management | USA | 29.53B | 28.09 | 92.87 | 1.20% | 696,889
396 | RTRX | Retrophin, Inc. | Healthcare | Biotechnology | USA | 1.04B | - | 20.95 | -0.62% | 405,474
397 | RXN | Rexnord Corporation | Industrials | Specialty Industrial Machinery | USA | 3.96B | 25.15 | 32.64 | 0.83% | 404,370
398 | RYAAY | Ryanair Holdings plc | Industrials | Airlines | Ireland | 16.80B | 85.39 | 89.75 | 1.70% | 508,473
399 | SAIL | SailPoint Technologies Holdings, Inc. | Technology | Software - Infrastructure | USA | 4.24B | 1110.24 | 45.52 | 0.75% | 820,329
400 | SBAC | SBA Communications Corporation | Real Estate | REIT - Specialty | USA | 33.85B | - | 302.11 | 2.75% | 631,832
I think the issue might have something to do with the async requests. There may not be enough of a delay, and as a result, Finviz may block some of them.
I've taken a look at Mario's code, and as long as I kept the async requests, the error kept occurring. At a lower level. the HTML returns b'Too many requests.'
instead of the table we want. For the example detailed above, there are 25 requests being made (all at the same time).
The best solution for this would be to remove async requests, and institute sequential requests with a set delay to avoid setting off any alarms. It might take a bit longer to get your data, but it should maintain the integrity. I've just test 5 second delays, which work, but I'm going to see if I can reduce it. I'm also going to add an error message to avoid the silent failure.