Twitter-Scraper

This is not a perfect scraper, so feel free to add improvements if you find any.

IMPROVEMENTS:

Improved error handling so that tweets are not rejected if certain fields are null, etc...
Leveraged the WebDriverAwait class to enable better detection of desired load states
Each record is saved while scraping instead of all at the end; minimizing data loss for a failed session.

NOTES AND THINGS TO THINK ABOUT:

The scroll_down_page function has an argument for num_seconds_to_load that represents the num of seconds that the program will wait until attempting to scroll again. I'm currently making 5 attemps with a pause between. You could also increase the number of max attempts and decrease the num_seconds_to_load. This could possibly speed up the scraping as you would be more likely to get to a successfull scroll down quicker.
The collect_all_tweets_from_current_view function has a lookback_limit argument that controls how many tweets are processed from each scroll. I've written more about this in the function docstring.
I've implemented WebDriverWait in several sections of this updated code. I think this is a much better solution than a hard-coded sleep call because it will only timeout after a certain period of time if specific conditions are not met. There are many other sections of this code that could be improved, I'm sure, by leveraging this class.
Feel free to replace the save_tweet_data_to_csv function with any other io option you want, such as a database save via pyodbc, sqlite3, or whatever you want really.
I encourage you to explore the "Advanced Search" functionality. Try adding your criteria and see how the url is built. You can then leverage this to make your searches more customized... with date ranges, special keywords, etc... --> https://twitter.com/search-advanced?

israel-dryer/Twitter-Scraper