TypeError: 'NoneType' object is not subscriptable

Question

TypeError: 'NoneType' object is not subscriptable

Closed this issue 4 years ago · 19 comments

python3 ./scrape.py -u jack

Traceback (most recent call last):
  File "./scrape.py", line 276, in <module>
    begin = datetime.strptime(args.since, DATE_FORMAT) if args.since else get_join_date(args.username)
  File "./scrape.py", line 256, in get_join_date
    date_string = soup.find("span", {"class": "ProfileHeaderCard-joinDateText"})["title"].split(" - ")[1]
TypeError: 'NoneType' object is not subscriptable

Answer 1 · 2020-06-03T01:58:38.000Z

shit.... i hope they didn't change up their front end... i just checked the HTML and it seems like they've discarded the coherent class names for some garbled stuff... class="css-901oao css-16my406 r-1re7ezh r-4qtqp9 r-1qd0xha r-ad9z0x r-zso239 r-bcqeeo r-qvutc0"...

will need to investigate this. if they changed this up, the soup parser will need to be updated so it can find the start date.

As a workaround, you can manually specify the --since parameter so that it doesn't try to figure it out.

Answer 2 · 2020-06-03T21:26:25.000Z

--since works, and it successfully opens Chrome and shows each page of search results, but it doesn't seem to find any tweets:

 python3 ./scrape.py -u jack --since 2020-01-01
[ scraping user @jack... ]
[ 0 existing tweets in jack.json ]
[ searching for tweets... ]
[ found 0 new tweets ]
[ retrieving new tweets (estimated time: 0 seconds)... ]
[ finished scraping ]
[ stored tweets in jack.json ]

cat jack.json
{}%

It's using the old Twitter UI but perhaps something changed?

Answer 3 · 2020-06-05T07:06:01.000Z

they've re-written their front end, so all the HTML parsing in this repo will need to be updated ):

tweet_selector = "li.js-stream-item" will need to be updated with the new identifier for tweets
soup.find("span", {"class": "ProfileHeaderCard-joinDateText"}) will need to be updated to find the classes css-901oao css-16my406 r-1re7ezh r-4qtqp9 r-1qd0xha r-ad9z0x r-zso239 r-bcqeeo r-qvutc0 instead

Answer 4 · 2020-06-07T21:56:53.000Z

All right, I updated it to work with the new CSS, both in terms of tweet scraping and figuring out user join date. Let me know if it works for you now

Answer 5 · 2020-06-07T22:00:14.000Z

python3 ./scrape.py -u jack --since 2020-01-01

[ scraping user @jack... ]
[ 0 existing tweets in jack.json ]
[ searching for tweets... ]
Traceback (most recent call last):
  File "./scrape.py", line 279, in <module>
    user.scrape(begin, end, args.by, args.delay)
  File "./scrape.py", line 101, in scrape
    self.__find_tweets(start, end, by, loading_delay)
  File "./scrape.py", line 203, in __find_tweets
    tweet_id = locate_tweet_id(tw)
  File "./scrape.py", line 170, in locate_tweet_id
    tweet_id_link = list(filter(lambda tw: "status" in tw.get_attribute("href"), subelements))[0]
IndexError: list index out of range

Answer 6 · 2020-06-07T22:20:23.000Z

just pushed again, should be good now

> ./scrape.py -u jack --since 2020-01-01                                                                                                                                             [5:04:58]
[ scraping user @jack... ]
[ 0 existing tweets in jack.json ]
[ searching for tweets... ]
no tweets in time period 2020-02-05 -- 2020-02-12
no tweets in time period 2020-04-15 -- 2020-04-22
[ found 128 new tweets ]
[ retrieving new tweets (estimated time: 12 seconds)... ]
- batch 1 of 2
- batch 2 of 2
[ finished scraping ]
[ stored tweets in jack.json ]

Answer 7 · 2020-06-07T22:35:24.000Z

Working now, thanks! Although I found slightly fewer tweets than you so that might be a bug (or something weird with Twitter)

Either way I've uploaded jack.json in case you want to look into that:

https://pastebin.com/raw/GeAzTQmN

python3 ./scrape.py -u jack --since 2020-01-01

[ scraping user @jack... ]
[ 0 existing tweets in jack.json ]
[ searching for tweets... ]
> no tweets in time period 2020-02-05 -- 2020-02-12
[ found 110 new tweets ]
[ retrieving new tweets (estimated time: 12 seconds)... ]
- batch 1 of 2
- batch 2 of 2
[ finished scraping ]
[ stored tweets in jack.json ]

Quick question, is there support for cookies? I'd like to scrape tweets from a private account but at the moment it doesn't look like that's possible.

Answer 8 · 2020-06-08T06:48:14.000Z

Quick question, is there support for cookies? I'd like to scrape tweets from a private account but at the moment it doesn't look like that's possible.

The only way to scrape a private account is to have Twitter API credentials for an account that is following the private account, otherwise the tweets just aren't visible.

I found slightly fewer tweets

I noticed this too... Sadly, scraping tweets can be hit-or-miss. I adapted and rewrote portions of some old code to create this repo, and the only sure-fire way to get better results has been to increase the --delay and decrease the --by parameter such that no tweets are missed.

If you want to further investigate the core logic of the scraping, you could look at these lines of code. I think there's definitely room for improvement, as the increment variable is a bit opaque (in terms of the while condition).

Answer 9 · 2020-06-08T12:46:24.000Z

The only way to scrape a private account is to have Twitter API credentials for an account that is following the private account, otherwise the tweets just aren't visible.

Ah, you're right. Well that sucks. from: seems to be limited to just a week. to: doesn't appear to have a restriction, so it's possible to find some older tweets that way, but doesn't help for tweets with no replies.

If you want to further investigate the core logic of the scraping, you could look at these lines of code. I think there's definitely room for improvement, as the increment variable is a bit opaque (in terms of the while condition).

I'll take a look and see :-)

Answer 10 · 2020-10-12T14:10:29.000Z

they've re-written their front end, so all the HTML parsing in this repo will need to be updated ):

tweet_selector = "li.js-stream-item" will need to be updated with the new identifier for tweets
soup.find("span", {"class": "ProfileHeaderCard-joinDateText"}) will need to be updated to find the classes css-901oao css-16my406 r-1re7ezh r-4qtqp9 r-1qd0xha r-ad9z0x r-zso239 r-bcqeeo r-qvutc0 instead

Hi Matt,

Thanks for the code. Really helpful. Though am unable to understand where I need to make these changes in the code. I changed the value for tweet_selector. However, I don't understand where I need to introduce the soup.find code and the classes part. Request you to help me out here.

Thanks in advance.

Answer 11 · 2020-10-12T14:13:07.000Z

Facing the same problem, the twitter page with the search query and the tweets shows up, but none of it gets captured in the generated json file.

Answer 12 · 2020-10-16T07:56:11.000Z

@nmalcolm can you help me out here, if you were able to run the script?

Answer 13 · 2020-10-16T08:32:05.000Z

Did it stop working again? If so, they must have changed up the CSS classes again. Sorry for the delayed reply, coursework is hitting hard lately. i’ll try and take a peek in the next 24 hours and run some tests

Answer 14 · 2020-10-17T00:39:50.000Z

Facing the same problem, the twitter page with the search query and the tweets shows up, but none of it gets captured in the generated json file.

I think I'm just going to re-write the Selenium portion to have more robust scraping. It's currently using auto-generated CSS tags, which are subject to change. It's a lame fix for when they switched up their front end CSS. I'll try to hit it this weekend, but I've got a lot on my plate.

Here's a primer on text extraction with Selenium though

Answer 15 · 2020-10-18T07:14:37.000Z

Facing the same problem, the twitter page with the search query and the tweets shows up, but none of it gets captured in the generated json file.

I think I'm just going to re-write the Selenium portion to have more robust scraping. It's currently using auto-generated CSS tags, which are subject to change. It's a lame fix for when they switched up their front end CSS. I'll try to hit it this weekend, but I've got a lot on my plate.

Here's a primer on text extraction with Selenium though

Thank you @MatthewWolff. Please don't prioritize this, whenever you're done with all the important stuff, and if you can find time for this later, then please help. I will meanwhile try to go through the primer and figure it out by myself. Thanks again for the code.

Thanks,
Sandeep

Answer 16 · 2020-10-21T01:17:03.000Z

@sandeep17oct Got around to peeking at it. They changed their CSS again, and it's auto-generated to begin with, so the former solution here wasn't very great. I've modified it to be more robust now—I limited Selenium usage and there is no longer any dependency on CSS classes, it just parses out all of the target user's tweet ids on a page.

Additionally, the driver now runs headless (yay), but you can turn on a debug option if desired.

Let me know if it works better for you.

Answer 17 · 2020-10-21T05:25:17.000Z

Modified scraper to no longer search using CSS tags, this bug can no longer occur

Answer 18 · 2020-10-21T07:41:43.000Z

Thank you @MatthewWolff for the quick help. Works like a charm. 💯 👍

Answer 19 · 2020-10-22T08:10:06.000Z

Hi @MatthewWolff, Quick question: have you tried scraping a handle with a very large number of tweets? I tried scraping sadhguruJV handle, which had about 10K tweets, got roughly 2.7K tweets. Also, it seems that the variation is larger, the longer the script runs. So, year 1 might fetch the accurate number of tweets, year 2 a few missing, year 3 a few more...and so on. Apart from the delay and increment variable (by), is there a way to tune this to reduce the misses?