the-convocation/twitter-scraper

Unable to get a user replies

louisgrasset opened this issue · 5 comments

I'm trying to get the lastest 50 tweets (expecting tweets and replies) but it sounds like I'm only able to get:

  • tweets
  • retweets.

I'm leveraging getTweets('userHandle', 50) to do so, no auth required.
Am I missing something? Is this a bug or an expected behavior?

From what I see, Twitter isn't allowing non-authenticated webapp to access replies, may this be the reason?

Twitter Webapp not returning replies

I believe that's probably what's happening — I'll have to check if we get more data when authenticated via the API method that uses.

I'd consider this a bug for now, but given that we're at the whims of the company formerly known as Twitter, this might have to become the new expected behavior, at least while unauthenticated.

I'd consider this a bug for now, but given that we're at the whims of the company formerly known as Twitter, this might have to become the new expected behavior, at least while unauthenticated.

It seems like even while authenticated replies don't show when using getTweets

You may be totally true, when I discovered that issue, I moved my code to use the following:
searchTweets(`from:@${handle}`, 50, SearchMode.Latest)

With authentication you can use fetchSearchTweets.
Something along the lines of this:

  const tweets = []
  const maxTweets = 50
  const query = `(from:${userHandle}) filter:replies`
  
  let cursor
  let nTweets = 0
  while (nTweets < maxTweets) {
    const res = await scraper.fetchSearchTweets(
      query,
      maxTweets,
      SearchMode.Latest,
      cursor
    )

    if (res.tweets.length === 0) break

    nTweets += res.tweets.length
    cursor = res.next
    tweets.push(...res.tweets)
  }