the-convocation/twitter-scraper

Expose cursor to caller

Closed this issue · 2 comments

0x62 commented

I'm working on an application that needs to be able to scrape large volumes of tweets. This necessitates splitting searches into batches. It would be useful to expose the cursor after completing a query, so that the same cursor could be reused later to resume the search.

karashiiro commented

The fetchSearchTweets function does return the cursor in the response — it's intended for runtimes that don't support async generators, but I believe it works for this use case as well.

  • 👍1
0x62 commented

At the moment it only returns the bottom cursor, but for my use-case I need both the bottom and top cursor (scrape historical, but also keep the results updated as new matches come in).