drawrowfly/tiktok-scraper

No more posts while gathering *lot* of posts based on hashtags

mandys opened this issue · 0 comments

Describe the bug
The script works fine if I am trying to pull around 500 hashtags ( most of the time ). But, if the hashtags have let's say about 1000+ results, the response is unpredictable. The script is looping fine, 30 results at a time.

For my use case: the hashtag has more than 10,000+ results. I have set number=1000.
Every single time it's looping, it stops around -
https://m.tiktok.com/api/challenge/item_list/?challengeID=&count=30&cursor=800&aid=1988&verifyFp=&_signature=

{
"statusCode": 0,
"cursor": "800",
"hasMore": false
}

I think in TikTok.js it reaches here
if ((updatedApiResponse && !result.itemList) || (!updatedApiResponse && !result.items)) {
throw new Error('No more posts');
}

In this scenario -

  1. The script should return the results since it went through 26 cycles before encountering no itemList. At least whatever it scraped should be returned ? Right now it just aborts saying "No more posts".

  2. Why is this stopping around 799 ( total ) results when the hashtag has 10,000+ results. In fact even on tiktok web interface, I wasn't able to consistently scroll past 500 odd results. They stopped loading post that. So, does tiktok not give back all the results ? Only upto a particular date ? Any idea ?

If fields below will be empty then issue will be Ignored and Closed

To Reproduce
Steps to reproduce the behavior:

  • Used OS
    [ Mac OS X ]

  • NodeJs Version (node -v)
    [ 12.16.1]

  • Scraper Version (tiktok-scraper --version)
    [1.4.36]

  • CLI or Module
    [ Module ]

  • Used method
    TikTokScraper.hashtag

  • Used options
    const posts = await TikTokScraper.hashtag(hashtag, {
    number: 700,
    sessionList: ['sid_tt=123;'],
    });

  • Do you run the scraper through Proxy, VPN, VPS?
    VPN and also through an AWS box

  • If scraping from a profile. Did you checked if profile is Public? Scraper won't work with Private profiles!
    [ Scraping hashtags which are public ]

  • Preferably provide URL to the profile,hashtag,video and etc page
    https://www.tiktok.com/tag/pandya

Screenshots
If applicable, add screenshots to help explain your problem.