No more posts while gathering *lot* of posts based on hashtags
mandys opened this issue · 0 comments
Describe the bug
The script works fine if I am trying to pull around 500 hashtags ( most of the time ). But, if the hashtags have let's say about 1000+ results, the response is unpredictable. The script is looping fine, 30 results at a time.
For my use case: the hashtag has more than 10,000+ results. I have set number=1000.
Every single time it's looping, it stops around -
https://m.tiktok.com/api/challenge/item_list/?challengeID=&count=30&cursor=800&aid=1988&verifyFp=&_signature=
{
"statusCode": 0,
"cursor": "800",
"hasMore": false
}
I think in TikTok.js it reaches here
if ((updatedApiResponse && !result.itemList) || (!updatedApiResponse && !result.items)) {
throw new Error('No more posts');
}
In this scenario -
-
The script should return the results since it went through 26 cycles before encountering no itemList. At least whatever it scraped should be returned ? Right now it just aborts saying "No more posts".
-
Why is this stopping around 799 ( total ) results when the hashtag has 10,000+ results. In fact even on tiktok web interface, I wasn't able to consistently scroll past 500 odd results. They stopped loading post that. So, does tiktok not give back all the results ? Only upto a particular date ? Any idea ?
If fields below will be empty then issue will be Ignored and Closed
To Reproduce
Steps to reproduce the behavior:
-
Used OS
[ Mac OS X ] -
NodeJs Version (node -v)
[ 12.16.1] -
Scraper Version (tiktok-scraper --version)
[1.4.36] -
CLI or Module
[ Module ] -
Used method
TikTokScraper.hashtag -
Used options
const posts = await TikTokScraper.hashtag(hashtag, {
number: 700,
sessionList: ['sid_tt=123;'],
}); -
Do you run the scraper through Proxy, VPN, VPS?
VPN and also through an AWS box -
If scraping from a profile. Did you checked if profile is Public? Scraper won't work with Private profiles!
[ Scraping hashtags which are public ] -
Preferably provide URL to the profile,hashtag,video and etc page
https://www.tiktok.com/tag/pandya
Screenshots
If applicable, add screenshots to help explain your problem.