Error API response does not contain a "content_html" field
cpuuntery opened this issue · 19 comments
I tried to download the comments from this video We tried Android 11!
which has 3,330 comments. and using node js LTS Version 12.18.0 running under Debian. the Scraper start fetch the comments but after some time it throws an error
API response does not contain a "content_html" field
the only solution I found is to try five or ten times in hope one time it will not throw an error
and the parameters I used are youtube-comment-scraper -f json -o comments.json 05X0RRmUtE0
and also the error happens with videos with more than 1,000 comments. videos with comments under 1,000 will not throw an error
same.... I guess this Error raised in phil's this library https://github.com/philbot9/youtube-comments-task
He obviously recognize this issue.
maybe few people facing this error.
I wanna scrape huge amounts of comments for NLP.
who can fix these error?
Gimme mercy, phil-san.
got this error as well
Same error on my end. Happens randomly. I've made a feature request here, which would help overcome this issue: #48
I am getting this error consistently on all videos now, and I do not receive any data at all.
Perhaps something has changed on YouTube's end.
Example:
https://www.youtube.com/watch?v=ecIWPzGEbFc
youtube-comment-scraper --stream -- ecIWPzGEbFc | tee output.json
✕ API response does not contain a "content_html" field
I am having the same problem
me too facing same isue
I am also facing the same issue.
@philbot9 can you tell me what is the solution to prevent this error "API response does not contain a "content_html" field"
my current solution
https://github.com/dddat1017/Scraping-Youtube-Comments
@philbot9, please help me. there is an error. previously it worked correctly. after the may 27 update It didn't work. always gives this error "API response does not contain a "content_html" field"
I want to get some youtube comments but your cli program doesn't work now. @philbot9 please give me a solution to get the data. error is "API response does not contain a "content_html" field"
my current solution
https://github.com/dddat1017/Scraping-Youtube-Comments
this is the alternative solution.it works for me.thank you @hissatupassenger
Hi! I have the same error in all the videos of a Youtube channel. Have somebody found a solution? Thank you!
https://github.com/dddat1017/Scraping-Youtube-Comments
Note, only the main comments will be extracted, not any of the replies that may be under the comment.
I'm glad this is a solution for some of you, but to me it's next to worthless without replies.
I'd still rather continue using this program, but I created an issue for that other one:
dddat1017/Scraping-Youtube-Comments#1
I saw it reading the issue, but I need replies too... I hope @philbot9 can find a solution, I'm not a JS expert and I didn't find a solution yet.
A couple of weeks ago Youtube's frontend was updated. It used to return plain html that already contained data such as search results and , I'm guessing, comments as well. Now it returns html with the navbar but no data on the body, I've seen people found the data on embeded script tags on the first html response but afterwards data is loaded with ajax calls. I noticed the change on my own library for scrapping search results and on other similar python packages.
It may be posible to locate the endpoints from which the ajax calls are being made and get the data using them.
Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader
Perhaps it can give an ideia of how to solve this current problem, since the request is apparently made to the same url, but the script above uses this url to scrape only some comments from livestreams... to download all comments and replies from regular videos, another (similar) url is used, which the author calls "old url" because it's somehow related to the old youtube API. Anyway, I'm not into js so I couldn't find out what's going with Phil's code or what has changed in the response and possible modifications to the request to solve this issue.
Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader
Thanks for this; it works for me.
It's nowhere near a replacement tool for philbot9's though, but I do hope it provides insight into any necessary changes.
Here is a simple and working (python) script to scrape comments AND replies, but unfortunately it has less features, e.g., no timestamp: https://github.com/egbertbouman/youtube-comment-downloader
Thanks, it's a good replacement for now