How is this data obtained from Thunderskill
axiangcoding opened this issue · 3 comments
How is this data obtained from Thunderskill, crawling html like scrapy? Is this part of the code public?
I am investigating the feasibility of a project, and I also need to obtain data from thunderskill, so I want to know how you did it
thanks
How is this data obtained from Thunderskill, crawling html like scrapy?
Yes, directly scrapped from HTML.
Is this part of the code public?
In the previous, I made it public, but Thunderskill developers use Cloudflare to counter the crawler, maybe because they saw the source code. So, I removed it to make sure the data can be retrieved in long term.
I am investigating the feasibility of a project, and I also need to obtain data from Thunderskill, so I want to know how you did it
You are free to use the data here. I believe it includes all information in Thunderskill.
Thank you for your answer! I have learned about how to crawling data from both thunderskill and gaijin official website, and I can export data from it.
However, I agree with what you said, to prevent abuse, we should reuse the data that has been generated
My project is currently under investigation. It is a community-like website aimed at Chinese users. But at present, I am the only one responsible for the development, which may consume a lot of time.
After completing 70% of the progress, I will open it up, maybe by then, I can have the honor to invite you to join my project
Sounds great! Good luck.