JustSySy/WebVideoBot

Web crawler.

Web Vedeo Bot

• Easy install, all dependencies are already included, just need to have a java environment.

Environment, Architecture

• Java1.8

• Crawler4j

• Spring Boot x H2 Db

Run

java -jar PornBot.jar

Database Description

http://localhost:8000/h2-console/

JDBC URL: jdbc:h2:~/porn/porn-db

User Name: sa

Password: empty

Record Table：

 Table_Name               ：PORN_RECORD
 viewKey                  ：The website's video unique key.
 imageUrl                 ：Image url of video.
 linkUrl                  ：Video jump to Website`s link
 videoUrl                 ：Video adrress.
 videoTitle               ：Title of video.
 videoDuration            ：Video click count.
 videoQuality             ：Defualt quality - 240, 480, 960, 1280p.
 download                 ：Has been downloaded. True or false.
 createdTime              ：The record created time.
 filePath                 ：The video downloaded path.

Configuration

 FILE_PATH                ：Video download path.
 VIDEO_DOWNLOAD_SIZE      ：Maximum download size of the video.(Byte)
 MAX_PAGE_SIZE            ：Crawling page size.
 CONCURRENT_THREAD_SIZE   ：Muilti-threads request amount.
 START_URL                ：Crawling url.
 DOWNLOAD_VIDEO           ：Download video. Y or N.

See Default Configuration