#Webp crawling application
Read files in url-test folder and invoke URLs
- Successful invoked URLs are stored in my.db
- Failed invoked URLs are stored in output/failed/* (with file rotation)
Clear my.db to run again (file checkpoint is cached here)
Simple web crawler written in Golang, using BoltDB and file rotation
Go
#Webp crawling application
Read files in url-test folder and invoke URLs
Clear my.db to run again (file checkpoint is cached here)