/web-crawler-golang

Simple web crawler written in Golang, using BoltDB and file rotation

Primary LanguageGo

#Webp crawling application

Read files in url-test folder and invoke URLs

  • Successful invoked URLs are stored in my.db
  • Failed invoked URLs are stored in output/failed/* (with file rotation)

Clear my.db to run again (file checkpoint is cached here)