kgretzky/dcrawl

Suggestion

Opened this issue · 1 comments

It would be nice to be able to choose User-Agent or, even better and not so complicated either, to create a random user agent for each requests. Something like they did with Go-Colly

Check it out at https://github.com/gocolly/colly/blob/master/extensions/random_user_agent.go

This require some code rewrite but it seems this repo is dead.