Please use a custom User Agent header
Closed this issue · 3 comments
mattdennewitz commented
Please use a custom User Agent header so that we know you're playing nice. Thanks!
michalczaplinski commented
Im guessing you mean modifying the default user agent string that is sent by urllib2 "Python-urllib/2.6" to something else? Excuse my ignorance, but what is a purpose of it other than allowing the server admins to see what app reads their page? Is there a commonly accepted practice on what should go in there, like author+app name?
mattdennewitz commented
So that we can see your scraper is exactly why I'm asking. I'll defer to you for what the string could be, but something as simple as michalczaplinski/pitchfork-1.0
would be helpful.
michalczaplinski commented
fixed! thanks for the feedback.