joshsh/ripple

52 - make use of robots.txt

Opened this issue · 0 comments

joshsh commented 
LinkedDataSail should request, parse, and comply with the robots.txt of domains which supply one.

Original link: http://code.google.com/p/ripple/issues/detail?id=52