NYULibraries/spatial_data_repository

block googlebot from indexing search results

Closed this issue · 1 comments

mnyrop commented 
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

User-agent: * 
Crawl-Delay: 10

Disallow: /catalog*
Disallow: /catalog?*
Disallow: /catalog/?*
Disallow: /catalog?q=*&
Disallow: /catalog/?q=*&
Allow: /catalog?q=*
Allow: /catalog/?q=*
Disallow: /?*
Allow: /?q=*