info.westberks.gov.uk
Closed this issue · 2 comments
PhilRumens commented
Our main information site is info.westberks.gov.uk
KevinJump commented
yeah the crawl currently finds around 95 docs from starting with www. but these are also on info so I might just change the primary domain and see what we get :)
we will still get a deflated page count because all pages follow the form index.aspx?articleid=
The crawler will crawl all pages with query string parameters, but it counts them as just one page. Its about knowing what is a page ( index.aspx?articleid=1 and index.aspx?articleid=2 ) and what is just rouge bits of query string (quite often &acceptcookie=1 or &send_to_friend) .
KevinJump commented
Updated, and latest db put onto site, so you can at least now see the crawl data