add option to normalize URLs so fragment identifiers are ignored

[robfentress]

This is a feature request. It would be good to add a configuration option to ignore fragment identifiers in URLs, so that pages are only scanned once.