[FEATURE] Crawl Sitemap.xml
Opened this issue · 0 comments
tomasnorre commented
Is your feature request related to a problem? Please describe
@benjaminkott suggested me today, to add the feature for crawler the sitemap.xml file.
This would remove some complexity from the crawler that calculates which pages to crawler a which not.
This will especially be helpful for sites generated by plugins like news. Where not every news have its own site.
The Crawler can handle this today, but with a little complexer PageTS.
The sitemap.xml crawling as an option would make this easier.