Add support for sitemap indices
Closed this issue · 0 comments
wjh18 commented
Is your feature request related to a problem? Please describe.
The sitemap parser only works with direct links to standalone sitemaps, not multiple sitemaps or sitemap indices.
Describe the solution you'd like
Modify the parser to traverse sitemap indices, parse URLs from child sitemaps and include those URLs in the request tasks.
Describe alternatives you've considered
Use a 3rd-party package like advertools which has this functionality out of the box. The downside to this is that there's no way to only install the sitemap parser from this package.
Additional context
n/a