lgraubner/sitemap-generator-cli

Too much ignore pages

bshafiei-ir opened this issue · 5 comments

bshafiei-ir commented

Hi
I want to crawl my website but i get 128 ignore pages and just 4 added page. What is problem?

I ran this command:
sitemap-generator -v -q https:\www.landa-sme.ir

result:
Added 4 pages, ignored 128 pages, encountered 0 errors.

dottodot commented

I'm having the same issue got 100's of pages and it only adds 13 random ones. Not sure what the issue is as it's detecting the urls just not adding them.

lgraubner commented

Interesting. I think it ignores them falsely because of connection issues.

@dottodot Can you provide an URL to check against?

dottodot commented

Yes this is the one I'm trying https://www.boxxerworld.com

bshafiei-ir commented

Interesting. I think it ignores them falsely because of connection issues.

@dottodot Can you provide an URL to check against?

And please check www.landa-sme.ir
thanks

thibaultmol commented

For me it just queries the index.php page a bunch and then claims it's done

~/Downloads >>> sitemap-generator https://wiki.openstreetmap.org/wiki/Main_Page -f osmwiki.xml -v                                                                                                                             
(node:1158170) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
[ ADD ] https://wiki.openstreetmap.org/wiki/Main_Page
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php