detect pagination options links
Opened this issue · 0 comments
kmike commented
Websites often provide links "show 20/50/100 results per page"; by following them crawler can get the same contents multiple times. It'd be nice to detect these links.
Opened this issue · 0 comments
Websites often provide links "show 20/50/100 results per page"; by following them crawler can get the same contents multiple times. It'd be nice to detect these links.