Is there a way to add own URLs down the road while crawling is taking place?

Question

Is there a way to add own URLs down the road while crawling is taking place?

Opened this issue 6 years ago · 1 comments

Right now, it only parses HTML to get the URLs, and while I have written code that parses JS(both inside an HTML file, and in asset files), and gets all those URLs, but now I want those URLs to be a part of this process. Is there a way to do so, like any workaround?

Answer 1 · 2019-03-04T16:24:59.000Z

can you provide an example of how that might work? are you suggesting perhaps a hook method where you can supply your own block to extract urls?