decrypto-org/spider

Paths & Base URLs not matching up

Closed this issue · 1 comments

By the pigeonhole principle, we should have at least one path per base URL, which was not the case in the latest scrape.
Plan: Do a lower level scrape and test on this one to see what happened.
If we find an issue, write a unit test (end to end) to reveal the issue and fix it.
Otherwise, do a larger depth scrape and check whether we then encounter the issue.

Could not replicate the behaviour