decrypto-org/spider

Filter by file type in the url

Opened this issue · 0 comments

jogli5er commented

We found that we hit a lot of image urls, which we won't store anyway, therefore we should filter by file type