Missing `follow redirects` feature

[bugbaba]

Hey,

While trying out this, I noticed that it is missing the follow redirects feature which both dirsearch and gobuster has.

It surely helps with the servers that redirects every request like:

1. port 80 redirecting request to 443
2. redirecting to add forward slash at the end.

Thanks

[Isona]

Hi,

Thanks for the issue, I actually had this feature in an earlier version of Dirble by enabling libcurl's follow redirects flag, but removed it in bf12ab4 for a few reasons.

These are:

1. Following redirects might take Dirble out of scope (including the https version of a website which may be different) and it feels better for it to continue scanning the sites which are in scope
2. It breaks directory detection - a directory is detected by checking if a request has been redirected from [url] to [url]/ (your second point), following a redirect would turn this into a different status code
3. Dirble displays the location that a redirect goes to in its report

If there are good use cases for redirect following then I would consider reimplementing it in the future, but feel like it would need a lot more time devoted to it than the previous solution of simply enabling the flag in libcurl.

[bugbaba]

Surely those are fair reasons for redirects on 10% of wordlist entries are ok.
But for endpoints that redirects to [url]/ for all entries in those case dirble kind of become's unusable.

Besides this, I did some benchmarks of dirble vs gobuster. Results were in favour of gobuster will be doing the same with dirsearch tomorrow.

What flags did you guys used to compare ? I am currently using with following command with same VPS, wordlist, thread count, URL endpoint.

./dirble -k -r -w wordlist -u https://domain.tld

[Isona]

Sorry, I'm still misunderstanding your point about when a [url] is redirected to [url]/, in these cases, Dirble will detect that they are a directory and add them to the scan queue, the scan of the directory should detect any indexes in the folder (wordlist dependent).

In the case of dirble vs gobuster, be sure to run with these flags to best emulate it:
dirble [url] --wordlist-split 10 -r
By default dirble only spawns 3 threads for each combination of folder/extension to allow for concurrent scans of multiple hosts/directories within its default 10 threads, changing the --wordlist-split argument to 10 causes it to spawn 10 threads for each combination of folder/extension.
In my testing, Gobuster was ~10% faster, though I believe this is because it does less processing on the results.

[bugbaba]

My bad, I was running with -r(recursive mode off) hence it was just printing the redirected as directories.

[Isona]

Thanks for this, I've updated the readme with an explanation of how scanning works, hopefully it will prevent future confusion (other people have expressed similar problems).