BUG - Inconsistent results over the same search

[w0rmr1d3r]

Describe the bug
Blackbird doesn't report consistent results over the same call and not finding true positives in some cases. I tried this with myself, and it cannot find my GitHub profile. Also in some cases, it reports 1 site and in others it reports several.

To Reproduce
Steps to reproduce the behavior:

1. Follow the instructions in the README to clone and install requirements for the project
2. Run blackbird with: python blackbird.py --username w0rmr1d3r
3. Wait for the results to appear
4. Run blackbird with: python blackbird.py --username w0rmr1d3r
5. The results are different

Expected behavior
Even after executing with a couple of seconds between the first and second execution, it should (from my perspective) give the same results.

Screenshots
Screenshot of showing my Pypi profile, but not GitHub:

Desktop (please complete the following information):

• macOS
• Python virtualenv with 3.10.13

Additional context
I saw there's the timeout option to pass, but it did no effect, still displayed inconsistent results. If this is an expected behavior, I haven't seen it described in the README.

PS: Thank you for such a great tool!! 🚀

[p1ngul1n0]

Thank you for reporting the issue. Could you please send me the log file? It is in the 'logs' folder.

[w0rmr1d3r]

Hi @p1ngul1n0 ,
Find it attached here:
blackbird.log

I saw most of the errors are:

root - ERROR - Error in Async HTTP Request [GET] WEBSITE | Cannot connect to host WEBSITE:443 ssl:False [nodename nor servname provided, or not known]

Replace WEBSITE with almost any site blackbird requests.

Hope it helps!

[p1ngul1n0]

It looks like that sending multiple requests simultaneously with aiohttp and asyncio result in this error due to macOS's limitation on the maximum number of concurrent HTTP requests.

To address this, I’ve published a new branch (fix/semaphore-fix) that utilizes Semaphore to limit the number of concurrent requests. This should help manage the load and prevent errors.

By default, the maximum number of concurrent requests is set to 50. However, you can customize this limit by using the --max-concurrent-requests argument.

python blackbird.py --username w0rmr1d3r --max-concurrent-requests 30

Please note that reducing the number of concurrent requests can make the search process slower, as fewer requests will be processed at the same time.

Please give this new branch a try and let me know if it resolves the issue.

[w0rmr1d3r]

hi @p1ngul1n0 ,

I did some tests with the new branch in the same environment I did the previous tests on.

I firstly tried what you mentioned:
python blackbird.py --username w0rmr1d3r --max-concurrent-requests 30

Saw similar (but better) results as without the option. Then tried:

python blackbird.py --username w0rmr1d3r --max-concurrent-requests 1 --timeout 10 --no-update

And brought better results. Of course, it was painful slow.

But if I adjusted the --max-concurrent-requests to 30 or even 10, a similar behavior happens as the initial description I opened.

In summary, the new option available does help improve the results. Could probably needed to be adjusted to really small values if using a Mac.

Different but related topic, the default value as per the description seems to be 10, while you said 50 in your last comment. Current info with the --help command:

 --max-concurrent-requests MAX_CONCURRENT_REQUESTS
                        Specify the maximum number of concurrent requests allowed. Default is 10.

Hope this helps.

[p1ngul1n0]

Hi @w0rmr1d3r,

Thanks for testing the new branch and providing detailed feedback. So it seems like adjusting the --max-concurrent-requests option does help improve the results, even though it's slower with lower values. This address the problem for now, although it’s not the most efficient solution.

I've updated the default value for --max-concurrent-requests to 30 instead of 50, based on your observations. I will also include this adjustment in the documentation, recommending that users experiencing similar issues try executing with a lower number of concurrent requests.

Thanks again for your valuable input!

[w0rmr1d3r]

Thank you @p1ngul1n0 for this great tool!
Feel free to close this issue/bug once you consider it solved

Cheers!