Could you possibly provide only unique proxies?
snroeust opened this issue · 3 comments
I noticed that some proxies appear multiple times in the list, which requires filtering out all multiple proxies during a proxy check. It's not a big deal, but it would be helpful if this list was already filtered.
What about IPsets instead? Firewall would dedup but not accept IP with ports.
To get a word count you can simply do
curl https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks5.txt | wc
Then to dedupe / sort and uniq the list like so
curl https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks5.txt | sort | uniq
If you really want to dedupe based on IP and not just port (for sanity) you can do this. It will identify the IP on the left and deduplicate based on that alone (keeping the first entry it finds of that IP) ensuring that you don't end up with any duplicate IPs .
curl https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/socks5.txt | tee >(awk -F: '{print $1}' | sort | uniq | while read ip; do grep -m 1 "$ip"; done)
@snroeust The proxies are already unique with IP:PORT combination. As the proxies are free, you might not want to miss out on any IP:PORT combination while fixing as we are unaware of which is the correct combination. However, if you still want to dedupe based on IP I will suggest going with @pentestfunctions solution on your side.