/blackweb

URLs Blacklist for Squid

Primary LanguageShellGNU General Public License v3.0GPL-3.0

License GitHub version

Blackweb es un proyecto que pretende recopilar la mayor cantidad de listas negras públicas de dominios (para bloquear porno, descargas, drogas, malware, spyware, trackers, bots, redes sociales, warez, venta de armas, etc), con el objeto de unificarlas y hacerlas compatibles con Squid-Cache (Tested in v3.5.x). Para lograrlo, realizamos una depuración de urls, para evitar duplicados, dominios inválidos (validación de ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), y un filtrado con listas blancas de dominios (falsos positivos, como google, hotmail, yahoo, etc), para obtener una mega ACL, optimizada para Squid-Cache, libre de "overlapping domains" (e.g: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").

Blackweb is a project that aims to collect as many public domain blacklists (to block porn, downloads, drugs, malware, spyware, trackers, Bots, social networks, warez, arms sales, etc.), in order to unify them and make them compatible with Squid-Cache (Tested in v3.5.x ). To do this, we perform a debugging of urls, to avoid duplicates, invalid domains (validation, ccTLD, ccSLD, sTLD, uTLD, gSLD, gTLD, etc), and filter with white lists of domains (false positives such as google , hotmail, yahoo, etc.), to get a mega ACL, optimized for Squid-Cache, free of overlapping domains (eg: "ERROR: '.sub.example.com' is a subdomain of '.example.com'").

FICHA TECNICA / DATA SHEET


ACL Black Domains txt size tar.gz size
blackweb.txt 2.405.211 56,6 MB 12,1 MB

DEPENDENCIAS / DEPENDENCIES


git squid bash tar zip wget subversion python

DESCARGA / DOWNLOAD


git clone --depth=1 https://github.com/maravento/blackweb.git

MODO DE USO / HOW TO USE


La ACL blackweb.txt ya viene optimizada para Squid-Cache. Descárguela y descomprimala en la ruta de su preferencia:

The ACL blackweb.txt is already optimized for Squid-Cache. Download it and unzip it in the path of your preference:

Download ACL
wget -q -N https://github.com/maravento/blackweb/raw/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
Checksum ACL
wget -q -N https://github.com/maravento/blackweb/raw/master/blackweb.md5
md5sum blackweb.txt | awk '{print $1}' && cat blackweb.md5 | awk '{print $1}'

ACTUALIZACIÓN / UPDATE


El script bwupdate.sh actualiza la ACL blackweb.txt, realizando la captura, depuración y limpieza de dominios, sin embargo puede generar conflíctos por errores en las FUENTES, por tanto deberá depurarlos manualmente. Tenga en cuenta que este script consume gran cantidad de recursos de hardware durante el procesamiento y puede tomar mucho tiempo.

The bwupdate.sh script updates blackweb.txt ACL, doing the capture, debugging and cleaning of domains, however it can generate conflicts for errors in the SOURCES, therefore you must manually debug conflicts. Keep in mind that this script consumes a lot of hardware resources during processing and it can take a long time.

wget -q -N https://github.com/maravento/blackweb/raw/master/bwupdate/bwupdate.sh && sudo chmod +x bwupdate.sh && sudo ./bwupdate.sh
Verifique la ejecución / Check execution

path: /var/log/syslog

Blackweb for Squid: Done 06/05/2017 15:47:14

Y verifique el contenido del archivo SquidError.txt en su escritorio para corregirlos / And check the contents of the SquidError.txt file on your desktop to fix them

Importante Antes de Usar / Important Before Use
  • Antes de utilizar bwupdate.sh debe activar la regla en Squid-Cache / You must activate the rule in Squid-Cache before using bwupdate.sh
  • bwupdate.sh debe ejecutarse en equipos de pruebas. Nunca en servidores en producción. / bwupdate.sh must run on test equipment. Never on servers in production.
  • bwupdate.sh no incluye sitios cloud (Mega, Dropbox, Pcloud, iCloud, etc) o de soporte remoto (Teamviewer, Anydesk, logmein, etc), excepto si ya vienen bloqueados desde las FUENTES. Para bloquearlos o excluirlos debe activar la línea según su elección: / bwupdate.sh does not include cloud sites (Mega, Dropbox, Pcloud, iCloud, etc) or remote support (Teamviewer, Anydesk, logmein, etc), except if they are already blocked from the SOURCES. To block or exclude them you must activate the line according to your choice:
# unblock
#sed '/^$/d; /#/d' {cloudsync,remoteurls}.txt | sort -u >> urls.txt
# block
#sed '/^$/d; /#/d' {cloudsync,remoteurls}.txt | sort -u >> bwtmp/bw.txt

Edite / Edit:

/etc/squid/squid.conf

Y agregue las siguientes líneas: / And add the following lines:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
acl blackweb dstdomain -i "/path_to_acl/blackweb.txt"
http_access deny blackweb
Edición / Edition

Blackweb contiene millones de dominios bloqueados, por tanto: / Blackweb contains millions of blocked domains, so:

acl whitedomains dstdomain -i "/path_to_acl/whitedomains.txt"
acl blackdomains dstdomain -i "/path_to_acl/blackdomains.txt"
acl blackweb dstdomain -i "/path_to_acl/blackweb.txt"
http_access allow whitedomains
http_access deny blackdomains
http_access deny blackweb

FUENTES / SOURCES


URLs Blacklists

Adaway

adblockplus malwaredomains_full

ABPindo indonesianadblockrules

Anti-WebMiner

anudeepND

BambenekConsulting

Capitole - Direction du Système d'Information (DSI)

Carl Spam

cedia.org.ec

chadmayfield (included: porn_all, porn top)

CHEF-KOCH BarbBlock-filter-list

Cibercrime-Tracker

crazy-max WindowsSpyBlocker

Dawsey21 List

Disconnect.me (included: simple_ad, simple_malvertising, simple_tracking)

dshield.org (included: Low, Medium, High)

ethanr dns-blacklists

firebog.net (included: AdguardDNS, Airelle-hrsk, Airelle-trc, BillStearns, Easylist, Easyprivacy, Kowabit, Prigent-Ads, Prigent-Malware, Prigent-Phishing, Shalla-mal, WaLLy3K)

hexxium

hostsfile.mine.nu

hosts-file.net (included: ad_servers, emd, grm, hosts, psh)

Joelotz URL Blacklist

Joewein Blacklist

KADhosts

Lehigh Malwaredomains

malc0de

Malwaredomain Hosts List

Malware-domains

Matomo-org referrer-spam-blacklist

MESD blacklists

mitchellkrogza (included: Badd-Boyz-Hosts, Hacked Malware Web Sites, Nginx Ultimate Bad Bot Blocker, The Big List of Hacked Malware Web Sites, Ultimate Hosts Blacklist)

notabug latvian-list

Oleksiig Blacklist

openphish

Passwall SpamAssassin (Server Down since Dec 2016). Last Update

Perflyst (included: android-tracking, SmartTV)

Quedlin blacklist

quidsup (included: notrack-blocklists, notrack-malware, trackers, qmalware)

Ransomware Abuse (included: CryptoWall, Locky, Domain Blocklist, Ransomware Abuse ,URL Blocklist ,TorrentLocker)

reddestdream

securemecca.net and hostsfile.org

Shallalist.de

Someonewhocares

squidblacklist.org (included: dg-ads, dg-malicious.acl)

StevenBlack (included: add.2o7Net, add.Risk, fakenews-gambling-porn-social, hosts, spam, uncheckyAds)

Stopforumspam Toxic Domains

tankmohit UnifiedHosts

Taz SpamDomains

vokins yhosts

Winhelp2002

Yoyo

zerodot1 CoinBlockerLists (included: Host, host_browser, host_optional, list, list_browser, list_browser_UBO)

Zeustracker

Discontinued URLs Blacklists

UrlBlacklist (Server Down since July 24, 2017)

Internal Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)

BlackTLDs

BlackURLs

CloudSync

Debug (Common Errors)

Invalid Domains/TLDs

RemoteURLs

WhiteTLDs

WhiteURLs

External Debugging (URLs/TLDs Whitelists, Invalid Domains, etc)

Central Repo Dead Domains

ipv6-hosts (Partial)

O365IPAddresses (Partial)

Ransomware Database

TLDs IANA

TLDs Mozilla Public Suffix

University Domains and Names Data List

Whoisxmlapi GTLDs

Wikipedia Top Level Domains

Internal Tools

Debugging list

httpstatus bash

External Tools

CTFR

httpstatus

Parse Domains (modified)

CONTRIBUCIONES / CONTRIBUTIONS


Agradecemos a todos aquellos que han contribuido a este proyecto. Los interesados pueden contribuir, enviándonos enlaces de nuevas listas, para ser incluidas en este proyecto / We thank all those who have contributed to this project. Those interested can contribute, sending us links of new lists, to be included in this project

DONACION / DONATE


BTC: 3M84UKpz8AwwPADiYGQjT9spPKCvbqm4Bc

LICENCIA / LICENCE


GPL-3.0

License maravento.com, gateproxy.com and dextroyer.com is licensed under a Creative Commons Reconocimiento-CompartirIgual 4.0 Internacional License.

© 2018 Maravento Studio

EXENCION DE RESPONSABILIDAD / DISCLAIMER


THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.