/Scam-Blocklist

Blocklist for newly created scam, phishing, and malware domains automatically retrieved daily using Google Search API, automated detection, and public databases.

Primary LanguageShellGNU General Public License v3.0GPL-3.0

Jarelllama's Scam Blocklist

Blocklist for newly created scam and phishing domains automatically retrieved daily using Google Search API, automated detection, and other public sources.

The automated retrieval is done daily at 16:00 UTC.

This blocklist aims to be an alternative to blocking all newly registered domains (NRDs) seeing how many, but not all, NRDs are malicious. A variety of sources are integrated to detect new malicious domains within a short time span of their registration date.

Download

Format Syntax
Adblock Plus ||scam.com^
Wildcard Domains scam.com

Statistics

Total domains: 163793
Light version: 14745

New domains from each source: *
Today | Yesterday | Excluded | Source
   38 |         - |      50% | Emerging Threats phishing
  175 |         - |      18% | FakeWebshopListHUN
   52 |         - |       4% | Google Search
  103 |         - |      10% | Greek Tax Scam
 1692 |         - |       8% | Jeroengui phishing feed
   35 |         - |       6% | Jeroengui scam feed
    3 |         - |       0% | Manual Entries
 3336 |         - |      18% | PhishStats
  917 |         - |       0% | PhishStats (NRDs)
 1152 |         - |       1% | Regex Matching (NRDs)
   13 |         - |       9% | aa419.org
  117 |         - |       2% | dnstwist (NRDs)
  229 |         - |      31% | guntab.com
   50 |         - |      14% | scam.directory
    5 |         - |      32% | scamadviser.com
    2 |         - |       6% | stopgunscams.com
 7002 |         - |      16% | All sources

* The new domain numbers reflect what was retrieved, not
 what was added to the blocklist.
* The Excluded % is of domains not included in the
 blocklist. Mostly dead, whitelisted, and parked domains.

Important

All data retrieved are publicly available and can be viewed from their respective sources.
Any data hidden behind account creation/commercial licenses is never used.

Domains over time (days)

Domains over time

Courtesy of iam-py-test/blocklist_stats.

Other blocklists

Light version

Targeted at list maintainers, a light version of the blocklist is available in the lists directory.

Details about the light version
  • Intended for collated blocklists cautious about size
  • Only includes sources whose domains can be filtered by date registered/reported
  • Only includes domains retrieved/reported from February 2024 onwards, whereas the full list goes back further historically
  • Note that dead and parked domains that become alive/unparked are not added back into the light version (due to limitations in the way these domains are recorded)
Sources excluded from the light version are marked in SOURCES.md.

The full version should be used where possible as it fully contains the light version and accounts for resurrected/unparked domains.

NSFW Blocklist

Created from requests, a blocklist for NSFW domains is available in Adblock Plus format here: nsfw.txt

Details about the NSFW Blocklist
  • Domains are automatically retrieved from the Tranco Top Sites Ranking daily
  • Dead domains are removed daily
  • Note that resurrected domains are not added back into the blocklist
  • Note that parked domains are not checked for in this blocklist
Total domains: 12083

This blocklist does not just include adult videos, but also NSFW content of the artistic variety (rule34, illustrations, etc).

Malware Blocklist

A blocklist for malicious domains extracted from Proofpoint's Emerging Threats rulesets can be found here: jarelllama/Emerging-Threats

Parts of the rulesets are integrated into the Scam Blocklist as well.

Sources

Retrieving scam domains using Google Search API

Google provides a Search API to retrieve JSON-formatted results from Google Search. A list of search terms almost exclusively found in scam sites is used by the API to retrieve domains. See the list of search terms here: search_terms.csv

Details

Scam sites often do not have long lifespans; malicious domains may be replaced before they can be manually reported. By programmatically searching Google using paragraphs from real-world scam sites, new domains can be added as soon as Google crawls the site. This requires no manual reporting.

The list of search terms is proactively maintained and is mostly sourced from investigating new scam site templates seen on r/Scams.

Active search terms: 13
API calls made today: 69
Domains retrieved today: 52

Retrieving phishing NRDs using dnstwist

New phishing domains are created daily, and unlike other sources that rely on manual reporting, dnstwist can automatically detect new phishing domains within days of their registration date.

dnstwist is an open-source detection tool for common cybersquatting techniques like Typosquatting, Doppelganger Domains, and IDN Homograph Attacks.

Details

dnstwist uses a list of common phishing targets to find permutations of the targets' domains. The target list is a handpicked compilation of cryptocurrency exchanges, delivery companies, etc. collated while wary of potential false positives. The list of phishing targets can be viewed here: phishing_targets.csv

The generated domain permutations are checked for matches in a newly registered domains (NRDs) feed comprising domains registered within the last 30 days. Each permutation is tested for alternate top-level domains (TLDs) using the 15 most prevalent TLDs from the NRD feed at the time of retrieval.

Active targets: 54
Domains retrieved today: 117

Regarding other sources

All sources used presently or formerly are credited here: SOURCES.md

The domain retrieval process for all sources can be viewed in the repository's code.

Automated filtering process

  • The domains collated from all sources are filtered against an actively maintained whitelist (scam reporting sites, forums, vetted stores, etc.)
  • The domains are checked against the Tranco Top Sites Ranking for potential false positives which are then vetted manually
  • Common subdomains like 'www' are stripped. The list of subdomains checked for can be viewed here: subdomains.txt
  • Only domains are included in the blocklist; URLs are stripped down to their domains and IP addresses are manually checked for resolving DNS records
  • Redundant rules are removed via wildcard matching. For example, 'abc.example.com' is a wildcard match of 'example.com' and, therefore, is redundant and removed. Wildcards are occasionally added to the blocklist manually to further optimize the number of entries

Entries that require manual verification/intervention are sent in a Telegram notification for fast remediations.

The full filtering process can be viewed in the repository's code.

Dead domains

Dead domains are removed daily using AdGuard's Dead Domains Linter.

Dead domains that are resolving again are included back into the blocklist.

Dead domains removed today: 4164
Resurrected domains added today: 515

Parked domains

From initial testing, 9% of the blocklist consisted of parked domains that inflated the number of entries. Because these domains pose no real threat (besides the obnoxious advertising), they are removed from the blocklist daily.

A list of common parked domain messages is used to automatically detect these domains. This list can be viewed here: parked_terms.txt

If these parked sites no longer contain any of the parked messages, they are assumed to be unparked and are added back into the blocklist.

Tip

For list maintainers interested in integrating the parked domains as a source, the list of daily-updated parked domains can be found here: parked_domains.txt (capped to newest 8000 entries)

Parked domains removed today: 343
Unparked domains added today: 123

As seen in

Resources / See also

Appreciation

Thanks to the following people for the help, inspiration, and support!

@T145 - @bongochong - @hagezi - @iam-py-test - @sefinek24 - @sjhgvr

Contributing

You can contribute to this project in the following ways: