samber/the-great-gpt-firewall

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

The Great GPT Firewall 📛

This collection is a curated list of websites that employ the robots.txt file to restrict access to AI Agents, AI crawlers and GPTs.

It will be updated monthly.

User agents & robots.txt

The robots.txt file allows website owners to control and limit the access of these user agents to certain areas of their website by specifying rules and directives.

# OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT
# https://platform.openai.com/docs/bots
User-agent: GPTBot

# ChatGPT plugins
# https://platform.openai.com/docs/bots
User-agent: ChatGPT-User

# OpenAI Search bot
# https://platform.openai.com/docs/bots
User-agent: OAI-SearchBot

# Google's web crawler: Bard, VertexAI, Gemini
# https://blog.google/technology/ai/an-update-on-web-publisher-controls/
User-agent: Google-Extended

# Apple's web crawler, dedicated to GenAI projects
# https://support.apple.com/en-us/119829
User-agent: Applebot-Extended

# Claude
User-agent: anthropic-ai

# Claude Bot
User-agent: ClaudeBot

# Claude web
User-agent: Claude-Web

# Cohere
User-agent: Cohere-ai

# Perplexity
User-agent: PerplexityBot

# Common Crawl
# https://commoncrawl.org/ccbot
User-agent: CCBot

# Omglibot: webz.io
# https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/
User-agent: Omgilibot
User-agent: Omgili
User-agent: Webzio-Extended

# Facebook: Llama
# https://developers.facebook.com/docs/sharing/bot/
User-agent: FacebookBot

# ByteDance: Duobao
User-agent: Bytespider

# Censorship area
Disallow: /

Disclaimer

Please note that this blocklist is intended for informational purposes only. Despite the provoking project name, it's fine to disallow web crawling and protect content ownership.

2024-05 update

Category: Press

• Scanned: 66
• ✅ Passing: 41 %
• 🔐 Blocked: 59 %
• ❓ Unknown: 0 %

Name	Country	Status
The Times	🇬🇧	🔐
BBC	🇬🇧	🔐
The Guardian	🇬🇧	🔐
The Economist	🇬🇧	🔐
Financial Times	🇬🇧	🔐
The Independent	🇬🇧	✅
The Telegraph	🇬🇧	🔐
Daily Mail	🇬🇧	🔐
The Sun	🇬🇧	🔐
Daily Mirror	🇬🇧	✅
Daily Express	🇬🇧	✅
Washington Post	🇺🇸	🔐
USA Today	🇺🇸	✅
Fox News	🇺🇸	✅
ABC News	🇺🇸	🔐
NBC News	🇺🇸	🔐
CBS News	🇺🇸	🔐
Los Angeles Times	🇺🇸	🔐
Chicago Tribune	🇺🇸	✅
New York Post	🇺🇸	🔐
New York Daily News	🇺🇸	✅
The New Yorker	🇺🇸	🔐
Vice	🇺🇸	✅
New York Times	🇺🇸	🔐
Wall Street Journal	🇺🇸	🔐
CNN	🇺🇸	🔐
El País	🇪🇸	✅
Süddeutsche Zeitung	🇩🇪	🔐
Der Spiegel	🇩🇪	🔐
Corriere della Sera	🇮🇹	🔐
La Repubblica	🇮🇹	🔐
Le Monde	🇫🇷	🔐
Libération	🇫🇷	🔐
Le Figaro	🇫🇷	🔐
20 Minutes	🇫🇷	🔐
Ouest France	🇫🇷	🔐
Le Parisien	🇫🇷	🔐
L'Equipe	🇫🇷	🔐
Le Point	🇫🇷	🔐
Marianne	🇫🇷	🔐
Le Nouvel Observateur	🇫🇷	🔐
L'Express	🇫🇷	🔐
France 24	🇫🇷	🔐
BFMTV	🇫🇷	🔐
CNews	🇫🇷	✅
Le Monde Diplomatique	🇫🇷	✅
Mediapart	🇫🇷	🔐
Courrier International	🇫🇷	🔐
Brut	🇫🇷	✅
IMDB	🌍	✅
Allocine	🇫🇷	✅
Fakt	🇵🇱	✅
Super Express	🇵🇱	✅
Gazeta Wyborcza	🇵🇱	🔐
Rzeczpospolita	🇵🇱	✅
Dziennik Gazeta Prawna	🇵🇱	✅
Polityka	🇵🇱	✅
Newsweek Polska	🇵🇱	✅
Gość Niedzielny	🇵🇱	✅
Sieci	🇵🇱	✅
Do Rzeczy	🇵🇱	✅
Twój Styl	🇵🇱	✅
Zwierciadło	🇵🇱	✅
Wysokie Obcasy Extra	🇵🇱	🔐
Pani	🇵🇱	✅
Elle	🇵🇱	✅

Category: Video on demand

• Scanned: 9
• ✅ Passing: 56 %
• 🔐 Blocked: 44 %
• ❓ Unknown: 0 %

Name	Country	Status
Prime Video	🌍	✅
Netflix	🌍	✅
Disney+	🌍	🔐
Hulu	🇺🇸	🔐
HBO Max	🇺🇸	✅
Canal+	🇫🇷	🔐
FranceTV	🇫🇷	✅
TF1	🇫🇷	🔐
6Play	🇫🇷	✅

Category: Music

• Scanned: 6
• ✅ Passing: 67 %
• 🔐 Blocked: 33 %
• ❓ Unknown: 0 %

Name	Country	Status
Soundcloud	🌍	🔐
Youtube	🌍	✅
Apple Music	🌍	✅
Spotify	🌍	🔐
Deezer	🇫🇷	✅
LastFM	🇬🇧	✅

Category: Podcast

• Scanned: 8
• ✅ Passing: 75 %
• 🔐 Blocked: 25 %
• ❓ Unknown: 0 %

Name	Country	Status
Google Podcasts	🌍	✅
Apple Podcast	🌍	✅
Spotify Podcaster	🌍	🔐
Buzzsprout	🌍	✅
Podbean	🌍	✅
Acast	🇬🇧	✅
AudioMeans	🇫🇷	✅
Radio France	🇫🇷	🔐

Category: X

• Scanned: 6
• ✅ Passing: 67 %
• 🔐 Blocked: 33 %
• ❓ Unknown: 0 %

Name	Country	Status
PornHub	🌍	🔐
YouPorn	🌍	🔐
Xnxx	🌍	✅
Xvideos	🌍	✅
Xhamster	🌍	✅
OnlyFan	🌍	✅

Category: Religion

• Scanned: 5
• ✅ Passing: 100 %
• 🔐 Blocked: 0 %
• ❓ Unknown: 0 %

Name	Country	Status
Bible	🇺🇸	✅
Bible gateway	🇺🇸	✅
Jehovah's Witnesses	🇺🇸	✅
Vatican	🇻🇦	✅
Islamweb	🌍	✅

Category: Social media

• Scanned: 13
• ✅ Passing: 31 %
• 🔐 Blocked: 62 %
• ❓ Unknown: 8 %

Name	Country	Status
Facebook	🌍	🔐
Instagram	🌍	🔐
Reddit	🌍	✅
Hacker News	🌍	❓
Lobsters	🌍	🔐
Pinterest	🌍	🔐
TikTok	🌍	✅
Twitter	🌍	🔐
LinkedIn	🌍	✅
Quora	🌍	🔐
VK	🇷🇺	✅
TripAdvisor	🌍	🔐
Yelp	🌍	🔐

Category: Artist

• Scanned: 42
• ✅ Passing: 71 %
• 🔐 Blocked: 19 %
• ❓ Unknown: 10 %

Name	Country	Status
Michael Jackson	🇺🇸	✅
Madonna	🇺🇸	✅
Taylor Swift	🇺🇸	🔐
Rihanna	🇺🇸	✅
Bruno Mars	🇺🇸	✅
Justin Bieber	🇺🇸	🔐
Beyoncé	🇺🇸	✅
Katy Perry	🇺🇸	🔐
Lady Gaga	🇺🇸	🔐
Hardwell	🇺🇸	✅
Dimitri Vegas & Like Mike	🇺🇸	✅
Kanye West	🇺🇸	❓
Black Eyed Peas	🇺🇸	❓
Imagine Dragons	🇺🇸	✅
Twenty One Pilots	🇺🇸	✅
Maroon 5	🇺🇸	🔐
Selena Gomez	🇺🇸	🔐
Usher	🇺🇸	🔐
Stromae	🇧🇪	❓
Aya Nakamura	🇫🇷	❓
Soprano	🇫🇷	✅
Johnny Hallyday	🇫🇷	✅
Grand Corps Malade	🇫🇷	✅
Zaho	🇫🇷	✅
Jean Louis Aubert	🇫🇷	✅
Camelia Jordana	🇫🇷	✅
Indochine	🇫🇷	✅
Tryo	🇫🇷	✅
David Guetta	🇫🇷	✅
Mc Solaar	🇫🇷	✅
Zaz	🇫🇷	✅
Christine and the Queens	🇫🇷	✅
Boulevard des Airs	🇫🇷	✅
Calogero	🇫🇷	✅
Hoshi	🇫🇷	✅
Avicii	🇸🇪	✅
Adele	🇬🇧	✅
Calvin Harris	🇬🇧	✅
Ed Sheeran	🇬🇧	✅
Arctic Monkeys	🇬🇧	✅
Coldplay	🇬🇧	✅
The Weeknd	🇨🇦	🔐

Category: Gov

• Scanned: 3
• ✅ Passing: 100 %
• 🔐 Blocked: 0 %
• ❓ Unknown: 0 %

Name	Country	Status
White House	🇺🇸	✅
Elysée	🇫🇷	✅
Europe	🇪🇺	✅

Category: Science

• Scanned: 28
• ✅ Passing: 89 %
• 🔐 Blocked: 11 %
• ❓ Unknown: 0 %

Name	Country	Status
Google Scholar	🌍	✅
Sci-Hub	🌍	✅
PubPeer	🌍	✅
Scopus	🇳🇱	🔐
Elsevier	🇳🇱	🔐
ScienceDirect	🇳🇱	✅
MDPI	🇨🇭	✅
Springer	🇩🇪	✅
Wiley	🇺🇸	✅
American Chemical Society	🇺🇸	✅
PubMed	🇺🇸	✅
Academia	🇺🇸	✅
Science	🇺🇸	🔐
ArXiv	🇺🇸	✅
American Physical Society	🇺🇸	✅
Mendeley	🇬🇧	✅
Nature	🇬🇧	✅
Taylor & Francis	🇬🇧	✅
Oxford University Press	🇬🇧	✅
Cambridge University Press	🇬🇧	✅
Royal Society of Chemistry	🇬🇧	✅
ResearchGate	🇩🇪	✅
BNF	🇫🇷	✅
Cairn	🇫🇷	✅
Persee	🇫🇷	✅
Gallica	🇫🇷	✅
HAL	🇫🇷	✅
OpenEdition	🇫🇷	✅

Category: Dev

• Scanned: 3
• ✅ Passing: 67 %
• 🔐 Blocked: 33 %
• ❓ Unknown: 0 %

Name	Country	Status
Github	🌍	✅
Gitlab	🌍	✅
Stack Overflow	🌍	🔐

Category: Other content

• Scanned: 19
• ✅ Passing: 74 %
• 🔐 Blocked: 26 %
• ❓ Unknown: 0 %

Name	Country	Status
Wikipedia	🌍	✅
Medium	🌍	🔐
Substack	🌍	✅
Common Crawl	🌍	✅
Internet Archive	🌍	✅
Wayback Machine	🌍	✅
Notion	🌍	✅
Weather	🇺🇸	🔐
AccuWeather	🇺🇸	✅
Météo France	🇫🇷	✅
Getty Images	🇺🇸	✅
Shutterstock	🇺🇸	🔐
Adobe Stock	🇺🇸	🔐
Unsplash	🇨🇦	🔐
Pexels	🇩🇪	✅
Pixabay	🇩🇪	✅
Flickr	🇺🇸	✅
500px	🇨🇦	✅
Giphy	🇺🇸	✅

Category: Other

• Scanned: 1
• ✅ Passing: 100 %
• 🔐 Blocked: 0 %
• ❓ Unknown: 0 %

Name	Country	Status
Indeed	🇺🇸	✅

WTF list

A.k.a: do they understand their business model? 💸

Name	Status
Getty Images	✅
Pexels	✅
500px	✅

Shame list

A.k.a: this is public interest. 🖕

Name	Status
Medium	🔐
Quora	🔐
Elsevier	🔐
Scopus	🔐
Science	🔐

🤝 Contributing

Looking for contributions:

• Enrich website database
• Chinese websites
• New categories

Please open issues!

• Ping me on Twitter @samuelberthe (DMs, mentions, whatever :))
• Fork the project
• Fix open issues or request new features

Don't hesitate ;)

Build

python -m venv venv
source ./venv/bin/activate
pip3 install -r requirements.txt
python3 scrape.py
# then copy the last version into readme

👤 Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

📝 License

Copyright © 2024 Samuel Berthe.

This project is MIT licensed.