/regex-for-OSINT

A list of application tools and information resources to help you effectively use regular expressions in OSINT (Open Source Intelligence)

MIT LicenseMIT

Regex for OSINT

A list of application tools and information resources to help you effectively use regular expressions in OSINT (Open Source Intelligence)

GitHub stars GitHub forks

@cyb_detective

If you are serious about OSINT but do not yet use regular expressions in your practice, I recommend that you read this article to realize all the amazing possibilities this technology offers:

How regular expressions can be useful in OSINT. Theory and some practice using Google Sheets

Table of contents

Theory

Best Regex Cheatsheets

Link Description
Devhints Simple universal regex cheatsheet
Remram44 Cheatsheet for Perl/PCR, Python's re, POSIX (BRE), POSIX extended (ERE)and Vim regex
DaveChild Regex Cheatsheet Posted on Cheatography cheatsheet with downloadable PDF version.

Regex online courses

Link Description
Regex Learn Complete 56 short online exercises and learn how to use regular expressions.
Regex One 25 lessons with interactive exercises

Regex Lists and Libraries

Link Description
UI Bakery Regex Library A short list of a couple dozen of the most useful and widely used regular expressions.
Regex101 16000+ regular expressions + editor/debugger with quick reference
XRegexp JavaScript regular expressions library
"Secret regex list" List of regular expressions to search for API keys and other juicy info. Useful when searching in public code repositories as well as when researching files on the computer/server that is your target.
Stackoverflow regex tag 250 000 + questions with examples of regex

Important:

If you happen to find some regular expression on the Internet and it doesn't work, check its compatibility with the application or programming language you are using. All the differences between the different types of regular expressions (PCRE, JavaScript, Python, POSIX ERE, etc.) are described in detail here

Regex generators and constructors

Link Description
Autoregex A tool that generates regular expressions against a verbal description using AI
GREX Command line tool written by Rust that generates a regular expression corresponding to entered string. There are dozens of different options, each of which can produce completely different results.
RegexpConstructor Online tool for creating regular expressions based on typed string. Very little automation, very many options and settings. The result depends almost entirely on the user's effort.
Regular expression builder Online tool for writing regular expressions by selecting values and parameters from a drop-down list.
Strings to regex Generates a regular expression matching a long list of string values.
Regex Help Generating regular expressions based on example strings.

Regex testers

Link Description
Regexly JS A tool for testing various JavaScript methods with a combination of "regular expression" + "string".

Understanding regex

Link Description
Regex Translator An online tool that generates a text description for any regular expression
Stanford regex analyzer Another online tool that generates a text description for any regular expression
Regexper A tool that converts a regular expression into a block diagram
Extend Class Regex Visualizer Test, debug and visaulize regex online

Practice

Important: Regular expressions are supported by a huge number of applications. Even Google Docs and Adobe Photoshop (in JavaScript automations) support them. Before you install any new application to use regular expressions, think about whether you can use something you already have.

Command line tools for searching and extracting data with regex

Link Description
FD Written in Rust analog of the standard Linux find utility. Faster and more User friendly
SD Written in Rust analog of the standard Linux sed utility. Faster and more User friendly
Awesome GREP A list of various tools for working with different file types that have similar functionality to the standard Linux grep utility
VIM The most powerful command line text editor with many functions for working with regular expressions.
AWK AWK is a domain-specific language designed for text processing and typically used as a data extraction and reporting tool

Text editors with regex support

Link Description
Notepad++
Sublime Text
Edit Pad
Atom
Visual Studio

Search engines with regex support

Link Description
Public WWW Source Code Search Engine. Find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code.
Grep.app Search across half of million Github repos
NetlasIO Search service for all devices connected to the Internet. You can search by IP addresses, port, protocols, certificates, vulnerability names and other parameters.
Sourcegraph Source code search engine
Live Grep Search across Github repos

Examples of OSINT tools with regex support

Link Description
WayMore Search archived links to domain in Wayback Machine and Common Crawl (+ Urlscan and Alien Vault OTX). Very extensive options for filtering search results by keywords (using Regexp), date and time, and other parameters.
Secret Finder Tool for find sensitive data (apikeys, accesstoken,jwt,..) or search anything with #regexp on #javascript files.
DumpsterDiver Tool can analyze big volumes of data and find some "secrets" in the files (passwords and hardcoded password, SSH, Azure and AWS keys etc). Uncompress archived files, support regular expressions and advanced search rules.
Rextract This extreme simple tool extracts the strings corresponding to a certain #regex from the html code of the list of URLs.
GREP for OSINT Set of very simple shell scripts that will help you quickly analyze a text or a folder with files for data useful for investigation (phone numbers, bank card numbers, URLs, emails and nicknames).

Google Chrome Extensions for searching and extracting data with regex

Link Description
Regex Checker Search and highlight (in webpage): Emails, Phone numbers, Dates, Prices, Adresses
find+ Regex Find-in-Page Tool
Chrome regex search Dynamically highlight regex matches on the web page

Don't forget to follow @cyb_detective