/MatterBot

MatterBot is a two-part extensible Python bot framework that provides a scheduled feed of information sources to your Mattermost channel(s), and listens in channel(s) for commands to trigger the appropriate module(s).

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

MatterBot

Code is GPLv3, (c) Arnim Eijkhoudt, 2022-2024.

  • Official github repository: https://github.com/uforia/MatterBot/
  • Pull/feature requests and comments are welcome: please open/post them on GitHub
  • If you are looking to deploy MatterBot for commercial purposes, please reach out to me via the uforia[@]dhcp[.]net email adress

Status

Code probably has bugs, but it is officially in a 'works for me' and 'works for others' state ;-)

Contents

MatterBot consists of two parts that can be run independently: matterbot and matterfeed. Both parts should be run within tmux or screen; the code itself is not daemonized (this may happen at some point in the future).

matterfeed Sources

Matterfeed reports news updates on a set schedule. The currently supported sources are listed in the table below:

Name Type API Key Required Paid Subscription
Aqua Security Blog RSS No No
Bruce Schneier's Blog RSS No No
CISecurity RSS No No
Cqure Blog RSS No No
CSHub (configurable list of CSHub feeds) RSS No No
DarkNet Blog RSS No No
DataBreachSecurity News RSS No No
DIVD CSIRT RSS No No
GBHackers News RSS No No
Group-IB Blog RSS No No
Kitploit Tool Updates RSS No No
KnowBe4 News RSS No No
KrebsOnSecurity Blog RSS No No
MajorLeagueHacking News RSS No No
Microsoft Vulnerability Reports RSS No No
NCSC Netherlands Advisories RSS No No
NCSC United Kingdom Advisories RSS No No
OpenCVE feed of CVEs RSS Yes No
Osint10x Blog / News RSS No No
PacketStorm Security RSS No No
RansomLook (with support for detection of keywords/regex) JSON No No
Ransomwatch JSON No No
Reddit (configurable list of subreddits) RSS No No
SANS Internet Storm Center RSS No No
SebDraven RSS No No
SecureList News RSS No No
SecurityAffairs News RSS No No
Spiceworks Tech News RSS No No
TheHackerNews News RSS No No
The Record Media RSS No No
Threatpost News RSS No No
TrendMicro Research RSS No No
Tripwire State of Security RSS No No
US-CERT National Cyber Awareness System (Advisories, Alerts, Analysis Reports, Current Activity) RSS No No
Velociraptor News/Updates RSS No No
WeLiveSecurity News RSS No No
WikiJS Page Updates WikiJS GraphQL Yes No

New Matterfeed modules can be created. A boilerplate example can be found in the modules directory.

matterbot Commands

The Matterbot component listens in a given set of channels (configurable per module) for user-definable commands, executes and returns the results of the module code. The currently supported commands are listed below:

Name Type Functionality / Use Case API Key Required Paid Subscription
A1000 ReversingsLabs Threat Intel Look up IPv4, IPv6, hosts/domains, URLs and MD5/SHA1/SHA256/SHA512 hashes Yes Yes
AbuseIPDB Threat Intel Look up IPv4, IPv6 and netblocks for indicators/reports of abuse Yes No, but higher tier API rate limits are only available to paid subscribers
AlienVault OTX Threat Intel Look up IPv4, IPv6, hostnames, domains, MD5/SHA1/SHA256 hashes and URLs No No, but some API limitations may apply
ASN WHOIS Threat Intel Look up Autonomous System Numbers and return the ownership, peering and location information No No
AttackMatrix Threat Intel Query an AttackMatrix instance for e.g. MITRE ATT&CK IDs, Actor- and TTP-overlap. Requires Python GraphViz bindings to display the accompanying Graph No, unless the AttackMatrix API is configured to require an API key No
Bootloaders Threat Intel Query 'Bootloaders' project for vulnerable/malicious bootloader information. Returns detailed information, hashes and detection rules No No
Broadcom Symantec Security Cloud (BSSC) Threat Intel Retrieve 'Threat Intel Insight' information for SHA256 file hashes, IPs, reputations, domains and URLs Yes Yes
Censys Threat Intel Query Censys for IPs and SHA256 certificate fingerprints. Query results are returned as the original Censys JSON blob Yes No: basic functionality
Yes: additional features, such as pagination
ChatGPT LLM / GPT queries Ask OpenAI's ChatGPT singular questions (no  support for chat history). Requires a paid subscription with sufficient credits Yes Yes
Diceroll Fun Roll any kind of dice combination: #d# format No No
DocGen Documentation Automatically create documentation with templated variables, rendering and more. Probably useless unless you have all the required materials. No No
Early Warning & Advisory (EWA) Threat Intel Create Early Warning & Advisory documents using the National Vulnerability Database (NVD) and WikiJS information. Requires pandoc,  pypandoc, LaTeX, a WikiJS instance and a CSS template for final rendering No (for NVD)
Yes (for WikiJS)		 No
GeoLocation Threat Intel Convert latitude/longitude  values into an address No No
GreyNoise Threat Intel Query the GreyNoise API for IP address reputation, such as whether an IP has been observed scanning the internet,  source & destination countries, fingerprints, ports scanned, whether it is benign or not, etc. Yes Yes: certain features require an additional subscription license, such as timeline and similarity features; see the GreyNoise website and API documentation or more information
GTFOBins Threat Intel Query the 'Get The F**k Break Out' [binaries project] for file information. Can be used to escape from restricted shells, escalate or maintain elevated privileges, transfer files, spawn bind and reverse shells, and facilitate the other post-exploitation tasks. Returns detailed information and code examples, which can be used in detection patterns No No
Hybrid-Analysis Threat Intel Look up IPs, hostnames, domains, URLs, MD5, SHA1, SHA256, Authentihash, Imphash, ssdeep hashes and VxFamily names, as well as known 'context' and 'similarity' Yes: 'vetted' API key strongly recommended to prevent hitting API limits No: basic functionality, Yes: additional features/details
IPLocation Threat Intel Look up general location information (country, ISP) for an IPv4 of IPv6 address No No
IPWHOIS Threat Intel Look up IP address information: ownership, ASN, geolocation information No No
LeakIX Threat Intel Find subdomains and look up possible information/data leaks for hosts and domains Yes: API key strongly recommended to prevent hitting API limits No: basic functionality, Yes: additional data
LOLBAS Threat Intel Query the 'Living Off The Land Binaries, Scripts and Libraries' project for file information. Returns detailed information and detection rules No No
LOLDrivers Threat Intel Query 'Living Off The Land Drivers' project for driver information. Returns detailed information, hashes and detection rules No No
Malpedia Threat Intel Look up malware families, threat actors and MD5/SHA256 malware hashes No: basic functionality
Yes: malware downloads		 No
MalwareBazaar Threat Intel Query MalwareBazaar for MD5/SHA1/SHA256 hashes of malware. Will also return include a downloadable malware sample, if available No No
MISP Threat Intel Wildcard-searching of a MISP instance for the given search terms. Returns links to the MISP Events where the search terms have been found Yes No
MWDB Threat Intel Query your own or the CERT.pl MWDB instance for information on malware samples, malware configs or text blobs extracted from malware Yes No
OpenCVE Vulnerability Management Query an OpenCVE instance for information about vulnerabilities. Yes No
Qualys Vulnerability Management Query the Qualys CSAM API for software/libraries present on systems. Extremely useful for Attack Surface Management / Vulnerability Management. Returns a collated dataset of found software and versions, as well as a CSV list of systems found Yes Yes: Qualys subscription required, as well as CSAM subscription and agent-based scans on hosts
RansomLook.io Threat Intel Search and view ransomware groups, markets, posts and telegram channels No No
RIPE WHOIS Threat Intel Look up IP address information: ownership, CIDR and geolocation information No No
Shodan Threat Intel Query Shodan for IP address or host information, as well as performing count and search queries. Results will include the original Shodan JSON blob as a download Yes No: basic functionality
Yes: pagination, search queries, etc.
SSLMate Threat Intel Look up SSL/TLS SHA256 hashes in the Certificate Transparency logs. Returns historic data, related hostnames, revocation status and validity times Yes No
ThreatFox Threat Intel Query ThreatFox for MD5/SHA1/SHA256 hashes, IP addresses No No
TLSGrab Threat Intel Connect to the given IP address + port, and attempt to retrieve the TLS certificate CNs. Note: this is an OPSEC risk, because the bot will actively attempt to connect to the host/port! No No
Tria.ge Threat Intel Search the tria.ge sandbox project for IPv4, IPv6, domains, urls, hashes, a Tria.ge ID, etc. Responses may include the malware sample, if available Yes No: basic functionality
Yes: additional information
Tweetfeed Threat Intel Query the Tweetfeed API for the given IoC/tag No No
Unprotect.it Threat Intel Search the Unprotect.it project for information on TTPs, code snippets and detection rules. Returns code snippets and detection rules as a download, if available No No
URLhaus Threat Intel Look up reputation info on URLhaus for URLs and MD5 / SHA1 / SHA256 URL hashes No No
VirusTotal Threat Intel Search VirusTotal for IP addresses, MD5/SHA1/SHA256 hashes, URLs and domains. Returned results will include maliciousness, TTP sets, malware family names, etc., if available Yes No: basic functionality
Yes: paid VT features, throttling limit removal, etc.
WikiJS Information Retrieval Search through WikiJS pages' contents for the given search terms. Returns links to the pages where the contents were found Yes Yes: currently requires a Microsoft Azure Search instance that indexes the WikiJS instance (Note: this is a WikiJS limitation!)

New Matterbot modules can be created. A boilerplate example can be found in the commands directory.

Requirements

Python

  • Tested with Python 3.10+, although earlier Python 3 versions might work (test at your own discretion). Most modern distributions should be able to run this.
  • Make sure to install the Python requirements (see requirements.txt).
  • For GraphViz support (e.g. AttackMatrix visual graph generation), you will need to install GraphViz for your distribution/OS. Make sure that it includes GTS (GNU Triangulated Surface) support.

Mattermost

  • A Mattermost instance.
  • A 'bot account' on that Mattermost instance. NOTE: Currently the bot requires an admin account (see open issues) on your Mattermost instance!
  • Remember to invite the bot to the correct channels, both for outputting the results from its feed parsing and so it can listen to commands!

matterfeed.py

General information

matterfeed.py goes through the modules directory and will run all detected modules every (by default) 5 minutes, outputting the results to the specified channels. Every module has its own custom configuration: you'll need to check the individual directories for more information. For example, the WikiJS module requires you to have a WikiJS instance with GraphQL API access, as well as a Microsoft Azure Search instance. You'll need to put the API key etc. in its configuration for it to work properly.

Getting started

  1. Copy config.defaults.yaml to config.yaml and edit the settings.
  2. For every module you want to use, check the respective configuration in modules/.../. If you do not want to use a feed module, the easiest way to disable it is to move the directory somewhere else (or delete it), so it will not be detected on startup.
  3. Start up the matterfeed.py and watch the logfile for errors.

matterbot.py

General information

matterbot.py goes through the commands directory and will start listening in every specified channel for every specified bind (command). Every module has its own custom configuration: you'll need to check the individual directories for more information. For example, the ChatGPT module requires you to have an OpenAI account with API access, and you'll need to put the API key etc. in the configuration for it to work.

Getting started

  1. Copy config.defaults.yaml to config.yaml and edit the settings.
  2. For every module you want to use, check the respective configuration in commands/.../. You must make create your own settings.py for every module in the commands/.../ directory you want to use! This is necessary so the bot can override the default configuration from defaults.py.. If you do not want to use a module, the easiest way to disable it is to move the directory somewhere else (or delete it), so it will not be detected on startup.
  3. Start up the matterbot.py.

Configuring Matterbot behavior

When doing threat intelligence investigations, it is crucial to observe Operational Security (OPSEC) best practices. Therefore, by default:

  1. MatterBot will not join any channels;

  2. MatterBot does not run with or need Mattermost Administrator access;

  3. MatterBot will not listen to any commands in a channel it gets added to.

Note: module configurations (e.g. in settings.py) may override this behavior!

Configuring channel module listeners

  1. After setting up a channel (public or private), manually add the MatterBot to that channel;

  2. Use the @map command (these triggers may be changed in the config..yaml) to list all available modules and whether they are currently enabled for the channel or not;

  3. Use @bind <modulename> or @unbind <modulename> to respectively enable and disable a specific module for a channel, or ...

  4. ... use @bind * / @unbind * to enable/disable all modules.

OPSEC considerations and best practices

  • Some modules will reach out to online internet services! Whether or not this is an OPSEC concern, may depend on your threat model, if you are using paid API access, etc. It is solely up to you to decide whether or not this is an acceptable risk! By using MatterBot, you agree not to hold the author(s) responsible/liable for any OPSEC failures or resulting damage;

  • Consider using separate Mattermost channels for:

    • projects;

    • incidents;

    • day-to-day discussions;

    • 'security clearance' levels;

    • etc.

  • Use the MatterBot modules binding/unbinding feature (see above) to only enable MatterBot modules that align with your 'risk appetite' for each individual Mattermost channel.

Writing your own module

  • For matterfeed.py, it is relatively simple to copy an existing module and alter it to your own needs. Make sure to update the pathlib construct to reflect the right module and directory names.

  • matterbot.py is a fully asynchronous setup, which has both advantages and limitations. The example command is a good place to learn more and start developing your own command handler. Pay particular attention to the description in the commands/example/command.py file for more information on how to get started and to avoid common pitfalls.

General known issues and to-do's

  • Code cleanups and optimizations
  • Better (generalized) logging and error handling

Acknowledgements

MatterBot would not be possible without the amazing work and/or generous help of others. If I have erroneously failed to list you here, please let me know! In alphabetical order, the people/organisations/companies I would particularly like to thank are:

  • Broadcom Symantec: For providing an API key that let me develop integration with Broadcom Symantec Security Cloud
  • CERT Polska (CERT.pl): For their awesome MWDB analysis/sandboxing platform and excellent API documentation
  • GTFOBins: The GTFOBins project https://gtfobins.github.io/, in particular AИDREA for providing a simple single file download upon request
  • LOLBAS: The LOLBAS project https://lolbas-project.github.io/#
  • LOLDrivers: The LOLDrivers project https://www.loldrivers.io/
  • Malpedia: Being an amazing community and accepting me into it many years ago https://malpedia.caad.fkie.fraunhofer.de/
  • MalwareBazaar: The author(s), for helping me iron out some bugs https://bazaar.abuse.ch
  • MISP: For being an absolutely amazing open-source platform for TI exchange https://misp-project.org
  • ReversingLabs: For providing an API key to Spectra Analyze and TICloud to develop the integration (particular thank you to D.H.!)
  • ThreatFox: The author(s), for helping me iron out some bugs https://threatfox.abuse.ch
  • Tycho van Marle: for his countless suggestions and contributions to the project
  • Unprotect.it: The author(s), for being receptive, kind and open to me including default (download) support for their project https://unprotect.it
  • URLhaus: The author(s), for helping me iron out some bugs https://urlhaus.abuse.ch

Additional thanks to AlienVault, Censys, Shodan, Tweetfeed, VirusTotal for providing good API documentation, letting me easily write plugins for their services.