Chan9390/osint-combiner

Combining OSINT sources in Elastic Stack

OSINT Combiner

Combining OSINT sources in Elastic Stack

This project contains:

• various Python3 scripts which gather data from OSINT sources, convert them so they fit into Elasticsearch and write the results to outputfiles/*;
• logstash config files which use the outputfiles as input for Elasticsearch.

Currently supported OSINT sources:

• Shodan.io
• Censys.io
• IpInfo by DutchSec

- Zoomeye is not yet supported because of limitations on their API. They don't respond on e-mails.

Requirements

• A Shodan.io key for API access and scan credits
• A Censys.io ID and KEY with SQL and Export privileges
• Project needs a text file named "config.ini" with the following content:

[osint_sources]

SHODAN_API_KEY: *{Shodan API key here}*

CENSYS_API_ID: *{Censys API ID here}* 

CENSYS_API_KEY: *{Censys Secret here}*

[elastic]

ELASTICSEARCH_IP: *{IP of Elasticsearch cluster here}*

X-PACK_ENABLED: *{Whether X-PACK is enabled (true/false}*

X-PACK_USERNAME: *{(optional) X-PACK SHIELD username here}*

X-PACK_PASSWORD: *{(optional) X-PACK SHIELD password here}*

[other]

INSTITUTIONS_FILE:  *{(optional) Path to CSV file containing institutions/organisations. Format is [name,CIDR] where
every CIDR belonging to an institution should be a separate entry. The scripts will combine multiple entries to one
institution with a lists of CIDRS}*

• The Python3 scripts need the following modules (can be installed with easy_install3 or pip3):
  • Shodan
  • Censys
  • Elasticsearch
  • Netaddr

How to use

You can run the following scripts:

• tofile-*.py files take arguments and can be runned automatically, for example with a CRON job. Run with the '-h' flag for more info;
• tofile-*-manually.py files will ask for user input interactively;
• convert-*.py files can convert the resulting files from tofile-*.py to Elasticsearch compatible files, if not already converted with the '-c' flag from tofile-*.py;
• Scripts in debugscripts/ can be used for debugging purposes.

The *.conf files are Logstash configuration files which you need to edit so the config will point to the right files and Elasticsearch index.

Elasticsearch needs a specific mapping to import the data from the scripts. Use the mapping in the Wiki.