MX Inference

MX Inference is a tool that uses DNS data and active port scanning data to infer the mail provider of a domain. This tool makes inference based on the MX records specified by the domain.

Installation

Clone our repository

git clone https://github.com/ucsdsysnet/mx_inference.git

Install OpenSSL and associated development libraries (Reference).

# CentOS:
$ yum install openssl-devel libffi-devel

# Ubuntu:
$ apt-get install libssl-dev libffi-dev

# OS X (with Homebrew installed):
$ brew install openssl

Install python libraries.

# Optionally: start a virtual environment
python3 -m venv .
source bin/activate

# Mandatory
python3 -m pip install -r requirements.txt

Tested Version

We tested our code with Python 3.6/3.8 and OpenSSL version 1.1.1i/1.0.2g on Ubuntu and Mac respectively.

Usage

Default Setup

We have some examples built in. You can simply run the following command (might take a while):

python3 demo_mx_inference.py

Sample output:

...
Domain: ucsd.edu
	ID:pphosted.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:1
		Suggested Company: ProofPoint
	...

Domain: netflix.com
	ID:google.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:2
		Suggested Company: Google
	...

Domain: gsipartners.com
	ID:google.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:2
		Suggested Company: Google
	...

# lodi.gov has two IDs because it has two MX records with same priority
Domain: lodi.gov
	ID:iphmx.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:1
		Suggested Company: Cisco
	
	ID:iphmx.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:1
		Suggested Company: Cisco
	...

Domain: jeniustoto.net
	Infered Provider ID: N/A


Domain: sgnetway.net
	ID:sgnetway.net, Type:OK, Source:Banner/EHLO, Debug MSG:Banner/EHLO ID Ok, Conf_Score:1
	...

Domain: bbw-chan.nl
	ID:bbw-chan.nl, Type:OK, Source:MX, Debug MSG:MX RD Ok, Conf_Score:1
	...

Domain: utexas.edu
	ID:utexas.edu, Type:OK, Source:TLS, Conf_Score:1
		Heuristics suggests new provider id: iphmx.com, reason: All IPs are in Cisco Ironport's AS
		Suggested company: Cisco
	...

Domain: summitorganization.org
	ID:secureserver.net, Type:OK, Source:Banner/EHLO, Conf_Score:1
		Heuristics suggests that this provider id might NOT be accurate, reason: FQDN (s132-148-130-121.secureserver.net) used by Banner/EHLO Indicates Potentially VPS
	...

Domain: arfonts.net
	ID:ovh.net, Type:OK, Source:Banner/EHLO, Conf_Score:1
		Heuristics suggests that this provider id might NOT be accurate, reason: FQDN (vps797297.ovh.net) used by Banner/EHLO Indicates Potentially VPS
	...

All scanning results are saved by default with the following name pattern under the project directory:

mx_inference-data-timestamp.csv

You can load the data saved locally and run the inference program on those domains.

python3 demo_mx_inference.py --load_data_from_path=/path/to/saved/data

Providers of domains

You can probe domains of your choice.

python3 demo_mx_inference.py --domains eng.ucsd.edu ucsd.edu

Sample output:

...
Domain: eng.ucsd.edu
	ID:google.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:1
		Suggested Company: Google
	...

Domain: ucsd.edu
	ID:pphosted.com, Type:OK, Source:TLS, Debug MSG:Cert ID Ok, Conf_Score:1
		Suggested Company: ProofPoint
	...

More verbose debug information

This is helpful when you are not getting expected results.

# Additional debug info
python3 demo_mx_inference.py --debug=1

# Very verbose debug info
python3 demo_mx_inference.py --debug=2

Play with different data sources for inference

Make inference based on TLS and MX

python3 demo_mx_inference.py --disable_banner

Make inference based on Banner/EHLO and MX

python3 demo_mx_inference.py --disable_tls

Other Arguments

--disable_tls: do not use TLS certificates for inference
--disable_banner: do not use banner/EHLO information for inference
--disable_heuristics: do not apply any heuristics
--heuristics_threshold: apply heuristics when confidence score <= threshold
--disable_heuristics_as: do not use heuristics that are based on AS information
--disable_heuristics_tls_pattern: do not use heuristics that are based on TLS FQDN pattern
--disable_heuristics_banner_pattern: do not use heuristics that are based on Banner/EHLO FQDN pattern
--map_id_to_company: Try mapping provider id to company. Default: True
--save_scan_data: Save scanned data of domains. Default: True
--debug: Debug information level. 0 = Minimum, 1 = Light, 2 = Verbose. Default: 0.

Note

This tool is not designed for large-scale analysis. Please use third-party datasets instead (e.g., OpenINTEL and Censys).
This tool is not tested with IPv6.
This tool does NOT infer the eventual mail provider used by end users of domain.
Use our heuristics with a grain of salt.

Project Structure

mx_inference
├── ...
├── lib/                       # Libraries
│   ├── certs/                 # CA/Intermediate Certificates 
│   ├── cert.py                # Handle certificates
│   ├── ds.py                  # Data structures
│   ├── extract_domain.py      # Extract domain from strings
│   ├── helper.py              # I/O helper functions
│   ├── heuristics.py          # Heuristics
│   ├── inference_funcs.py     # Inference functions
│   ├── network_lib.py         # Network functions
│   ├── preprocess.py          # Preprocessing certs
│   └── union_find.py          # Union find algorithm
├── config/                     
│   └── config.py              # Configurations
└── inference.py               # High-level wrapper for inferring mail providers of a domain

Extending This Work

How do I import my own data? If you have some data and want to use our tool, you add your own load data function in helper.py. An example function load_domain_data_from_path_format_censys can be found in helper.py. Data associated with each domain is loaded in demo_mx_inference.py by calling load_domain_data_from_path_format_censys.
How do I add other information for inference (e.g., rDNS of an IP)? If you find the information we use (i.e., MX, Banner/EHLO, TLS) for inference is not satisfying, you can modify the data structures defined in ds.py and scanning functions defined in network_lib.py.
How do I add my own heuristics? You can find our heuristics in heuristics.py. Heursitic functions are used by _infer_provider_id_for_mx_of_one_domain function in inference.py.

Cite Our Paper

@inproceedings{MxInfer,
  title={Who's Got Your Mail? Characterizing Mail Service Provider Usage},
  author={Liu, Enze and Akiwate, Gautam and Jonker, Mattijs and Mirian, Ariana and Savage, Stefan and Voelker, Geoffrey M.},
  booktitle={ACM Internet Measurement Conference (IMC'21)},
  publisher={ACM},
  year      = {2021},
  address   = {Virtual Event},
  month     = {November}
}

Bugs and Issues

This software is used and maintained for a research project and likely will have many bugs and issues. If you want to report any bugs or issues, please do it through the Github Issue Page.

sumanthvrao/mx_inference