Spam Classifier
This is a spam classifier that uses naive Bayesian probability. I created it as a proof of concept spam filter for a college course.
The classifier first takes a body of known spam and ham (non-spam) emails to evaluate. Then, it evaluates each email in a test body of emails as spam or ham, with the difference between ham and spam only known to the classifier for the purpose of calculating the success rate.
Several parameters can be changed to optimize the effectiveness of the classifier. By tweaking these parameters, rates in the upper 90% range for both spam and ham classification can be reached.
Included is a sample of Apache SpamAssassin’s public corpus of spam and ham emails for testing. I've also included the paper I wrote and the presentation I created for the project. These both detail the implementation details of the naive Bayesian probabilistic approach to spam filtering and its effectiveness.
Usage is documented in the help output from python spam_classifier.py -h:
usage: spam_classifier.py [-h] [-init_prob_spam INIT_PROB_SPAM]
[-occurence_threshold OCCURENCE_THRESHOLD]
[-score_threshold SCORE_THRESHOLD]
[-phrase_length PHRASE_LENGTH]
spam_examples ham_examples spam_test ham_test
Naive Bayes Spam Filter
positional arguments:
spam_examples spam example messages folder
ham_examples ham example messages folder
spam_test test messages folder
ham_test test messages folder
optional arguments:
-h, --help show this help message and exit
-init_prob_spam INIT_PROB_SPAM
initial probability of a message being spam (0-1)
-occurence_threshold OCCURENCE_THRESHOLD
number of times a word must appear in spam and ham
messages overall
-score_threshold SCORE_THRESHOLD
spam score above which a message must be rated to be
considered spam (0-1)
-phrase_length PHRASE_LENGTH
maximum length of word phrases considered (higher
phrase lengths will impact performance)
Example
As an example, this is what the tweaked input values for the supplied test data look like:
python spam_classifier.py spam_examples ham_examples spam_test ham_test
-init_prob_spam 0.7 -occurence_threshold 5 -phrase_length 5
After evaluating the contents of the emails (which can take a few seconds given that a phrase length of 5 was used), the classification success rates are displayed:
Spam success rate: 98.00%
Ham success rate: 99.50%