Email Bounces Processing System with Rails plugin¶ ↑

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP’s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

Package Structure¶ ↑

This package consists of two parts:

Perl scripts to process incoming email:
- bounces processor – could be assigned to process all your bounce emails
- feedback loops messages processor – more specific for Scribd, but still - could be modified for your needs.
Rails plugin to work with mailing blacklists

Database structure¶ ↑

Our blacklists database structure could be used as an example for your own tables or you can use something really different. In our mailing-related database we have two tables:

mailing_domains - this table stores a list of all mailing domains we’ve ever seen in our blacklists/bounces.
- id - auto-incrementing primary key
- name_crc32 - CRC32 hash of the domain name (used as a key for faster domains lookups)
- name - domain name
mailing_blacklist - major blacklisting table used for keeping track of all blacklisting events in the system and all emails related to those events.
- id - auto-incrementing primary key
- domain_id - foreign key to associate a record with a domain
- user_crc32 - CRC32 hash of a username part (username@domain.tld) of an email
- user - username part (username@domain.tld) of an email
- source - shows where a record came from (bounce, unsubscribe request, honeypot email or something else)
- level - one of the two possible blacklisting event levels (hard or soft)
- reason - short string description of a reason for blacklisting.
- created_at - creation timestamp

Bounces processing script¶ ↑

One of the first things I’d recommend you to do (even before you’ll start using your blacklists in your mailing code) is to set up bounces processing and configure your MTA appropriately.

1. Create bounces user account¶ ↑

Create some non-privileged user account which will be used to process all incoming bounce emails.

2. Check-out the code¶ ↑

You can get the latest version of this package from our github account:

$ git clone git://github.com/kovyrin/bounces-handler.git

To run the script you’ll need to install the following CPAN modules (at least): DBI, DBD::mysql, Mail::DeliveryStatus::BounceParser, Email::MIME.

3. Set up your DB¶ ↑

When you have the latest version of our code, you need to set up the database for our scripts.

The easiest way to do so it to import ‘db/schema.sql’ to your database or use ‘db/migrate’ files to add an appropriate migrations to your Rails application.

4. MySQL configuration¶ ↑

When your DB is ready to use, change mysql credentials in your ‘email-processor/bounce-processor.pl’ file.

5. Set up your MTA¶ ↑

Next step you need to do is to make your MTA forward all incoming bounces to your script. On a postfix-based mail machine (and some other MTAs) you can use .forward file in your bounces user home directory. For example:

bounces@mta:~$ cat .forward 
| /home/bounces/bounces-handler/email-processor/bounce-processor.pl

bounces@mta:~$

This file says your MTA to send all incoming emails for the ‘bounces’ user to the STDIN of the specified script. When this file is in place, you can test your configuration by sending your emails to this user and checking the result. When you’re ready, make your MTA send all bounces to this user and let it start collecting blacklisted/dead/bad email addresses for you.

Feedback Loops Processing Script¶ ↑

Along with the bounces handling script, we have a simple scaffold-like script to handle ISP’s feedback loops emails (when they forward you all emails on which users clicked ‘Report as Spam’).

To use this script, you need to do all the stuff I’ve explained above for the bounces-processor, but forward your feedback loops emails to the feedback-loop-processor.pl script.

Rails plugin¶ ↑

Our Rails plugin has a few parts in it:

models - there are two models in the plugin and each of them is related to one of the tables we use for our blacklists (see above)
- MailingDomain - this model is related to mailing_domain table and has a set of methods to manage/find domains.
- MailingBlacklist - more sophisticated and powerful model which is used to manage all blackisting from the Rails side (see sources and specs for details)
ActionMailer expension - it is a simple class extension which gives you one class method ‘filter_blacklisted(level)’ which should be used in all mailer classes where you would like to use our blackists to filter out emails from your mailings.

To install the plugin, you can use the following command:

./script/plugin install git://github.com/kovyrin/bounces-handler.git

This will install the whole package in your vendor/plugins directory.

Example Mailer¶ ↑

Here I’d like to show you how to use our rails plugin in your mailers:

class MyMailer < ActionMailer::Base
  filter_blacklisted :hard # this will remove all hard-blacklisted emails from your mailings (bounced emails, etc)

  # This mail will be sent to all specified emails but those who has been hard-banned in our blacklists
  def notification(email)
    ...
  end

  # This mail will be sent to all users (because we're manually unbanning them before doing the actual delivery)
  def forgot_password(email)
    MailingBlacklist.unban!(email)
    ...
  end
end

Other Examples¶ ↑

In your system you may want to have some places where you’d like to perform ban/unban actions from your code (for example, out-out controller where users could manually unsubscribe from all your mailings).

To ban some email, you should use one of the following calls:

MailingBlacklist.ban!(email, level, reason)
MailingBlacklist.ban!(email, level, reason, source)

Possible level values:

:hard - for bounces, feedback loop emails and other places where you want to be sure any of your emails won’t be send to an address.
:soft - for unsubscribe link clicks, etc, where some emails exists, but you want to ban it from the system for some reason (unsubscribe link click, for example).

Possible source values, default one is :other :

:bounce - used for bounce emails, could be hard (if an address does not exist), or soft (mailbox quota, etc).
:unsubscribe - used for user unsubscribe requests
:honeypot - used for email addresses which are known honeypots for spammers
:other - used for other situations (you can explain them in your reason field)

If you’d like to unban some email, you just call:

MailingBlacklist.unban!(email)

In this case we remove all blacklisting records for this email and let you send any emails you want (but, if we’ll receive some bounces again, we’ll ban it again).

If you need to know if some email is banned or not, you should call one of the following methods:

MailingBlacklist.banned?(email)
MailingBlacklist.banned?(email, level)
MailingBlacklist.hard_banned?(email)
MailingBlacklist.soft_banned?(email)

In our code we use the following idea: if you ask for hard-bans, then we return true only if a given address is hard-banned, but if you ask for any ban (or a soft ban) and given address has a hard ban in place, then we’ll return true as well (to make sure you won’t send your email there).

Authors and credits¶ ↑

All the code in this package has been developed by Alexey Kovyrin for Scribd.com and is released under the GPLv2 license. For more details, see LICENSE file.

For the bounces processing code we’ve used Mail::DeliveryStatus::BounceParser CPAN module by Meng Weng Wong.

Links¶ ↑

Author’s Blog:: blog.kovyrin.net Scribd Site:: www.scribd.com Bounces Parsing Module:: search.cpan.org/~freeside/Mail-DeliveryStatus-BounceParser-1.4/BounceParser.pm