/rb-Sisimai

Mail Analyzing Interface for email bounce: A Ruby library to parse RFC5322 bounce mails and generating structured data as JSON from parsed results. Ruby version of Sisimai: an error mail analyzer.

Primary LanguageRubyOtherNOASSERTION

License Coverage Status Build Status Codacy Badge Ruby Gem Version

What is Sisimai

Sisimai is a Ruby library for analyzing RFC5322 bounce emails and generating structured data from parsed results. The Ruby version of Sisimai is ported from the Perl version of Sisimai at github.com/sisimai/p5-Sisimai.

Key Features

  • Convert Bounce Mails to Structured Data
    • Supported formats are Ruby(Hash, Array) and JSON(String)
  • Easy to Install, Use.
    • gem install
    • git clone & make
  • High Precision of Analysis
    • 2 times higher than bounceHammer
    • Support 22 known MTAs and 5 unknown MTAs
    • Support 22 major MSPs(Mail Service Providers)
    • Support 2 major Cloud Email Delivery Services(JSON format)
    • Support Feedback Loop Message(ARF)
    • Can detect 29 error reasons

Command line demo

The following screen shows a demonstration of Sisimai at the command line using Ruby(rb-Sisimai) and Perl(p5-Sisimai) version of Sisimai.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

Install

From RubyGems

$ sudo gem install sisimai
Fetching: sisimai-4.22.0.gem (100%)
Successfully installed sisimai-4.22.0
Parsing documentation for sisimai-4.22.0
Installing ri documentation for sisimai-4.22.0
Done installing documentation for sisimai after 6 seconds
1 gem installed

From GitHub

$ cd /usr/local/src
$ git clone https://github.com/sisimai/rb-Sisimai.git
$ cd ./rb-Sisimai
$ sudo make depend install-from-local
gem install bundle rake rspec coveralls
...
4 gems installed
bundle exec rake install
sisimai 4.22.0 built to pkg/sisimai-4.22.0.gem.
sisimai (4.22.0) installed.

Usage

Basic usage

make() method provides feature for getting parsed data from bounced email messages like following.

#! /usr/bin/env ruby
require 'sisimai'
v = Sisimai.make('/path/to/mbox')       # or path to Maildir/

# If you want to get bounce records which reason is "delivered", set "delivered"
# option to make() method like the following:
v = Sisimai.make('/path/to/mbox', delivered: true)

if v.is_a? Array
  v.each do |e|
    puts e.class                # Sisimai::Data
    puts e.recipient.class      # Sisimai::Address
    puts e.timestamp.class      # Sisimai::Time

    puts e.addresser.address    # shironeko@example.org # From
    puts e.recipient.address    # kijitora@example.jp   # To
    puts e.recipient.host       # example.jp
    puts e.deliverystatus       # 5.1.1
    puts e.replycode            # 550
    puts e.reason               # userunknown

    h = e.damn                  # Convert to HASH
    j = e.dump('json')          # Convert to JSON string
    puts e.dump('json')         # JSON formatted bounce data
  end
end

Convert to JSON

Sisimai.dump() method provides feature for getting parsed data as JSON string from bounced email messages like following.

# Get JSON string from parsed mailbox or Maildir/
puts Sisimai.dump('/path/to/mbox')  # or path to Maildir/

# dump() method also accepts "delivered" option like the following code:
puts Sisimai.dump('/path/to/mbox', delivered: true)

Read bounce object

The way to read a bounce object retrived from Cloud Email Services as JSON using their API is the following code. This feature is available at Sisimai 4.20.0 or later.

#! /usr/bin/env ruby
require 'json'
require 'sisimai'

j = JSON.load('{"notificationType"=>"Bounce", "bounce"=>{"...') # JSON String
v = Sisimai.make(j, input: 'json')

if v.is_a? Array
  v.each do |e|
    ...
  end
end

As of present, Only Amazon SES and SendGrid are supported.

Callback feature

Beginning with Sisimai 4.19.0, make() and dump() methods of Sisimai accept a Lambda (Proc object) in hook argument for setting a callback method and getting the results generated by the method via Sisimai::Data.catch method.

#! /usr/bin/env ruby
require 'sisimai'
callbackto = lambda do |v|
  r = { 'x-mailer' => '', 'queue-id' => '' }

  if cv = v['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    r['queue-id'] = cv[1]
  end
  r['x-mailer'] = v['headers']['x-mailer'] || ''
  return r
end

list = ['X-Mailer']
data = Sisimai.make('/path/to/mbox', hook: callbackto, field: list)
json = Sisimai.dump('/path/to/mbox', hook: callbackto, field: list)

puts data[0].catch['x-mailer']      # Apple Mail (2.1283)

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

One-Liner

% ruby -rsisimai -e 'puts Sisimai.dump($*.shift)' /path/to/mbox

Output example

[{"recipient": "kijitora@example.jp", "addresser": "shironeko@1jo.example.org", "feedbacktype": "", "action": "failed", "subject": "Nyaaaaan", "smtpcommand": "DATA", "diagnosticcode": "550 Unknown user kijitora@example.jp", "listid": "", "destination": "example.jp", "smtpagent": "Email::Courier", "lhost": "1jo.example.org", "deliverystatus": "5.0.0", "timestamp": 1291954879, "messageid": "201012100421.oBA4LJFU042012@1jo.example.org", "diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "filtered", "token": "ce999a4c869e3f5e4d8a77b2e310b23960fb32ab", "alias": "", "senderdomain": "1jo.example.org", "rhost": "mfsmax.example.jp"}, {"diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "userunknown", "timestamp": 1381900535, "messageid": "E1C50F1B-1C83-4820-BC36-AC6FBFBE8568@example.org", "token": "9fe754876e9133aae5d20f0fd8dd7f05b4e9d9f0", "alias": "", "senderdomain": "example.org", "rhost": "mx.bouncehammer.jp", "action": "failed", "addresser": "kijitora@example.org", "recipient": "userunknown@bouncehammer.jp", "feedbacktype": "", "smtpcommand": "DATA", "subject": "バウンスメールのテスト(日本語)", "destination": "bouncehammer.jp", "listid": "", "diagnosticcode": "550 5.1.1 <userunknown@bouncehammer.jp>... User Unknown", "deliverystatus": "5.1.1", "lhost": "p0000-ipbfpfx00kyoto.kyoto.example.co.jp", "smtpagent": "Email::Sendmail"}]

Sisimai Specification

Differences between Ruby version and Perl version

The following table show the differences between Ruby version of Sisimai and Perl version of Sisimai. Information about differences between Sisimai and bounceHammer are available at Sisimai | Differences page.

Features Ruby version Perl version
System requirements Ruby 2.1 - 2.4 Perl 5.10 -
JRuby 9.0.4.0-
Analytical precision ratio(2000 emails)[1] 1.00 1.00
The speed of parsing email(1000 emails) 3.30s 2.33s
How to install gem install cpanm, cpm
Dependencies (Except core modules) 1 module 2 modules
LOC:Source lines of code 12600 lines 9300 lines
The number of tests(spec/,t/,xt/) directory 196800 tests 204600 tests
License BSD 2-Clause BSD 2-Clause
Support Contract provided by Developer Coming soon Available
  1. See ./ANALYTICAL-PRECISION

Other specification of Sisimai

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be parsed

Bounce mails which could not be parsed by Sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be parsed using Sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

Related Sites

See also

Author

@azumakuniyuki

Copyright

Copyright (C) 2015-2017 azumakuniyuki, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.