Sisimai is a Ruby library for analyzing RFC5322 bounce emails and generating structured data from parsed results. Ruby version of Sisimai is ported from Perl version of Sisimai at https://github.com/sisimai/p5-Sisimai/ .
Sisimai(シシマイ)はRFC5322準拠のエラーメールを解析し、解析結果をデータ構造に 変換するインターフェイスを提供するRubyライブラリです。 https://github.com/sisimai/p5-Sisimai/で公開しているPerl版シシマイから移植しました。
- Convert Bounce Mails to Structured Data | エラーメールをデータ構造に変換
- Supported formats are Perl and JSON | Perlのデータ形式とJSONに対応
- Easy to Install, Use. | インストールも使用も簡単
- gem install
- git clone & make
- High Precision of Analysis | 高い解析精度
- 2 times higher than bounceHammer | 解析精度はbounceHammerの二倍
- Support 21 known MTAs and 5 unknown MTAs | 26種類のMTAに対応
- Support 21 major MSPs(Mail Service Providers) | 21種類の著名なMSPに対応
- Support Feedback Loop Message(ARF) | Feedback Loopにも対応
- Can detect 27 error reasons | 27種類のエラー理由を検出
More details about system requirements are available at available at Sisimai | Getting Started page.
- Ruby 2.1.0 or later
- Also works on JRuby 9.0.0.0 or later
% sudo gem install sisimai
Fetching: sisimai-4.16.0.gem (100%)
Successfully installed sisimai-4.16.0
Parsing documentation for sisimai-4.16.0
Installing ri documentation for sisimai-4.16.0
Done installing documentation for sisimai after 6 seconds
1 gem installed% cd /usr/local/src
% git clone https://github.com/sisimai/rb-Sisimai.git
% cd ./rb-Sisimai
% sudo make depend install-from-local
gem install bundle rake rspec coveralls
Successfully installed bundle-0.0.1
Parsing documentation for bundle-0.0.1
Done installing documentation for bundle after 0 seconds
Successfully installed rake-10.5.0
Parsing documentation for rake-10.5.0
Done installing documentation for rake after 1 seconds
Successfully installed rspec-3.4.0
Parsing documentation for rspec-3.4.0
Done installing documentation for rspec after 0 seconds
Successfully installed coveralls-0.8.10
Parsing documentation for coveralls-0.8.10
Done installing documentation for coveralls after 0 seconds
4 gems installed
bundle exec rake install
sisimai 4.14.2 built to pkg/sisimai-4.16.0.gem.
sisimai (4.14.2) installed.make() method provides feature for getting parsed data from bounced email messages like following.
#! /usr/bin/env ruby
require 'sisimai'
v = Sisimai.make('/path/to/mbox') # or path to Maildir/
# If you want to get bounce records which reason is "delivered", set "delivered"
# option to make() method like the following:
v = Sisimai.make('/path/to/mbox', delivered: true)
unless v.void
v.each do |e|
puts e.class # Sisimai::Data
puts e.recipient.class # Sisimai::Address
puts e.timestamp.class # Sisimai::Time
puts e.addresser.address # shironeko@example.org # From
puts e.recipient.address # kijitora@example.jp # To
puts e.recipient.host # example.jp
puts e.deliverystatus # 5.1.1
puts e.replycode # 550
puts e.reason # userunknown
h = e.damn # Convert to HASH
j = e.dump('json') # Convert to JSON string
puts e.dump('json') # JSON formatted bounce data
end
end
# Get JSON string from parsed mailbox or Maildir/
puts Sisimai.dump('/path/to/mbox') # or path to Maildir/
# dump() method also accepts "delivered" option like the following code:
puts Sisimai.dump('/path/to/mbox', delivered: true)[{"recipient": "kijitora@example.jp", "addresser": "shironeko@1jo.example.org", "feedbacktype": "", "action": "failed", "subject": "Nyaaaaan", "smtpcommand": "DATA", "diagnosticcode": "550 Unknown user kijitora@example.jp", "listid": "", "destination": "example.jp", "smtpagent": "Courier", "lhost": "1jo.example.org", "deliverystatus": "5.0.0", "timestamp": 1291954879, "messageid": "201012100421.oBA4LJFU042012@1jo.example.org", "diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "filtered", "token": "ce999a4c869e3f5e4d8a77b2e310b23960fb32ab", "alias": "", "senderdomain": "1jo.example.org", "rhost": "mfsmax.example.jp"}, {"diagnostictype": "SMTP", "timezoneoffset": "+0900", "reason": "userunknown", "timestamp": 1381900535, "messageid": "E1C50F1B-1C83-4820-BC36-AC6FBFBE8568@example.org", "token": "9fe754876e9133aae5d20f0fd8dd7f05b4e9d9f0", "alias": "", "senderdomain": "example.org", "rhost": "mx.bouncehammer.jp", "action": "failed", "addresser": "kijitora@example.org", "recipient": "userunknown@bouncehammer.jp", "feedbacktype": "", "smtpcommand": "DATA", "subject": "バウンスメールのテスト(日本語)", "destination": "bouncehammer.jp", "listid": "", "diagnosticcode": "550 5.1.1 <userunknown@bouncehammer.jp>... User Unknown", "deliverystatus": "5.1.1", "lhost": "p0000-ipbfpfx00kyoto.kyoto.example.co.jp", "smtpagent": "Sendmail"}]上記のようにSisimaiのmake()メソッドをmboxかMaildirのPATHを引数にして実行すると 解析結果が配列で返ってきます。
% ruby -rsisimai -e 'puts Sisimai.dump($*.shift)' /path/to/mboxThe following table show the differences between Perl version of Sisimai and Ruby version of Sisimai.
| Features | Ruby version | Perl version |
|---|---|---|
| System requirements | Ruby 2.1-2.3 | Perl 5.10 - |
| JRuby 9.0.4.0- | ||
| Analytical precision ratio(2000 emails)[1] | 1.00 | 1.00 |
| The speed of parsing email(1000 emails) | 3.30s | 2.33s |
| How to install | gem install | cpanm |
| Dependencies (Except core modules) | 1 modules | 2 modules |
| LOC:Source lines of code | 11000 lines | 8000 lines |
| The number of tests in t/, xt/ directory | 94300 tests | 170000 tests |
| License | BSD 2-Clause | BSD 2-Clause |
| Support Contract provided by Developer | Coming soon | Available |
- See ./ANALYTICAL-PRECISION
公開中のPerl版Sisimai(p5-Sisimai)とRuby版Sisimai(rb-Sisimai)は下記のような違いが あります。
| 機能 | Ruby version | Perl version |
|---|---|---|
| 動作環境 | Ruby 2.1-2.3 | Perl 5.10 - |
| JRuby 9.0.4.0- | ||
| 解析精度の割合(2000通のメール)[1] | 1.00 | 1.00 |
| メール解析速度(1000通のメール) | 3.30秒 | 2.33秒 |
| インストール方法 | gem install | cpanm |
| 依存モジュール数(コアモジュールを除く) | 1モジュール | 2モジュール |
| LOC:ソースコードの行数 | 11000行 | 8000行 |
| テスト件数(t/,xt/ディレクトリ) | 94300件 | 170000件 |
| ライセンス | 二条項BSD | 二条項BSD |
| 開発会社によるサポート契約 | 準備中 | 提供中 |
- ./ANALYTICAL-PRECISIONを参照
The following table is the list of MTA/MSP:(Mail Service Provider) modules. More details about these modules are available at Sisimai | Parser Engines page.
| Module Name(Sisimai::) | Description |
|---|---|
| MTA::Activehunter | TransWARE Active!hunter |
| MTA::ApacheJames | Java Apache Mail Enterprise Server |
| MTA::Courier | Courier MTA |
| MTA::Domino | IBM Domino Server |
| MTA::Exchange | Microsoft Exchange Server |
| MTA::Exim | Exim |
| MTA::IMailServer | IPSWITCH IMail Server |
| MTA::InterScanMSS | Trend Micro InterScan Messaging Security Suite |
| MTA::MXLogic | McAfee SaaS |
| MTA::MailFoundry | MailFoundry |
| MTA::MailMarshalSMTP | Trustwave Secure Email Gateway |
| MTA::McAfee | McAfee Email Appliance |
| MTA::MessagingServer | Oracle Communications Messaging Server |
| MTA::MFILTER | Digital Arts m-FILTER |
| MTA::Notes | Lotus Notes |
| MTA::OpenSMTPD | OpenSMTPD |
| MTA::Postfix | Postfix |
| MTA::Qmail | qmail |
| MTA::Sendmail | V8Sendmail: /usr/sbin/sendmail |
| MTA::SurfControl | WebSense SurfControl |
| MTA::V5sendmail | Sendmail version 5 |
| MTA::X1 | Unknown MTA #1 |
| MTA::X2 | Unknown MTA #2 |
| MTA::X3 | Unknown MTA #3 |
| MTA::X4 | Unknown MTA #4 qmail clones |
| MTA::X5 | Unknown MTA #5 |
| MSP::DE::EinsUndEins | 1&1: http://www.1and1.de |
| MSP::DE::GMX | GMX: http://www.gmx.net |
| MSP::JP::Biglobe | BIGLOBE: http://www.biglobe.ne.jp |
| MSP::JP::EZweb | au EZweb: http://www.au.kddi.com/mobile/ |
| MSP::JP::KDDI | au by KDDI: http://www.au.kddi.com |
| MSP::RU::MailRu | @mail.ru: https://mail.ru |
| MSP::RU::Yandex | Yandex.Mail: http://www.yandex.ru |
| MSP::UK::MessageLabs | Symantec.cloud http://www.messagelabs.com |
| MSP::US::AmazonSES | AmazonSES(Sending): http://aws.amazon.com/ses/ |
| MSP::US::AmazonWorkMail | Amazon WorkMail: https://aws.amazon.com/workmail/ |
| MSP::US::Aol | Aol Mail: http://www.aol.com |
| MSP::US::Bigfoot | Bigfoot: http://www.bigfoot.com |
| MSP::US::Facebook | Facebook: https://www.facebook.com |
| MSP::US::Google | Google Gmail: https://mail.google.com |
| MSP::US::Office365 | Microsoft Office 365: http://office.microsoft.com/ |
| MSP::US::Outlook | Microsoft Outlook.com: https://www.outlook.com/ |
| MSP::US::ReceivingSES | AmazonSES(Receiving): http://aws.amazon.com/ses/ |
| MSP::US::SendGrid | SendGrid: http://sendgrid.com/ |
| MSP::US::Verizon | Verizon Wireless: http://www.verizonwireless.com |
| MSP::US::Yahoo | Yahoo! MAIL: https://www.yahoo.com |
| MSP::US::Zoho | Zoho Mail: https://www.zoho.com |
| ARF | Abuse Feedback Reporting Format |
| RFC3464 | Fallback Module for MTAs |
| RFC3834 | Detector for auto replied message |
上記はSisimaiに含まれてるMTA/MSP(メールサービスプロバイダ)モジュールの一覧です。
Sisimai can detect the following 27 bounce reasons. More details about reasons are available at Sisimai | Bounce Reason List page.
| Reason(理由) | Description | 理由の説明 |
|---|---|---|
| Blocked | Blocked due to client IP address | IPアドレスによる拒否 |
| ContentError | Invalid format email | 不正な形式のメール |
| Delivered[1] | Successfully delivered (> v4.16.0) | 正常に配信された(> v4.16.0) |
| ExceedLimit | Message size exceeded the limit(5.2.3) | メールサイズの超過 |
| Expired | Delivery time expired | 配送時間切れ |
| Feedback | Bounced for a complaint of the message | 元メールへの苦情によるバウンス |
| Filtered | Rejected after DATA command | DATAコマンド以降で拒否された |
| HasMoved | Destination mail addrees has moved | 宛先メールアドレスは移動した |
| HostUnknown | Unknown destination host name | 宛先ホスト名が存在しない |
| MailboxFull | Recipient's mailbox is full | メールボックスが一杯 |
| MailerError | Mailer program error | メールプログラムのエラー |
| MesgTooBig | Message size is too big(5.3.4) | メールが大き過ぎる |
| NetworkError | Network error: DNS or routing | DNS等ネットワーク関係のエラー |
| NotAccept | Destinaion does not accept any message | 宛先ホストはメールを受けとらない |
| OnHold | Deciding the bounce reason is on hold | エラー理由の特定は保留 |
| Rejected | Rejected due to envelope from address | エンベロープFromで拒否された |
| NoRelaying | Relay access denied | リレーの拒否 |
| SecurityError | Virus detected or authentication error | ウィルスの検出または認証失敗 |
| SpamDetected | Detected a message as spam | メールはスパムとして判定された |
| Suspend | Recipient's account is suspended | 宛先アカウントは一時的に停止中 |
| SyntaxError | syntax error in SMTP (> v4.17.0) | SMTPの文法エラー(> v4.17.0) |
| SystemError | Some error on the destination host | 宛先サーバでのOSレベルのエラー |
| SystemFull | Disk full on the destination host | 宛先サーバのディスクが一杯 |
| TooManyConn | Connection rate limit exceeded | 接続制限数を超過した |
| UserUnknown | Recipient's address does not exist | 宛先メールアドレスは存在しない |
| Undefined | Could not decide the error reason | バウンスした理由は特定出来ず |
| Vacation | Auto replied message | 自動応答メッセージ |
Sisimaiは上記のエラー27種を検出します。
- This reason is not included by default | このバウンス理由は標準では解析結果に含まれません
The following table shows a data structure(Sisimai::Data) of parsed bounce mail. More details about data structure are available at available at Sisimai — Data Structure of Sisimai::Data page.
| Name | Description | 値の説明 |
|---|---|---|
| action | The value of Action: header | Action:ヘッダの値 |
| addresser | The From address | 送信者のアドレス |
| alias | Alias of the recipient | 受信者アドレスのエイリアス |
| destination | The domain part of the "recipinet" | "recipient"のドメイン部分 |
| deliverystatus | Delivery Status(DSN) | 配信状態(DSN)の値 |
| diagnosticcode | Error message | エラーメッセージ |
| diagnostictype | Error message type | エラーメッセージの種別 |
| feedbacktype | Feedback Type | Feedback-Typeのフィールド |
| lhost | local host name(local MTA) | 送信側MTAのホスト名 |
| listid | List-Id: header of each ML | List-Idヘッダの値 |
| messageid | Message-Id: of the original message | 元メールのMessage-Id |
| reason | Detected bounce reason | 検出したバウンスした理由 |
| recipient | Recipient address which bounced | バウンスした受信者のアドレス |
| replycode | SMTP Reply Code | SMTP応答コード |
| rhost | Remote host name(remote MTA) | 受信側MTAのホスト名 |
| senderdomain | The domain part of the "addresser" | "addresser"のドメイン部分 |
| softbounce | The bounce is soft bounce or not | ソフトバウンスであるかどうか |
| smtpagent | MTA name(Sisimai::MTA::, MSP::) | MTA名(Sisimai::MTA::,MSP::) |
| smtpcommand | The last SMTP command in the session | セッション中最後のSMTPコマンド |
| subject | Subject of the original message(UTF8) | 元メールのSubject(UTF-8) |
| timestamp | Date: header in the original message | 元メールのDate |
| timezoneoffset | Time zone offset(seconds) | タイムゾーンの時差 |
| token | MD5 value of addresser and recipient | 送信者と受信者のハッシュ値 |
上記の表は解析後のバウンスメールの構造(Sisimai::Data)です。
Bounce mails which could not be parsed by Sisimai are saved in the directory
set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you find any
bounce email cannot be parsed using Sisimai, please add the email into the directory
and send Pull-Request to this repository.
解析出来ない__バウンスメール__はset-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet
ディレクトリにはいっています。もしもSisimaiで解析出来ないメールを見つけたら、
このディレクトリに追加してPull-Requestを送ってください。
- libsisimai.org | Sisimai | The successor to bounceHammer, Library to parse bounce mails
- GitHub | github.com/sisimai/rb-Sisimai
- Perl verson | Perl version of Sisimai
- RFC3463 - Enhanced Mail System Status Codes
- RFC3464 - An Extensible Message Format for Delivery Status Notifications
- RFC3834 - Recommendations for Automatic Responses to Electronic Mail
- RFC5321 - Simple Mail Transfer Protocol
- RFC5322 - Internet Message Format
Copyright (C) 2015-2016 azumakuniyuki, All Rights Reserved.
This software is distributed under The BSD 2-Clause License.