imap-filter

Intoduction

imap-filter is a Ruby implementation of an IMAP filtering application. it can handle multiple IMAP accounts, and create IMAP folders automatically where none exists.

The imap-filter DSL makes it easy to filter. You can also do “dry-runs” to make sure what happens is what is expected.

Installation

This is a Ruby CLI application. If you are running Linux, most likely you have Ruby already installed. If you are running MacOSX or Windows, you may have to Install Ruby. We support Ruby 2.2 and later.

Simply, do:

gem install imap-filter

or if in a Gemfile:

gem 'imap-filter'

By default,

Documentation

I think that to make this documenation truly “complete”, I would wind up writing more lines than in the imap-filter code itself!!! I will attempt to be as concise as possible, and besides me writing a thick manual, no one wants to read such either.

Tutorial and Examples

Let’s go though the steps with just one Email account, then we’ll add a second account. Then you can look at the sample default.imap file, and go beyond.

This tutorial assumes you have both a Gmail and a Yahoo account. Please feel free to substitute with your own accounts.

First, install imap-filter as described in Installation.

Next, create a default.imap file with Emacs, vi, nano, or whatever suits your fancy. Place in it the following:

account :gmail do
  login 'youruserid@gmail.com', 'yourpassword'
  serv "imap.gmail.com"
  auth :plain
end

And obviously adjust your userid and password to your own. Now let’s try to run your new script:

imap-filter filter default.imap -t

Note that the ‘-t’ is the test flag. It will simply test the connections to your accounts and nothing more.

And nothing happens. Why not? Very simple. The directive of this file must be activated. Edit your default.imap and add “activate :all” to the very end:

account :gmail do
  login 'youruserid@gmail.com', 'yourpassword'
  serv "imap.gmail.com"
  auth :plain
end

activate :all

Now run the command again:

imap-filter filter default.imap -t

Ah, something happens now! If all goes well, you should see the following:

====== Test Accounts
  gmail...SUCCESS, delim /

The “delim” is the mailbox delimiter for your paricular vendor. Most places use the forward slash (/). Some might use a period (.). This lets you know what to use, so make particular note of that.

If something is wrong with your settings, you might see an error message similar to:

====== Test Accounts
  gmail...FAILED: Lookup failed xxxxxxxxxxxxx

And so check your login line to make sure your userID and password is correct. For some vendors, the userID is your complete email address. For others, it might just be what procedes the ‘@’. Please check with your vendor to see what it is for you. Usually, your vendor will have IMAP Client instructions on their website.

We use ssl connections by default, and it would be extremely insecure not to. But you can turn off the ssl if need be. You can also change your port settings accordingly, if your vendor uses a non-standard port. For example,

account :gmail do
  login 'youruserid@gmail.com', 'yourpassword'
  serv "imap.gmail.com"
  auth :plain
  ssl true
  port 993
end

activate :all

You may set ssl to “false” or make the port “2001”. You have full control over the settings. auth could be :login or :plain, so try :login if :plain doesn’t work.

There are more extensive authenication methods that IMAP can use, but we do not support them at this time. If you have need for that support, please let me know!!!!!

Now, let’s assume you have your settings working. Now what? There are no filters specified, which is the entire point of this program!!! So let’s add one:

account :gmail do
  login 'youruserid@gmail.com', 'yourpassword'
  serv "imap.gmail.com"
  auth :plain
end

filter :github, 'gmail:INBOX' do
  search do
    from 'github.com'
    unseen
  end

  cp 'gmail:test/Github'
end

activate :all

Note that the “actvate :all” is always the last entry. The filter directive is fairly sophisticated, supporing most of what is possible with IMAP search. We will document this more throughly at a later date.

In this filter example, we are receiving email notifications from github which is cluttering up the Inbox. This filter will copy the “unseen”, or unread ones to the “test/Github” mailbox.

Notice in the string, we also specify the target account label. This is required, and will allow you to specify alternate accounts very easily.

The mailbox will automatically be created if it does not exist yet.

So now, let’s run it with the following commands. First, we wish to do a sanity check, so we will do a dry run:

imap-filter filter default.imap -u -v1

The “-u” is the alias for “–dryrun” The “-v1” is the verbosity switch. While doing your setups and debugging your scripts, you will want to use “-v1” or even “-v2” verbosity settings.

If you actually have a match for the search critieria, you may see a list of messages that matched the search scroll by. In my test case:

...
##Subject: Re: [jekyll/jekyll] jekyll serve can't work (#5216)
##Subject: [jruby/jruby] Fix Socket.tcp connect_timeout option (#4061)
##Subject: Re: [ansible/ansible-modules-core] Add state=empty into file module (#902)
##Subject: [ansible/ansible] Publish modules to Galaxy? (#17027)
##Subject: [ansible/ansible-modules-core] Add support for password aging on Solaris (#4372)
##Subject: Re: [jruby/jruby] Fix Socket.tcp connect_timeout option (#4061)
##Subject: Re: [tmux/tmux] Keeps getting "lost server" in tmux 2.1 and 2.2. Installed using Homebrew (#498)
##Subject: [jekyll/jekyll] Add link to Staticman (#5224)
##Subject: Re: [ansible/ansible] Ansible facts override special variables (#16935)
##Subject: Re: [elixir-lang/elixir] Introduce List.pop_at/3 (#5118)
...

Which are all unread or “unseen” as per our specificaions. Now no modifications were peformed due to the “-u” dry run switch.

Now that we are feeling a bit more confident, let’s run the same command without the “-u” setting:

imap-filter filter default.imap -v1

And now you will find that the filter actually executed what we wanted. A new test/Gitub mailbox was created, and all the matching messages are now copied there. Neat, huh?

Well, that’s it for the tutorial. Please have a look at the example.

There’s a shorthand for the filters that may interest you for the majority of your cases. For example:

filter :slashdot, 'gmx:INBOX', from: 'slashdot' do
  mark :seen
  cp 'google:INBOX'
  mv 'gmx:Slashdot'
end

Uses the short-hand of “from: ‘slashdot’” when all you need to do is search a single field, like in this case, the From field.

Others are obvios: to:, cc:, subject: will all work here.no-expand

Environment variables

Variable	Description
IMAPF_IMAP_FILE	pathname to the default .imap file. The default is ./default.imap

Accessing your Gmail

Google may automatically block this application by default. Please visit the follow page to understand how to allow imap-filter to access your Gmail account.

https://support.google.com/accounts/answer/6010255

Automatic blocks might happen while you are testing your Email accounts

Depending on the vendor, while you are setting up your scripts, you will be doing many tests. Some vendors like Yahoo might block the app if it sees what it consideres to be “too much” activity. If you see this happen, wait about 30 minutes before trying again. Should work in most cases.

DSL

account :redbird do
  login "you@redbird.org", "password"
  serv "imap.redbird.org"
  ssl false
  port 993
end

account :yahoo do
  login "you@yahoo.com", "password"
  serv "imap.yahoo.com"
end

account :google do
  login "you@google.com", "password"
  serv "imap.google.com"
end

filter :workmail, redbird.inbox, from: 'sally' do
  cp yahoo.girlfriend
  mv google.inbox
end

filter :maillist, google.inbox, from: 'github' do
  mv google.groups.github
end

Reference Documents

https://tools.ietf.org/html/rfc3501 https://www.ietf.org/rfc/rfc2822.txt

Release Notes and Known Issues

Release Notes

Version	Breif	Description
0.1.4	Dangling command removed	There was a list command that I had not implemented yet, but Thor now complains when it sees that. So the dangling command was removed.
0.1.0	election day bug fixed	This was an issue with moves not deleting the sources, and had to do with state information not existing in the duplicate delegator object. This snafu has been dealt with. No more duplicate delegators
0.0.4	“Too many errors” fixed	The “duplicate mailbox” error was being counted by some providers as being “too much” so now we check first before attempting to create a new mailbox.
0.0.3	Fixed bug with timeout	On some lengthy remove copy/move operations, the source account can timeout. We do a simple NOOP to keep the connection alive.
0.0.2	Initial Release

Known Issues

Date	Description
2017-09-25	No spec coverage. I know, I know…
2017-09-25	There is a problem connecting to Yahoo, and it looks like an OAuth support issue from my end, as they apparenly no longer support the “simple” login anymore. Issue is being worked on on a branch for now.
2016-11-08	US election day bug :p – messages on move are not being moved, only copied with v0.0.4. Refactoring is broken. Use 0.0.3 for now. Fix is coming soon.
2016-11-05	Aside from being Guy Fawkes Day, the fix to “Too Many Errors” involved caching the list of mail boxes. If something else creates a mailbox while this is running, there is a potential race condition, but the name collision, if any, will only result in a “duplicate mailbox” error.
2016-09-25	Remote copy / moves of HTML-based email does not work cleanly.
2016-10-03	Timeout errors still persist in some cases.
	Need to capture all errors and provide human-readable responses, unless a certail verbosity level has been set.

Contributing to imap-filter

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet. |
Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it. |
Fork the project. |
Start a feature/bugfix branch. |
Commit and push until you are happy with your contribution. |
Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally. |
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it. |

Copyright

Scratchpad

This section is my personal scratchpad. Should be of no revelance to anyone else. Please ignore what’s in this section.

imap = Net::IMAP.new(‘imap.gmail.com’, ssl: true) imap.authenticate(‘PLAIN’, ENV[‘GOOGLE_EMAIL’], ENV[‘GOOGLE_PASS’]) imap.select(‘Shelby’) seq = imap.search([‘ALL’])acc.imap.fetch(1..2, ‘BODY[TEXT]’) – the entire text, flags as well. acc.imap.fetch(1..2, ‘BODY[HEADER.FIELDS (SUBJECT)]’)IOError: closed stream from 2.3.1/lib/ruby/2.3.0/openssl/buffering.rb:322:in `syswrite’

flajann2/imap-filter