Net::IMAP thread deadlock bug
Closed this issue · 5 comments
Any clues how to workaorund this ?
[19:36:41] [info] [<] INBOX/supply-infreq/di.se: fetching latest message flags...
[19:36:41] [info] [>] INBOX/supply-infreq/di.se: fetching latest message flags.../usr/lib/ruby/1.9.1/thread.rb:68:in sleep': deadlock detected (fatal) from /usr/lib/ruby/1.9.1/thread.rb:68:in
wait'
from /usr/lib/ruby/1.9.1/monitor.rb:97:in wait' from /usr/lib/ruby/1.9.1/net/imap.rb:1170:in
send_literal'
from /usr/lib/ruby/1.9.1/net/imap.rb:1373:in send_data' from /usr/lib/ruby/1.9.1/net/imap.rb:1145:in
send_data'
from /usr/lib/ruby/1.9.1/net/imap.rb:1072:in block (2 levels) in send_command' from /usr/lib/ruby/1.9.1/net/imap.rb:1070:in
each'
from /usr/lib/ruby/1.9.1/net/imap.rb:1070:in block in send_command' from /usr/lib/ruby/1.9.1/monitor.rb:190:in
mon_synchronize'
from /usr/lib/ruby/1.9.1/net/imap.rb:1064:in send_command' from /usr/lib/ruby/1.9.1/net/imap.rb:635:in
append'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:78:in block in append' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:188:in
safely'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:72:in append' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:230:in
block in copy_messages'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:104:in block in each_db_message' from /var/lib/gems/1.9.1/gems/sequel-3.9.0/lib/sequel/dataset/actions.rb:15:in
each'
from /var/lib/gems/1.9.1/gems/sequel-3.9.0/lib/sequel/dataset/actions.rb:15:in all' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:104:in
each_db_message'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:188:in copy_messages' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:72:in
block in copy_all'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in block in each_mailbox' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in
each_value'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in each_mailbox' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:60:in
copy_all'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/bin/larch:118:in <module:Larch>' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/bin/larch:9:in
<top (required)>'
from /var/lib/gems/1.9.1/bin/larch:19:in load' from /var/lib/gems/1.9.1/bin/larch:19:in
[19:36:41] [info] imap.1and1.co.uk/INBOX/supply-infreq/direktronik -> imap.gmail.com/INBOX/supply-infreq/direktronik
...
[19:36:41] [info] [<] INBOX/supply-infreq/di.se: fetching latest message flags...
[19:36:41] [info] [>] INBOX/supply-infreq/di.se: fetching latest message flags.../usr/lib/ruby/1.9.1/thread.rb:68:in sleep': deadlock detected (fatal) from /usr/lib/ruby/1.9.1/thread.rb:68:in
wait'
from /usr/lib/ruby/1.9.1/monitor.rb:97:in wait' from /usr/lib/ruby/1.9.1/net/imap.rb:1170:in
send_literal'
from /usr/lib/ruby/1.9.1/net/imap.rb:1373:in send_data' from /usr/lib/ruby/1.9.1/net/imap.rb:1145:in
send_data'
from /usr/lib/ruby/1.9.1/net/imap.rb:1072:in block (2 levels) in send_command' from /usr/lib/ruby/1.9.1/net/imap.rb:1070:in
each'
from /usr/lib/ruby/1.9.1/net/imap.rb:1070:in block in send_command' from /usr/lib/ruby/1.9.1/monitor.rb:190:in
mon_synchronize'
from /usr/lib/ruby/1.9.1/net/imap.rb:1064:in send_command' from /usr/lib/ruby/1.9.1/net/imap.rb:635:in
append'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:78:in block in append' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:188:in
safely'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:72:in append' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:230:in
block in copy_messages'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:104:in block in each_db_message' from /var/lib/gems/1.9.1/gems/sequel-3.9.0/lib/sequel/dataset/actions.rb:15:in
each'
from /var/lib/gems/1.9.1/gems/sequel-3.9.0/lib/sequel/dataset/actions.rb:15:in all' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap/mailbox.rb:104:in
each_db_message'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:188:in copy_messages' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:72:in
block in copy_all'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in block in each_mailbox' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in
each_value'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch/imap.rb:126:in each_mailbox' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/lib/larch.rb:60:in
copy_all'
from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/bin/larch:118:in <module:Larch>' from /var/lib/gems/1.9.1/gems/larch-1.1.0.dev.20100209/bin/larch:9:in
<top (required)>'
from /var/lib/gems/1.9.1/bin/larch:19:in load' from /var/lib/gems/1.9.1/bin/larch:19:in
[19:36:41] [info] imap.1and1.co.uk/INBOX/supply-infreq/direktronik -> imap.gmail.com/INBOX/supply-infreq/direktronik
Please ignore this bugreport.
The error occured while the source server was slow to respond, today I don't have this error.
Unfortunately, this is a bug in Ruby's Net::IMAP library that Larch has no control over. Glad to hear you eventually got past it, though.
FYI I didn't really give a complete picture regarding the error, I'm sorry about that.
Here's what happened:
- I ran the migration using larch
- it failed/stopped with the errormessage in the above bug report
- I re-ran larch 3 times, and it always stopped at the same message (slightly different debug output afaic remember)
- I used Thunderbird to manually, via IMAP, move the last message in the debug output
- Manual move worked
- Re-ran, and now the migration ran to completion
(emails: 16436, folders: 354, mailbox transferred/size: 928MB)
The single message that blocked the migration, had a weird (dingbat looking) character in the subject.
Yep. If Net::IMAP's response parser gets into an unexpected state due to a malformed or otherwise unanticipated response, it can cause a deadlock like this. Skipping the offending message or moving it out of the way with another mail client can get things moving again.
Is it possible to scan the source mailbox beforehand, for these kinds of malformed messages, i.e. can we detect them somehow ?
The previous migration took 8 hours, and I would like to not monitor it, rather manually move malformed emails beforehand.