Create callback for "overdue" work

[zorkian]

Right now Marshal is vulnerable to a type of situation where, if you get a message that causes your processing to break, your committed offset will never be advanced since that message will end up the "oldest outstanding" and Marshal will not advance.

One solution to this is to be able to give Marshal a callback and a timeout and, when we detect that a message has been outstanding for to too long, Marshal will call you and give you a copy of the message. You can then take whatever action you want (such as moving the message to a new queue or firing an exception or etc).

The proposed implementation:

• New consumer option MessageTimeout which specifies the number of seconds before we will let you know a message has "timed out". This is measured by the time from the message being returned in the Consume call.
• New consumer option MessageTimeoutCallback which is called with a *proto.Message. When this callback returns, Marshal will consider the message to have been committed and knows it can advance the offset.

Thoughts?

cc @DrTall @basharal

[DrTall]

Sounds reasonable. I wonder if anybody will ever want a way to shut down the original processing from inside MessageTimeoutCallback, to make it atomic? If this is implemented in the most straightforward way, it would be possible for the offset to be committed while the timeout callback is running. But I suppose that if you are ALO then this should be acceptable.

[zorkian]

Migrated dropbox#4