aodn/python-aodncore

SMTP notification fails for very large collections (e.g. ARGO)

Opened this issue · 0 comments

Deleted user commented

The notify runner needs to ensure that emails are limited in size, probably by truncating the output in case of very large collections.

There are two possible causes of unlimited size:

  1. the ZIP file containing check results: https://github.com/aodn/python-aodncore/blob/master/aodncore/pipeline/steps/notify.py#L255
  2. the actual body of the email which contains a table containing each file (I suspect this is the cause in this case, because the failures are not "check" failures): https://github.com/aodn/python-aodncore/blob/master/aodncore/pipeline/steps/notify.py#L229

Possible solutions are to put the table in a ZIP attachment to reduce the size (maybe only for very large collections?). The other factor bloating the email size is that fact that the email contains both a plain text AND an HTML "part", in an effort to be friendly for multiple email clients, each containing the table separately. We could possibly remove one of the parts (the plain text?) and only send HTML.

One of the main hurdles to overcome will be to identify what maximum size to allow and how to limit the message size, bearing in mind that the message length is something like 37% larger when it is BASE64 encoded.

2018-04-10 00:03:26,755 SYSINFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] get_child_notify_runner -> 'EmailNotifyRunner'
2018-04-10 00:03:26,755 INFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] email recipients: [u'laurent.besnard@utas.edu.au']
2018-04-10 00:03:37,438 ERROR tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] (552, '5.3.4 Message size exceeds fixed limit', u'AODN <info@aodn.org.au>')
Traceback (most recent call last):
  File "/mnt/ebs/aodn-pipeline/local/lib/python2.7/site-packages/aodncore/pipeline/steps/notify.py", line 311, in run
    error_dict = self._send(recipient_addresses, message)
  File "/mnt/ebs/aodn-pipeline/local/lib/python2.7/site-packages/aodncore/pipeline/steps/notify.py", line 280, in _send
    message.as_string())
  File "/usr/lib/python2.7/smtplib.py", line 736, in sendmail
    raise SMTPSenderRefused(code, resp, from_addr)
SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', u'AODN <info@aodn.org.au>')
2018-04-10 00:03:37,442 ERROR tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] notifications failed to the following recipients: [(u'email:laurent.besnard@utas.edu.au', SMTPSenderRefused(552, '5.3.4 Message size exceeds fixed limit', u'AODN <info@aodn.org.au>'))]
2018-04-10 00:03:37,573 SYSINFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] ArgoHandler transitioned to state: HANDLER_NOTIFIED_ERROR
2018-04-10 00:03:37,577 INFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] handler result for input_file '/mnt/ebs/processing/ARGO/0be9d7a0-5601-42f4-bf55-22f7b1388743/argo_rsync.20180406-211801.rsync_manifest': ERROR
2018-04-10 00:03:37,582 SYSINFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] ArgoHandler transitioned to state: HANDLER_COMPLETED_ERROR
2018-04-10 00:03:37,649 SYSINFO tasks.ARGO[0be9d7a0-5601-42f4-bf55-22f7b1388743] IncomingFileStateManager transitioned to state: FILE_IN_ERROR