Mail client IP marked with RCVD_IN_ZEN_LASTEXTERNAL and SHORTCIRCUIT
bjoernv opened this issue · 12 comments
Currently spamassassin-dqs does not always ignores the mail client (sender) IP. Mail client IP addresses are often marked in blocklists like ZEN. May be the bug is triggered by special Received: header formats.
The following mail shows the problem. The senders IP 5.61.176.191 is found in ZEN (can be shown with "spamassassin --debug").
Received: from fra1frontrelay14.vodafonemail.de (fra1prox51.fra-mediabeam.com [10.110.1.51]) by fra1checkrelay06.fra-mediabeam.com (8.15.2/8.15.2/Debian-10) with ESMTPS id 13PKjZmL025258 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for <example@vodafone.com>; Sun, 25 Apr 2021 22:45:36 +0200
Received: from smtpout2.vodafonemail.de (fra1prox21.fra-mediabeam.com [10.110.1.21]) by fra1frontrelay14.vodafonemail.de (8.15.2/8.15.2/Debian-10) with ESMTPS id 13PKjZQ3011373 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT) for <example@vodafone.com>; Sun, 25 Apr 2021 22:45:35 +0200
Received: from smtp.vodafone.de (smtpa07.fra-mediabeam.com [10.2.0.38]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id F3BCF1244F4; Sun, 25 Apr 2021 22:45:34 +0200 (CEST)
Received: from [192.168.1.2] (unknown [5.61.176.191]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.vodafone.de (Postfix) with ESMTPSA id C4E6A140258; Sun, 25 Apr 2021 20:45:33 +0000 (UTC)
Date: Sun, 25 Apr 2021 22:45:33 +0200
Subject: test
From: <example@vodafone.com>
Content-Type: text/plain; charset=utf-8
test
The mail gets these flags from spamassassin:
X-Spam-Flag: YES
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=100.5 required=5.0 tests=RCVD_IN_ZEN_LASTEXTERNAL,
SHORTCIRCUIT shortcircuit=spam autolearn=disabled version=3.4.6
X-Spam-Report:
* 100 SHORTCIRCUIT Not all rules were run, due to a shortcircuited
* rule
* 0.5 RCVD_IN_ZEN_LASTEXTERNAL The last untrusted relay is listed in
* Spamhaus ZEN
Hi,
this is not a bug in SpamAssassin or the plugin, but a misconfiguration on your side.
You need to configure your system to avoid RBLs for SMTP submissions.
this is not a bug in SpamAssassin or the plugin, but a misconfiguration on your side.
You need to configure your system to avoid RBLs for SMTP submissions.
No, you are wrong here. The SMTP submission side is managed by Vodafone. Vodafone is a big telecommunication company like AT&T or German Telekom. I can not change their mailservers. Spamassassin is running on the receiver side.
This is the data flow for the example mail:
Thunderbird mailer (at home with IP 192.168.1.2] (unknown [5.61.176.191] --> SMTP submission with SMTP AUTH (Vodafone, no Spamassassin --> SMTP receiver (my company mail server with Spamassassin) or my local Linux desktop SMTP server (with Spamassassin, mail fetched by fetchmail - in this example)
Please paste a complete sample on pastebin so I can take a look at it
Here is a complete sample
https://paste.opensuse.org/c24fe76b
This is the data flow:
Thunderbird mailer (at home with IP [192.168.1.2] (unknown [5.61.176.191])) From: bjoernv@arcor.de, To: bjoernv@arcor.de --> Vodafone (MX for arcor.de, smtp.vodafone.de) --> Fetchmail to local Sendmail (at home with IP 192.168.1.2)
Thanks, I just ran spamassassin on my test server (SA 3.4.6 with latest plugin installed) and this is the result:
Content preview: test 1
Content analysis details: (-3.0 points, 5.0 required)
pts rule name description
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-2.0 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail
provider
[bjoernv[at]arcor.de]
I really think that there is some misconfiguration on your side since I cannot reproduce the error
pts rule name description
What do you mean?
It's the standard output when running spamassassin -t
I'll close this issue as it is not a bug of the plugin. If you need help with spamassassin I suggest you to post your issues on the mailing list spamassassin-users: https://cwiki.apache.org/confluence/display/SPAMASSASSIN/MailingLists
Output of spamassassin -t is:
Pkte Regelname Beschreibung
---- ---------------------- --------------------------------------------------
100 SHORTCIRCUIT Not all rules were run, due to a shortcircuited
rule
0.5 RCVD_IN_ZEN_LASTEXTERNAL The last untrusted relay is listed in
Spamhaus ZEN
RCVD_IN_ZEN_LASTEXTERNAL is defined in /etc/mail/spamassassin/sh.cf from spamassassin-dqs. So only spamassassin-dqs users can see the issue.
I know it is defined there, but it uses standard SA configuration parameters, it's not a lookup done by the plugin itself.
The only thing you can do is fix your configuration
Personally I doubt, that Spamassassin does not understand the Received: header of the Vodafone mail servers correctly. /usr/lib/perl5/vendor_perl/5.32.1/Mail/SpamAssassin/Message/Metadata/Received.pm (from Spamassasin base) has a lot of rules for mail servers from Gmail, AOL, etc. I will try to find the bug here. spamassassin-dqs needs to correctly distinguish between ESMTPSA (SMTP with authentication) Received: and other Received: headers and this does not work correctly for Vodafone.
It's not the job of a rule to distinguish between submissions or email received from external servers.
Other than that, there are hundreds of customers running spamassassin-dqs and nobody complained about this issue, and since vodafone.de is a pretty big provider if there were a bug I bet we would have ton of reports.
Now I found the main reason for the problems. zendqs-lastexternal selects an IP using this code (I added some debugging statements).
/usr/lib/perl5/vendor_perl/5.32.1/Mail/SpamAssassin/Plugin/DNSEval.pm
# use the external IP set, instead of the trusted set; the user may have
# specified some third-party relays as trusted. Also, don't use
# @originating; those headers are added by a phase of relaying through
# a server like Hotmail, which is not going to be in dialup lists anyway.
dbg("dns: MY fullexternal IPs: ".join(", ", @fullexternal));
@ips = $self->ip_list_uniq_and_strip_private(@fullexternal);
if ($1 eq "lastexternal") {
dbg("dns: MY ips IPs: ".join(", ", @ips));
@ips = (defined $ips[0]) ? ($ips[0]) : ();
dbg("dns: MY lastexternal IPs: ".join(", ", @ips));
} else {
pop @ips if (scalar @ips > 1);
}
For the Vodafone example mail it only finds on external IP (5.61.176.191) in the following list: 10.110.1.51, 10.110.1.21, 10.2.0.38, 5.61.176.191
Unfortunately IP 5.61.176.191 is my dialup IP and is listed in ZEN. Shortcircuit makes Spamassassin fail this mail as Spam directly.
Vodafone has a mail header configuration issue. The following Vodafone header is wrong because it resolves the host smtp.vodafone.de with the private IP 10.2.0.38.
Received: from smtp.vodafone.de (smtpa07.fra-mediabeam.com [10.2.0.38]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 6EC5B12153A for bjoernv@arcor.de; Tue, 27 Apr 2021 19:43:59 +0200 (CEST)
This is the corrected header
Received: from smtp.vodafone.de (smtp.vodafonemail.de [2.207.150.234]) by smtpout2.vodafonemail.de (Postfix) with ESMTP id 6EC5B12153A for bjoernv@arcor.de; Tue, 27 Apr 2021 19:43:59 +0200 (CEST)
If I manually correct the Vodafone header (can be also done automatically, then Spamassassin selects the IP 2.207.150.234 with zendqs-lastexternal and the mail is not marked as Spam anymore
I found these solutions until Vodafone fixes their mailservers:
- Comment out zendqs-lastexternal in /etc/mail/spamassassin/sh.cf from spamassassin-dqs or
- Decrease the score for RCVD_IN_ZEN_LASTEXTERNAL and comment out RCVD_IN_ZEN_LASTEXTERNAL in the Mail::SpamAssassin::Plugin::Shortcircuit block in /etc/mail/spamassassin/sh.cf from spamassassin-dqs