Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373
jbouwh opened this issue · 16 comments
When processing the opendmarc history file to be imported into opendmarc-import warnings appear to stderr.
To make sure the database is not the cause I ran dpkg-reconfigure opendmarc to recreate the opendmarc database.
The warning reoccurs every 21 lines of the .
It seems the requests table is not populating. So there is no way reports can be sent.
Records still get imported into the database though
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 22.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 22.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 43.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 43.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 64.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 64.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 85.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 85.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 106.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 106.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 127.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 127.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 148.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 148.
<13>Oct 2 15:11:51 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 169.
<13>Oct 2 15:11:51 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 169.
Running opendmarc on debian 11.0:
libopendmarc-dev/stable 1.4.0~beta1+dfsg-6 amd64
Headers and development libraries for the OpenDMARC library
libopendmarc2/stable,now 1.4.0~beta1+dfsg-6 amd64 [installed,automatic]
Library for DMARC validation and reporting
opendmarc/stable,now 1.4.0~beta1+dfsg-6 amd64 [installed]
Milter implementation of DMARC
In the mean while I have upgraded opendmarc to 1.4.1.1-1 (the testing version for debian). I will report back if there is any difference. A least it seems the opendmarc-import script was not changed for some time. So it could be caused bij changes in the history file format.
A test file wih history file (changed the domains):
job 5619C60507
reporter mta.receiver-domain.com
received 1633209492
ipaddr 1.2.3.4
from hotmail.com
mfrom example-mail.com
dkim example-mail.com selector1 0
spf 0
pdomain example-mail.com
policy 15
rua mailto:d@rua.example.com
pct 100
adkim 114
aspf 114
p 110
sp 0
align_dkim 4
align_spf 4
arc 7
arc_policy 2 json:[{ "i": 1, "d":"example.com", "s":"arcselector1", "ip":"" }]
action 2
will give the following log errors:
<13>Oct 2 23:18:49 dmarcgenreports: Use of uninitialized value in numeric ne (!=) at /usr/sbin/opendmarc-import line 371, <STDIN> line 21.
<13>Oct 2 23:18:49 dmarcgenreports: Use of uninitialized value in concatenation (.) or string at /usr/sbin/opendmarc-import line 373, <STDIN> line 21.
<13>Oct 2 23:18:49 dmarcgenreports: opendmarc-import: failed to retrieve table ID:
Same on my bullseye host (problem started after dist-upgrade from buster).
Got opendmarc-import 1.4.1 into /usr/local and tried it with history file copied into /tmp
/usr/local/sbin/opendmarc-importstats
opendmarc-import-new: started at Sun Oct 3 11:27:28 2021
opendmarc-import-new: connected to database
opendmarc-import-new: updating at line 22
opendmarc-import-new: failed to insert message: Powtórzone wystąpienie '1-2021-10-02 08:36:45-A688C973204' dla klucza 'reporter'
(duplicated occurrence for key)
History record was:
job A688C973204
reporter my.mta.pl
received 1633156605
ipaddr x.x.x.x
from domain.pl
mfrom z.domain.pl
dkim z.domain.pl m3 0
spf 0
pdomain domain.pl
policy 18
rua mailto:dmarc-reports@domain.pl
pct 100
adkim 114
aspf 114
p 110
sp 0
align_dkim 5
align_spf 5
arc 7
arc_policy 2 json:[]
action 2
So upgrading to version from Debian testing may change nothing as opendmarc probably will not change any parts of perl libs and new import script did not solve my problem.
diff /usr/sbin/opendmarc-import /usr/local/sbin/opendmarc-import-new
241c242,243
< !defined($ipaddr_id))
---
> !defined($ipaddr_id) ||
> !defined($request_id))
Same on my bullseye host (problem started after dist-upgrade from buster).
Got opendmarc-import 1.4.1 into /usr/local and tried it with history file copied into /tmp
/usr/local/sbin/opendmarc-importstats opendmarc-import-new: started at Sun Oct 3 11:27:28 2021 opendmarc-import-new: connected to database opendmarc-import-new: updating at line 22 opendmarc-import-new: failed to insert message: Powtórzone wystąpienie '1-2021-10-02 08:36:45-A688C973204' dla klucza 'reporter'(duplicated occurrence for key)
History record was:
job A688C973204 reporter my.mta.pl received 1633156605 ipaddr x.x.x.x from domain.pl mfrom z.domain.pl dkim z.domain.pl m3 0 spf 0 pdomain domain.pl policy 18 rua mailto:dmarc-reports@domain.pl pct 100 adkim 114 aspf 114 p 110 sp 0 align_dkim 5 align_spf 5 arc 7 arc_policy 2 json:[] action 2So upgrading to version from Debian testing may change nothing as opendmarc probably will not change any parts of perl libs and new import script did not solve my problem.
diff /usr/sbin/opendmarc-import /usr/local/sbin/opendmarc-import-new 241c242,243 < !defined($ipaddr_id)) --- > !defined($ipaddr_id) || > !defined($request_id))
I think there is a new database schema, did you update to the latest schema?
Can you try replacing Debian’s opendmarc-import with one produced directly from this repository and see if the problem goes away?
This may or may not be related to Debian-specific mystery patch ticket193.patch. This patch has been in Debian for a while but it may have become out of sync. (And frankly I don’t know why it is there or what it does.) There is also another open Debian bug, 955867. Help with handling these issues is welcome.
Just guessing though.
Can you try replacing Debian’s
opendmarc-importwith one produced directly from this repository and see if the problem goes away?This may or may not be related to Debian-specific mystery patch ticket193.patch. This patch has been in Debian for a while but it may have become out of sync. (And frankly I don’t know why it is there or what it does.) There is also another open Debian bug, 955867. Help with handling these issues is welcome.
Just guessing though.
I'll have a look tomorrow. the database patch I already noticed. Without it it would break the db upgrade.
Can you try replacing Debian’s
opendmarc-importwith one produced directly from this repository and see if the problem goes away?
Got opendmarc-import and opendmarc-reports from 1.4.1 and everything returned to work as before upgrade to bullseye:
/etc/cron.daily/opendmarc-importstats:
opendmarc-import: started at Mon Oct 4 08:12:24 2021
opendmarc-import: connected to database
opendmarc-import: updating at line 22
opendmarc-import: updating at line 43
opendmarc-import: updating at line 64
opendmarc-import: updating at line 86
opendmarc-import: updating at line 108
opendmarc-import: updating at line 130
opendmarc-import: updating at line 151
opendmarc-import: updating at line 171
opendmarc-import: terminating at Mon Oct 4 08:12:26 2021
/etc/cron.daily/opendmarc-reports:
opendmarc-reports: started at Mon Oct 4 08:12:31 2021
opendmarc-reports: selected 9 domain(s)
opendmarc-reports: sent report for dom.pl to dmarc-reports@dom.pl (2.0.0 Ok: queued as 0C93B1892B5)
opendmarc-reports: sent report for google.com to mailauth-reports@google.com (2.0.0 Ok: queued as 74F621892B7)
opendmarc-reports: sent report for github.com to dmarc@github.com (2.0.0 Ok: queued as 9ED141892B8)
opendmarc-reports: sent report for github.com to d@rua.agari.com (2.0.0 Ok: queued as D7E9D1892B9)
opendmarc-reports: sent report for dom.pl to dmarc@dom.pl (2.0.0 Ok: queued as 760931892BA)
opendmarc-reports: terminating at Mon Oct 4 08:12:33 2021
So can confirm proper working with perl 5.32.1-4+deb11u1.
This may or may not be related to Debian-specific mystery patch ticket193.patch. This patch has been in Debian for a while but it may have become out of sync. (And frankly I don’t know why it is there or what it does.) There is also another open Debian bug, 955867. Help with handling these issues is welcome.
I think it was the problem.
Thx4help&support.
I am sure this has to do with the patch. I had to subtitute @SQLBACKEND to mysql and @VERSION to 1.4.1 to make it all work.
Now the records import correctly. Not sure why this patch was applied.
The database patch inserts 1970-01-01 00:00:01 and a time_zone. We must make sure that we insert supported values, see:
https://mariadb.com/kb/en/timestamp/
Further a new patch is needed to undo the disrupture.
@thegushi Can you organize that the right actions will be organized?
Conclusion: The debian patch breaks opendmarc-import but fixes the database schema.
@jbouwh We now know that this is a bug in the Debian package, so it needs to be handled in Debian bugs, not here. I can open a ticket.
edit: Here it is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995694
@jbouwh We now know that this is a bug in the Debian package, so it needs to be handled in Debian bugs, not here. I can open a ticket.
edit: Here it is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995694
Right, what I can tell is that the database schema needs at least the adjustments for setting 1970-01-01 00:00:00 as a default value, since this is not a supported value. This is some thing the patch also addresses, but needs to be changed here.
Right. I suggest a PR is opened to apply the neede schema changes. 1970-01-01 00:00:01 would be fine. Further we should be helped with a better documentation of the debian patch. Needed changes to source or schema should be updated in the source i suppose.
Closing this issue as this is clearly stale.