Full zone update brakes a zone and next incremental updates do not add new indicators

[Homas]

1. Bad full zone update

2019-09-10T04:41:29.231269+00:00 Zone "notracking.ioc2rpz" serial 1568047260, refresh time 604800 current status ready
2019-09-10T04:41:29.231634+00:00 Updating zone "notracking.ioc2rpz" full
2019-09-10T04:41:29.231856+00:00 Source  "notracking_hosts" was expired in cache
2019-09-10T04:41:29.457620+00:00 Source: "notracking_hosts", size: 2.25/MB (2358005), MD5: "d7768fc5f62ceff4c3feaa92a056c845"
2019-09-10T04:41:29.634436+00:00 Source: "notracking_hosts", got 41487 indicators, clean time 0
2019-09-10T04:41:29.685434+00:00 Source  "notracking_domains" was expired in cache
2019-09-10T04:41:30.289340+00:00 Source: "notracking_domains", size: 5.33/MB (5592314), MD5: "fd98ddd2a22f3563feba8c1cf26ad5c0"
2019-09-10T04:41:30.722246+00:00 Source: "notracking_domains", got 92379 indicators, clean time 0
2019-09-10T04:41:31.082647+00:00 Delete old records from zone "notracking.ioc2rpz".  before 133964 after 1033
2019-09-10T04:41:33.940637+00:00 Live zone "notracking.ioc2rpz", 267732 rules, 133866 IOCs
2019-09-10T04:41:33.940923+00:00 Zone "notracking.ioc2rpz" updated in 0 seconds, new serial 1568090460, 267732 rules, 133866 indicators.

2. Incremental update

2019-09-11T04:41:29.231298+00:00 Start incremental update Zone "notracking.ioc2rpz" serial 1568133660 full refresh time 43200, Ctime 1568176889 cache <<"true">> status ready
2019-09-11T04:41:29.231634+00:00 Process PID <0.17606.7> incremental update "notracking.ioc2rpz" started
2019-09-11T04:41:29.258359+00:00 Updating zone "notracking.ioc2rpz" inc. Last IXFR update 43200 seconds ago, last non-zero update 43200 seconds ago
2019-09-11T04:41:30.496608+00:00 Source: "notracking_hosts", size: 2.25/MB (2359116), MD5: "3c06772a834447f362aff0f04528a33e"
2019-09-11T04:41:30.653933+00:00 Source: "notracking_hosts", got 41505 indicators, clean time 0
2019-09-11T04:41:31.515626+00:00 Source: "notracking_domains", size: 5.34/MB (5594792), MD5: "7d5690dabc07983e470107731795948c"
2019-09-11T04:41:31.935628+00:00 Source: "notracking_domains", got 92424 indicators, clean time 0
2019-09-11T04:41:32.096719+00:00 Fetching zone "notracking.ioc2rpz" from ets
2019-09-11T04:41:32.201353+00:00 Rebuilding AXFR zone "notracking.ioc2rpz". New IOCs 132951
2019-09-11T04:41:32.225914+00:00 Zone "notracking.ioc2rpz", # of rules 2066, # of IOCs 1033
2019-09-11T04:41:32.226278+00:00 AXFR zone "notracking.ioc2rpz" was rebuilded. 2066 rules 1033 indicators. Parsed 133929 indicators.
2019-09-11T04:41:32.235334+00:00 Zone "notracking.ioc2rpz" records before 1033 after 1033.
2019-09-11T04:41:32.235449+00:00 Process PID <0.17606.7> incremental update "notracking.ioc2rpz" finished in 0 seconds

[Homas]

Optimization and the zone clean up may cause the issue.

Delete old records from zone "notracking.ioc2rpz".  before 133964 after 1033

[Homas]

Added some extra debug logs

[Homas]

2019-09-13T05:57:29.418938+00:00 Source: "phishtank", size: 45.93/KB (47033), MD5: "833805aa833eea1922675d76715d15f5"
2019-09-13T05:57:29.422352+00:00 Source: "phishtank", got 1769 indicators, clean time 0
2019-09-13T05:57:29.482811+00:00 AXFR update ets. Zone "phishtank.ioc2rpz". Before 1772 After 1819
2019-09-13T05:57:29.482960+00:00 AXFR update ets. Zone "phishtank.ioc2rpz". Indicators 1769
2019-09-13T05:57:29.565056+00:00 Delete old records from zone "phishtank.ioc2rpz".  before 1819 after 47

In write_db_record(ets,Zone,IOCs,axfr) function insert_new was replaced by insert to update the serial. Check the performance impact.

%  [ets:insert_new(rpz_ixfr_table, {{ioc,Zone#rpz.zone,IOC},Zone#rpz.serial,IOCExp}) || {IOC,IOCExp} <- IOCs, (IOCExp > CTime) or (IOCExp == 0)],
  [ets:insert(rpz_ixfr_table, {{ioc,Zone#rpz.zone,IOC},Zone#rpz.serial,IOCExp}) || {IOC,IOCExp} <- IOCs, (IOCExp > CTime) or (IOCExp == 0)],

[Homas]

Fixed in 1.0.0.3-2019091501.